You should be able to safely measure the floating linear power supply, as long as you know what you're doing and you're sure that the supply is in fact floating.
So first step is to make sure that the supply is floating. It would be simple to use a multimeter to confirm that there isn't a conductive path between the power supply's rails and ground. Is that is true, you can simply connect oscilloscope probe's ground connector to some point in the circuit. Often this would be the circuit's negative line, but does not need to be.
If the power supply isn't floating (or to say more clearly is grounded), you'd have to connect the oscilloscope's probe ground to the grounded rail of the power supply. That would be usually the negative rail, but could be positive, so to be 100% sure, you'd need to confirm the ground connection with a multimeter.
Do note that once you've connected the ground clip of the probe to some part of the circuit, that part is now ground referenced! That is important, because the ground clip of the other probe is also connected to ground and if you touch another part of the circuit with it, you'll short it to ground, which could have very negative consequences.
Here's a diagram of usual probe connections inside of an oscilloscope:
So if you for example connect the ground clip of one probe to negative rail of the supply and the other to positive, you'd have a short-circuit.
Now about the actual measurement itself:
First step would be to see if the probe can handle the voltages and to determine the appropriate probe setting. Usually 10x attenuation is used on probes, since that represents what is usually negligible load on the power supply and provides more bandwidth for the oscilloscope.
After that, connect the ground clip of the probe to the power supply and the probe tip to the point which you want to measure. Some sources recommend that the tested device be powered-off during the connection and that to me looks like a good idea since it minimizes the chances to make a short somewhere where it shouldn't be while connecting the probe. Once you connect the probe, check that the probe is properly connected and isn't touching anything it shouldn't be, like heatsinks (which might be connected to power supply's negative side).
Next, activate the oscilloscope and make sure that the probe attenuation factor is set at the same setting as seen on the probe. Next make sure that the probe coupling setting is correct. It shouldn't be set to ground and it should be set to DC. More about that is in the manual under To Set up the Vertical System.
Next step would be to set the oscilloscope trigger voltage for the connected probe a little bit higher (or lower) than the power supply's nominal voltage. This should make the scope trigger on ripple.
After that, turn on the power supply. You should be able to see a (more or less) flat line representing the supply's output voltage on screen and you may see some interference riding on that voltage.
Next part is a bit more difficult to explain and is a bit more experimental, but once you do it a few times, it will be easy.
The idea is to zoom in on the interference you see. You could try with automatic measurements and see how they work out. In case they don't show what you want to see, I'll explain how to do it manually. The whole story is explained in the horizontal and vertical settings part of the manual. Basically you use the scale knob to zoom in on the wave you see and then you use the position knob to set the wave on the center. I usually first adjust vertical settings, then horizontal and repeat the procedure until I can clearly see the ripple. Once you see it, you can measure the ripple using the graticule or you can use cursors. Cursor use is explained in the example 5 at the end of the manual for the scope and in the To Measure with Cursors section. When you're using graticule, you simply look up how much time or volts each division represents and then multiply the number of occupied divisions by the value you have. Cursor measurement will usually provide you with more precise result.
So far I haven't mentioned the math menu, because there is no need to use it. You definitely need to reference some point in the circuit to the oscilloscopes ground, since the scope does all measurement with respect to ground. If you connect one probe to the positive rail of the power supply and the second to negative and subtract them, you'll gt same result as if you measured against the probe clip's ground.
Do note that in the case of the isolated linear power supply, you can't get ground loop and have noise, since there will be no current going from power supply's ground through the oscilloscope's ground to the main ground, because the PSU itself isn't ground-referenced and there's no closed loop for current to go through.
A bit about AC coupling: As Vorac says, if you set the probe to AC coupling, you'll remove low frequency signals. This includes the DC component of power supply voltage, which will leave you with only the ripple. This way, you can avoid the need to use vertical position controls to bring the noise into view because it will be already centered to zero volts, so you can just zoom in on it.
Another handy thing are the trigger settings. You can also set filtering to triggering circuit, so that it will work on AC, DC, low or high frequencies. AC trigger coupling will remove all signals under 10 Hz from trigger circuit, so slow periodic signals won't interfere with trigger. LF reject will block all signals under 8kHz and HF reject will block all signals above 150 kHz. This can sometimes be useful if you're trying to focus on only one component of the signal and trigger on it.
Somethings not quite right there.
I have the 200MHz SD8202 which has the same manual so I assume the procedures are the same. It has a 5V pk-pk compensation output. I'm guessing you have adjusted the vertical position to zoom in at the top of the waveform.
I just tried this and I get similar to your top picture but with a bit less noise (did you connect the ground clip too? Was it bandwidth limited to 20MHz like the second waveform?)
All I can think of trying right now is running the self calibration routine (remove probes, press "Utility" button, press "Function" (H1 button), choose "Adjust" with "M" knob, press "Self Cal" (H2 button)
Let this complete (couple of minutes) and try the probe compensation again. Let us know how it goes.
EDIT - What do you get with all the waveform on the screen? (e.g. 1V/div)
It's possible the front end is slightly different in your model (or conditions/part tolerances are different), and it's at the limit of the vertical amplifier as Zebonaut proposes.
I'd also try a divider on the compensation signal to make it so the full waveform is on the screen at 20mV/div, just in case it's a problem specific to that gain setting.
At 5V pk-pk, to bring it down to say, 100mV pk-pk you need a 50:1 divider. Something like 50k and 1k will get "close enough" (98mV pk-pk) Or use a potentiometer and just adjust visibly.
If it still looks the same like this then I would say there is a problem.
Best Answer
You should be safe repairing the wires and calling it a day.
When you connected the second ground alligator clip, you made a short at the supply output. The full 25A passed through your scope, but I doubt it was damaged, and I doubt the cables were damaged too.
Inside the scope there is at least a plane on the pcb dedicated to ground, it should be more than capable to carry that current for a limited period of time, at least.
Coax cables have a fairly thick shielding, which can carry some current without being damaged. I sometimes use them to power things with no problem. I am not sure of the current rating, I would guess less than 10A but not much less, so they can probably withstand 25A for a short time.
The weak point is the tiny cable that connects the probe to the alligator clip, and it fried.
You already tested your probes with the 1kHz source, if possible I would try some higher frequencies and I would check also that the measured voltage is ok, but I think it is very unlikely you made more damage than some molten plastic.