You should be able to safely measure the floating linear power supply, as long as you know what you're doing and you're sure that the supply is in fact floating.
So first step is to make sure that the supply is floating. It would be simple to use a multimeter to confirm that there isn't a conductive path between the power supply's rails and ground. Is that is true, you can simply connect oscilloscope probe's ground connector to some point in the circuit. Often this would be the circuit's negative line, but does not need to be.
If the power supply isn't floating (or to say more clearly is grounded), you'd have to connect the oscilloscope's probe ground to the grounded rail of the power supply. That would be usually the negative rail, but could be positive, so to be 100% sure, you'd need to confirm the ground connection with a multimeter.
Do note that once you've connected the ground clip of the probe to some part of the circuit, that part is now ground referenced! That is important, because the ground clip of the other probe is also connected to ground and if you touch another part of the circuit with it, you'll short it to ground, which could have very negative consequences.
Here's a diagram of usual probe connections inside of an oscilloscope:
So if you for example connect the ground clip of one probe to negative rail of the supply and the other to positive, you'd have a short-circuit.
Now about the actual measurement itself:
First step would be to see if the probe can handle the voltages and to determine the appropriate probe setting. Usually 10x attenuation is used on probes, since that represents what is usually negligible load on the power supply and provides more bandwidth for the oscilloscope.
After that, connect the ground clip of the probe to the power supply and the probe tip to the point which you want to measure. Some sources recommend that the tested device be powered-off during the connection and that to me looks like a good idea since it minimizes the chances to make a short somewhere where it shouldn't be while connecting the probe. Once you connect the probe, check that the probe is properly connected and isn't touching anything it shouldn't be, like heatsinks (which might be connected to power supply's negative side).
Next, activate the oscilloscope and make sure that the probe attenuation factor is set at the same setting as seen on the probe. Next make sure that the probe coupling setting is correct. It shouldn't be set to ground and it should be set to DC. More about that is in the manual under To Set up the Vertical System.
Next step would be to set the oscilloscope trigger voltage for the connected probe a little bit higher (or lower) than the power supply's nominal voltage. This should make the scope trigger on ripple.
After that, turn on the power supply. You should be able to see a (more or less) flat line representing the supply's output voltage on screen and you may see some interference riding on that voltage.
Next part is a bit more difficult to explain and is a bit more experimental, but once you do it a few times, it will be easy.
The idea is to zoom in on the interference you see. You could try with automatic measurements and see how they work out. In case they don't show what you want to see, I'll explain how to do it manually. The whole story is explained in the horizontal and vertical settings part of the manual. Basically you use the scale knob to zoom in on the wave you see and then you use the position knob to set the wave on the center. I usually first adjust vertical settings, then horizontal and repeat the procedure until I can clearly see the ripple. Once you see it, you can measure the ripple using the graticule or you can use cursors. Cursor use is explained in the example 5 at the end of the manual for the scope and in the To Measure with Cursors section. When you're using graticule, you simply look up how much time or volts each division represents and then multiply the number of occupied divisions by the value you have. Cursor measurement will usually provide you with more precise result.
So far I haven't mentioned the math menu, because there is no need to use it. You definitely need to reference some point in the circuit to the oscilloscopes ground, since the scope does all measurement with respect to ground. If you connect one probe to the positive rail of the power supply and the second to negative and subtract them, you'll gt same result as if you measured against the probe clip's ground.
Do note that in the case of the isolated linear power supply, you can't get ground loop and have noise, since there will be no current going from power supply's ground through the oscilloscope's ground to the main ground, because the PSU itself isn't ground-referenced and there's no closed loop for current to go through.
A bit about AC coupling: As Vorac says, if you set the probe to AC coupling, you'll remove low frequency signals. This includes the DC component of power supply voltage, which will leave you with only the ripple. This way, you can avoid the need to use vertical position controls to bring the noise into view because it will be already centered to zero volts, so you can just zoom in on it.
Another handy thing are the trigger settings. You can also set filtering to triggering circuit, so that it will work on AC, DC, low or high frequencies. AC trigger coupling will remove all signals under 10 Hz from trigger circuit, so slow periodic signals won't interfere with trigger. LF reject will block all signals under 8kHz and HF reject will block all signals above 150 kHz. This can sometimes be useful if you're trying to focus on only one component of the signal and trigger on it.
Best Answer
Your Keysight scope is sampling at 1 MSa/s. I can't quite make out the Rigol's sample rate, but it looks like 5 MSa/s. That might account for some of the difference. Regardless, neither of those sample rates is high enough to correctly show a 20 MHz signal. You're seeing garbage on both scopes; it's just different kinds of garbage.