Electronic – Massive pk-pk ripple on output using isolated DC-DC converter module

There is of course no single answer to what "decent" power supply noise is. That's like asking what a decent car is without telling us whether its for driving around a racetrack or back country dirt roads.

Whether the values you mention are decent depends on how that power rail will be used. What you really seem to be asking is just from the power supply's point of view whether these value seem reasonable or not. 20mV for a generic bench top supply sounds quite reasonable to me, and so does 7mV for a on-board boost converter (in fact that's actually quite good compared to a lot of them).

Your circuit, however, may have a different opinion. If the 5V supply is just powering digital circuitry, then it's a lot cleaner than it needs to be. Even 100mVpp ripple would be tolerable.

If you're powering sensitive analog circuitry, then 7mV could be large. In that case the frequency content of the ripple also matters. Most analog ICs have a power supply rejection spec. There is active electronics in the IC to make its operation somewhat independent of the power supply voltage. However, that electronics can only react to noise up to some frequency. The frequency requirements to get the specified power supply rejection ratio is rarely specified. It's a good practice to put a ferrite bead or small chip inductor followed by a ceramic cap to ground on the power leads of analog parts. This will attenuate the high frequencies of the noise, with the remaining low frequencies hopefully in the range the part can handle and reject actively.

Some parts are much more susceptible to this than others. The first time I used one of the Freescale multi-axis accellerometers there was a lot of noise on the output. The power supply noise actually seemed to be amplified onto the output. Adding the aforementioned chip inductor in series with cap to ground on the power lead helped a lot to clean up the output signal.

To answer your last question, the normal way to look at power supply noise is exactly what you did. AC couple the scope input, crank up the gain, and look at the size of the resulting mess.

To answer the basic question of your "not referenced to mains earth" (i.e. floating) supply, it will be safe to probe this with your oscilloscope. You should use a good quality double insulated transformer with low capacitive coupling between windings though.
You can think of the supply as like a battery.
Be aware that when you connect the probe ground lead, the supply then becomes referenced to mains earth.

Voltage is always relative to something, you cannot just say "this point is at 10V", rather "this point is +10V relative to this point" or, "this point is -5V relative to this point". The reference point is usually called "circuit ground", note that this point does not have to be the same as "earth ground" (i.e. mains earth)

The main issue with scopes is when you have a supply that has it's circuit ground referenced to earth ground and not at the same potential (and low impedance - capable of supplying a fair amount of current)
Because the scope probe ground is directly (i.e. low impedance) connected to mains earth, you cannot connect it to anything referenced to earth and not at at the same potential (i.e. 0V)
You can connect it to the un-referenced supply, as this is just like connecting it to one terminal of a battery (then the other side of the battery becomes +/- the battery voltage relative to mains earth)

Many bench supplies have an un-referenced output, but also have an earth terminal you can use if you wish to tie the output to earth ground. If you tie the positive terminal to earth, the supply is negative relative to the earth terminal, and vice versa. You could do this with your supply if you wish. In the image below the centre green terminal is chassis (mains earth) ground. The datasheet explains the use of the terminal.

EDIT - To try and explain the low impedance floating ground issue, have a look at this circuit, an unregulated dual polarity supply (around +/-16V/15A):

Here is the current through the probe ground lead:

Everything here is fine, as the supply has no low impedance reference connection with mains earth, so you could connect the probe ground to any of the terminals and get the same result. There is a tiny leakage current through Rleak and Rleak2, which is normal (I've left out capacitive leakage)

Now what happens if we connect earth ground (see 0ω Rearth is added) - not to circuit ground, but to the negative supply (so it's no longer the negative supply - it could be e.g. chassis ground) Now our circuit ground is floating 16V above mains ground, and is low impedance.

Now look at the current through the probe ground lead:

There is a large current flowing (i.e the full current the supply can deliver), which is only limited by the supply transformer's output winding resistance. This is not good ;-)

This is the same as just connecting the probe ground to the V+ rail of any circuit with it's ground tied to mains earth (through a low impedance).
However, it shows us that circuit ground is not always at 0V relative to mains earth, so we must be careful and check before connecting the probe ground.