DIY Ring Light: Grounding Bulbs in Series

To answer the basic question of your "not referenced to mains earth" (i.e. floating) supply, it will be safe to probe this with your oscilloscope. You should use a good quality double insulated transformer with low capacitive coupling between windings though.
You can think of the supply as like a battery.
Be aware that when you connect the probe ground lead, the supply then becomes referenced to mains earth.

Voltage is always relative to something, you cannot just say "this point is at 10V", rather "this point is +10V relative to this point" or, "this point is -5V relative to this point". The reference point is usually called "circuit ground", note that this point does not have to be the same as "earth ground" (i.e. mains earth)

The main issue with scopes is when you have a supply that has it's circuit ground referenced to earth ground and not at the same potential (and low impedance - capable of supplying a fair amount of current)
Because the scope probe ground is directly (i.e. low impedance) connected to mains earth, you cannot connect it to anything referenced to earth and not at at the same potential (i.e. 0V)
You can connect it to the un-referenced supply, as this is just like connecting it to one terminal of a battery (then the other side of the battery becomes +/- the battery voltage relative to mains earth)

Many bench supplies have an un-referenced output, but also have an earth terminal you can use if you wish to tie the output to earth ground. If you tie the positive terminal to earth, the supply is negative relative to the earth terminal, and vice versa. You could do this with your supply if you wish. In the image below the centre green terminal is chassis (mains earth) ground. The datasheet explains the use of the terminal.

EDIT - To try and explain the low impedance floating ground issue, have a look at this circuit, an unregulated dual polarity supply (around +/-16V/15A):

Here is the current through the probe ground lead:

Everything here is fine, as the supply has no low impedance reference connection with mains earth, so you could connect the probe ground to any of the terminals and get the same result. There is a tiny leakage current through Rleak and Rleak2, which is normal (I've left out capacitive leakage)

Now what happens if we connect earth ground (see 0ω Rearth is added) - not to circuit ground, but to the negative supply (so it's no longer the negative supply - it could be e.g. chassis ground) Now our circuit ground is floating 16V above mains ground, and is low impedance.

Now look at the current through the probe ground lead:

There is a large current flowing (i.e the full current the supply can deliver), which is only limited by the supply transformer's output winding resistance. This is not good ;-)

This is the same as just connecting the probe ground to the V+ rail of any circuit with it's ground tied to mains earth (through a low impedance).
However, it shows us that circuit ground is not always at 0V relative to mains earth, so we must be careful and check before connecting the probe ground.

What the chunk of metal does is to capacitively couple your circuit to ground. Have a look at this post: Grounding the chassis with capacitor , where the poster notice the difference between connecting directly to ground and connecting via a capacitor.