I think what you did was close, but I wouldn't leave the whole chassis floating. I'm assuming this isolation transformer comes with a standard 3 prong outlet for the output side. Just disconnect the ground connection on this outlet, but continue to let the input side ground be connected to the chassis.
Generally you want as little as possible floating at some arbitrary voltage. There will be plenty of other things around at the normal ground potential, so the chassis of the transformer isn't going to make anything worse.
I agree with Dave in that I don't understand why isolation transformers come with the grounds connected. I got one that way too many years ago, but didn't expect it. I actually blew out a fuse before realizing that the grounds were connected together inside the box. I don't know what they think isolation means, but different from what I had assumed.
By the way, beware of inrush currents on a 500 W isolation transformer. All the ones I've seen are toroidal cores. These can retain residual magnetism depending on what part of the power cycle they were turned off at. If this happened to be at one peak and you turn it on next time near the other peak, there will be a very large inrush. I once blew a 30 A breaker with such a transformer, even though I had switched it on a number of times previously on the same circuit. Unfortunately this was on a weekend and the breaker was in a different locked room. Argh. Often you want to adjust the voltage anyway when you're debugging power circuits where you want isolation. In that case, put the variac before the isolation transformer instead of after it. If you bring the variac voltage up and down with the knob, there won't be much inrush.
What might be the reason then for this amplifier?
The possible reason is that your motor controller (Variable frequency drive) is connected to AC mains power and may not have satisfactory protection to offer a signal that controls it. This might mean that the VFD can inject (unintentionally) noise and interference onto its input relative to ground (earth). This may upset the PC DAQ output or indeed cause it to fail or maybe just give the PC a few resets once per year.
There may also be a safety issue - the spec for the isolation device says it has a 250VAC operational isolation and is capable of being "tested" to 3.5kVAC. This sounds to me like it would protect against risk of electric shock and provide decent isolation for noise and ground loops from the VFD.
Best Answer
It sounds like you're talking about an oscilloscope or something similar. In that case, the channel-to-channel isolation describes the isolation between each channel and every other channel. The channel-to-ground isolation describes the isolation between each channel and ground. These are different, but related, concerns.
Channel-to-channel isolation describes how much voltage differential there can be between two inputs to the scope.
Channel-to-ground isolation describes how much voltage there can be between any input and the ground input to the scope, usually through the power connector.
The "techniques" involved aren't different at all. When two things are to be isolated, make sure there aren't any conductive paths between them, and make sure that all barriers between them (optocouplers, transformers, air distance, creepage distance, insulation, etc.) are rated for the isolation voltage you want.
Now, if all channels are isolated from each other, they must also all be isolated from ground; if they were all referenced to ground, they would all also be referenced to each other! It is, however, possible to have all channels isolated from ground while still being referenced to some other common node. This scenario can be useful, depending entirely on what it is you want to measure. If all the signals being measured are referenced to the same voltage, which is not earth, then your scope also having isolation to ground but not channel-to-channel would be fine.
As for crosstalk, that would require more details about what you're doing. I don't think there's a general answer there.