Electronic – Using an earthed function generator with an earthed oscilloscope and ground loops

Yes, it sounds like (a little confusing) you have a ground loop problem, and yes they can matter, especially when trying to measure small analog signals. If all grounds tie back to the same outlet strip via relatively short line cords, then it would probably be OK. However, you say that this cryostat thing (whatever that is) is connected separately to building ground, so that is obviously not the case and it's confusing therefore why you brought it up.

In general, it's good to convert analog signals to digital as close as possible to the source, then ship around digital signals. Those are much easier to isolate, like via opto-couplers, pulse transformers, radio, etc. In other words, a old fashioned A/D card in the computer is not the best overall architecture from a system level point of view.

However, look at the A/D card carefully. Most likely it can be configured for single ended and differential operation. This is a case where you want differential inputs. The cryostat thingy may produce a ground referenced signal, but take its ground and output signal as being differential. This will essentially subtract the ground offset from the signal before converting it.

This trick will only work up to some frequency, probably a few kHz or low 10s of kHz. It should work pretty well in subtracting off any ground signal due to 60 Hz or 50 Hz power line return currents accross ground paths in the loop. Sharp common mode spikes can still confuse the diff amp in the A/D and show up as noise in the final output. It's worth a try though. If it's not good enough, go back and convert to digital at the sensor, then opto-isolate the digital telemetry signal.

What you describe does create a ground loop, but it's unlikely to cause problems because the loop is so small. The ground loops that cause issues typically involve loops measured in meters, not centimeters. That said, the definition of "small" depends on the frequencies used in your circuit, so if you're working with very high frequencies (maybe, over 100 MHz), or your mezzanine board is especially large, you may need to reconsider this.

Just like how when you're laying out a circuit board, you normally want to lay out separate ground regions in a star topology, with a single connection point; if you could arrange it, you'd also want to move your two mezzanine connectors to the same end of your board to create a single connection area for your ground connections.

But, if you simply can't change the locations of the connectors, it's almost certainly better to have ground pins in both connectors. The reason is if you remove the ground pins from one of the connectors, then the return currents for any ground-referenced signals on the other pins have to travel around a larger loop back to their origins through the other connector (the one that still has ground pins), and this is likely to cause emissions or signal integrity problems.