Yes, it sounds like (a little confusing) you have a ground loop problem, and yes they can matter, especially when trying to measure small analog signals. If all grounds tie back to the same outlet strip via relatively short line cords, then it would probably be OK. However, you say that this cryostat thing (whatever that is) is connected separately to building ground, so that is obviously not the case and it's confusing therefore why you brought it up.
In general, it's good to convert analog signals to digital as close as possible to the source, then ship around digital signals. Those are much easier to isolate, like via opto-couplers, pulse transformers, radio, etc. In other words, a old fashioned A/D card in the computer is not the best overall architecture from a system level point of view.
However, look at the A/D card carefully. Most likely it can be configured for single ended and differential operation. This is a case where you want differential inputs. The cryostat thingy may produce a ground referenced signal, but take its ground and output signal as being differential. This will essentially subtract the ground offset from the signal before converting it.
This trick will only work up to some frequency, probably a few kHz or low 10s of kHz. It should work pretty well in subtracting off any ground signal due to 60 Hz or 50 Hz power line return currents accross ground paths in the loop. Sharp common mode spikes can still confuse the diff amp in the A/D and show up as noise in the final output. It's worth a try though. If it's not good enough, go back and convert to digital at the sensor, then opto-isolate the digital telemetry signal.
The designer has tried to indicate on the schematic the way the grounds should be separated, and done a reasonable job with the standard symbols available to him.
There ought to be a detailed description and written guidelines in the datasheet, and recommended PCB layouts either there, or in a separate Application Note (if you look up this chip on the TI website, the relevant App Notes should be easy to find)
But basically, the IC contains both a high gain amplifier with a sensitive input, and a high current switch, capable of generating a lot of noise. With incorrect grounding, high currents in the ground wires can generate unwanted signals on the amplifier input, causing instability or poor voltage regulation.
The solution is to - as far as practical - provide two separate grounds; one quiet one for sensitive signals (denoted by "earth ground" ) and one for high currents (denoted by chassis ground, which doesn't have to be connected to the actual chassis!) The two MUST be tied together - at one, carefully chosen point, sometimes called a "star earth" (useful search term for further reading!)
Thus R1 and R2 provide the voltage feedback to the error amplifier. You don't want to inject large errors via R2, so it is returned to the quiet ground. The error amplifier will take its reference from the "GND" pin (again on the quiet ground)
Now...
Switching current through L imposes a huge AC current waveform on Vin, and generates a huge AC current on Vout respectively. These currents are communicated to ground via C1 and C2 respectively.
In fact the power side of this circuit can be read as one continuous loop GND -> C1 -> L1 -> (switch inside chip between L and Vout) -> C2 -> GND.
This loop is the most important part of the circuit and must be kept as small as possible. Best thing to do is to put the GND leads of C1 and C2 right next to each other - virtually all the AC current goes from one C pin directly to the other. The other connections (PGND, VAUX via C3) are less important but go to this point too.
And one (reasonably thick) trace from here to the low noise ground will carry relatively little current, with relatively little noise on it.
Learning to read this high current path and keep it separate from low noise ground will go a long way to making your switchers trouble free.
Best Answer
@rdtsc provided a link to the manual which covers what to do with the grounding. You are going to connect DC negative to chassis ground with heavy gauge wire (see manual), and you are going to have to decide what "chassis" means in your case. Maybe if this box of yours is metal, you can just bond the chassis ground to the metal box. But this is not really the important part. More importantly, make sure you use a GFI somewhere between the inverter and your load. Inside the inverter, since it is a UL 458 inverter, there is a relay which connects green wire to neutral only when the inverter is supplying power. When it is in standby or pass-through mode (when you supply AC to the inverter so it can recharge its batteries) the connection from neutral to GND is open. In either case, there is exactly one place where GND is bonded to neutral. Your 12 battery is not a shock hazard. It stores an impressive amount of energy, and can be a fire hazard if there is a short, so make sure you have proper fuses. But if you screw up the AC grounding, you can introduce a shock hazard. Luckily it is not that complicated. Just read the manual again.