OK, answering my own question because I finally got it : the FIFO feature of the DMA was not enabled, so any interruption or memory transfer was delaying access to the memory bus by the DMA.
Enabling it is much better now. This has been done with this :
// Enable FIFO (see p190 of ref manual)
DMA2_Stream1->FCR=DMA_SxFCR_DMDIS; // disable direct mode
You are more-or-less correct on the "maximum frequency", but what you can see depends on the order of the low pass filter they're using. It's quite likely you could still see some significant signal above the noise floor at (say) 600Hz.
Generally what the spec means that your signal will be reduced to half (-3dB) at 400Hz. There may be significant attenuation to the signal at lower frequencies than 400Hz, again depending on the order of the anti-aliasing filter.
The resolution (24nV/bit) actually means very little (marketing aside) in this particular case. That's because the quantization noise is almost certainly not going to be dominant here. If you want to know the AC performance you need to look at the noise power spectral density (PSD) graph when connected to a source of the appropriate impedance (because there will be current noise as well as voltage noise).
A very, very good amplifier might have a white noise density of 1 or 2 nV/\$\sqrt{Hz}\$, so at any bandwidth greater than a couple tens of Hz the amplifier noise will dominate over the quantization noise. There will also be a flicker noise corner frequency below which the noise increases as \$1\over f\$. That may be out of the bandwidth of your signals. A crappier amplifier with high input impedance might be tens of nV/\$\sqrt{Hz}\$. The source will likely also have some Johnson-Nyquist noise- at room temperature, even a perfect 1K resistor will have noise of 4.1 nV/\$\sqrt{Hz}\$, so for a 200Hz bandwidth, it will have 56nV RMS noise, or almost 400nV p-p. This is not because of any manufacturing issue with resistors (although it's quite possible to make worse resistors than theoretical), it's a fundamental thermodynamic gotcha of the universe. See the fluctuation dissipation theorem.
Here's a decent introductory summary of noise.
Edit: According to their datasheet, the noise from 0.1Hz to 10Hz (9.9Hz BW) is 400nV (16 LSB), implying a noise density of 127 nV/\$\sqrt{Hz}\$. That's probably below the 1/f corner frequency so it's necessarily as bad as it sounds. Keep in mind that at higher bandwidths, the noise will be MUCH more than 16 LSBs, but there's not enough information provided to tell just how much worse (worst case, 400Hz BW noise would be about 100 LSBs).
So, as you can see, the resolution means almost nothing here, the lower order bits will be mostly noise.
Best Answer
The question was answered in the STM32 Q&A forums with a link to this document.
Basically the limit is the maximum resolution according to the registers (4096x2048) but what is more important is the bandwidth to the framebuffer memory (typically in a connected SDRAM). So it seems that the resolution should be possible but we're already quite near the limit.