TI's Analog Applications Journal includes an article about noise analysis for an op amp driving an ADC as in your application. The author uses \$F_{\text{L}} = 0.1\text{Hz}\$ with the following rationale:
When we think about noise at these low frequencies, we may jump to the conclusion that we should take this formula down to a very low frequency, such as 0.0001 Hz (0.0001 Hz = 1 cycle per 2.8 hours). However, at frequencies
lower than 0.1 Hz, which is one cycle every 10 seconds, it is very possible that other things such as temperature, aging, or component life are changing in the circuit. Realistically, low-frequency noise from the amplifier will probably
not appear at this sample speed; but changes in the circuit, such as temperature or power supply voltage, may.
Additionally, the OPA625 datasheet specifies the input noise voltage starting at \$f = 0.1\text{Hz}\$.
\$F_{\text{L}} = 0.1\text{Hz}\$ seems like a good choice since all three documents use it.
If you want to know what the bandwidth is, it should be in the datasheet, if its not then it might be hard to find it. One thing you might be able to do is limit the bandwith yourself.
Measuring noise bandwidth is not going to work with a DC volt meter, its measuring near 0Hz (whatever the update rate is). RMS values measure noise across many frequencies and use a bandpass filter.
There are two problems of measuring resistor thermal noise:
1) Other noise sources are going to make it difficult, meters, cables and amplifiers are all noise sources and need to be accounted for when measuring noise
2) The bandwidth of the noise and sampling of the noise needs to be accounted for because we cannot measure the distribution of white noise directly (that would involve measuring across all times to infinity, but we can approximate it).

The noise of a resistor looks like this:
$$ \frac{v_n^2}{\Delta f} = 4k_BTR$$
The voltage source is modeled as the noise power and that power changes depending on how much noise sources frequency you sample, like you did in the oscilloscope example.
No matter the instrument, you will need to know the bandpass of the noise or you will not be able to determine the noise power.
It might also be worth noting that you can ground out the inputs and measure the baseline noise (of the pre-amplifier or meter) if your using them and find out how much noise you have from the system. Then connect the resistor that you wish to measure, the RMS noise from the resistor and the system will add by the sum of the squares.
If you want to measure noise with a DC voltmeter you would need a pre amplifier and a way to convert the total noise power to an analog measurement at DC. There are circuits you could use to do this, but most of them would be hard to build yourself, and probably not worth your time. Some manufactures make IC's such as this one(AD737)
that can calculate the RMS power, then run that value into the DC Meter.
If the meter has an RMS (AC) button, then just use the AC button after the amplifier and use a preamplifier, the bandwidth of the RMS measurment must be known to calculate the noise power from the resistor.
If your just using a DC meter, your limited on what noise you can measure based on the accuracy of the meter.

Here are some other good resources:
Noise sampling
Low level measurements Handbook
Best Answer
the ktC noise formula simply happens when you insert the noise bandwidth of an RC filter, B=1/(4RC), into the formula for voltage square over a resistor. The R cancels out.
So, no, there's correctly no bandwidth in that formula, because the bandwidth is "hidden" in the C.
The wikipedia article on Johnson-Nyquist Noise actually containst this explanation, and a lot more stuff that you might want to know about kTC noise, considering you seem to be coming at this from a non-intuitive, formularistic side. It's usually easier to understand what's happening than to have all the right formulas exactly right :)