The first thing to do when looking to design a filter for a signal is to obtain the spectrum of the signal. This can be done in different ways. For example, you can measure or capture the signal with a scope. Most modern scopes can calculate an FFT of captured data and give a spectrum.
For a design though, it is nice to make an idealized waveform and then write the Fourier series of it. This way you get only the signal and no unwanted stuff, and so get a clean spectrum which will just be the coefficients of the harmonics. Then you can truncate or alter harmonic content (coefficient amplitudes) according to some filter type. Reconstruct a filtered signal from those altered coefficients and the filter effects will be evident.
From the comments of having a 5MHz carrier with ~1MHz BPSK, it is likely that many harmonics will be needed if a good representation of the original signal is desired. It wouldn't be surprising if 20 to 40 harmonics were needed for good representation. That would be a filter that started to roll off at > 20MHz, or maybe 40MHz. That's only 2 or 3 octaves from the 2 meter band (~150MHz). A filter with 1st order roll off isn't going to do that. You're going to need a filter with at least 3rd order performance for something like that. It's possible to get 2nd order performance out of a single stage Sallen-Key low pass filter.
As to the design shown in the schematic, it looks like the amplifiers are OK for what's being asked of them. LC filters are sensitive to termination. Best performance is when they are terminated into their characteristic impedance. The LC shown has \$Z_o\$ of 45 Ohms, and when used alone shows a resonant lobe of about 6dB. You can see the effects of impedance matching in case 4 when the LC is terminated into U2 with 100 Ohms. For a better match, R8 and R7 values could be halved, or L1 could be doubled and C5 could be halved.
Usually when active filters are used, inductors are not. It's because inductors are not needed to get complex poles and high Q roll offs, amplifier gain (and sometimes a little positive feedback, as in Sallen-Key) give that. Passive filters are good if the filter will be someplace where there is no bias voltage, or where frequencies and or filter order is high.
First of all, the circuit drawn is a high-pass filter.
At very high frequencies, the inductor has very high impedance and thus, the output \$V_{ab}\$ should essentially equal the input.
At very low frequencies, the inductor has very low impedance and thus, the output should be essentially zero.
The frequency where the output voltage is \$\frac{1}{\sqrt{2}}\$ times the input voltage is given by
$$f_0 = \frac{R}{2\pi L} $$
So, either the circuit drawn does not reflect the circuit measured or something went terribly wrong with your measurement apparatus (or operator of said apparatus).
For completeness, the transfer function is, by inspection:
$$H(j\omega) = \frac{j\omega L/R}{1 + j\omega L/R}\quad,\quad\omega = 2\pi f$$
$$|H(f)| = \frac{2\pi fL/R}{\sqrt{1 + (2\pi fL/R)^2}} $$
.
.
Best Answer
You don't have a whole lot of signal there at the higher attenuations so I think those are just artifacts of the measurement.
If the behavior deep into the stop band of your MFB filter is really of keen interest, try to increase the signal levels to as high as your ADC can handle without clipping. You have only 16 bits so the the quantization noise will limit you around -95dB. There is probably significantly more noise than that in the ADC. If that doesn't work well enough for your requirements, get a better ADC or even try a sound card.