Electronic – Help interpreting strange spectrum

Well, you're not going to be making RF measurements up to 30 GHz without spending a bunch of money, so either path is big bucks.

Typically, Spectrum analyzers are used to do frequency domain measurements. You'll get a display of power vs frequency on the display. The controls in the SA are setup for relevant things, Center frequency, bandwidth, resolution bandwith, signal powers in dBm/dBc etc.

Digital oscilloscopes don't directly have sampling rates to directly sample a 30 Ghz signal, so they'll undersample and assume that the signal repeats. probably a safe assumption, although with no front end filters built into them, you've got dynamic range issues, as well as aliasing concerns that aren't present in a Spectrum Analyzer. You won't directly get spectral plots out of a Digital oscilloscope, you'll need to do an FFT on that. Now, that opens up a can of worms. FFT bin width/windowing function selection, etc. All stuff that can be worked through, but another question to deal with.

You won't get eye diagrams out of a spectrum analyzer, it's a useless measurement @ RF. That's a demodulated signal measurement.

Ultimately, if you want time domain data, then use an oscilloscope. If you want Spectral information, use a spectrum analyzer.

But why is the first entry in the FFT results not the fundamental?

Typically the 0'th entry stores the DC term. In some schemes, the [N/2]'th entry might be the DC term and the [N/2+1]'th entry would be the fundamental.

I applied an LC filter, by iterating through the FFT results and multiplying each bin by the filter gain at that frequency. I then did an inverse FFT, ... The result has both real and imaginary components.

Did you consider the phase effect of the filter? You should be multiplying by a complex value accounting for both the magnitude and phase response of the LC filter.

Even then, due to finite precision issues, you will often still get non-zero imaginary components in the inverse-transformed time-domain signal. But usually these are very small, for example on the order of \$10^{-15}\times\$ the real components. These are usually dealt with by simply ignoring them (taking the real part of the result only). Some tools even offer a built-in function (Mathematica's Chop, for example) to eliminate these artifacts.