I would like to perform a FFT on a signal with equally sampled values of which some are missing.
(Actually, they are not even missing, but simply erroneous, so I can't use them, but I can detect when I have a wrong samplepoint.)
Is there a way to compensate for those missing values and get a good transformation? I suppose this is a frequent problem, but so far I only found this and somebody interpolating the missing values.
As far as I understand it, if my signal is, say, 100 HZ and the frequencies I care about are, say, 5 Hz, following the Nyquist theorem a 10Hz signal should be enough, so I could downsample the signal in a way that removes the missing points.
So my question is, is there a way to use a fourier transformation (preferably FFT) on a signal with missing values and get an (at least almost) correct spectrum for small frequency values while retaining the other information? How about other transformations (orthogonal like wavelets, etc.), is it the same problem or a completly different?
Edit:
Thanks for all the input, I will definitly review the literature, I was just hoping there was a good starting point or standard solution I had missed so far.
One commenter said I didn't provide enough information, so I'll add some. The signal processing should be real time (best case), so more processing time would be a big issue. The data cannot be recorded continuously and then sampled as the recording method per definition uses sampling (like a camera).
I am looking for a recurring signal with a frequency which is between 15 and 100 times smaller than the sampling rate, though most times it's rather 20-40 times smaller. The recurring signal itself however has higher frequency components, so it's not exactly staying below my sampling frequency.
My actual problem, for which I was trying to gather ideas with this question, is that at random points in time I get baseline shifts (inherent, no physical solution possible). They can be detected as they are way greater than the actual signal and others are working on ways to compensate them algorithmically. However, during signal analysis, as of now, I have thos shifts of baseline that may or may not happen fast.
Those shifts can either produce spikes and effectively killing any interesting signal or be more or less filterable, talking for example a ramp signal overimposed on a part of the original signal.
I'd like to retain as much data as possible – and as I'm trying to work real time, just using a transformation on the part before the shift and another one on the part after the shift would be a bad solution.
I do know the base frequency range (in which the signal is repeated) and more or less the signal form (however not exactly amplitude and length, it may be a little shorter or longer losely depending on the reptition frequency) I am looking for.
Best Answer
Unless your signal has zero mean, I would avoid zeroing the missing samples because that would introduce bias in the estimation of the mean, the power and the spectrum of the signal. Instead, the best estimation of the missing samples is the average value of your time series.
However, using the mean isn't good either. The subset of replaced samples would have a constant value. That would solve the estimation bias problem for the average value (f=0), but not for the remaining spectrum. In other words: the replaced samples don't have the statistical and spectral characteristics required.
How can we improve this? Interpolation of the missing samples is an option.
The subset of interpolated samples will also have the same average value of the signal of interest, along with some variability around it. If the wrong samples are randomly spaced, then the subset of interpolated samples will have a variance (power) similar to that of the signal of interest.
Interpolation implies some degree of smoothing. We can think of that as some kind of low-pass filtering on the subset of interpolated samples. Thus, we could mitigate the interpolation-induced distortion with oversampling.
So far, we've dealt with the problem at sample level. But there are other options. If we are estimating the spectrum using non-parametric methods based on signal segments (Barlett, Welch...), then we could discard the segments containing the wrong samples. Discarding segments will increase the variance of the estimation, but will not introduce estimation bias at any frequency.