You need to step back and deal with this noise properly. Trying to cover it up in the firmware is the last resort. Something is making a mess somewhere.
Start by turning off everything except the receiver module and look at its output with a scope. Make sure the scope probe is properly grounded right at the receiver module ground. Make sure the logic analyser, the arduino, etc, are all off. Power the receiver from batteries if you have to. Is the noise now present on the receiver output? If so, then all the stuff that's off isn't at fault. Shut off as much other equipment in the vicinity as you can manage. Put a antenna tuned to your frequency (you only said ISM band, which doesn't tell us the frequency) on the receiver. Make a 1/2 wavelength dipole from wire if you need to. Keep poking around to find the source of the noise, which in this case is either bad module hookup or actual RF interference in your band. If the latter, there is little you can do about it.
If the signal is clean with just the module, then slowly add back other parts of the circuit one by one. Each time watch the output of the receiver on the scope. At some point you'll get the noise back, but now you'll know the source.
The best way to deal with any noise is to eliminate it at the source. Once noise gets added to your signal, there are likely parts of the noise that can't be distinguished from valid signal anymore. No amount of post processing can get rid of such noise.
I can't answer this authoritatively but my gut tells me your spec is going to be "very difficult".
In particular, your transition band of 50 MHz is only 0.02 decades at 1 GHz, so you're looking for a drop of 714 dB/decade between your pass band edge and your rejection band. Which implies something like a 71-pole filter, requiring 71 active elements.
For reference, here's what can be done with a reasonable number of elements:
(Graph from TI's App Guide "Op-Amps for Everyone") The graph is in terms of "normalized frequency", meaning you can scale the filter elements in such a way as to make a frequency of "1" on the graph correspond to any frequency you choose, for example 1 GHz in your case.
At lower frequencies, we normally construct multi-pole filters by cascading 1 and 2-pole active sections to obtain some desired response.
At 1 GHz, you may, just be able to do that using rf amplifiers to buffer between stages. But more likely, you'll be stuck falling back on older techniques of constructing an LC ladder to get an approximation of the response you want. The problem with this technique is it tends to make the filter response more sensitive to small variations in the component values, caused by manufacturing differences or temperature sensitivities.
Using microstrip elements, you might have less trouble with L and C variability, but you're likely to find that the range of L and C values required are outside of what can be sensibly constructed in microstrip. In addition, my (very limitted) experience suggests that microstrip filters are only likely to be effective over about an octave frequency range. So if you want a 1 GHz LPF, you might find you get an unwanted blocking band below 500 MHz, or an unwanted pass-band above 2 GHz. In any case you don't want to jump in to designing microstrip filters without access to some kind of reasonable CAD tool. Agilent's ADS or Genesys jump to mind. Genesys would be particularly helpful for you, if you can get access to it, because it provides special tools for generating filter designs given a spec like you've given in your question.
Of course, a combination of lumped and microstrip elements is also possible.
Edit:
One reasonable design approach would be to use a tool like Matlab or Octave to see what kind of filter (Butterworth, Chebychev, etc, and how many poles) can come close to meeting your requirements. If you have access to a good library, look for a book with a title like "filter design handbook". This will give you lookup tables for the pole and zero locations of various types of filter of different orders. This will make it "easy" to calculate the response even if you don't have a high-priced tool like Matlab with the right toolbox to get the filter parameters from software.
Then, once you know where you want your poles and zeros, use a tool like ADS, or Genesys, or even SPICE, to design a filter using real L and C elements to create the mathematical response you optimized in Matlab. Then, be sure to do a sensitivity analysis to be sure the response stays in spec under normal variation of the part characteristics. Finally, depending on the L and C values you come up with, decide whether you want to implement some or all of those elements in microstrip instead of with discrete components. If you do decide to use microststrip, then use an rf design tool like ADS or Genesys (those are just two tools I've used myself, but there are others that could do this) to simulate and optimize the microstrip layout to achieve the behavior you want.
Another late note: You can see in the graph that for a Chebychev filter, the slope immediately after cut-off is steeper than the eventual slope of the skirt, so my statement of needing a 71-pole filter is probably too strong. But nonetheless, its clear you need at least 10 poles to meet your spec, and doing that with only passives is very challenging because of the stage-to-stage interactions and the required tight tolerances on the component values.
Best Answer
This is done with amplitude modulation. The intensity of the laser is adjusted linearly with the amplitude of the input signal. At the other end, a wideband optical detector converts this back into a electrical signal. This works even up to relatively high frequencies (many GHz) because the frequency of the light used is around 200 THz.