Sinusoidal frequency doubling

Question 1: Assuming that the oscillator (perhaps programmable) is given, what kind of hardware would you suggest for (2) and (3) ?

For 2: All-pass (phase-shifting) filter, a.k.a. Hilbert transformer

For 3: Balanced modulator

Question 2: Does this setup have significant shortcomings (beside cost of components?)

Yes. While direct conversion to complex baseband is regularly done in the digital domain, it can be quite finicky to get the same concept working well (and reliably) in the analog domain. It's nearly always easier to do the detection at an intermediate RF frequency.

Question 3: is this observation [about square-wave local oscillators] correct?

Yes, except that you want a bandbass filter that's centered on the carrier frequency, not a lowpass filter. Sometimes this filter has a bandwidth of approximately 2B, in which case it needs to be tuned along with the local oscillator. This sort of setup is called a preselector, and is commonly used in superheterodyne receivers, such as those used for the AM and FM broadcast bands.

But sometimes, a fixed bandpass filter that covers the entire band of interest is used instead. This works as long as none of the unwanted "image" bands ever falls into the band of interest. This is more common in 2-way VHF communications systems, such as those used for air traffic control and public service (police, fire).

Your root problem is likely "First, I tried to apply the sampling and filtering to obtain the modulating signal."

Your first step to obtain the modulating signal should be envelope detection.

You should also be aware that your ratio of carrier frequency to data rate (125k/20k) is fairly low. This is a key issue because the resonant circuits in RFID often have narrow bandwidth. (Due to their high Q-factor)

So check that your resonant circuits (TX and RX) have sufficient bandwidth to pass your data signals without too much distortion! You can do that by modulating switch SW1 in your diagram, increasing from a low rate and then noticing where the received signal power is down to 1/2 (or amplitude down by 1/1.414)

If at all possible, start with a very low data rate, e.g. 1k bits/s. That will make your life a lot easier and quickly provide valuable insight. Then start tweaking up the data rate!