Electronic – How does 3D Time-of-Flight (TOF) camera work

As pointed out in the comments, the TSOP32538 expects a 38 kHz carrier wave. Per page 5 of the datasheet:

This series is designed to suppress spurious output pulses due to noise or disturbance signals. The devices can distinguish data signals from noise due to differences in frequency, burst length, and envelope duty cycle. The data signal should be close to the device’s band-pass center frequency (e.g. 38 kHz) [...] Some examples which are suppressed are: [...] Continuous signals at any frequency.

The table on the same page specifies a minimum burst duration of 6 carrier cycles, which would imply 158 us, but the note in Fig. 1 says this is only a recommendation. Shorter bursts, even bursts of only one cycle, are clearly detectable based on your Receiver Output (the empty spaces in the middles of modulations are long enough that each modulation is interpreted as two bursts).

However, the detection is inconsistent most likely because single pulses or pulses deviating too far from the carrier period may be classified as noise and suppressed. The widths of the pulses shown in the scope pictures vary from 25 to 50 us - it would take a good timing diagram to determine whether there is any pattern as to which durations are being recognized and which are not.

Further considerations:

1. The modulation pulses are too short to modulate more than two cycles of 38 Hz carrier, so even if you could add it, I am not sure it would help much. There appear to be higher-speed infrared receivers outside digikey.
2. Adjacent pulses that are detected are combined into bursts, and single pulses tend to be wider than pair. Even with consistent detection, this will make it difficult to distinguish the modulations, unless you can figure out whether there were one or two pulses based on the output burst width.

If you have no influence on the signal and cannot find a less suppressive model, there are many guides to making IR receiver circuits, though the result will not be nearly as compact.

Your root problem is likely "First, I tried to apply the sampling and filtering to obtain the modulating signal."

Your first step to obtain the modulating signal should be envelope detection.

You should also be aware that your ratio of carrier frequency to data rate (125k/20k) is fairly low. This is a key issue because the resonant circuits in RFID often have narrow bandwidth. (Due to their high Q-factor)

So check that your resonant circuits (TX and RX) have sufficient bandwidth to pass your data signals without too much distortion! You can do that by modulating switch SW1 in your diagram, increasing from a low rate and then noticing where the received signal power is down to 1/2 (or amplitude down by 1/1.414)

If at all possible, start with a very low data rate, e.g. 1k bits/s. That will make your life a lot easier and quickly provide valuable insight. Then start tweaking up the data rate!