Electronic – Simple RFID reader – AM demodulator not working

I realize that full RFID is overkill for your application, so I hope you won't mind me linking to pages that are focused on RFID when I think the principles also apply to your system. You might also want to look at Acousto-magnetic tags and other electronic article surveillance tags which I hear are simpler and may be lower cost than full RFID systems.

I'm really just speculating, but I suspect that the controller you disassembled transmits at many frequencies, and each of the tags resonates at some unique frequency. Perhaps something like this:

How do I design the controller antenna ?

Entire books have been written on antenna design.

Have you seen "AN710: Antenna Circuit Design for RFID Applications" ? It's part of the "Microchip microID 125 kHz RFID System Design Guide".

What circuit does the controller use to detect the tag?

Perhaps the controller uses coil-driver and envelope-detection circuit like one of these:

from scanlime: SIMPLEST RFID READER?.

similar circuits: "Simple Low Cost UHF RFID Reader"; "Proximity Security System"; "Arduino based RFID reader"; etc.

When we look at just the inductor+cap tank circuit:

drive ---- coil ---(sense point)--- capacitor ----GND.

The sense point voltage has very large amplitude swings when everything is in resonance (perhaps several times the peak-to-peak voltage of the drive signal), and much smaller amplitude swings when it is out of resonance.

How does it work?

I speculate that the controller you disassembled works in one of these three ways:

frequency vs impedance

The controller puts some signal, perhaps a squarewave, on one side of the coil+capacitor tank circuit, and measures the analog amplitude of the response at the connection between the coil and the capacitor -- perhaps using an envelope-detection circuit as above. After driving the circuit a few dozen cycles at one frequency, pausing, then driving a few cycles at some other frequency, randomly jumping between frequencies until they have all been measured -- or perhaps gradually sweeping the frequency in a chirp -- the controller has measured the analog response at many different frequencies. Since the effective inductance of the coil depends on which coil is close by, each tag (hopefully) produces a unique frequency-vs-response-amplitude graph.

resonance detection

Perhaps the controller has some sort of feedback oscillator with a frequency influenced by the particular tag in the range. The controller occasionally pulses the oscillator circuit to give it a bit of a shove, and then the effective inductance of the coil -- which depends on which tag is close by -- controls the frequency of the feedback oscillator. The oscillator frequency measured by the controller is (hopefully) unique for each tag.

pulse and listen:

Perhaps the controller sends out a sharp pulse or two, exciting the LC resonance of any tag in range, and then tries to measure the frequency of the return signal. The frequency heard by the controller is (hopefully) unique for each tag.

At the very least, you're going to have to put some capacitors across the motor terminals, AND from each motor terminal to its metal case. Even the cheapest toys have such capacitors, or they would get in trouble with the FCC pretty quick. In addition, you're going to have to consider the fundamental frequency of the PWM signals you're feeding the H-bridge, along with all its harmonics. These harmonics will vary from "all odd" harmonics with a 50% duty cycle, to a complex blend of even and odd harmonics near the very low and high duty cycles. Chances are one of these harmonics is at or very close to the frequency your RFID reader is looking for.