Electronic – How should I test a coaxial cable

The coax cable shielded the signal line between the antenna and the receiving circuit, which is how it reduces extraneous pickup. Note that this does nothing about extraneous signals picked up by the antenna itself.

The reason it attenuated signals from the "antenna" is because it adds some capacitance from the signal line to ground. It looks like what you are calling a antenna is really just a capacitive pickup. The capacitance from the source signal to this pickup will be small, so the small capacitance in the coax cable is large enough to form a significant voltage divider.

I think that the massive reflection from the unterminated far end is masking anything you might expect to see. I also think you might benefit from putting the measurement circuit inside a Wheatstone bridge arrangement. The bridge stimulus will be the driving signal - feed it via (say) 2x 47 ohm resistors to (a) the terminated cable and (b) to a reference impedance to ground.

The reference impedance to ground will be 45 ohms (possibly adjustable to compensate for the basic cable being a few ohms different). Use a differential amplifier to measure the difference voltage between what is applied to the cable (via one 47R) and what is applied to the reference resistor (45R) by the other 47R.

^{simulate this circuit – Schematic created using CircuitLab}

With a perfect far-end termination and perfectly homogeneous cable, there should be zero signal from the difference amplifier. Deforming the cable should produce a small reflection and this should be detectable.