Electronic – 9-pin micro-d connector for LVDS signals

LVDS signals are generally incompatible with transformers/magnetics. The Texas Instruments application note does not have any examples of this sort.

Here is a similar discussion, see this Xilinx thread.

The LVDS will likely work over 100ft of RJ45/CATwhatever connection, you just need no magnetics and correct termination scheme on driver and receiver sides. However, hot plug/reconnect could be problematic due to possibility of cable charge up and ESD killing your driver/receivers.

I think you are getting somewhat off track.

In comments you say you don't know the signal encoding and it may have some DC (or very low frequency content). In that case, a simple series capacitor termination will block some of your desired signal, probably causing worse distortion than what you are encountering from the cable loss.

How would I size them, a decade below signaling frequency?

If you did this, you would be basically trying to make the filter not affect any of your signal frequencies. It wouldn't have any equalizing effect on the frequencies you are actually using.

Rather than just a series capacitor, maybe try a simple de-emphasis filter:

^{simulate this circuit – Schematic created using CircuitLab}

The resistors provide some attenuation (de-emphasis) for low-frequency signal content, while the capacitors let high-frequency content (whatever has made it through the transmission line) pass through.

Appropriate values for the filter capacitor and resistor depend on the characteristics of the transmission line between the transmitter and receiver as much as they do on the characteristics of the receiver or the signal. You'll have to experiment (or simulate) to find the best values for your situation.