Electronic – Distortion due to transmission line effects of low pass filtering

You are misapplying the resistor.

The problem is that the input of the receiving chip has a higher impedance than the characteristic impedance of the transmission line. Putting in a series resistor at the destination won't help much with the initial problem. Instead, you should use a shunt resistor to ground.

If you want to use a series resistor, it needs to be at the sending end. Because the driver has a low impedance relative to the transmission line, you get a discontinuity there too, and an inverted reflection results. If you put a series resistor at the driver matching the line impedance, then that is absorbed, and you eliminate the ringing caused by back-and-forth propagation of inverted reflections which is so evident on your scope.

An interesting case is a series resistor at the driver, and a high impedance receiver input without a terminator. Initially, the series resistor and line impedance form a voltage divider, and only half the applied step goes down the line. When it gets to the "open" end formed by the receiver, it reflects in phase, doubling the voltage back to the original. When the reflection gets back to the transmitter, it is absorbed by the receiver. Gradually the line voltage floats up to the drive voltage, as the situations becomes more like DC than like a step function.

I'd consider doing it digitally because the cable losses won't affect the analogue signal. You'd need 16 bit resolution - this gives a resolution of 15 \$\mu V\$ in 1 volt and you'd need to sample at 250k Sps to give your DAC filter a chance to reproduce 100 kHz.

Keeping it as tight as possible means probably a 20 bit word where the first 4 bits are a constant header so that you can recognize where the MSb is going to be. You could just about do it with 18 bits but 20 bits will play safe.

So it's 20 bits sent 250,000 times per second - that's a data rate of 5 Mbps - how far can you get this down decent coax? The sort of coax I've used in the past has transmitted 40M bpsec for 350m so maybe at one-eighth the data rate you'd get 8 x 350m = 2.8 km. The coax was 12mm diameter stuff and 75 ohms but I can't remember the part number - I know I went for the "fattest" coax I could fit in the cable glands and hoped it would be OK (and it was).

Looks like a couple of repeaters would help at 2km and 4km. Screened twisted pair might be even better than coax. Try looking up twinax.

Sending it analogue will be a problem unless you are prepared to suffer degradation of the signal in the frequencies above 100 kHz as the cable characteristic impedance changes from resistive-capacitive to purely resistive - you could try pre-emphasis and de-emphasis to help. Noise will surely be a problem over that distance and maybe someone else who has tried this can offer some advice. External pick-up is an unknown quantity but at least for digital you have a threshold that noise has to exceed.