Your problem is likely in using the wrong baudrate: 112500 is most likely a mistake, the standard rate in that range is 115200 (a search of the manufacturer's website for this number finds many hits, but none for 112500). You could also not be producing the programmed baud rate on one end or the other due to divider granularity; sometimes changing the oversampling of the UART can help. EDIT: Specifically, a value of 8 (divisor of 9) will get you 111111 baud from 8MHz if you set the "double speed" mode bit.
Secondarily, you have a problem such as lack of a common ground between the boards, or haven't grounded the scope to them, thus resulting in the distorted waveform. It's not clear yet if that is what the receiver sees, or if it's merely a measurement mistake in applying the scope.
Additionally, have you verified that the un-named external board also runs without a serial level translator? Most modular inter-board serial communications is at RS232 levels and logically inverted from the logic-level signals, though there are exceptions.
I would guess that it could be lack of supply stiffness with the MAX device.
Are the pump caps 0.1 uF as shown.
And what cap is on the 3V3 rail and how far away is it from the IC.
If the power supply is not stiff enough either because the "pump" lacks energy or the source rail is noisy then something like you are seeing could happen.
Pin 2 and pin 6 (V+/V-) should have some ripple at the pump frequency but no signal frequency noise to speak of.
What supply voltage are you actually using?
What is the drive voltage on the low voltage side?
It may well not matter but the TI data sheet in figure 4 bottom of page 7 shows differing cap values for different supply voltages.
The Maxim data sheet shows the same values in table 2 at the top of page 12. Maxim say these are minimum capacitor values. TI says that the same values are typical values.
Note that the spurious output is unipolar - it's not that it's low - the scope ground is at half waveform height and the proper signal is ~+/- 5V, as it should be, but in spurious mode it's -5/0V. This implies that the + side pump isn't working or that Murphy has proclaimed a bank holiday. I'd start by checking voltages on C3 & C4, supply capacitor size and correct capacitor values.
Best Answer
100 ohm twisted pair is a poor match for rs232 drivers (which are about 1000 ohms) , the main limitation is the drivers capability to charge and discharge the line.
In my experience on long runs the main problem is dielectric absorbtion during the idle time causing the start bit to be distorted, this can be mitigated somewhat by fitting a diode at the receiver end to limit the negative voltage
So what you want most is low output impedance. voltage levels are less important.
Here the max3232's 35mA short-circuit current beats the max232's 10mA soundly.