Electronic – UART signal is “rounded”

Edit: The level information was confirmed to be okay (TTL out inverted-> TTL in inverted on an FTDI USB bridge) but I've left it in at the end of this answer in case anyone else needs it.

The easiest way to deal with this sort of troubleshooting is to break it down and confirm the bits are working. An oscilloscope (or logic analyzer) will confirm that the timing is correct and that the signal is not inverted and is the proper levels. That eliminates potential issues with BRG setting, clock source or frequency settings and so on.

If there is a mismatch between the number of start and stop bits or length at receiver and transmitter, then it can show up when characters are sent one after the other with no space, so it can help to space them out (it also makes it easier to interpret the logic analyzer or 'scope images). Not in this case, since it's interrupt-driven but it's even possible the buffer could be overwritten.

I have had good luck using Realterm as a terminal program, also Teraterm and Putty to a lesser extent.

Original guess as to problem (left for historical reasons).

If you are not using an RS-232 level converter, that could be your problem. A typical unit is the MAX232.

As well as shifting levels to appropriate voltages, it inverts the signal.

Most RS-232 receivers will respond to 0/5V signals, but the inverted signal will lead to incorrect data being received.

From the datasheet linked above:

The chip contains charge pumps to get +/- 8.5V signals from a single 5V supply, with only a few external ceramic or electrolytic capacitors.

You could try it out with an inverter if you don't have any of these chips around.

There's more to serial than just the baud rate. RS-232 is a framed protocol. That is, each byte of data is framed by identifying markers - namely the start and stop bits. Only when a valid RS-232 packet delineated by these marker bits has been detected will it be sent to your laptop as a valid byte.

Unless your unknown data stream happens to be RS-232 in its format the FT232 chip won't be able to identify valid packets and thus won't be able to tell you what's going on.

There are other FT* chips available that can work with arbitrary data streams or raw IO pins, and you may be able to find a chip that is a suitable match, but finding one in a simple to use dongle may be more tricky.

You really want something more generic, like a Bus Pirate, but in fact anything which is able to inspect the state of digital signal lines fast enough and convert that into a form the computer can read (USB) would do the job.

I often use a generic microcontroller based board to do the job (I use 80MHz PIC32MX or 200MHz PIC32MZ chips and the chipKIT platform mostly). These boards have the advantage that you can use them for other things as well. Arduino is popular, but the low end ones may be a bit slow for reliable sensing without using fancy tricks (interrupts etc).