Here is how I would suggest going about debugging this problem:-

1. Check the connections.. if you've mixed up signal ground and chassis ground (shield) you could get funny behavior just like you describe. Pin 7 on the DB-25 is the one you want to connect to DE-9 pin 5. I'll assume you're using a commercial adapter/cable or have one wired like this.

   

   Maybe check the transmit out line to ensure it's idling at -12 ( < -3 anyway).

2. Make sure the instrument is set to RS-232, and not current loop (Jumper 7,8 and 1,2).

3. Try the most common serial port setting (9600/8-N-1), 9600 baud, NO parity, 1 stop bit switch 1, 2, 3 OFF switch 4, 5 ON switch 6 ON switch 7, 8 ON

   Make SURE you understand which is 'OFF' and which is 'ON'. Don't laugh, I've seen this happen more than once, and some types of DIP switches are kind of ambiguously marked. If in doubt, an ohmmeter across the switch.. in fact you could check all of the switches to ensure they're working.

   I like Realterm as a terminal program. It's free, and easy to change the parameters. Try it on something known working first of course, and look for an online tutorial. Really worth it.

4. If still gibberish after that, get a scope screen cap or photo and post it.

5. If you have a continuing need for a logic analyzer, this one is very cheap and relatively useful. Not as good as my Tektronix boat anchor, but this one will fit in a coat pocket. But I'd get a scope first if you don't have one. Even the worst scope is better than nothing.

Update

As I originally predicted, the wiring is proprietary and the flow control signals are a mess. @SofaKng has reversed engineered the official cable and produced this table (reproduced here):

RJ45 pin,   DB9 pin (female)
----------------------------
  1           1
  2           6 + 8
  3           2
  4           5
  5           5
  6           3
  7           4
  8           7

Original

You missed a key detail. Their RJ-45 patch cable is null modem wired. That means it is reversed. You got all of your pairs backwards (you connected TX to TX, RX to RX, etc...)

Assuming you are using a straight-through patch cable...

• as you indicated that you are with your comment "standard TIA-568B"
• you must use TIA-568B ordering on both ends of your patch cable

...it goes this way:

Device Signal, RJ-45 Pin#, DB9 Pin#, PC Signal
----------------------------------------------
> DTR 1 --- 6 DSR   
> GND 2 --- 5 GND
> RTS 3 --- 8 CTS
> TxD 4 --- 2 RXD
> RxD 5 --- 3 TXD
> DSR 6 --- 4 DTR
> GND 7 --- 5 GND <-- note: repeats, bussed to device #2
> CTS 8 --- 7 RTS
> RI  9 --- 9 RI  <-- doesn't actually fit in an RJ-45 (only 8 signals), probably safe to ignore if I read their diagram correctly

The DB9 on a PC is pinned out this way (see figure). Note how it is null-modem reversed from the table listing you provided. That's because their RJ-45 cable is null-modem reversed to cancel it out. Tricky and silly, but that's how they chose to implement it.

Some other thoughts...

Actually, I noticed from tracing in the diagram that they do some very strange stuff with the flow-control signals. For example, they short CTS and DSR on the PC side, but not on the other side. They route RTS on the PC side to DSR on the device side. And other weirdness.

This may be bad documentation, but I suspect they have implemented custom firmware/software that makes use of the flow-control signals in non-standard ways as a means of ensuring that you only buy and use their cables and adapters.

I would suggest that you make two half cables. On one end go RJ-45 to unterminated wire and the other go DB9F to unterminated wire. Then you can twist your way through all of these weird configurations until you get it right. I would start with my suggested mapping. If that doesn't work, report back and I'll give you my mapping for all of their weirdness in the flow-control lines.

Good luck! =)