This sounds like it should be easy to diagnose. You say the signals on the RS-485 bus look OK, so check the next point. Keep following the signal until you find it not right somewhere.
The next logical point to test after the RS-485 bus would be when it gets converted back to a single-ended signal that presumably gets presented to UART receive pin of the device. Does that look right? If not, there is a problem with the RS-485 receiver circuitry. If so, then either the device is not working or you have a protocol problem.
In general, it's good to keep dividing up the suspect parts of the system to isolate what is not working, or to test them in isolation. For example, you could either try to tap the UART line into the device and see if you get what is expected by looking at the bytes on a PC with a low level program, or you could try a program on the PC to inject what you think is the right sequence of bytes and see if the device responds as you expect.
Chances are that if the UART signal at the device looks OK (including the obvious issue of baud rate), then you have a protocol problem. In other words, somehow you're not sending the right data to the device. Early on in bringing up a system like this, I often use a low level test program on the PC that sends and receives individual bytes. I can type byte values in decimal, hex, and as ASCII character, and it displays everything it receives likewise. This can be very useful for verifying that you really do understand the protocol the device expects.
Do some of these experiments and tell us what you find. It would also help to provide some information on this device you are trying to talk to from the micro.
You can find the required voltage difference at the sending and receiving ends in the RS485 standard. As a second best, read the specs of some RS485 transceiver.
What is probably the problem in your project (but it is difficult to reconstruct the original signal after a half-garbling filter) is that the common of your sender and receiver differ too much. That is, there are two requirements for successful RS485 communication:
- the voltage difference between D+ and D- must be OK. This is probably not your problem.
- the voltage difference between D+ and ground, and D- and ground must be in the range acceptable by the receiver. If they are not, you have a ground potential difference problem. To solve it, you can either reduce the difference, or find a receiver that can tolerate a higher difference.
The voltage levels for RS485, from The RS485 design guid :
Note that this picture is about the differential voltages, it omits the ground connection.
Best Answer
You have basically two questions in one.
1. Common ground
With 1000m between the nodes it is practically unavoidable to have ground potential out of transceiver's common mode range at some point. Therefore, you need at least one more conductor in your wiring for common ground. This, however, creates ground loop problems. So, you have to make environmental assessment first:
a) If this is permanent indoor installation then TIA/EIA-485-A recommended solution is to connect common ground wire to local ground at each node with 100 Ohm resistor.
b) If this is outdoor installation and especially if hot-plugging of the network nodes is expected, then isolated transceivers should be used, with common wire grounded at one side only.
I strongly recommend reading the RS-485 Design Guide by Texas instruments. Unfortunately, TIA standards are not free, but here you can find very good overview of the key points in TIA-485.
2. Power Supply
I am not sure what you mean when you say "ensure a 5v ground on both ends". A ground is a ground and 5V is 5V. In this context 5V is a supply voltage referenced to the ground. Since you have to run common ground wire between the nodes anyway (see above), an obvious solution would be to add one more conductor with power supply for all the nodes. But this is not a practical solution. The voltage drop and power losses over 1000m will be huge.
So, now again you need to make an assessment of how your network will be actually used:
a) If you go with non-isolated network (case 1.a above) and your nodes require very little current (like various sensors, for example) then one common solution is to have higher voltage power supply line along with data and ground wires. Inject 12-24V at single point and then use POL converters to bring it down to 5V at each node.
b) If you go with non-isolated network but your nodes require local power supplies anyway, then there is no point in added costs of central power distribution. Simply power nodes and their transceivers from local power supplies.
c) In case of isolated network (case 1.b) use isolated transceivers with built-in power converters, like MAXM22511 for example.