Electronic – 3 wires on a (bipolar) DC plug – what are they for

There are two issues with crosstalk between conductors of a cable: capacitive coupling and inductive coupling.

Inductive coupling happens because current flowing thru a wire creates a circular magnetic field around that wire. This also works in reverse. If a wire is subjected to a changing circular magnetic field around it, a voltage will be induced. Therefore, if one wire is carrying a changing current and another wire is close enough so that part of the resulting circular magnetic field also surrounds this second wire, then a voltage is induced in the second wire.

The most effective way to combat this problem is to make sure a equal and opposite return current flows such that the magnetic fields from the two currents cancel out. The best way to ensure this is coaxial cable. Outside the cable, the two equal and opposite currents cancel and there is no net magnetic field.

Inside a multi-wire cable like you describe, this is usually done with twisted pair. At any one point, the two conductors are next to each other. Far enough away the two currents cancel, but up close they don't. Another wire running closer to one wire of the pair will preferentially pick up signal from that wire. This is why the wires are twisted together. A straight wire next to a twisted pair will alternately be nearer one conductor then the other of the twisted pair. Each induces the opposite voltage since the direction of current is opposite in the two wires of the twisted pair. However, these average out to zero over any whole number of twists. In a long enough cable they usually average out well enough.

There is another problem with twisted pair though. Suppose you have multiple twisted pair in the same cable, which is probably the case in the cable you describe. If the twists of one pair are in sync with the twists of another pair, then the induced voltage no longer cancels out in the long run. This is why cables with multiple twisted pair usually have a different twist pitch for each pair. Let's say one pair has 11 twists/foot and another 13 twists/foot. Over any one foot the induced coupled voltage cancells out again. Take a look at the CAT5 cable spec and you will see the different twist pitches for each of the four pairs are carefully specified.

Capacitive coupling is because there is some finite capacitance between every two conductors in the universe. For things far enough apart this can usually be ignored. However, different conductors in a multi-conductor cable are up against each other for a long distance such that capacitive coupling can't be ignored. The best defense against capacitive coupling is a shield, but those are expensive and you say your cable doesn't have any internal shields. Twisted pair again helps here. There will still be capacitive coupling between pairs, but with the right twisting strategy as described above there won't be much preferential coupling to one conductor of the pair. In other words, twisting will cause all the coupling to be common mode with the differential mode mostly cancelling out.

So to finally get to some kind of answer, make sure each signal is carried by a separate twisted pair with the forward and return current of that signal all being carried by that pair. Then treat any signal as differential. Realize there will be common mode noise added to each pair and deal with it accordingly. The external world will also have noise, and due to twisting, this noise will show up as common mode on each pair. 10 base-T and later ethernet, for example, is transformer coupled at each end partly for this reason and also to avoid ground loops.

All that said, I think running stepper motor coil currents thru the same cable as signals is just nuts. This is going to cause trouble. I would seriously look into sending control signals and power for the steppers, but put the stepper drivers close to the stepper motors. That also allows for a local power reservoir cap to make the supply low impedance again after the long cable run. Having the resistance and inductance of the cable between the stepper driver and the stepper motor is asking for more trouble.

Yes you can use normal CAT5 ethernet cables for other things. These are cables with 4 separate twisted pair in them, with a reasonably controlled impedance and a different twist pitch for each pair so that crosstalk will be mostly common mode. Note that this is not using ethernet as Wouter pointed out, only wiring commonly used to implement ethernet. Ethernet itself implies a lot more than just the cable.

Since you have multiple twisted pair in the same cable and these can crosstalk to each other in common mode, you want to send data differentially over each pair. This is also what ethernet does. There are differential line driver and receiver chips out there, which would be a good idea to use if you want to transport individual digital signals from one end to the other. If you have signals that are already inherently differential, like CAN for example, then you use the pairs directly.

Look up what the impedance of CAT5 cable is and try to terminate the ends with that. I know ethernet uses 50 Ω terminations at the receiving end, but I wouldn't just copy that since it may include some other compromises.

One problem of using CAT5 cable with RJ-45 connectors is that it will look like ethernet and someone is going to plug it in to a ethernet port, or plug ethernet into your device. If you are going to use CAT5 cable for something other than ethernet, I recommend to use something other than RJ-45 connectors on the ends.