There are two issues with crosstalk between conductors of a cable: capacitive coupling and inductive coupling.
Inductive coupling happens because current flowing thru a wire creates a circular magnetic field around that wire. This also works in reverse. If a wire is subjected to a changing circular magnetic field around it, a voltage will be induced. Therefore, if one wire is carrying a changing current and another wire is close enough so that part of the resulting circular magnetic field also surrounds this second wire, then a voltage is induced in the second wire.
The most effective way to combat this problem is to make sure a equal and opposite return current flows such that the magnetic fields from the two currents cancel out. The best way to ensure this is coaxial cable. Outside the cable, the two equal and opposite currents cancel and there is no net magnetic field.
Inside a multi-wire cable like you describe, this is usually done with twisted pair. At any one point, the two conductors are next to each other. Far enough away the two currents cancel, but up close they don't. Another wire running closer to one wire of the pair will preferentially pick up signal from that wire. This is why the wires are twisted together. A straight wire next to a twisted pair will alternately be nearer one conductor then the other of the twisted pair. Each induces the opposite voltage since the direction of current is opposite in the two wires of the twisted pair. However, these average out to zero over any whole number of twists. In a long enough cable they usually average out well enough.
There is another problem with twisted pair though. Suppose you have multiple twisted pair in the same cable, which is probably the case in the cable you describe. If the twists of one pair are in sync with the twists of another pair, then the induced voltage no longer cancels out in the long run. This is why cables with multiple twisted pair usually have a different twist pitch for each pair. Let's say one pair has 11 twists/foot and another 13 twists/foot. Over any one foot the induced coupled voltage cancells out again. Take a look at the CAT5 cable spec and you will see the different twist pitches for each of the four pairs are carefully specified.
Capacitive coupling is because there is some finite capacitance between every two conductors in the universe. For things far enough apart this can usually be ignored. However, different conductors in a multi-conductor cable are up against each other for a long distance such that capacitive coupling can't be ignored. The best defense against capacitive coupling is a shield, but those are expensive and you say your cable doesn't have any internal shields. Twisted pair again helps here. There will still be capacitive coupling between pairs, but with the right twisting strategy as described above there won't be much preferential coupling to one conductor of the pair. In other words, twisting will cause all the coupling to be common mode with the differential mode mostly cancelling out.
So to finally get to some kind of answer, make sure each signal is carried by a separate twisted pair with the forward and return current of that signal all being carried by that pair. Then treat any signal as differential. Realize there will be common mode noise added to each pair and deal with it accordingly. The external world will also have noise, and due to twisting, this noise will show up as common mode on each pair. 10 base-T and later ethernet, for example, is transformer coupled at each end partly for this reason and also to avoid ground loops.
All that said, I think running stepper motor coil currents thru the same cable as signals is just nuts. This is going to cause trouble. I would seriously look into sending control signals and power for the steppers, but put the stepper drivers close to the stepper motors. That also allows for a local power reservoir cap to make the supply low impedance again after the long cable run. Having the resistance and inductance of the cable between the stepper driver and the stepper motor is asking for more trouble.
If you already have a ground connection with one or more other wires, then use the otherwise unused wires to make a more solid ground connection.
If there is otherwise no ground connect, like with ethernet for example, then I agree with Keelan in that you shouldn't create one. That could cause trouble with ground loops. In that case, either leave the extra line floating or connect it to ground at one end only.
Best Answer
What I'd probably do is to use the shield for one connection (ring) and the put the two inner conductors in parallel for the tip connection.
It costs you nothing to use both inner conductors except for stripping and soldering two wires instead of one. It gets you lower resistance, which might be needed in a long cable. It also gets you a little redundancy - if one wire breaks the other is still connected.