There are two issues with crosstalk between conductors of a cable: capacitive coupling and inductive coupling.
Inductive coupling happens because current flowing thru a wire creates a circular magnetic field around that wire. This also works in reverse. If a wire is subjected to a changing circular magnetic field around it, a voltage will be induced. Therefore, if one wire is carrying a changing current and another wire is close enough so that part of the resulting circular magnetic field also surrounds this second wire, then a voltage is induced in the second wire.
The most effective way to combat this problem is to make sure a equal and opposite return current flows such that the magnetic fields from the two currents cancel out. The best way to ensure this is coaxial cable. Outside the cable, the two equal and opposite currents cancel and there is no net magnetic field.
Inside a multi-wire cable like you describe, this is usually done with twisted pair. At any one point, the two conductors are next to each other. Far enough away the two currents cancel, but up close they don't. Another wire running closer to one wire of the pair will preferentially pick up signal from that wire. This is why the wires are twisted together. A straight wire next to a twisted pair will alternately be nearer one conductor then the other of the twisted pair. Each induces the opposite voltage since the direction of current is opposite in the two wires of the twisted pair. However, these average out to zero over any whole number of twists. In a long enough cable they usually average out well enough.
There is another problem with twisted pair though. Suppose you have multiple twisted pair in the same cable, which is probably the case in the cable you describe. If the twists of one pair are in sync with the twists of another pair, then the induced voltage no longer cancels out in the long run. This is why cables with multiple twisted pair usually have a different twist pitch for each pair. Let's say one pair has 11 twists/foot and another 13 twists/foot. Over any one foot the induced coupled voltage cancells out again. Take a look at the CAT5 cable spec and you will see the different twist pitches for each of the four pairs are carefully specified.
Capacitive coupling is because there is some finite capacitance between every two conductors in the universe. For things far enough apart this can usually be ignored. However, different conductors in a multi-conductor cable are up against each other for a long distance such that capacitive coupling can't be ignored. The best defense against capacitive coupling is a shield, but those are expensive and you say your cable doesn't have any internal shields. Twisted pair again helps here. There will still be capacitive coupling between pairs, but with the right twisting strategy as described above there won't be much preferential coupling to one conductor of the pair. In other words, twisting will cause all the coupling to be common mode with the differential mode mostly cancelling out.
So to finally get to some kind of answer, make sure each signal is carried by a separate twisted pair with the forward and return current of that signal all being carried by that pair. Then treat any signal as differential. Realize there will be common mode noise added to each pair and deal with it accordingly. The external world will also have noise, and due to twisting, this noise will show up as common mode on each pair. 10 base-T and later ethernet, for example, is transformer coupled at each end partly for this reason and also to avoid ground loops.
All that said, I think running stepper motor coil currents thru the same cable as signals is just nuts. This is going to cause trouble. I would seriously look into sending control signals and power for the steppers, but put the stepper drivers close to the stepper motors. That also allows for a local power reservoir cap to make the supply low impedance again after the long cable run. Having the resistance and inductance of the cable between the stepper driver and the stepper motor is asking for more trouble.
If both devices have their own ground reference then you should only attach the shield at one end. This way you do not produce what is called a ground loop.
When you do attach it I would attach it to the chassis as this is the return path you want noise to take, the chassis to earth, if you attach the device that I assume is inside the metallic case and then that attaches out to the earth reference you are bring noise inside the box that you do not need to.
If one device is operating lets say PoE and has no earth shield reference you want its shielding/chassis connected to the shield to act as its connection to earth for noise mitigation.
Best Answer
You can treat the 4- twisted-pairs inside a standard CAT-5 cable as simple unshielded twisted pairs for audio applications. If both the sending end and receiving end are fully balanced, you don't need to do anything else. You can try using the 4th pair as a ground conductor but even that is not normally necessary.
On the other hand: if either the sending end or receiving end are un-balanced, you will need to play around to see what works for you. I would normally insert audio balancing transformers at the receiving end - this works well even in the presence of significant common-mode noise.
I use CAT-5 cable to connect RTS Adam Intercom Panels from remote locations back to the Intercom Frame. A standard Adam analog port uses 3- pairs: balanced audio to the frame, balanced audio from the frame, RS-485 data for key-panel data. Even with cable runs that exceed 2000 feet, I find that I don't need a ground conductor between the Intercom frame and the key-panel. My concern, of course, is keeping the RS-485 data transceivers operating within their common-mode range. Just hasn't been a problem - ever.
In your situation (short cable run within a single room: I wouldn't expect any problems whatsoever. This assumes that you are talking about balanced audio signals. I'm making that assumption because you are talking about using standard mic cables with XLR connectors.