What's the data rate of your RS-485? Since your cable isn't twisted pair you may have to limit the speed to 100kbps and use short cables.
But IMO the mismatched impedance isn't the real problem.
"The characteristic impedance of an RS-485 cable is 120 ohm according to the standard, but cables designed for high frequency operation usually has an impedance between 135 and 165 ohm, which limits the capacitance to approximately 30 pF/m." (source).
You don't mention voltage or current for the motor's power, but at 200W both may be too high to avoid trouble. A high current will inductively couple with the data wires, a high voltage will do the same capacitively. I expect you may see serious disturbances.
The cable just doesn't seem to be fit for the application. It should have been a double twisted pair at least, preferably with different twist rates.
edit
All will depend on how the motors behave and what your data look like. Have you had problems yet with the deployed cables? Eight noisy amperes coupled over a long stretch doesn't sound good. Bad looking waveforms in itself are not necessarily a problem if the receiver can properly detect edges. What are you sending, UART format? If you have continuously glitches sooner or later they might be seen as a start bit, and you'll have a ghost byte in between your actual data. Not only the data format is important, also what happens at the higher protocol levels. Is there a checksum or CRC? Acknowledge? Can faulty received data be retransmitted? If you would encounter communication problems I think this is the level where you can attack them.
It's also possible that for the installed cables no problems are reported at all. The 57kbps may help here. And maybe there is an error detection/correction mechanism in the protocol.
Plant leakage is a well known effect - The system of cables and connectors in distribution systems is known as "plant" for some reason - and causes signal to leak out and be received via antennas.
In an ideal coax, the wave travels between the conductors and imposes local current eddies and voltage nodes along the length of the transmission line, but most notably on the inner surface of the outer conductor and on the outer surface of the inner conductor (the skin effect). Again ideally, no signals that travel on the outside of the outer conductor can interact with the signals on the inner surface of the outer conductor.
Plant leakage arises from the divergence from this ideal, when you use wound or wrapped coax, poor connector termination and also mismatched termination loads.
In your drawing above, assuming the termination also is properly shielded, the return signal should travel along the inner surface of the outer connector if the system is ideal. So you shouldn't actually see any radiation. Evidence of this is how non-coaxial waveguides (TEM- mode) work - everything flows along the inner surface and there is very low leakage.
If this is beyond a hypothetical situation and you are seeing leakage, it can arise from the quality of the coax, if it is solid foil or braided, and that depends on the frequency you're driving it at too (wavelength vs. mesh size). Connector shield continuity, termination matching all are radiation points. Also, the attached eqt. on the receiving end may inject a current mode return signal into the grounded shield causing the whole outer surface to act as an antenna and if this is a signal derived from the feed signal (amplifier power rail bounce as one example) then it might appear that the coax is leaking.
In general cable television plant leakage arises because of the less expensive cables they use (they have many many miles/KM installed cost really matters), the difficulty in maintaining a connector over many years and damage, nicks and knicks etc.
There are some systems that use gas dielectric and solid conductors that transport 10's of MW to antennae with almost no leakage so ideal performance is approachable.
Best Answer
While often shielded, CAT5 can be of the UTP type, meaning unshielded twisted pair. The "true" RS-485 cable you link to has two twisted pairs and a shield. If I remember right, CAT5e (and above) has a shield, at least most cables I have seen have - the exact standard can vary. I guess those would do the job just nice. Just make sure you use the twisted pair like in the original RS-485 cable (your data sheet looks like [1 White/Orange Stripe] and [2 Orange/White Stripe] are a twisted pair, and [3 Blue/White Stripe] is a third, untwisted wire.)
There seems to be a small impedance mismatch (100 Ohms for CAT5, 120 Ohms for RS-485). This will cause reflections at the driver and at the receiver, but I am pretty sure your application will still work. While you may read that 120 Ohms is typical for RS-485, the termination network uses 120 Ohms between the differential pair and 2 * 680 Ohms to VCC and GND.
Source
Thus, the value of the termination that the cable "looks into" is smaller than 120 Ohms anyway: (120||(680+680)) Ohms = 110 Ohms.
If you have the chance to test, you could go for it. I would probably not even hesitate and use STP CAT5(e). If your contract says you pay money for every minute your installation fails, you probably want to use cables with the proper specification. (The latter will still not make sure nothing ever fails, but you're probably in a better position if you can blame it on the "true" cable instead of having someone else blaming your "wrong" cable. But you see how this last paragraph has nothing to do with physics...)