Electronic – Terminating shielded/screened twisted pair correctly

Simple Answer: No. It won't work. Not even close.

Long Answer:

There is a lot more to Ethernet, Cat-5 cables, and networking than just having wire that connects Point A to Point B.

Let's start with Ethernet. You have not mentioned what kind of Ethernet you are talking about. The commonly used 100Base-T Ethernet (a.k.a. 100 megabit Ethernet) uses 2 pairs of wire (4 conductors total). 1000Base-T (a.k.a. Gigabit Ethernet) uses 4 pairs of wires (8 conductors).

Cat-5 cable is made from 4 pairs of wire, with each pair twisted in a very specific way. Each pair is twisted at a different rate from the other pairs, to control the crosstalk between pairs. The insulation dielectric and thickness is carefully controlled, along with the twist rate, to control the impedance between the wires. Cat-5 (and Cat-5e, and Cat-6, etc.) has very specific specifications that must be met to be called Cat-5.

What you are calling "12 Core Coax" does not appear to be coax at all, using the traditional meaning of the word "coax". Rather, it appears to be normal 12-conductor cable with a wire braid shield. The link you provided has no specifications on things like twist rate, impedance, crosstalk, capacitance, bandwidth, etc.

Your first question, "Can I use this 12-conductor cable for Ethernet?", must be answered with a no. We have no way to compare Cat-5 with the cable you want to use. Your cable has no specs, while Cat-5 has very specific specs.

Your next question, "Can I fit two Ethernet links into 12 conductors?", is a definite maybe. I am, of course, ignoring the signal quality of that cable and only looking at the number of conductors. 100 Mbps Ethernet requires 4 conductors. 4+4=8. 8 <= 12. So yes, you can put 100 Mbps on 12 conductors. But Gigabit Ethernet requires 8 conductors. 8+8=16. 16>12. So there is no way to put two Gigabit links on a 12 conductor cable.

Next, you mention using some of the "cores as a common earth". Ethernet is transformer isolated at both ends of the cable and thus does not have a "common earth". If you want to get super technical about things, each pair of wires is in essence its own common earth. But don't read too much into that because you'll just get more confused.

I know what you're probably thinking at this point. You're thinking that you are only running slow-old-100-Mbps-Ethernet and all of this impedance controlled, twisted stuff can't be that important. And that you'll just chance it and run 100 Mbps Ethernet over the "12-core Coax" and hope for the best. I have talked to many people in this same position. People who ignored the advise of experts (not just me) and tried to run Ethernet over non-standard cables. All of these people have deeply regretted that choice-- in some cases they "lost" thousands of dollars in rewiring costs, and in other cases they lost their jobs. Don't do it!

But here is one solution that might work for you (and I have actually done this with success): Use standard Cat-5 or Cat-5e cable and run two 100 Mbps links over it. Two pairs for the first link, two more pairs for the second link. To make things better, Cat-5/5e cable is probably a lot cheaper than that 12-conductor stuff you found. Just make sure that when you select the wires for each pair of signals that you actually select pairs of wires! Just taking a Cat-5 cable and not paying attention to which wires are twisted together is guaranteed failure.

This is a difficult one to answer, mostly because RF and EMI are so incredibly non-intuitive. One might say that if someone claims to understand EMI then they most certainly do not understand EMI. I do not claim to completely understand EMI. I know a lot about it, but I have some holes in my knowledge. Consider that when reading my answer.

My main concern is that LVDS, and really any other differential signaling method that does not use isolation transformers, is not perfectly differential. There are mismatches in the differential drivers that cause common mode "noise" on the diff-pair. This common-mode noise also has a signal return path, which would be on the GND or shield in this scenario. The problem with having the shields disconnected at one end is that this signal return path would be on the power cable-- causing a huge loop area and resultantly huge EMI. While the common mode noise return current is small, the loop area his large, and so this must be accounted for in the design.

In one design of mine, I ran some 2.5 GHz signals over an 18" SATA cable. For those who don't know, a SATA cable has two diff-pairs in it and two shields. Both shields are connected together at the ends. There are no GND wires in the cable other than the shields. In my design, the shields were connected to signal GND at both ends. This design worked great, and is in volume production right now. It complies with FCC Class B, and the equivalent CE version, for electro-magnetic-compliance including radiated emissions, RF susceptibility, and ESD susceptibility.

Going on with the SATA comparison, all SATA motherboard/drives connect the shields at both ends, and they work fine at high speeds. SATA cables are available in length of about 6 inches to 2 feet-- similar to what the OP is using. Systems with SATA meet the more stringent EMC regulations. And they are shipped in the tens to hundreds of millions of units per year.

Were I designing this system, I would connect the shields at both ends. There are millions of modern systems that show this works.