Electronic – CAN bus layout design

This is a difficult one to answer, mostly because RF and EMI are so incredibly non-intuitive. One might say that if someone claims to understand EMI then they most certainly do not understand EMI. I do not claim to completely understand EMI. I know a lot about it, but I have some holes in my knowledge. Consider that when reading my answer.

My main concern is that LVDS, and really any other differential signaling method that does not use isolation transformers, is not perfectly differential. There are mismatches in the differential drivers that cause common mode "noise" on the diff-pair. This common-mode noise also has a signal return path, which would be on the GND or shield in this scenario. The problem with having the shields disconnected at one end is that this signal return path would be on the power cable-- causing a huge loop area and resultantly huge EMI. While the common mode noise return current is small, the loop area his large, and so this must be accounted for in the design.

In one design of mine, I ran some 2.5 GHz signals over an 18" SATA cable. For those who don't know, a SATA cable has two diff-pairs in it and two shields. Both shields are connected together at the ends. There are no GND wires in the cable other than the shields. In my design, the shields were connected to signal GND at both ends. This design worked great, and is in volume production right now. It complies with FCC Class B, and the equivalent CE version, for electro-magnetic-compliance including radiated emissions, RF susceptibility, and ESD susceptibility.

Going on with the SATA comparison, all SATA motherboard/drives connect the shields at both ends, and they work fine at high speeds. SATA cables are available in length of about 6 inches to 2 feet-- similar to what the OP is using. Systems with SATA meet the more stringent EMC regulations. And they are shipped in the tens to hundreds of millions of units per year.

Were I designing this system, I would connect the shields at both ends. There are millions of modern systems that show this works.

Yes, use twisted pair. Failing to do so for a differential signal is self-defeating. Don't forget to include 120-ohm termination resistors at each end of the transmission line.

The less connectors in any transmission system the better. Connectors are almost guaranteed to present an impedance discontinuity, and hence will cause reflections.

Transmission line stubs of any length are also a source of reflections and (if I'm reading your description correctly) you have transmission line stubs of several metres. The longer the stub, the worse the impact of the reflections.

Reflections are bad because they can cause destructive interference, i.e. they can corrupt any transmitted data.

I'd use a longer primary (twisted pair) cable, less connectors and shorter stubs. Run the primary cable as close to each node as you can, even if it means that the cable has to be longer.