Electronic – When is a PCIE endpoint/card powered

Looking through the datasheet it looks like the CC6678 can operate in root complex mode so you should be able to connect the two. I did a similar board for testing a large batch of pcie cards.

Lanes

If I were going to try it I'd start with a 4 layer board. You're going to want nice 100Ohm differential pairs to connect your lanes together. It only supports 1 or two of them so that shouldn't be a problem. Just run them over a GND plane on layer two. Just follow good practice here for routing HS signals. Oh and Tx -> Rx, and watch your polarities :)

Clock

I saw the clock generator you picked from your other post, I'd probably put a 1->2 fanout buffer and drive one to each of your pcie slots.

Reset

You're going to need to generate a reset pulse for the cards as usually the mother board would do that. You could just slap down a simple supervisor that monitors 12V and tie it's reset output to both connectors. Maybe add a pushbutton too so you can reset manually during debug. I used a Linear LTC2916CDDB-1#TRMPBF.

Power

You're going to need 12V and 3.3V to power the connectors. I just used a standard desktop pc power supply because I had a bunch and they're easy to get. I just used a standard mother board power supply connector (Molex 44206-0007) and I put a little toggle switch from PS_ON to GND to turn things on and off. Oh and throw in a few 100uF tantalums on each rail for good measure.

Don't be nervous keep the lanes short, match the impedances and you should be fine.

Here's a pinout if you need one:

This sounds a lot like how high-throughput routers are designed. One of the main problems in router design is exactly the "memory matrix" or "cross-bar backplane" that you describe. The large companies in the space (Juniper, Cisco, F5, etc) do not use PCIe for this purpose.

I think a better interconnect might be HyperTransport (which is AMDs solution to the same problem that Intel later developed as QuickConnect for.)

Note that in a PC, the PCIe bus sits "after" a HyperTransport bus, and thus HT has lower latency and higher throughput than PCIe can get to RAM (which in turn sits on the other end of the CPU these days.)