What a coincidence. I'm doing exactly that right now.
There are lots of ways to do this, but controlling the impedance is critical. The problem with using an unshielded ribbon cable is that it is hard to manage the impedance. It can be done, but finding a good cable supplier is hard, especially one that can guarantee the impedance. On the surface this looks like a good solution, but the difficulty of suppliers and stuff makes this less attractive.
Another way to do it is to use a shielded ribbon cable. I don't like this approach because the cable is expensive and terminating the shield is next to impossible.
The Flex-PCB is a good solution, but the NRE is high. If this solution is within your budget it might be the best thing.
A "quick and dirty and cheap" solution would be to use a couple of SATA cables! The impedance is right, they are cheap and widely available, and more importantly they work. Of course you will be limited in the number of signals so don't expect to get more than a PCIe x1 connection plus a reset signal (no power, JTAG, etc.).
The cat6a specifications are for 100m and 10Gb ethernet (so that's fine?)
I think what you're trying to say with this is that if 10G Ethernet can transmit 100 m over Cat6A cable, then it should be possible to transmit 3.2 Gb/s over 50 m with the same cable.
The difference between what it sounds like you want to do and how 10GbE does things is that the Xilinx serial IO, if I recall correctly, outputs a single 3.2 Gb/s serial data stream over a single pair of wires.
10GbE uses several tricks to get the maximum data rate through the longest copper cable.
First, they use all 4 pairs in the Cat6A cable to transmit the 10 Gb/s. That means that each pair is only transmitting 2.5 Gb/s.
Second, they use pre-emphasis encoding to maximize the usable bandwidth of the cable. Basically they enhance the high frequency portion of the transmitted signal. The transmitted signal then doesn't look like a clean data signal. But when its transmitted through the cable, the high-frequency portion is attenuated, and the received signal is closer to the ideal wave shape.
Third, they use error detecting and error correcting codes to allow error-free data transmission even when the cable degrades the signal enough to cause some errors in the raw bit stream.
Fourth, they use a 16-level pulse-amplitude modulation (PAM), instead of simple on-off coding, to send 4 bits of data for every symbol transmitted over the wire.
These last two methods are possible to improve the data rate due to the Shannon Theorem, which says that the maximum possible data transmission rate through a channel is determined both by the bandwidth of the channel and the signal-to-noise ratio in the channel.
I don't think any of this means that what you're proposing is utterly impossible. For example, the 2.5 Gb/s per pair data rate of 10 GbE actually becomes something like 3.125 Gb/s per pair when you include encoding overhead. But doing the PAM encoding to follow the 10GbE model is likely to require a specialized chip for both the transmitter and receiver, and some detailed design work to get it to work.
One possibility is, can you simply packetize this data up and actually send it over a 10 GbE link? That would allow you to use mostly commodity hardware to keep costs down, and also use a "proven" solution to reduce your risks. Some Xilinx FPGAs include a full Ethernet MAC that should enable this solution, but I don't know if its available at the price point you're trying to work at.
Best Answer
The maximum frequency is mostly related to the frequency-dependent loss characteristics of the cable. Eventually you get to a frequency where you simply don't get enough signal at the other end to use.
All of these tend to increase with frequency.
This is why we generally switch to other technologies at very high frequencies: waveguides for microwave radio equipment and optical fibers for high-speed data.