The entry level DSO Rigol 1052E (the one I own and 100 MHz capable with software change) uses an Analog Devices AD9288. This is a dual channel ADC with 8 bit parallel outputs and samples at either 40 or 100 million samples per second (depending on speed grad of chip). Although the Rigol is a 1 Gig samples per second, so I'm not sure if they are multiplexing these or what exactly is giving them 10x the samples of the single chip.
The AD9288 has bit-per-stage pipeline type converter for the 5 MSB bits and uses a 3-bit flash for the final 3 LSB. This makes sense, as the higher magnitude should be easier to convert fast with pipelines. As your ADC speeds go up, the number of bits sampled via flash conversion will increase, as steven said.
(Stream of consciousness notes and comments)
Well, I/Q is the way to go. Is this 100M Bits or Bytes/Second? There's a big difference. @ 100MBit/Sec you might be able to use an wifi approach. I don't think wifi will support 100MByte/Sec.
If you have to roll your own system, then you'll need to worry about framing, error correction, synchronization, filtering, etc.
Is this intended to be a full duplex system? in other words, do you need to transmit from both sides of the communications link?
You'll need a fair amount of modulation bandwidth, probably 50 MHz or more, so what frequency band are you planning on using, and how far does this need to go. There's been a lot of work in the 60GHz band recently for short distance, and there's plenty of data bandwidth there, that might be something to think about.
Best Answer
There's the sequel High Speed Signal Propagation: Advanced Black Magic.