How to increase data transmission speed between cc2500 and cc2511

In order to transmit more than 64 bytes, you basically have to fill up the FIFO with the first 64 byte chunk, then check for the TX FIFO emptying. Place the radio in TX mode and as its transmitting, the TX FIFO gets emptied as bytes are sent. Once it reaches a threshold(watermark), you fill up the rest of the FIFO with more packet bytes.

In order to detect the FIFO threashold, you have to configure the device correctly to send large packets as well as the watermark and then either:

1) Poll the device to see the number of bytes in the TX FIFO 2) Configure one of the GDO interrupts to trigger when watermark is reached.

Take a look at this and App Note AN049, which should have an example of how to do this.

Does your hardware works with a lower data rate? Can you connect the two units with a coax cable and two 10dB attenuators (One on each unit) and verify a link at 500kbps? Do you get CRC or framing errors from the devices? Do you have a wifi network on channel 1, bluetooth tranceiver, other 2.4GHz devices near your units?

Edit: Did you check the crystals? They should be within 80ppm of each other. For 2 nodes with 27MHz crystals of the CC2500 it means a maximum of 200Hz difference between the 2 units. The CC25xx requires a maximum 40ppm error from its timing source (Crystal or oscillator). Try generating a CW signal and measure the difference with a spectrum analyzer.

To make measurement of the crystal easier, you can output its frequency/192 to GDO0.