I suppose that your CAN devices won't be sending data all the time. Implement a buffer (e.g. a circular buffer) and fill it while you receive data, then send out the data via your serial port.
Once the buffer overflows, make sure you signal that to the host as well.
Compression only makes sense when you know your data. If the data is random, you could use e.g. RLE, which is a quite simple type of encoding and doesn't require much calculation power.
But as I said before, I don't think you will need that.
Per Olin Lathrop's suggestion, I'll expand on bit-stuffing.
CAN uses NRZ coding, and is therefor not happy with long runs of ones or zeroes (It loses track of where the clock edges ought to be). It solves this potential problem by bit-stuffing. When transmitting, if it encounters a run of 5 successive ones or zeros it inserts a bit of the other polarity, and when receiving, if it encounters 5 successive ones or zeroes it ignores the subsequent bit (unless the bit is the same as the previous bits, in which case it issues an error flag).
If you are sending all zeroes or all ones for your test data, a string of 64 identical bits will result in the insertion of 12 stuffed bits. This will increase total frame length to 140 bits, with a best-case frame rate of 874 frames / sec. If the data bits are the same as the MSB of the CRC, you'll get another stuffed bit there, and the frame rate drops to 868 frames/ sec. If the CRC has long runs of ones or zeroes, that will reduce the frame rate even further. The same consideration applies to your identifiers.
A total of 16 stuffed bits will produce an ideal frame rate of 850.3 frames/sec, so you ought to consider it. A quick test would be to use test data with alternating bits, and see what happens to your frame rate.
Best Answer
From Controller Area Network Physical Layer Requirements
Ultimately, the answer to the question is how the CAN protocol is implemented at a physical level. Change that protocol and a higher data rate can be used.
From Understanding Microchip’s CAN Module Bit Timing:
Two masters on either end of the CAN bus must be able to communicate and arbitrate which one has the bus, while each are on the bus at the same time.
If the bus length is 30 m, the time it takes to propagate the signal over the bus is: $$t_{BUS} = 30\ m @ 5.5\ ns/m = 165\ ns$$
Assuming the input comparator delay is \$t_{CMP}\$ = 40 ns and the output driver delay is \$t_{DRV}\$ = 60 ns for all devices.
The round trip time for a bit on the physical bus will be:
$$t_{PROP} = 2(t_{BUS} + t_{CMP} + t_{DRV}) = 2 (165\ ns + 40\ ns + 60\ ns) = 530\ ns$$ $$TQ = 530\ ns/6 = 88.33\ ns $$ $$t_{BIT} = 10\times TQ = 883.3\ ns $$ $$ f = 1/t_{BIT} = 1 / 883.3\ ns = 1.13\ MHz $$
The maximum rate is governed by bus length, line capacitance, connected nodes and the drivers selected by the protocol. In principle at 30 m, CAN (ISO 11898) could do 1.13 Mbps if everything was perfect.
The longer the bus, the slower the data rate. But a shorter bus would mean a higher rate. CAN bit rate vs. bus length:
Both referenced documents go into this at greater length.
CAN (ISO 11898) can transfer up to 8 bytes at 1 Mbps with a protocol based in the 80's. With today's vehicles, there is a need to transfer more data (64 byte packets), but at 1 Mbps, 64 bytes would take up to large of a time slot, possibly delaying vital data.
CAN FD (Flexible Data-Rate) is an extension to the original CAN bus protocol (ISO 11898-1). It is meant to run on existing CAN buses and eventually replace CAN.
The protocol starts out at the CAN 1 Mbps (500 kbps, etc.), with the possible arbitration process between multiple CAN and CAN FD masters, but when the CAN FD master obtains the bus, the data transfer rate accelerates to 5 Mbps to CAN FD slaves. At this rate, 64 bytes can be transferred in less time than an 8 byte CAN 1 Mbps packet. This means there is no timing conflict with existing CAN transfers. Once the CAN FD master gives up the bus, any CAN or CAN FD master can obtain bus.
From CAN FD EXPLAINED - A SIMPLE INTRO (2019):
The true answer depends upon the 1 Mbps arbitration process for a 40m CAN bus, but once the bus is obtained the bandwidth can be accelerated dependent on bus length, line capacitance, number of connected nodes and the drivers. The CAN FD bandwidth is 3-8 times the classic CAN bandwidth.