There is a small group of us working on Amazon Web Services (AWS), we use mac books as our desktop machines and VNC into servers running on AWS. The problem started when we upgraded our VPN from PPTP to L2TP (Centos 6 with Openswan). My colleague who set it up had no problems with the new VPN. However, within a few minutes of connecting my VNC viewer sessions went dark and reported:
RFB Protocol Error: Bad hextile data
or
RFB Protocol Error: Bad xrle data
and other of the encoding protocols used by RFB (Remote Frame Buffer). This continued every minute or so depending on activity, with a delay while it corrected itself of up to 10 seconds. Clearly unusable. I went back to the PPTP VPN and continued without problem. Yesterday however, we connected another colleague to the L2TP VPN and he saw the same issues as me, so now we have to fix it.
The only discernible difference was that the two of us having problems with the VPN have mid-2014 15-inch Mac Book Pro with retina display (bought early 2015 with Yosemite) whereas my colleague that does not have this problem has a Mac Book Air (13-inch, Mid 2013) upgraded to Yosemite.
Searches for RFB protocol error have turned up nothing and other than that I don't even know what to search for or even ask about.
Any help appreciated.
Best Answer
The most likely cause of this setting if you are using a VPN would be an MTU size set too large, causing packet fragmentation.
When this occurs some protocols will encounter problems, and is known to cause some issues with the RFB protocol.
If you have the PV AMI, then it may be worth testing if adjusting the MTU on both sides of the connection to a lower value would prevent problems.