Electronic – Calculate sum check

It sounds like you're using your XBee in 'data mode', which simulates a serial (stream) connection. You could be observing missing bytes for a number of reasons, including actual dropouts (though the XBee should account for that with retransmits) or bugs in the transmitting or receiving code, or simply a failure to read data from the XBee serial port fast enough.

XBees also typically have a packet-based API mode, however, which lets you send individually addressed packets of data to receivers, with variable acknowledge, retransmit, etc, settings. This lower level protocol is intrinsically packet based, so there's no need to implement your own packetization, though it may be worthwhile to add an application-level checksum such as a CRC to verify the integrity of your data.

There are a number of different XBees and XBee clones out there; check the manual for your particular model for API documentation.

How do you (or would you) design your systems protocol?

In my experience, everyone spends a lot more time debugging communication systems than they ever expected. And so I strongly suggest that whenever you need to make a choice for a communication protocol, you pick whichever option that makes the system easier to debug if at all possible.

I encourage you to play with designing a few custom protocols -- it's fun and very educational. However, I also encourage you to look at the pre-existing protocols. If I needed to communicate data from one place to another, I would try very hard to use some pre-existing protocol that someone else has already spent a lot of time debugging.

Writing your own communication protocol from scratch is highly likely to slam against many of the same common problems that everyone has when they write a new protocol.

There's a dozen embedded system protocols listed at Good RS232-based Protocols for Embedded to Computer Communication -- which one is the closest to your requirements?

Even if some circumstance made it impossible to use any pre-existing protocol exactly, I am more likely to get something working more quickly by starting with some protocol that almost fits the requirements, and then tweaking it.

bad news

As I have said before:

Unfortunately, it is impossible for any communication protocol to have all these nice-to-have features:

• transparency: data communication is transparent and "8 bit clean" -- (a) any possible data file can be transmitted, (b) byte sequences in the file always handled as data, and never mis-interpreted as something else, and (c) the destination receives the entire data file without error, without any additions or deletions.
• simple copy: forming packets is easiest if we simply blindly copy data from the source to the data field of the packet without change.
• unique start: the start-of-packet symbol is easy to recognize, because it is a known constant byte that never occurs anywhere else in the headers, header CRC, data payload, or data CRC.
• 8-bit: only uses 8-bit bytes.

I would be surprised and delighted if there were any way for a communication protocol to have all of these features.

good news

What are the other possible techniques/solutions exist to address the problem?

Often it makes debugging much, much easier if a human at a text terminal can replace any of the communicating devices. This requires the protocol to be designed to be relatively time-independent (doesn't time-out during the relatively long pauses between keystrokes typed by a human). Also, such protocols are limited to the sorts of bytes that are easy for a human to type and then to read on the screen.

Some protocols allow messages to be sent in either "text" or "binary" mode (and require all possible binary messages to have some "equivalent" text message that means the same thing). This can help make debugging much easier.

Some people seem to think that limiting a protocol to only use the printable characters is "wasteful", but the savings in debugging time often makes it worthwhile.

As you already mentioned, if you allow the data field to contain any arbitrary byte, including the start-of-header and end-of-header bytes, when a receiver is first turned on, it is likely that the receiver mis-synchronizes on what looks like a start-of-header (SOH) byte in the data field in the middle of one packet. Usually the receiver will get a mismatched checksum at the end of that pseudo-packet (which is typically halfway through a second real packet). It is very tempting to simply discard the entire pseudo-message (including the first half of that second packet) before looking for the next SOH -- with the consequence the receiver could stay out of sync for many messages.

As alex.forencich pointed out, a much better approach is for the receiver to discard bytes at the beginning of the buffer up to the next SOH. This allows the receiver (after possibly working through several SOH bytes in that data packet) to immediately synchronize on the second packet.

Can you point to the cons in the above list which can be easily worked around, thus removing them?

As Nicholas Clark pointed out, consistent-overhead byte stuffing (COBS) has a fixed overhead that works well with fixed-size frames.

One technique that is often overlooked is a dedicated end-of-frame marker byte. When the receiver turned on in the middle of a transmission, a dedicated end-of-frame marker byte helps the receiver synchronize faster.

When a receiver is turned on in the middle of a packet, and the data field of a packet happens to contain bytes that appear to be a start-of-packet (the beginning of a pseudo-packet), the transmitter can insert a series of end-of-frame marker bytes after that packet so such pseudo-start-of-packet bytes in the data field don't interfere with immediately synchronizing on and correctly decoding the next packet -- even when you are extremely unlucky and the checksum of that pseudo-packet appears correct.

Good luck.