I would use HDLC. I have had good luck with it in the past. I would for a point to point serial just use the Asynchronous framing and forget about all of the other control stuff as it would probably be overkill.
In addition to using HDLC for the framing of the packet. I format my packet like the following. This is how options are passed using 802.11
U8 cmd;
U8 len;
u8 payload[len];
The total size of each command packet is len +2
You then define commands like
#define TRIGGER_SENSOR 0x01
#define SENSOR_RESPONSE 0x02
The other advantage is that you can add new commands and if you design your parser correctly to ignore undefined commands then you will have some backwards compatibility.
So putting it all together the packet would look like the following.
// total packet length minus flags len+4
U8 sflag; //0x7e start of packet end of packet flag from HDLC
U8 cmd; //tells the other side what to do.
U8 len; // payload length
U8 payload[len]; // could be zero len
U16 crc;
U8 eflag; //end of frame flag
The system will then monitor the serial stream for the flag 0x7e and when it is there you check the length to see if it is pklen >= 4 and pklen=len+4 and that the crc is valid. Note do not rely on just crc for small packets you will get a lot of false positives also check length. If the length or crc does not match just reset the length and crc and start with decoding the new frame. If it is a match then copy the packet to a new buffer and pass it to your command processing function. Always reset length and crc when a flag is received.
For your command processing function grab the cmd and len and then use a switch to handle each type of command. I also require that a certain events send a response so the system behaves like a remote procedure call that is event driven.
So for example the sensor device can have a timer or respond to a command to take a reading. It then would format a packet and send it to the PC and the PC would respond that it received the packet. If not then the sensor device could resend on a timeout.
Also when you are doing a network transfer you should design it as a network stack like the OSI modle as Foredecker points don't forget about the physical layer stuff. My post with the HDLC is the data link layer and the RPC and command handling is the Application Layer.
Best Answer
Is this a full-duplex or half-duplex device? Can you control request-to-send and monitor clear-to-send on both ends of the serial line? Is the serial line point-to-point (like RS-232) or multi-drop (like RS-485)? It will make a (albeit small) difference if you are going to interfere with other already connected devices while negotiating with a newly connected one.
If you think of the handshake process like a modem negotiating a link layer protocol, it uses a standard set of messages to describe the type of communications it would like to have and waits for an "ack" from the other end. In your case I recommend having a "let's talk" standard message that your head end generates with the range of bit rates and waits for the ack from the device.
I also recommend reducing the number of configuration options for the device. Forget about variable data bits, parity bits, and stop bits. The serial communications world is no longer as unstable as it was back in the 70's. Just use 8 data bits, no parity, one stop bit and vary the bit rate. A CRC on the end of messages provides plenty of error-checking.