Electronic – How to create a safe UART communication protocol

I dont understand your confusion. You want to communicate between the two controllers with UART, then ofcourse both of them need UART capabilities and code for this. Your computer has drivers and software for dealing with this, so your microcontroller also needs a driver for it.

Your ATmega32 need UART TX functionality, while AT90USB1287 need UART RX functionality. Having already written both TX and RX for your ATmega32, writing the UART RX routine for AT90 should be a breese. You could add a bitvalue of some kind, identifying that the UART RX on AT90 is coming from your ATmega32, and directly pass it through to USB.

EDIT: Code from AVRFreaks that is written for at90usb1287

#include <avr/io.h> 
#include <util/delay.h> 

#define RXC      (RXC1) 
#define TXC      (RXC1) 
#define UCSRC    (UCSR1C) 
#define UCSRB    (UCSR1B) 
#define UCSRA    (UCSR1A) 
#define UDR      (UDR1) 
#define UBRRL    (UBRR1L) 
#define UBRRH    (UBRR1H) 
#define UBRR     (UBRR1) 
#define UDRE     (UDRE1) 
#define RXCIE    (RXCIE1) 
#define TXEN     (TXEN1) 
#define RXEN     (RXEN1) 
#define UCSZ0    (UCSZ10) 
#define UCSZ1    (UCSZ11) 
#define UMSEL0   (UMSEL10) 
#define UMSEL1   (UMSEL11) 

#define FRAMING_ERROR (1<<FE1) 
#define PARITY_ERROR (1<<UPE1) 
#define DATA_OVERRUN (1<<DOR1) 
#define DATA_REGISTER_EMPTY (1<<UDRE1) 
#define RX_COMPLETE (1<<RXC1) 

#define FOSC 16000000 

typedef unsigned char       Uchar; 
typedef unsigned long int   Uint32; 

#define Wait_USART_Ready() while (!(UCSR1A & (1<<UDRE1))) 

#define lowByte(w) ((uint8_t) ((w) & 0xff)) 
#define highByte(w) ((uint8_t) ((w) >> 8)) 

int main(void) 
{ 
/* Disable clock division */ 
clock_prescale_set(clock_div_1); 
   UBRRH = (Uchar)((((Uint32)FOSC)/((Uint32)USART_BAUDRATE*16)-1)>>8); 
   UBRRL = (Uchar)(((Uint32)FOSC)/((Uint32)USART_BAUDRATE*16)-1) & 0x0ff; 

   UCSRB |= (1 << RXEN) | (1 << TXEN); 
   UCSRC |= (1 << UCSZ1) | (1 << UCSZ0); 

   char ReceivedByte; 
   for (;;) // Loop forever 
   { 
      while ((UCSRA & (1 << RXC)) == 0) {}; // Do nothing until data have been received and is ready to be read from UDR 
      ReceivedByte = UDR; // Fetch the received byte value into the variable "ByteReceived" 

      while ((UCSRA & (1 << UDRE)) == 0) {}; // Do nothing until UDR is ready for more data to be written to it 
      UDR = ReceivedByte; // Echo back the received byte back to the computer 
   } 
}

First I recommend a protocol, a frame structure, ideally a start pattern that is not found in the data or if found wont be repetative, or have a start pattern and a checksum. In any case you have to do your system engineering/design to make sure that you are not operating on the wrong bytes or bytes in the wrong position.

Second, do your system engineering, what is your data rate what is the processing and execution rates. If for example you can process the four bytes worth of data within one byte transmission time with enough time to spare, then that would make for a simpler solution. You obviously have to store the 4 bytes of information somewhere be it in the uart if the uart has a fifo deep enough or outside the uart if not.

Well, actually does the host have to be constantly transmitting something or is it, or can the protocol be one burst of a packet to you then you burst an answer back? The question for your system design is do I have to be handling new bytes coming in while I am processing an existing request?

If you have to be handing data while processing other, and it takes more than one byte time to process a packet, even if so it is probably better to plan for a circular buffer, the size you likely have to tune because you probably dont have enough resources to make it big enough to not care. Also you have to plan to insure that while processing the current data you are collecting the new data (if the protocol requires you to handle more than one at a time). Interrupts may come into play here, but are tricky to get working, interrupts are not a required solution, but often how folks choose to do it.

if you go with a packet/frame approach, then you definitely need a fifo that is collecting at least a packets worth of data, each new byte that comes in shifts the data through a window (or better use a circular buffer and shift your head and tail pointers around) once you have a start pattern and at least the right number of bytes for the packet, then checksum it if that is how you choose to implement, and if passes process it, if not dont toss the whole thing just move the head pointer around to the next start pattern if there is any in the accumulated data, otherwise match it to the tail and wait for more.

if you are sending data back, same deal what is the protocol for that, frame/data format? Assumed to be the same baud rate, so if as you mention you accept four and send four back, it takes the same time in each direction, even if it takes a small number of mcu clock cycles to process the data, it still takes a lot of clock cycles to send it back out at that baud rate, if the hardware does not have a tx fifo and you protocol is not half duplex then you will need to be handling waiting for a tx slot to open, moving the next byte in there from some buffer/ring. the waiting can be polling or interrupts, same answer a above, doesnt HAVE to be either kind, can be what you are comfortable with and what works, for TX it is a lot easier unless the protocol has a timeout, as you cannot lose bytes on the way out, if your code isnt fast enough you simply add gaps. On the RX side if the world you can lose bytes if you are not fast enough.

All of this is very basic stuff and hopefully not stuff you already knew and had figured out. Start simple, send one blob of data from the pc, somehow indicate from the mcu to your debug solution (for me it is printing stuff out a uart or blinking an led) that you have the data. Either then develop the response with bogus data, or develop the processing step next. Basically do your system engineering, break the problem down into bite sized chunks, define the interfaces between the chunks, and develop each chunk and then glue them together, the boundaries between the chunks are opportunities if needed to place test code/instrumentation to test that chunk. Some of these chunks might be able to be developed on a host computer using a common programming language (C is a good one) with a test fixture around the code, that would only be for generic processing of course, if you have any mcu/board hardware specific resources required then you would either have to simulate that on a host (more code to develop and test to wrap around the code under test). That doesnt necessarily mean a simulator for that environment but finding/developing one might be useful. It all depends on how hard the problem becomes.

Performance, when I said a bytes worth of time, understand that simply adding or removing a line of code can change the performance of the code by a noticeable amount 10% 20% or more. Even the same code with its alignment changed (more or fewer instructions in the bootstrap) can and sometimes will change the performance of the code as much as 10% or 20% or more. Simply clocking the mcu up by a factor of N to get more performance does not make the same code run N times faster, esp if the flash is slower than the max speed of the mcu clock and you have to put additional wait states as you increase the mcu speed to try to compensate for processing requirements. Consider running from sram if you have enough if you run into this but there are no guarantees that the ram or memory interface was designed to run without wait states for higher clock speeds either. Likewise the cpu clock might be able to go one rate, but that doesnt mean the peripheral busses are on that clock. the only way through this is to test and measure (Michael Abrash Zen of Assembly language, look past the x86 nature of that book at the core concepts of finding cycle stealers, can be found free on github afaik)

very short answer, do your system engineering, does the hardware have a buffer, what other resources do you have. how much storage do you need for the communications in order to not lose data or get behind. Obviously you have to store up some number of bytes before you can do the processing so you need at least that much storage. How much beyond that is up to your protocol and analyzing how long it should take to do things. Or design the protocol to be a half duplex handshake and take as much time as you want.