VHDL USB UART Problem

Your implementation looks basically sound. Your 2nd process essentially counts off whole-bit times; then your 1st process detects the leading edge of the start bit, synchronizes the whole-bit counter to that edge, and you then sample the data every time the phase point of the whole-bit counter indicates the middle of the bit has arrived. Implementations vary, but the algorithm is classic - get to the center of the start bit, then sample at regular bit times thereafter.

Your error examples are interesting though. Bearing in mind that the LSB comes first, it looks like your system usually gets the first two, often three bits correct. And in every example, the MSB is incorrectly '1', which suggests the possibility that that '1' is in fact the stop bit.

All in all, it looks quite like a rate problem, specifically, like the sampling rate is just a bit too low compared to the actual data rate. I think I'd try 01000000 for a test vector; if a slow sampling rate is the problem, you're apt to read it as 00100000. Just for laughs you could try dialing down the whole-bit counter range from 434 to say 430 or 425 and see what happens.

And lastly, in the 'have you tried unplugging it and plugging it back in' department, (1) no doubt you've already verified that the oscillator in the real system is in fact the same 50MHz you simulated at, and not something like 48MHz, and (2) you've fed the test signal to some other piece of equipment known to function correctly at 115.2, and vetted that the signal source isn't running fast.

As I stated in my comment, mem is an inferred level-sensitive latch (for rest of the answer I will use latch for anything level-sensitive, and flop for anything edge-sensitive). Latches are inferred in always blocks that are not edge-triggered (posedge/negedge) and is not assigned to a deterministic value if all logical branches.

Here is your original logic reformatted with my comments:

always @(data_trigger or reset) // <-- poorly inferred latch
begin
    if (reset) // Asynchronous reset because 'reset' is in the sensitivity list
    begin
        for (n = 0; n < 10; n = n+1) mem[n] = 8'h00;
        i = 0;
        checksum_buff = 8'h00;
    end     
    else if (data_trigger) // <-- Also asynchronous
    begin 
        if ( i < 10) 
        begin
            mem[i] = input_data;
            i = i + 1; // <-- as a latch will cause feedback loop on FPGA, will not be detected on in Simulations
        end
        else 
        begin
            checksum_buff = input_data;
            i = 0; // <-- also will cause a feedback loop to 'if(i<10)' in FPGA
        end
    end
end 

Except for synchronous logic, synthesizers do not look at the sensitivity list. Therefore input_data and i are actually part of the sensitivity list during synthesis. Simulation does use the sensitivity list, therefore it will ignore changes to input_data and i. This is a place where simulation and synthesis will mismatch.

i is part of the real sensitivity and it is updated on every pass of the always block. This creates an asynchronous feedback will data_trigger is high and reset is low. If you were to have added i to your sensitivity list, your simulation would likely hang from an infinite loop. On FPGA there is a slight propagation delay so you will results of seemingly random data. If you were measuring power you would also see high power consumption in this condition.

If you look through your log files there may be a warning about the asynchronous feedback.

i needs to be a flop. mem and checksum_buff should be flops too. From an eyeball review of your code, it looks like everything is running on the posedge clk domain. Therefore I suggest you change the code to:

always @(posedge clk)
begin
    if (reset)   
    begin
        for (n = 0; n < 10; n = n+1) mem[n] <= 8'h00;
        i <= 0;
        checksum_buff <= 8'h00;
    end     
    else if (data_trigger)
    begin 
        if ( i < 10) 
        begin
            mem[i] <= input_data;
            i <= i + 1;
        end
        else 
        begin
            checksum_buff <= input_data;
            i <= 0;
        end
    end
end

With this reset will also be synchronous. If you really want asynchronous reset, change it to always @(posedge clk or posedge reset). If you really want data_trigger to be the clocking event (I don't recommend it) then change the posedge clk to posedge data_trigger AND replace else if (data_trigger) with else. Any posedge/negedge signal referenced in the block is treated as an asynchronous trigger. You have this issue in your checksum256 module.

Verilog combinational logic should be written in always @* or the synonymous always @(*). You will run into code that give a list of signals instead of a literal *; don't do that. That style is for legacy Verilog-1995. Since Verilog-2001 @*/@(*) is the preferred strategy to limit the risk an incomplete sensitivity list. The other thing for proper combinational logic is that each left-hand-side (LHS) signal must be assigned to a determinate value for all possible logic branches. Assigning it to itself doesn't count. Assigning it a determinate value at the top of an always block before an if or case gives it a default assignment. Combinational logic should be assigned with blocking statements (=).

Synchronous logic must be have a clocking trigger posedge or negedge. Do not reference the clock signal in the body of the of the always block. Up to two additional posedge or negedge signals can be in the sensitivity list for asynchronous reset/preset, and asynchronous must assign registers to constants. Not all FPGAs support flops with asynchronous reset/preset; some have a limited number. Check the manual for your board. Synchronous logic should be assigned with non-blocking statements (<=).

Latches usually should be avoided. Unfortunately they can be unintentionally inferred with what should be combinational logic. A latch happens when not all logical branches assign the signal to a deterministic value. If you need a latch, keep it logic simple and assign with non-blocking.

always @* begin
  if (enable) Q <= D;
end

If you have SystemVerilog you can be more explicit by replacing always @* with always_comb and always_latch. This way your synthesizer can do a better job warning you about unintentional latches.

FYI: Verilog non-blocking was inspired/copied form VHDL. The usage of blocking and non-blocking should be similar between the two.