Electronic – Map n states to hardwired outputs

Another answer: Stop using interrupts.

People jump to use interrupts too easily. Personally, I rarely use them because they actually waste a lot of time, as you are discovering.

It's often possible to write a main loop which polls everything so rapidly that's it's latency is within spec, and very little time is wasted.

loop
{
    if (serial_bit_ready)
    {
        // shift serial bit into a byte
    }

    if (serial_byte_ready)
    {
        // decode serial data
    }

    if (enough_serial_bytes_available)
    {
        // more decoding
    }        

    if (usb_queue_not_empty)
    {
        // handle USB data
    }        
}

There might be some things in the loop which happen far more often than others. Perhaps the incoming bits for example, in which case, add more of those tests, so that more of the processor is dedicated to that task.

loop
{
    if (serial_bit_ready)
    {
        // shift serial bit into a byte
    }

    if (serial_byte_ready)
    {
        // decode serial data
    }

    if (serial_bit_ready)
    {
        // shift serial bit into a byte
    }

    if (enough_serial_bytes_available)
    {
        // more decoding
    }        

    if (serial_bit_ready)
    {
        // shift serial bit into a byte
    }

    if (usb_queue_not_empty)
    {
        // handle USB data
    }        
}

There might be some events for which the latency of this approach is too high. For example, you might need a very accurately timed event. In which case, have that event on interrupt, and have everything else in the loop.

In the last few pictures, it looks like you've configured the "sub flag" as an output, not an input. Since there's nothing driving that net, it's resulting in an error.