PIC16f877: MPLAB X error 1250 (Could not find space for variable)

TRISB1 needs to be set to 1 to configure RB1 (RX) as an input. I'm not sure what the default is, so it may be ok.

You need to clear the receive interrupt flag (RCIF) by reading the receive register (RCREG). In addition, since the receive register is doubled-buffered, you may need to read it more than once.

So your interrupt routine needs to look more like this:

extern interrupt isr(void)
{
    while (RCIF)
    {
        char ch;

        RB5 = 1;
        ch = RCREG;    // normally would go into an array and increment a counter
    }
}

I don't know if that is your only problem, since you indicate you are not getting into the interrupt routine at all. But the above is the correct way to read the characters out of the receive buffer.

======================================

EDIT:

Don't know if this will help or not, but in this post, before enabling interrupts, the code clears out the FIFO first. (Their code also clears out the RCIF flag, but since it is readonly on your chip, that isn't needed.)

ch = RCREG;    // clear FIFO  
ch = RCREG;
ch = RCREG;

// then enable interruupts ...

Unless you are power-constrained, make it as fast as possible. The 16F877 is a old part and slow compared to many newer and cheaper parts available today. Its top speed is 20 MHz clock, which results in 5 MHz instruction rate. All 18F can go at least 2x as fast, and the 24H and 33F another 4x beyond that and manipulate 16 bit quantities per instruction.

If you need to save power, then the '877 make little sense, so if you're using the '877, then maxxing it out is a no-brainer. Use a 20 MHz crystal and you're done.