I don't know this specific MCU, but the question is fairly generic.
The obvious answer to all problems like these is to keep the ISRs as slim as possible. At most, they should stuff data into a ring buffer, which is later processed by the main program. (A MCU with DMA would have been even better, but I don't think you have DMA on MSP430?)
If the ISR is still too slow after such optimizations, then you have no other option but to do as you suggest: enable the global interrupt mask at the top of the UART ISR and let the higher priority interrupt take precedence. Be aware however, that when you allow more interrupts to come on top of an already executing ISR, you allow more stack depth.
Do I also need to unmask the serial RX interrupt at the end?
I assume that you have to do that from the ISR no matter the nature of the application? That's how most MCUs work. And yes, if you touch the global interrupt mask, you will have to clear the specific interrupt after you are done serving it, i.e. after you have copied the received data into local variables.
After the timer ISR returns, will the serial ISR continue, and then return to the main loop?
If you have changed the global interrupt mask, then yes.
Are there any potential race conditions I need to consider?
You always have to consider such whenever sharing data between an ISR and something else. It doesn't matter if the ISR only writes and the main program only reads etc, unless you can guarantee that each access is atomic, which you usually can't unless you write the code in assembler.
In a high level language, you may have to use some semaphore variable. How this is implemented is application-dependent. It particularly depends on if you can afford to miss out some data from the ISR or if you have to catch all data.
Is there a better solution?
DMA or a multi-core MCU.
Best Answer
In most normal cases this either can't happen or there is no harm. Every architecture I can think of disables interrupts in such a way the the interrupt that was just taken can't happen again until the software re-enables it -- usually. Some processors have a non-maskable interrupt, which may be handled differently. Ignore those for now.
On simple processors that have a single interrupt, interrupts are usually globally disabled when a interrupt is taken. That allows the code immediately at the start of the interrupt to know it can't be interrupted. That is useful since often various things need to be done in the interrupt routine that must appear to be atomic. Many processors have a means to re-enable interrupts combined with returning from the interrupt routine in such a way to guarantee that the interrupt routine need not be written to support re-entrance. For example, the PIC 16 has a global interrupt enable bit (GIE in INTCON). This automatically gets cleared when the single interrupt is taken, and the special instrution RETFIE can be used to return from the interrupt and set GIE at the same time.
Things get a little more complicated on processors that have multiple interrupt priorities. The purpose of priorities is specifically to allow high priority interrupts whether in a lower priority interrupt routine or not. There is usually a field in some register that is the current priority level. When a interrupt is taken, the existing priority level is saved along with other state (like the return address), then the priority is bumped so that only higher priority interrupts can occur. Any one interrupt routine can't be re-entered unless the code deliberately diddles the priority level, but all but the higher priority interrupt routines have to be written considering that they can be interrupted. This is usually not a problem in preforming operations that need to appear atomic since the other higher priority interrupts will usually deal with different hardware and state. It does mean though that you can't rely on sequential instruction timing in low priority interrupts. These are all things that need to be taken into account during the system level design of the software.
Now back to non-maskable interrupts (often called NMI). These are by definition interrupts that can't be turned off ("masked" off) by the software. This means that on some architectures the NMI interrupt routine could possibly be called re-entrantly. This is something you have to be aware of as the system designer. Usually you connect the NMI input to a signal that you know can't trigger with a very short interval. Since the NMI interrupt handler is the highest priority in the system, you also know it won't be interrupted and therefore can know that it will always execute within some maximum time.
Also keep in mind what a interrupt really is. You may think of being "in" a interrupt in some code, but often interrupt code is nothing special to the processor. To use the PIC 16 example again, interrupt is not a lasting condition but a single event. When the interrupt condition occurs and GIE is set, the processor effectively executes a call to location 4 and clears GIE. That's it. The processor is done with the interrupt. Whether you view code at that location as a interrupt handler or whether you think it is "in" a interrupt is strictly your own abstraction. Ordinary foreground code can clear GIE too, so running with GIE off doesn't make something interrupt code. If you happen to execute RETFIE some instructions after enterting the routine at 4, execution will return to where it was when the interrupt occurred and GIE is re-enabled. To you that may be "leaving" the interrupt routine, but the processor does nothing different before or after that instruction and has no state telling it that is is "in" interrupt code.