Why kernel code/thread executing in interrupt context cannot sleep

On a multiprocessor, a context switch can certainly happen while an interrupt handler is executing. In fact, it would be difficult to prevent.

On a single-CPU machine, by definition it can only be running one thread of control at a time. It only has one register set, one ALU, etc. So if the interrupt handler is running there simply are no resources with which to execute a context switch.

Now, if you mean, can the interrupt handler actually call the context switch code and make one happen, well, I suppose on some systems that could be made to work. But for most, this wouldn't have much value and would be difficult to arrange. The CPU is running at elevated priority, and this priority cannot be lowered or synchronization between interrupt levels is lost. Critical sections in the OS are already synchronizing against interrupt execution and this would introduce complexities. Furthermore, a context switch happens by changing stacks, much like in a threaded user mode program, so it's hard to imagine how this might happen when the interrupt stack is needed for a return from the interrupt.

Don't use semaphores in interrupt context, use spin_lock_irqsave instead. quoting LDD3:

If you have a spinlock that can be taken by code that runs in (hardware or software) interrupt context, you must use one of the forms of spin_lock that disables interrupts. Doing otherwise can deadlock the system, sooner or later. If you do not access your lock in a hardware interrupt handler, but you do via software interrupts (in code that runs out of a tasklet, for example, a topic covered in Chapter 7), you can use spin_lock_bh to safely avoid deadlocks while still allowing hardware interrupts to be serviced.

As for point 2, make your set_intr and clear_intr require the caller to lock the spinlock, otherwise you'll find your code deadlocking. Again from LDD3:

To make your locking work properly, you have to write some functions with the assumption that their caller has already acquired the relevant lock(s). Usually, only your internal, static functions can be written in this way; functions called from outside must handle locking explicitly. When you write internal functions that make assumptions about locking, do yourself (and anybody else who works with your code) a favor and document those assumptions explicitly. It can be very hard to come back months later and figure out whether you need to hold a lock to call a particular function or not.