Electronic – Can the ARM processor be switched to unprivileged mode inside handler mode

In the ARM documentation it is stated that:

The processor supports two operation modes, Thread mode and Handler
mode. Thread mode is entered on reset and normally on return from an
exception. When in Thread mode, code can be executed as either
Privileged or Unprivileged.

Handler mode will be entered as a result of an exception. Code in
Handler mode is always executed as Privileged, therefore the core will
automatically switch to Privileged mode when exceptions occur.

But it doesn't specify if I can just switch back to unprivileged mode inside an interrupt handler, or if the processor will just ignore that. I don't have an M3 handy with me to test at the moment so I am wondering if anyone knows the answer or is able to test?

Basically my scenario is I would like to be able to implement asynchronous "event handlers" from user code that are triggered by some hardware activity, but I want these to run in unprivileged mode, the obvious way to do this is to just have a loop like this in thread mode after setting up the kernel:

while (true) {
    __WFI();
    if (event_triggered) {
        run_event_handler();
    }
}

But the downside is that this code will be constantly running and doing the check whenever any interrupt happens, even if no event handler actually needs to run!

Whereas instead I could simply have a purely interrupt-driven design where whenever the kernel does something that (it knows) will trigger a user event, it can raise, say, a low-priority PendSV which then runs them as needed… but of course that would then run in a privileged context. So can I switch back to unprivileged here?