How to set a hard Apache timeout when using mod_proxy

I have a web service that must always respond to requests in under 5 seconds. I'm using Apache to proxy requests to a cluster of backend workers. When all the workers are full, Apache queues up requests and waits for a worker to be free.

I have set Timeout 5 in my VirtualHost

mod_proxy appears to take over the Timeout directive and use it to limit how long a worker can take to respond to a request. This means that requests can back up in the queue, and they don't get timed out in 5 seconds. I need to respond with a 504 if I can't turn a request around in under 5 seconds total, the time spent in the proxy worker isn't what I want to limit, it's the total open connection time. Alternatively, if there was a way to limit the time spent waiting for a free worker, that might work, but none of the balancer directives seems to do that (I've tried a lot of them)

How can I make Apache time out requests in 5 seconds, no matter what is going on in my worker cluster?