Apache reverse proxy causing ~40 second delay on first access after idle

I have a simple Apache 2.2 reverse proxy setup on OSX with three virtual hosts all providing SSL and Open Directory authentication before forwarding everything to their respective back end servers.

For each of the three virtual hosts things work fine and seem fast enough, but if any of them is idle for more than 10 minutes or so (i don't have an accurate timing of this)from any given host, the first call takes almost a minute, then subsequent calls are <1s. This delay is definitely at the proxy, I see no traffic on the back end server until the last second of the wait.

What could I be doing wrong here? WHat can cause a reverse proxy to delay like this? Is it refreshing a cache? How do I go about debugging it?

My virtual hosts all look about the same and the rest of the apache config is bare bones, only the modules I need and very basic setup…

ServerRoot "/usr"
User _www
Group _www
DefaultType text/html

Listen 80
Listen 443
Listen 6660

DocumentRoot /Volumes/data/httpd

# Basic Requirements for a proxy
LoadModule mime_module libexec/apache2/mod_mime.so
LoadModule log_config_module libexec/apache2/mod_log_config.so
LoadModule env_module libexec/apache2/mod_env.so
LoadModule proxy_module libexec/apache2/mod_proxy.so
LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so
LoadModule authz_host_module libexec/apache2/mod_authz_host.so

# Needed for OD Authentication
LoadModule apple_auth_module libexec/apache2/mod_auth_apple.so

# Needed for SSL (duh)
LoadModule ssl_module libexec/apache2/mod_ssl.so

ErrorLog /var/log/apache2/error_log
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog /var/log/apache2/access_log common

<IfModule mod_ssl.c>
    SSLSessionCache shmcb:/var/run/ssl_scache(512000)
    SSLSessionCacheTimeout 300
    SSLMutex file:/var/log/apache2/ssl_mutex
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin
    AddType application/x-x509-ca-cert crt
    AddType application/x-pkcs7-crl crl
</IfModule>


NameVirtualHost *:80
NameVirtualHost *:443
NameVirtualHost *:6660


<VirtualHost *:443>
ServerName mysite.server.com

SSLEngine On
SSLProxyEngine On
SSLCipherSuite "ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM"
SSLProtocol -ALL +SSLv3 +TLSv1
SSLProxyProtocol -ALL +SSLv3 +TLSv1
SSLCertificateFile "/etc/certificates/mysite.crt"
SSLCertificateKeyFile "/etc/certificates/mysite.key"

<Location />
    Order deny,allow
    Deny from all
    AuthType Basic
    Require group <my access group>
    Allow from <my test ip>

    Satisfy Any
    AuthName "MyGroup"
</Location>

SetEnv proxy-chain-auth On
ProxyPreserveHost On
ProxyPass / http://backend.mysite.com/ retry=0 keepalive=On
ProxyPassReverse / http://backend.mysite.com/
</VirtualHost>