Firewall – Subversion server behind firewall and Apache reverse proxy exhibits intermittent hanging

dmzfirewalllocal-area-networkreverse-proxysvn

Here's my situation:

1) DMZ: I have an self-signed SSL Cert on an Apache server (our bastion host) configured to run as a reverse proxy for 7 other LAN servers (subversion, ldap, jenkins, confluence, jira, mapi, etc).

2) Firewall: Between the DMZ bastion host (which is on DMZ subnet 192.168.1.X) I have a Cisco firewall configured to let (specific) traffic originate from the DMZ Bastion host side into specific LAN hosts. The LAN subnet is 192.168.50.X. Router is Cisco RV042.

3) LAN: I have 7 servers all running various apps on UBuntu with iptables enabled via ufw.

4) Subversion: One of the 7 servers is running Subversion 1.5.4 and exposes an HTTP port to the bastion host. Very much like this article discusses.

Everything has been working terrifically for years except for ONE THING which I cannot seem to solve: All HTTPS Subversion commands that run through the bastion host to the LAN subversion server fail if nobody has used the subversion server for a few hours/days (not exactly sure).

This is causing a real problem because remote developers make a bunch of local changes, commit…and then Eclipse hangs, has to be manually killed, the client sources cleaned up, etc….a real hassle. Then I get a call…and I navigate to the bastion host and try to view some sources, which after a few clicks starts working. Then the developer's next attempt to commit always works.

Here's what I've tried:

1) Firewall Off: If I disable the firewall on the Cisco router, then it always works…all the time, but we have no DMZ/LAN security!

2) LAN Subversion: It always works if you hit the Subversion LAN server directly.

3) Firewall configuration changes: When the firewall is enabled, I can create a rule to let ANY DMZ->LAN traffic pass and the problem still happens. In effect, the firewall is on but completely opened up and the problem still happens. It's as if the router's firewall between the bastion host and the LAN subversion server requires a statefull conversion originating on the LAN side, but I absolutely have tested that the firewall is open (if it was not it would NEVER work…like I said it can work for days/weeks if used frequently).

4) MTU Mismatch: Found an article which suggest this could be the problem, but the MTU on both the bastion and subversion server is 1500 acccording to ifconfig.

5) Misc bastion host Apache config changes…have tried dozens of things here to no avail. Here is the Apache config from /etc/apache2/sites-enabled/default-ssl on the bastion host (reverse proxy) I've been running for the past year:

ProxyPass /svn http://virt-svn-srv.mycomp.int/svn keepalive=on
<Location /svn>
    ProxyPassReverse http://virt-svn-srv.mycomp.int/svn
    SetEnv force-proxy-request-1.0 1
</Location>

I am really at my wits end on this…all suggestions are welcome.

Best Answer

This looks like an issue with timeouts (firewall deleting sessions from the session table).

I'm not entirely familiar with subversion and Eclipse but;

If it opens a connection with the subversion server at start up, then tries to use the same session to commit code after 60 minutes (Cisco's) default timeout setting, then it will fail.

Your firewall will drop the packets stating it doesn't find the matching session.

You can try and add a class-map and a policy-map to match traffic going to your subversion server.

access-list acl-timeout-subv extended permit tcp any host [subversionIP] eq [subversionPort]

then

class-map cls-timeout-subv
match access-list acl-timeout-subv
exit

then

policy-map timeout-subv
class cls-timeout-subv
set connection timeout idle 08:00:00
exit

This will set all traffic going to your subversion on the specified port with a timeout of 8 hours.

Related Topic