Apache reverse proxy sometimes not working

apache-2.4confluencejirareverse-proxy

I am using a digital ocean droplet (Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-52-generic x86_64)) to host both jira and confluence.
They are started on the same ip but different ports. I wanted to access them using jira.team.domain.com and confluence.team.domain.com so I went for the reverse proxy solution, using apache 2.4.7.

Things went really good and I had them working pretty fast. The issue is that after a couple of days, for a certain amount of time, the reverse proxy is not working and I get 'hostname not resolved' in the browser. I checked and the jira and confluence apps are accessible at the ip:port addresses (ports 8081 and 8091). After a while, don't know exactly how much, it starts working again.

The setup is the following:

Jira server.xml contains two connectors:

            <Connector port="8080"

               maxThreads="150"
               minSpareThreads="25"
               connectionTimeout="20000"

               enableLookups="false"
               maxHttpHeaderSize="8192"
               protocol="HTTP/1.1"
               useBodyEncodingForURI="true"
               redirectPort="8443"
               acceptCount="100"
               disableUploadTimeout="true"
               proxyName="jira.team.domain.com"
               proxyPort="80"/>

            <Connector port="8081"

               maxThreads="150"
               minSpareThreads="25"
               connectionTimeout="20000"

               enableLookups="false"
               maxHttpHeaderSize="8192"
               protocol="HTTP/1.1"
               useBodyEncodingForURI="true"
               redirectPort="8443"
               acceptCount="100"
               disableUploadTimeout="true"/>

Confluence server.xml also has 2 connectors:

<Connector port="8091" connectionTimeout="20000" redirectPort="8443"
            maxThreads="200" minSpareThreads="10"
            enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
            protocol="org.apache.coyote.http11.Http11NioProtocol" />

<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
            maxThreads="200" minSpareThreads="10"
            enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
            protocol="org.apache.coyote.http11.Http11NioProtocol"
            proxyName="confluence.team.domain.com" proxyPort="80" />

and the /etc/apache2/sites-enabled/000-default.conf looks like this:

<VirtualHost *:*>
  ServerName localhost
  # DocumentRoot /var/www/html

  <Proxy *>
    Require all granted
  </Proxy>

  # ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://jira.team.domain.com:8080/
  ProxyPassReverse / http://jira.team.domain.com:8080/
</VirtualHost>

<VirtualHost *:*>
  ServerName confluence.team.domain.com
  DocumentRoot /var/www/html
  <Proxy *>
    Require all granted
  </Proxy>

  ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://confluence.team.domain.com:8090/
  ProxyPassReverse / http://confluence.team.domain.com:8090/
</VirtualHost>

Could anyone help me solve this issue?

Best Answer

This sounds somewhat similar to issues I've had reverse proxying Atlassian applications characterized by intermittent issues with Apache while the Tomcat instances seem just fine. The "hostname not resolved" in the browser doesn't jive with that so perhaps not, but you can give these a shot. It won't hurt anything to try.

If the Apache server itself is occasionally having trouble doing a DNS query that would be pretty simple to resolve with the hosts file. Make sure the fqdn is in the /etc/hosts assigned to either the loopback (127.0.0.1) or whatever interface you have JIRA/Confluence listening on:

127.0.0.1 localhost jira.team.domain.com confluence.team.domain.com

If you assign it to the loopback you'll need to make sure JIRA and Confluence are listening on the loopback interface (or all of the interfaces).

A problem I've seen a few times (typically returning a 503 error to the user) is Apache detecting some kind of issue with the backend. When it does it backs off for 60 seconds by default. This can be a helpful clue that there's a resource contention issue with JIRA and/or Confluence so it shouldn't be ignored, but 60 seconds of downtime every time the backend gets flakey is a tad much. You can disable this by configuring the retry parameter on the ProxyPass directives:

<VirtualHost *:*>
  ServerName localhost
  # DocumentRoot /var/www/html

  <Proxy *>
    Require all granted
  </Proxy>

  # ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://jira.team.domain.com:8080/ retry=0
  ProxyPassReverse / http://jira.team.domain.com:8080/
</VirtualHost>

<VirtualHost *:*>
  ServerName confluence.team.domain.com
  DocumentRoot /var/www/html
  <Proxy *>
    Require all granted
  </Proxy>

  ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://confluence.team.domain.com:8090/ retry=0
  ProxyPassReverse / http://confluence.team.domain.com:8090/
</VirtualHost>

From the mod_proxy docs:

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache httpd will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

https://httpd.apache.org/docs/2.4/mod/mod_proxy.html

If that doesn't work try using your browser's Developer Tools to see exactly what is or isn't being returned by Apache and what headers are being set. Taking a look at the Apache logs using tail -f /path/to/logs/goes/here while you mash the refresh button wouldn't hurt either.

Related Topic