Other processes take over port 80 when restarting Apache – why, and how to solve

centos5httpport

I have a CentOS 5.5 server running Apache on port 80 as well as some other applications. All works fine until I for some reason need to restart the httpd process. Doing so returns:

sudo /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

First I thought perhaps httpd had frozen and was still running, but that was not the case. So I ran netstat to find out what was using port 80:

sudo netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 *:7203                      *:*                         LISTEN      24012/java          
tcp        0      0 localhost.localdomain:smux  *:*                         LISTEN      3547/snmpd          
tcp        0      0 *:mysql                     *:*                         LISTEN      21966/mysqld        
tcp        0      0 *:ssh                       *:*                         LISTEN      3562/sshd           
tcp        0      0 *:http                      *:*                         LISTEN      3780/python26

Turns out that my python process had taken over listening to http in the instant that httpd was restarting. So, I killed python and tried starting httpd again – but ran into the same error. Netstat again:

sudo netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 *:7203                      *:*                         LISTEN      24012/java          
tcp        0      0 localhost.localdomain:smux  *:*                         LISTEN      3547/snmpd          
tcp        0      0 *:mysql                     *:*                         LISTEN      21966/mysqld        
tcp        0      0 *:ssh                       *:*                         LISTEN      3562/sshd           
tcp        0      0 *:http                      *:*                         LISTEN      24012/java

Now my java process had taken over listening to http. I killed that too and could then successfully restart httpd.

But this is a terrible workaround. Why will these python and java processes start listening to port 80 as soon as httpd is restarted? How to solve?

Two other comments. 1) Both java and python processes are started by apache from a php script. But when apache is restarted, they should not be affected. And 2) I have the same setup on two other machines running Ubuntu and there's no problem there.

Any ideas?

Edit:

The Java process listens to port 7203 and the python process supposedly doesn't listen to any port. For some reason, they start listening to port 80 when apache is restarted. This hasn't happened before. On Ubuntu it runs fine. For some reason, on my current CentOS 5.5 machine, this problem arises.

Best Answer

The problem might be the way how Apache starts the sub-processes. They might get spawned by forking and letting the cloned Apache processes become other processes. Every clone inherits all open file-handles of the parent process, including the open handle of TCP port 80.

Netstat only shows one application associated with the open file handle, while three processes keep the handle open.

Possible solutions to your problem:

  1. If it is important to keep the sub-processes running during a Apache restart, the simplest solution would be to start the 3 Processes as separate system-services.

  2. If they depend on a running Apache, a 'apache stop' command should also terminate them. This could be achieved by editing the /etc/init.d/apache script.

  3. If you are forced to start them from Apache, you need to start them as real "daemon processes"!

I wrote a daemon creator for a PyQt-app inspired by http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/. You could start this script from Apache, call exec_as_daemon for each of your sub-processes and close it.

# Copyright: (c) 2011 phobie <pydaemon.phobie@safersignup.com>
# License: CC-by-sa 3.0
import os
def exec_as_daemon(s_path_cmd, l_args=[]):
    i_pid = os.fork()
    if i_pid != 0:
        # Always remember to gobble your zombie children
        os.wait()
        # Back to parent
        return
    # Detach from parent
    os.setsid()
    # Do not block any mounts
    os.chdir('/')
    # Reset file creation rights
    os.umask(0)
    i_pid = os.fork()
    if i_pid != 0:
        # Close the direct child process
        os._exit(0)
    # Get the maximum count of open file handles
    try:
        import resource
        i_fd_max = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
        if i_fd_max == resource.RLIM_INFINITY:
            i_fd_max = 1024
    except ImportError:
        i_fd_max = 1024
    # Try to close all possible file handles
    for i_cur_fd in range(0, i_fd_max):
        try:
            os.close(i_cur_fd)
        except OSError:
            pass
    # Assosiate STDIN with /dev/null
    os.open(os.devnull if hasattr(os, "devnull") else '/dev/null', os.O_RDWR)
    # STDOUT to /dev/null
    os.dup2(0, 1)
    # STDERR to /dev/null
    os.dup2(0, 2)
    l_args.insert(0, s_path_cmd)
    # Let the clone become a other process
    os.execv(s_path_cmd, l_args)

if __name__ == '__main__':
    exec_as_daemon('/usr/bin/java', ['-jar', '/usr/local/bin/apache_helper_daemon.jar'])
    exec_as_daemon('/usr/local/bin/apache_helper_daemon.py')
Related Topic