Linux – how to find out all the PID of Apache httpd processes that are in reading state

apache-2.2httpdlinux

The Apache web server in my RHEL shows many requests in R (..reading..) state. I want to find the client IPs that are making Apache wait in reading state. Specifically: I want to find out all the client IPs that are spending too long to send the request.

The server-status module is not enough. Server status does not show the client information when the PID is in reading state.

Best Answer

Try using netstat program with the -l flag to get a list of listening process. You probably want to run it with sudo so that you can use the -p flag to get process PIDs. You might also want the -t flag to only show tcp sockets instead of tcp and udp. Sometimes the -n flag is nice to show only port numbers and IPs without resolving them to services and names.

After that it's a matter of greping for just your apache process and then extracting the PID from the output columns:

sudo netstat -lntp | grep httpd | awk -F '[/ ]*' '{print $7}'

Related Solutions

Linux Server Performance – What Limits Maximum Number of Connections?

I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout seconds for more connections to become available. The conntrack table is what the kernel uses to track the state of connections so once it's full, the kernel starts dropping packets and printing this in the log:

Jun  2 20:39:14 XXXX-XXX kernel: ip_conntrack: table full, dropping packet.

The next step was getting the kernel to recycle all those connections in the TIME_WAIT state rather than dropping packets. I could get that to happen either by turning on tcp_tw_recycle or increasing ip_conntrack_max to be larger than the number of local ports made available for connections by ip_local_port_range. I guess once the kernel is out of local ports it starts recycling connections. This uses more memory tracking connections but it seems like the better solution than turning on tcp_tw_recycle since the docs imply that that is dangerous.

With this configuration I can run ab all day and never run out of connections:

net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768    61000

The tcp_max_orphans setting didn't have any effect on my tests and I don't know why. I would think it would close the connections in TIME_WAIT state once there were 8192 of them but it doesn't do that for me.

Apache filling up with connections in state “Reading Request”

What is your workers.properties file content? Do you try to use socket_timeout?

  socket_timeout    0   Socket timeout in seconds used for the communication
                        channel between JK and remote host. If the remote host
                        does not respond inside the timeout specified, JK will
                        generate an error, and retry again. If set to zero
                        (default) JK will wait for an infinite amount of time on
                        all socket operations.

By default the socket_timeout is infinite.

Best Answer

Related Solutions

Linux Server Performance – What Limits Maximum Number of Connections?

Apache filling up with connections in state “Reading Request”

Related Topic