We have this apache process which becomes io-bound time to time.
Using atop, we can see it is a write operation.
Using lsof -p <PID> we can see a list of files open by the httpd process. First we thought "log" files must be the problem. So we turned them off just to test. However write operations still continues.
We will continue testing a few other things. For instance we use php session variables a lot. Maybe php session files are getting all the writing.
But is there a way to quickly identify files which get written to by the httpd process? This way we can focus our efforts on those files.
UPDATE:
We used the strace command as suggested. Here are two lines from the output.
write(23, "\27\0\0\0\3SET CHARACTER SET utf8", 27) = 27
write(23, "\17\0\0\0\3SET NAMES utf8", 19) = 19
We do not have a mysql process on this server. So is strace also showing what is being written to an ethernet port?
UPDATE2:
During high io load, the process which consumes most of the write resources gives the following output to strace -e trace=write -p <PID>:
— SIGCHLD (Child exited) @ 0 (0) —
write(9, "!", 1) = 1
write(19, "OPTIONS * HTTP/1.0\r\nUser-Agent: Apache (internal dummy connection)\r\n\r\n", 70) = 70
However I cannot figure out where these are being written to.
UPDATE3:
Tried the following command which will show me all files open for writing:
lsof | grep -e "[[:digit:]]\+w"
and below are the only two lines I cannot check. Other files do not show any significant write activity. Could the items below temporarily being written to the hard drive?
httpd 14173 apache 9w FIFO 0,6 205676000 pipe
httpd 14173 apache 1w CHR 1,3 3346 /dev/null
REASON FOUND:
Finally we found out that the reason for io-boundness were the php session files. Moving the php session files to another hard drive also moved io-boundness to that hard drive immediately. Probably we will move the sessions to redis or another memory based system.
Interestingly, even though the answers below are technically correct, they did not lead us to the session files and kept us looking at different points of the system.
Best Answer