running out of ideas to explore. First off – let me warn you – I'm a programmer, not a systech 🙂
Here is the situation.
Dedicated server (LAMP) running a fair amount of sites. mySQL server is on a seperate box.
Last couple weeks, performance has been steadily degrading to the point where I can no longer even remote into the box.
Looking into mod_status, there are a fair amount of processes taking up CPU resources. However, the URLs are all different… there is not a common pattern – so I can't narrow anything down to a particular script that might be getting stuck.
PHP is ran as cgi.
Majority of the sites that are taking a while to run are using the cakephp framework
Restart the server, we are down within a few minutes again…
Crossed an error that said /var/tmp/ was full and couldn't write sessions. However, there was still room? Lack of inodes perhaps? Currently in the process of having someone walk down to the box and clear tmp.
Could the lack of ability to write sessions be causing the php processes to hang forever, and eventually clog everything up?
Any other ideas that I might want to explore? I have been monitoring the sql server to see if it is returning huge datasets in any of the queries, and there is nothing notable in there….
It's only 11:21am here and I already need a drink 🙂
Best Answer
Especially if the filesystem in
/var/tmpisext3(check that withmountcommand), and the mount point is not very big, it really can run out of inodes. And yes, that would cause the symptoms you describe us. Keep your eye ondf -icommand.If that's the case, make somehow
/var/tmpbigger and if possible, switch it to some other filesystem, sayXFS, which has dynamic (=infinite) inodes.I feel for you. Here pal, have a drink.