Linux – Get high I/O rate alert from linode.com,how to location problem

linodelinux

Today I get an alert mail from linode.com "Your Linode, Randy, has exceeded the notification threshold (1000) for disk io rate by averaging 1571.81 for the last 2 hours.".

Last time I get alert like that is several month ago, I keeping get alerts, but at the same time my website network traffic never grow. I try everything, but can not location the problem. Finally, I guess the growing log file size is the problem. I use cronolog cut log file, and problem resolved.

Now,I/O rate chart shows I/O rate is keeping growing too, how to I location problem? Is there anything can help me?

My server is Ubuntu 8.04.4 LTS/apache/php/mysql at linode.com.

Best Answer

You can determine which process is thrashing your disk using the iotop package. http://ubuntuforums.org/showthread.php?t=876738

Also, you can change the I/O rate alert threshold in the Settings tab of the Linode web interface.

Related Solutions

Linux – Minimal rate and default class problem for HTB

Since you didn't include the classifiers it's hard to deduct what traffic you exactly mean in each class. For instance, outgoing http or ssh traffic is very important for interactivity, incoming http not so much.

I would guarantee certain bandwidth for each service by saying: I have x kbps for incoming httpd traffic, and it gets divided equally among the users. If you have 10 or 100 users, it's fair." If you have high priority users or low priority users in each of these services you need to have additional classes and classifiers for them.

(Also I hope you know you can only shape outgoing traffic from an interface, NOT the incoming. That means if you want to limit the uplink, you have to work either with the outgoing interface to the internet or use the Intermediate Queuing Device. The lartc.org Guide is a very good resource.)

Linux – iptables rate-limit module problem

Your rule does not appear to specify any particular origin. After accepting 20 NEW connections in a minute, it stops accepting NEW connections.

You need to use the recent module in order for iptables to remember where the connections are coming from and blocking people that connect too fast from the same address. This takes two rules: one for iptables to "learn" the address, and then one for iptables to see how many times that address has hit the server in the specified time:

/sbin/iptables -A INPUT -p TCP -m state --state NEW -d xx.xxx.xxx.xx --dport 80 -m recent --set
/sbin/iptables -A INPUT -p TCP -m state --state NEW -d xx.xxx.xxx.xx --dport 80 -m recent --update --seconds 60 --hitcount 20 -j DROP

This article goes into more details.

Best Answer

Related Solutions

Linux – Minimal rate and default class problem for HTB

Linux – iptables rate-limit module problem

Related Topic