We have a high traffic website, at peak it has 1000 concurrent users, and in minimum it has 100 users at the same time. In average it has 40,000 to 100,000 visit a day. The problem is sometimes it load very slow(we named this time as disaster time 🙂 ), In in that time when we try to load website with Firefox, it shows waiting...(I tried it with many providers around the world)
We monitor the server at disaster times , CPU load , Memory Usage are normal. Also slow query log of MySQL doesn't any query up to 1 sec. Apache hasn't any errors. iotop doesn't show anything that causes this disaster.
It is very interesting that disaster time and peak times don't have any relations. Sometimes disaster happen at 300 concurrent user and another time different. I can't find any relation between them.
How can I trace the packets at disaster time? I want to know this disaster is our Data Center's fault (such as upstream or firewall) or our server fault(such as Apache configuration, web application or anything else that I don't know).
For additional data just add a comment, then I edit my question to provide the data that you need to answer.
Best Answer
The number of concurrent users / visits has nothing to do with the capacity/performance of the system - it's all about concurrent connections and what those requests are doing.
Adding request response times to your server log would be a start - if these don't reflect the problem then the problem is likely on the network. I notice you make no reference to your webserver logs in your question - did you check them?
You consider that you have high traffic volumes, and your question implies you only have a single server. Why? (multiple servers would add complications to this specific such as load distribution, but would also also simplify much of the diagnostics, however it's a no-rainer for performance and availabiltiy).
Tracking the number of connections and their state also provides essential data in diagnosing the problem.
With a packet capture program - this can be running anywhere from the client to the server. I use wireshark (available on Linux, MSWindows and others)
It would hae been useful if you'd mentioned what version/MPM your server is using and what OS it is running on.