I've got a network of static file serving servers. I use nginx to serve the files, and munin to monitor the network traffic. I want to know the output bandwidth of the server that goes to each IP address downloading a file to evaluate the bandwidth each Internet Service Provider in my country is downloading from my servers. The average output bandwidth of servers is about 700MB/s (9 servers, most of them have 4 1Gbits/s ports bonded). How can I do this?
Nginx – Monitoring network output bandwidth of server for each IP address
monitoringmuninnginxntop
Related Solutions
To analyze which device is generating/receiving the most traffic you are going to need network equipment that can collect these statistics.
As mentioned in a previous answer the easiest way to do this is with a managed switch which will allow you to poll the traffic on each port.
Another method is a router that can either collect traffic accounting data on a per MAC address level or export network traffic in "NetFlow" format which can be analyzed by a tool such as NFSen
The final method is going to be installing some sort of packet sniffer (e.g. a PC running Wireshark). In order to get this to work on an unmanaged switch you will need to either install the sniffing PC in-line to the traffic (i.e. two network cards configured for L2 forwarding) or install some form of Ethernet tap between the switch and router.
The less managed network hardware you have in the network the more networking knowledge you will need to collect the statistics.
The issue was me assuming that the linode.com graph peaks were true peaks. It turns out the graph uses 5 minute average data points, thus my peak appeared to be 24mbits when in fact I was hitting the 50mbit cap.
Now that they have raised it to 100mbits, my benchmarks immediately went up to the new outbound traffic limit.
If only I had noticed that earlier! A lot of my reasoning hinged on the idea that I was not hitting an outbound traffic limit due to that graph.
Now, I peak at 370 requests per second, which is right under 100mbps at which point I start getting a "backlog" of requests, and response times start going up.
I can now increase maximum concurrency by shrinking the page; with gzip enabled I get 600RPS.
I do still run into problems when I suddenly peak and the backlog of pending requests (limited by bandwidth) start to accumulate, but that sounds like a different question.
It's been a great lesson in optimization / reading this data / narrowing down the possible problems. Thank you so much for your input!
Best Answer
The first approach that comes to mind is to have your web servers simply log the requests they handle and use a log parser to generate usage statistics. The combined log format is a standard commonly supported by log parsing web analytics software.
If your web servers run as cluster rather than have each server maintain their own log files which you would need to merge to get aggregated results, you can use the syslog protocol to have all nodes transmit their log entries to a central logging host and generate usage statistics there.
The relevant settings are documented in the nginx manual.
This older question mentions some of the web analytics software, as does Wikipedia