One of my development servers (CentOS 6) has suddenly seen a huge spike in inbound network traffic
It's slowing things down to a crawl, SSH is taking >10 seconds to log in and there is a delay when typing, websites are timing out, Nagios is upset because NRPE checks keep timing out (this is my Nagios host) so there seems to be a sudden huge storm of nework traffic, but I can't figure out where it's coming from. The server has a public IP so is directly accessible, it runs a very restrictive IPTables rule-set (Only allowing 80, 443 and a couple of other utility ports for things like Jenkins). I've tried using tools like iftop but they're not showing anything out of the ordinary. Not sure if this is because IPTables is blocking the connections so they're not showing up, but since I'm not sure whether these are external devices trying to connect to my server, or something else. It seems strange that it's making SSH slow and other services non-responsive but the network traffic started about the same time and I first started getting issues. Where do I look to figure out where this traffic is coming from and how to stop it? I don't have direct access to any of the routers but I can do whatever I want on the server. I looked in /var/log/messages and there are a lot of weird messages about DNS I've never seen before but they don't appear to be errors, just over-verbose logging (see below).
Standard useful stuff;
[sr@ns309372 ~]$ sudo uptime
23:51:41 up 6:30, 3 users, load average: 0.03, 0.12, 0.11
[sr@ns309372 ~]$ sudo free -m
total used free shared buffers cached
Mem: 3920 2197 1722 0 103 1060
-/+ buffers/cache: 1032 2887
Swap: 1019 0 1019
[sr@ns309372 ~]$ sudo tail -n 30 /var/log/messages
Apr 18 23:11:08 ns309372 named[2451]: success resolving 'ftp.halifax.rwth-aachen.de/A' (in 'rwth-aachen.de'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:10 ns309372 named[2451]: success resolving 'deneb.dfn.de/AAAA' (in 'dfn.de'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:10 ns309372 named[2451]: success resolving 'ns1.leaseweb.nl/AAAA' (in 'leaseweb.nl'?) after disabling EDNS
Apr 18 23:11:15 ns309372 named[2451]: success resolving 'ns4.leaseweb.net/AAAA' (in 'leaseweb.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:22 ns309372 named[2451]: success resolving 'pkg.jenkins-ci.org/A' (in 'jenkins-ci.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/A' (in 'ovh.net'?) after disabling EDNS
Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/AAAA' (in 'ovh.net'?) after disabling EDNS
Apr 18 23:33:54 ns309372 named[2451]: success resolving 'vs1.nagios.org/A' (in 'nagios.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:36 ns309372 named[2451]: success resolving 'ns.ripe.net/A' (in 'ripe.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:36 ns309372 named[2451]: success resolving 'dns1.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/A' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:38 ns309372 named[2451]: success resolving 'sec1.apnic.net/AAAA' (in 'apnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:39 ns309372 named[2451]: success resolving 'sec3.apnic.net/AAAA' (in 'apnic.net'?) after disabling EDNS
Apr 18 23:34:40 ns309372 named[2451]: success resolving 'sec3.apnic.net/A' (in 'apnic.net'?) after disabling EDNS
Apr 18 23:34:40 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:02 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:56 ns309372 named[2451]: success resolving 'bitbucket.org/A' (in 'bitbucket.org'?) after disabling EDNS
Apr 18 23:48:26 ns309372 named[2451]: success resolving '113.155.23.94.in-addr.arpa/PTR' (in '155.23.94.in-addr.arpa'?) after disabling EDNS
Apr 18 23:48:29 ns309372 named[2451]: success resolving '8.137.145.217.in-addr.arpa/PTR' (in '217.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:29 ns309372 named[2451]: success resolving '10.169.216.196.in-addr.arpa/PTR' (in '169.216.196.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:29 ns309372 named[2451]: success resolving 'ns2.lacnic.net/AAAA' (in 'lacnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/AAAA' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/A' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:34 ns309372 named[2451]: success resolving 'ns2.afrinic.net/A' (in 'afrinic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:04 ns309372 named[2451]: success resolving 'ns2.ecogeek.org/A' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:05 ns309372 named[2451]: success resolving 'ns1.ecogeek.org/AAAA' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:05 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
[sr@ns309372 ~]$ sudo ifconfig
eth0 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51
inet addr:188.165.192.119 Bcast:188.165.192.255 Mask:255.255.255.0
inet6 addr: fe80::227:eff:fe0b:8651/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:456082 errors:0 dropped:91 overruns:0 frame:0
TX packets:821015 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:59793427 (57.0 MiB) TX bytes:1008283171 (961.5 MiB)
Interrupt:43 Base address:0xc000
eth0:0 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51
inet addr:94.23.155.32 Bcast:94.23.155.32 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:43 Base address:0xc000
eth0:1 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51
inet addr:94.23.155.113 Bcast:94.23.155.113 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:43 Base address:0xc000
eth0:2 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51
inet addr:178.32.48.78 Bcast:178.32.48.78 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:43 Base address:0xc000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:169675 errors:0 dropped:0 overruns:0 frame:0
TX packets:169675 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:172646550 (164.6 MiB) TX bytes:172646550 (164.6 MiB)
[sr@ns309372 ~]$ sudo sar -n DEV 1 3
Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64 (ns309372.ovh.net) 18/04/12 _x86_64_ (2 CPU)
23:57:35 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
23:57:36 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:36 dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:36 eth0 13.00 8.00 1.11 5.08 0.00 0.00 0.00
23:57:36 tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:36 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:36 ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:36 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
23:57:37 lo 10.00 10.00 2.92 2.92 0.00 0.00 0.00
23:57:37 dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:37 eth0 11.00 8.00 0.91 3.47 0.00 0.00 0.00
23:57:37 tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:37 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:37 ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:37 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
23:57:38 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:38 dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:38 eth0 7.00 9.00 7.54 1.33 0.00 0.00 0.00
23:57:38 tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:38 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
23:57:38 ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
Average: lo 3.33 3.33 0.97 0.97 0.00 0.00 0.00
Average: dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 10.33 8.33 3.19 3.30 0.00 0.00 0.00
Average: tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
I have a number of 'virtual' interfaces to support multiple IPs, only one physical interface
Best Answer
I would do two things:
Log traffic in iptables. You create a rule and use the
LOG(uses kernel log system) andULOG(directs logs to sockets instead of the kernel system) targets. So you would probably be interested in all-A INPUTpackets being sent to-j LOGwith perhaps--log-prefix "incoming packets "and--log-level 6(it follows syslog levels)Monitor your processes to see if any are the recipient of the vast bandwidth. I suggest the use of nethogs. It may be that traffic is making it past iptables, but not being written to disk but instead being immediately discarded by some oddball process.