Running top
on our server gives us
load average: 68.67, 63.48, 60.30
We suspect this is from too many httpd connections.
running:
netstat -tun 2>/dev/null | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
gives us (all IPs converted to private addresses):
418
176 192.168.1.1
41 192.168.1.2
8 192.168.1.3
5 192.168.1.4
5 192.168.1.5
4 192.168.1.6
2 192.168.1.7
2 192.168.1.8
2 192.168.1.9
2 127.0.0.1
1 servers)
1 Address
1 192.168.1.10
1 192.168.1.11
As you can see the 192.168.1.1
(converted from WAN address, just for here) it appears has 176 connections going to our server. A remote look up of this IP brings it back resolving to a DDOS service.
We've run
sudo iptables -I INPUT -m iprange --src-range 192.168.0.0-192.168.0.255 -j DROP
to try to drop all ranges associated with their full range but requests are still showing up when running the netstat command.
Is there something wrong with the IPtables command or netstat command?
We ran
sudo service iptables save
sudo service httpd restart
to store it and make it active and then
sudo iptables --list
to confirm it was added, which it was. Not sure if there's something we're missing. Thanks.
UPDATE
Running iptables -L -nv
shows
pkts bytes target prot opt in
30179 1793K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 source IP range STARTRANGE-ENDRANGE
Does this mean 30179
requests were blocked?
Also our IPtables looks like this (STARTRANGE/ENDRANGE are actual quad octet addresses)…
Chain INPUT (policy DROP)
target prot opt source destination
DROP all -- anywhere anywhere source IP range STARTRANGE-ENDRANGE
DROP all -- anywhere anywhere source IP range STARTRANGE-ENDRANGE
DROP all -- anywhere anywhere source IP range STARTRANGE-ENDRANGE
ACCEPT tcp -- anywhere anywhere tcp dpt:http
ACCEPT tcp -- anywhere anywhere tcp dpt:https
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:mysql
DROP all -- anywhere anywhere source IP range STARTRANGE-ENDRANGE
Shortened Netstat output (command run: netstat -n | grep '192.168'
):
tcp 0 1 OUR_SERVER_IP:44531 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44675 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44600 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44587 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44641 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44578 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44626 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44604 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44541 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44678 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44625 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44661 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44543 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44602 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44644 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44580 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44688 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44683 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44588 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44556 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44681 192.168.1.1:443 SYN_SENT
tcp 0 1 OUR_SERVER_IP:44631 192.168.1.1:443 SYN_SENT
Best Answer
Now you've finally posted the
netstat
output in question (thank you for that), we can see that the connections it's picking up almost certainly aren't from the remote server, as you'd supposed. Instead, your server is trying hard to initiate connections to that remote server on port 443 (HTTPS). That's why they aren't being blocked by anINPUT
chain rule; the first packet is outbound - only the response the remote server generates is blocked by theINPUT
rule, leaving the connection inSYN_SENT
until it times out.Use of
netstat -apn
has shown that it's the HTTPD server on your system that's making these connections. You don't know of any reasons why your server should be doing this, so you're going away to take a long hard look at its setup.