Over the last day or so, one of instances has had a large amount of bandwidth consumed. This means that we have nearly exceeded our allowance (roughly the same amount inbound and outbound).
Looking though the logs, the only thing I can see is lots and lots of 400 172 errors in the nginx access.log with the same text string.
I have changed nginx to a different port, implemented fail2ban but since the traffic is coming from different IPs this isn't working. I have also got our VPS provider to change our VPS' IP.
Fail2ban is currently dropping all connections to Port 80 which isn't ideal as we'd like to use this port.
Is there anything we can do to improve the situation? If we are dropping suspect traffic will this still count towards our allowance?
More Info
I managed to get more details by changing the nginx error log level.
The only error that seems to be occurring is cleint sent an invalid request while reading client request line.
The domain is new and has not being used before ( it is a brand new subdomain on one of long term existing domains).
I wil check whether it is using the same path.
Also is there any reason why its increasing outboubd traffic is it just because the inbound packets are being blocked?
Best Answer
In general, any traffic that reaches you or that reaches the gateway (when outbound from your host) counts against your bandwidth allowance. This is true even if the activity is insubstantial or consists only of traffic blocked by your host firewall.
Dropping the malicious traffic as soon as possible is ideal. However, it sounds like you might have a problem involving a hosting plan that has large bandwidth but small transfer quotas - you are experiencing a fundamental problem with such plans, if this is the case.
What you do from here depends on what the traffic is like. If it is only a few hosts, consider blocking those hosts using the VPS provider's firewall, if they make this option available. If it is a search engine robot, add a robots.txt prohibiting them from crawling whatever path is generating the errors.
172in your logs isn't part of the error code - these lines only mean your server returned the error code400 Bad Requestand the error page it sent back was 172 bytes in length.Since it is error 400, it's most likely not a search engine (unless you are running a strange CGI application), but without knowing what the query string (including method) was, it's hard to say what is happening. But, try addressing it based on what it looks like the "attacker" is trying to do.
Blocking the requests in nginx is unlikely to change much.
Instead, you can consider blocking these requests at the network layer by dropping offending traffic; this means the originator of the traffic will end up with a bunch of open connections. If the velocity does not change, odds are fairly good it is malicious. You might consider sending an abuse complaint to the originating ISP if it is all coming from the same network and it looks like a flood rather than an automated process trying to do something. It might be as simple as someone else having had your domain name before you.