Increased number of 408 Request Timeouts for null on a webserver

apache-2.2logging

In my log watch records there has been a steadily growing progression of these:

  408 Request Timeout
      null: 694 Time(s)

On my webserver.

Here are some of what look like the contributing requests from the /var/log/apache2/access.log access log:

ip - - date requestsfor"-"? httpcode bytes referrer useragent
75.149.117.146 - - [28/Jan/2013:17:49:47 -0500] "-" 408 0 "-" "-"
65.55.215.247 - - [28/Jan/2013:17:57:40 -0500] "-" 408 0 "-" "-"
205.157.206.75 - - [28/Jan/2013:18:00:21 -0500] "-" 408 0 "-" "-"

Normal access request examples of course have a lot more relevant info like this:

ip - - date request-for httpcode bytes referrer useragent
66.251.23.171 - - [28/Jan/2013:17:45:41 -0500] "GET /images/al/al-mb0608tn.jpg HTTP/1.1" 200 4085 "http://example.com/brands.php?F=S&BrandCode=AL" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

See a larger sampling of my access log here (with a few normal get requests that 408 in with the rest)

I did reverse IP lookups on the ips, and they seem to come from diverse locations in the US and Canada. That could just mean that there is a proxy involved, I suppose? There is a large block from:

96.42.74.117 - - [18/Feb/2013:02:55:58 -0500] "-" 408 0 "-" "-"

That repeats frequently.

I hesitate to jump to conclusions that this is an attack as opposed to a fault, but the number of probes recorded has been steadily increasing at around the same time, e.g. logwatch also says

 A total of 125 sites probed the server
    107.22.9.89
    108.132.76.100
    108.172.60.59
    108.226.133.142
    12.166.56.82
    12.54.94.24

…. on and on with a list of various ips that used detected probes against the server. There is some overlap of the ips listing as "probing" the server and the ips that hit the access log with nulls, so that may suggest an attack, but since the server has to serve requests, it'll be hard to tell a legit timeout from a DOS request timeout attack if that's what is going on here.

How do I debug this problem, or if it is an attack, deal with this attack?

Best Answer

From some researching I did, these are some possibilities leading to 408 message. You can test each to identify the relevant one for your case.

1) Very low Apache TIMEOUT value - your web server could be ending the session before the client even get the chance to send a request.

too many 408 error codes in access log

2) browser predictive optimization - you can easily test this with different browsers. Simply do tail -f /var/log/apache2/access.log, while using relevant key words which would bring up your site on first search page, hover over the link pointing to your site, check if the site preview appears.....all these without clicking on the link which opens your website.

http://forum.linode.com/viewtopic.php?f=10&t=8048

3) denial of service attack - ddos warms such as slow loris can open too many connections to the server without sending a single bye tying up all the Apache processes.

http://blog.spiderlabs.com/2011/07/advanced-topic-of-the-week-mitigating-slow-http-dos-attacks.html

A quote from above linke -

"The trick is to open a connection to the server but not send a single byte. Opening the connection and waiting requires almost no resources by the attacker, but it permanently ties up one Apache process to wait patiently for a request. Apache will wait until the timeout expires, and then close the connection. As of Apache 1.3.31, request-line timeouts are logged to the access log (with status code 408). Request line timeout messages appear in the error log with the level info. Apache 2 does not log such messages to the error log, but efforts are underway to add the same functionality as is present in the 1.x branch."

Related Topic