In summary, I had a case where all Apache threads were hung because all of them were waiting for a TCP ACK from clients after having sent the HTTP page, and because of that, the Apache threads were waiting 300s (Timeout value of conf) before going to the next request. And then the same thing happened.
The things happened as is: there was a sudden peak of traffic, apache & db servers load goes up as expected, and at some point, every apache thread goes into this state, and the apache & db load goes to 0. And it stayed this way for hours. After a apache restart, pages are processed normally again, load goes up, and again this happen and load goes to 0.
Now, if this was an attack or a consequence of some bad soft/hardware is yet to be known.
To the details: When everything is hung, you go see the server-status page of apaches, and all threads are marked as "W" (working), and you see the timer SS (time the request is being processed) going up up to 300s, and then go next to the next HTTP request and start again to 300s.
In the socket part via netstat, we see all the sockets from these Apache thread hung in CLOSE_WAIT with a high Send-Q value (packets sent not acknowledged).
Using strace, we indeed see Apache doing a poll() of 300s on the socket, waiting for the packets to be acknowleged.
Now, whether it is an attack or some bad network configuration that lost the packets, my question is: how do we prevent this? It seems to be a particular nasty kind of attack.
I am aware about the slow loris attack, when you make a HTTP request very slowly, and this can be mitigated if you have a CDN, a reverse proxy, …
But for this particular case, I am not seeing something that can prevent that?
How would you prevent this to happen?
Thanks!
Best Answer
put a litespeed infront of apache. That can solve your problem. That is exactly what we do for slow request, reflection and regular ddos problems. Solve means at least at some scale i meant.
Edited Section:
Sure,
Litespeed is a web server/load balancer a type of reverse proxy also rate base limiting aviable ddos attack mitigation software. We had the same type of attacks you meantioned and much more worse close to 400.000 http request per second based attacks. We install a litespeed server with 24 core 128 gb ram litespeeds runs in mounted ram and operation system runs in raid0 striped ssd disks. And that solved and stop this attack.
Also via configuration and setup you can stop the slow loris attack. We still have a load balancer, but the incoming traffic passes through first the litespeed then proxeyed to our hw loadbalancer.