I am running memcached on my server and when it hits 600+ req/s it becomes unstable and causes a big load of problems. It appears when the request rate gets that high, my PHP applications at random times are unable to connect to the memcache server, causing slow load times which makes nginx and php-fpm freak out and I receive a bunch of 104: Connection reset by peer errors in my nginx logs.
I would like to point out that in my memcache server I have 'hot objects' – objects that at times receive 90% of the memcache requests. I also noticed when so many requests hit a single object, it slightly adds a little more load time to the overall page (when it manages to load).
I would greatly appreciate any help to this problem. Thanks so much!
Best Answer
This really sounds like an issue with the networking layer. When you run into this issue can you grab the output of netstat -ano and see how many connections are in certain buckets? If you see a ton of connections that aren't in ESTABLISHED but rather in TIME_WAIT/FIN_WAIT etc you probably need to enable time wait reuse and recycling. From:
http://www.speedguide.net/articles/linux-tweaking-121