If your keys have unequal access patterns you will see unequal traffic to each memcached node. e.g. If you have 2 keys, one of which a is get/set 500 times per second and one b which is get/set 250 times per second then the node which contains a will have twice as much traffic as the node which contains b.
In my case, we had 8 memcached nodes with a few thousand keys. One of those keys was doing about 800 gets/sec at peak traffic and almost every other key was doing less than 1 get/sec. The memcached node which had the busy key exhibited significantly higher traffic than the others.
If you want to balance the traffic equally to each of your memcached nodes then you either need to:
- Play games with your keying to make sure that your busy keys are spread out properly.
- Switch to using repcached or Membase to replicate the keys across multiple nodes
We've encountered this as well, but have managed to dig into it and work out exactly what's going on. The symptoms we encountered was that memcache (with a reasonably large memory allocation running across multiple servers) started to evict content. This is undesirable, as it can adversely affect current visitors on the site.
By monitoring network traffic, we saw messages from PHP to Memcache as follows:
set memc.sess.key.abcdabcdabcdabcdabcdabcd 0 0 1823 data...
It's the second zero that causes the problems - this dictates the length of time that memcache caches the item. By setting it to zero, memcache never expires this item. In your case this meant users could return hours later and continue accessing your site. In our case, memcache was filling up and causing desired data to be evicted.
I dug further, and it boils down to the PHP memcached extension. As of 1.0.2 (which we're running), this code reads:
sess_lifetime = zend_ini_long(ZEND_STRL("session.gc_maxlifetime"), 0);
if (sess_lifetime > 0) {
expiration = time(NULL) + sess_lifetime;
} else {
expiration = 0;
}
In this excerpt, it's ZEND_STRL("session.gc_maxlifetime") which isn't returning the expected value. This has been reported as a bug to PHP, and a fix to the memcached library is described at https://bugs.php.net/bug.php?id=59641.
I've deployed this patch, reviewed network traffic, and found that it does set the expiry time as expected.
Best Answer
if you build memcached from source http://code.google.com/p/memcached/downloads/list
there is a memcached-debug version you can run.
Then rather than the usual memcached server you could just do something like: /usr/local/sbin/memcache-debug -v 9 > /tmp/memcached.log
I think it shows every transaction and result code.
Another way,would be to have just a teensy bit of wrapper code in the application that logs each put and get and the result.