PHP Session Storage in Fault Tolerant Memcached Pool

memcachedphp-fpmphp-memcached

I recently had the opportunity to move a web application from using a Nginx proxy "loadbalancer" to an F5 loadbalancer. Unfortunately during that migration it became clear that the memcached session storage needed to move from the Nginx proxy server to "somewhere". My thinking is that I should put memcached on all 3 of the web servers (the servers that sit behind the F5 in a pool) and use php-memcache or php-memcached to save sessions. Here's the trouble:

I've tried both php-memcache and php-memcached and can neither one to behave properly if one of the servers goes down. My latest attempt was with this configuration:

memcached version 2.2.0 with the configuration settings:

session.save_handler = memcached
session.save_path    = "172.29.104.13:11211,172.29.104.14:11211"

I have nothing special in memcached.ini other than extension=memcached.so.

With this configuration on both server 1 and 2 (I removed 3 temporarily to test), I point JMeter at the F5 VIP and start traffic. I can see memcached.log (the daemon) on both systems, though haven't spent time to decipher, start running.

Then if I stop one of the memcached daemons, traffic begins failing and my return is

session_start(): Write of lock failed

by the memcached that is left remaining.

At the end of the day my goal is simple – I need to be able to a) not run memcached on a single server (single point of failure), and the cluster needs to be resilient to a failure of a pool member.

I've also tried php-memcache but it too fails. For php-memcache the configuration looks like this:

memcache version 3.0.8 (beta) with the configuration settings:

 
session.save_handler = memcache
session.save_path    = "tcp://172.29.104.13:11211, tcp://172.29.104.14:11211"

and in memcache.ini:

extension=memcache.so
[memcache]
memcache.dbpath="/var/lib/memcache"
memcache.maxreclevel=0
memcache.maxfiles=0
memcache.archivememlim=0
memcache.maxfilesize=0
memcache.maxratio=0
memcache.hash_strategy=consistent
memcache.allow_failover=1
memcache.session_redundancy=2

The error here is simply invalid session token (implying to me that the server that was remaining didn't have the session token actually stored, meaning, replication of saving the session wasn't active).

I have not looked at putting session persistence back on the F5, though as a last resort I could do so, and clients trying to connect to the lost member would have to reauthenticate.

Best Answer

It is much simpler to have the clients store session state for you in cookies. This means no session store at all server-side, only a few microseconds of CPU usage to decrypt & verify the cookie. As CPU is by far the most abundant resource in a datacenter, this scheme will perform much better than a look up from memcached or any other server session storage.

See https://github.com/ascorbic/php-stateless-cookies for one implementation, there are many others kicking around. Note the session data should be encrypted but must be authenticated via HMAC or an AEAD cipher. Do not write this code yourself unless you are a cryptographer; use a well-vetted cryoto library.

Facebook uses this technique so any server can answer any user request, even if the user session started in another data center.

Related Solutions

Linux PHP Memcached – Memcache vs Memcached: Key Differences Explained

The short answer: Either one is what you are looking for, but my first choice would be memcache (the first one you listed), purely based on its correct use of nomenclature.

Now here's how I came to that conclusion:

Here is a quick backgrounder in naming conventions (for those unfamiliar), which explains the frustration by the question asker: For many *nix applications, the piece that does the backend work is called a "daemon" (think "service" in Windows-land), while the interface or client application is what you use to control or access the daemon. The daemon is most often named the same as the client, with the letter "d" appended to it. For example "imap" would be a client that connects to the "imapd" daemon.

This naming convention is clearly being adhered to by memcache when you read the introduction to the memcache module (notice the distinction between memcache and memcached in this excerpt):

Memcache module provides handy procedural and object oriented interface to memcached, highly effective caching daemon, which was especially designed to decrease database load in dynamic web applications.

The Memcache module also provides a session handler (memcache).

More information about memcached can be found at » http://www.danga.com/memcached/.

The frustration here is caused by the author of the PHP extension which was badly named memcached, since it shares the same name as the actual daemon called memcached. Notice also that in the introduction to memcached (the php module), it makes mention of libmemcached, which is the shared library (or API) that is used by the module to access the memcached daemon:

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

This extension uses libmemcached library to provide API for communicating with memcached servers. It also provides a session handler (memcached).

Information about libmemcached can be found at » http://tangent.org/552/libmemcached.html.

In summary, both are functionally the same, but they simply have different authors, and the one is simply named more appropriately than the other.

Memcached – Efficient Session Sharing with Pool of Memcache Daemons

Disclaimer: You'd be mad to listen to me without doing a tonne of testing AND getting a 2nd opinion from someone qualified - I'm new to this game.

The efficiency improvement idea proposed in this question won't work. The main mistake that I made was to think that the order that the memcached stores are defined in the pool dictates some kind of priority. This is not the case. When you define a pool of memached daemons (e.g. using session.save_path="tcp://192.168.0.1:11211, tcp://192.168.0.2:11211") you can't know which store will be used. Data is distributed evenly, meaning that a item might be stored in the first, or it could be the last (or it could be both if the memcache client is configured to replicate - note it is the client that handles replication, the memcached server does not do it itself). Either way will mean that using localhost as the first in the pool won't improve performance - there is a 50% chance of hitting either store.

Having done a little bit of testing and research I have concluded that you CAN share sessions across servers using memcache BUT you probably don't want to - it doesn't seem to be popular because it doesn't scale as well as using a shared database at it is not as robust. I'd appreciate feedback on this so I can learn more...

Ignore the following unless you have a PHP app:

Tip 1: If you want to share sessions across 2 servers using memcache:

Ensure you answered Yes to "Enable memcache session handler support?" when you installed the PHP memcache client and add the following in your /etc/php.d/memcache.ini file:

session.save_handler = memcache

On webserver 1 (IP: 192.168.0.1):

session.save_path="tcp://192.168.0.1:11211"

On webserver 2 (IP: 192.168.0.2):

session.save_path="tcp://192.168.0.1:11211"

Tip 2: If you want to share sessions across 2 servers using memcache AND have failover support:

Add the following to your /etc/php.d/memcache.ini file:

memcache.hash_strategy = consistent
memcache.allow_failover = 1