Why does static vs dynamic process management affect memory usage so much for php-fpm

php-fpm

I recently migrated a client to an EC2 instance running Nginx + PHP-FPM. When I first setup the server I had set pm=static with 40 worker processes. After a week or so I decided to experiment with pm=dynamic with a max of 200 and a min of 30 workers.

What I noticed was that with a static setting, 40 processes took up about 2.3GB of memory, whereas with dynamic, I saw spikes of 60 processes using only 1.2GB of memory.

See the chart below from New Relic, my annotations in red.

new relic memory chart comparing static vs dynamic pm

You can see that during the day on 11/25 I changed from static to dynamic and restarted php-fpm. After that we can see that 48 processes only take 990MB, and 60 processes only took 1.2GB of memory.

What might contribute to this disparity between static and dynamic management? Could it be that with dynamic, I have set max requests to 50? Perhaps with static memory usage was due more to memory leaks rather than to something internal with php-fpm?

Best Answer

pm.max_requests is not specific to any Process Manager (pm) mode. Its benefits are to respawn worker processes after the specified amount of requests they handle individually.

It could avoid memory leaks in extreme cases, but usually it just frees up memory allocations accumulated when memory-hungry scripts were executed. Memory allocations pile up and the total amount of memory allocated grows, but is only freed on exit (thus when respawn).

As you put it, you seem not to have activated pm.max_requests when using pm static. You should have seen a difference, and certainly more than a flat line.

pm dynamic has the added benefit of stopping workers depending on the number of idle processes amongst them (pm.min_spare_servers, pm.max_spare_servers), which are a kind of a metric measuring instant load. Stopping useless processes frees the associated allocated memory at the cost of process handling (CPU), which pm.min_spare_servers counteracts ensuring a safety cushion in case of spikes, maintaining idle workers ready to process requests.

Now, if you really are looking after your memory, pm ondemand is more aggressive, (de)spawning processes as (not) needed (anymore). This mode is the closest to the edge, since it won't absorb spikes as well as pm dynamic, but it is the leanest on memory consumption (with the counterpart of using more CPU for process management purposes).

TL;DR

Memory allocations stack up for any worker process. If a particular request was memory-hungry, it will top the processing worker memory allocation, and this won't be freed until the process stops.

Use pm.max_requests to recycle processes, whatever mode you use.

pm.dynamic will recycle processes on another criterion which is load, and will kill processes when too much of them are idling. Having a higher processes turnover is more likely to prevent them being too much memory-hungry, at the cost of more CPU cycles used for processes management.

Related Topic