How to monitor the backlog on a Redis pubsub subscription

redis

As documented in an issue on Google Code, when a Redis pubsub client can't keep up with the rate at which messages are being PUBLISHed to the channel, Redis maintains an ever-growing message backlog for that client, using more and more memory as time goes on.

I'd like to detect when this is happening and get alerts over it, or at least be able to check which pubsub subscribers have backlogs and how big they are. Is there a way to do this?

Best Answer

The bug described in https://code.google.com/archive/p/redis/issues/525 no longer exists. While there's no documented way to get advanced warning that a PubSub client is not keeping up, Redis will kill connections from slow clients to protect itself from running out of memory and inform you of this via its log file. Per the docs:

Output buffers limits

... it is possible that a client sends more commands producing more output to serve at a faster rate at which Redis can send the existing output to the client. This is especially true with Pub/Sub clients in case a client is not able to process new messages fast enough.

[This] will cause the client output buffer to grow and consume more and more memory. For this reason by default Redis sets limits to the output buffer size for different kind of clients. When the limit is reached the client connection is closed and the event logged in the Redis log file.

There are two kind of limits Redis uses:

  • The hard limit is a fixed limit that when reached will make Redis closing the client connection as soon as possible.

  • The soft limit instead is a limit that depends on the time, for instance a soft limit of 32 megabytes per 10 seconds means that if the client has an output buffer bigger than 32 megabytes for, continuously, 10 seconds, the connection gets closed.

Different kind of clients have different default limits:

...

  • Pub/Sub clients have a default hard limit of 32 megabytes and a soft limit of 8 megabytes per 60 seconds.

...

It is possible to change the limit at runtime using the CONFIG SET command or in a permanent way using the Redis configuration file redis.conf. See the example redis.conf in the Redis distribution for more information about how to set the limit.

Related Topic