Redis for logging

loggingredis

I am thinking of using Redis for web application logging purposes. I googled that there are people using this approach by dumping the logs into a Redis queue/list, and then a scheduled worker to write to disk.

http://nosql.mypopescu.com/post/8652869828/another-redis-use-case-centralized-logging

I wish to seek understanding that why not directly use Redis to persist to disk? If I have allocated a small server which Redis will write to, separated from the database, app server, is it feasible to use Redis to persist the logs directly?

I also need help in querying Redis by datetime, user, etc. For example, each log is as follow.

datetime=>2012-03-24 17:45:12
userid=>123
message=>test message
category=>my category

How can I query for results within a datetime range, by a specific user, of a particular category?

Thanks!

Best Answer

You need to keep in mind that Redis is an in-memory database (even if it can persist the data to disk). The data you put in Redis have to fit in memory.

The proposal in the article you mention is about using Redis as a distributed queuing system. Worker processes dequeue the items and write them to disk, so there are not that many items in Redis memory. This design has a flaw: if the worker processes cannot write the data fast enough to disk, Redis memory consumption will explode - so it has to be limited by configuration (Redis maxmemory parameter) or software (trim the queue at insert time, or empty the queue when it is full).

Now your proposal does not really work since all the data you write in Redis will be kept in memory (even if they are persisted to disk by Redis itself).

Another point is you cannot query Redis. Redis is not a relational database, it supports no ad-hoc query mechanism, only commands involving previously defined access paths. If you want to search data with different parameters, you have to anticipate all the possible searches and build the relevant data structures (set, sorted sets, etc ...) at insert time.

Another store (MongoDB, or a relational database) will probably be a much better fit for your use case.

Related Topic