MongoDB – Proper Way to Rotate Logs

While the world is divided on whether the humble named pipe is friend or foe, it is probably the simplest solution to your problem. It does have a few drawbacks (in that you need to create the pipes ahead of time), but it eliminates the need for a cron and allows you to used the logging pipe-filter of your choice.

Here's an example using cronolog on access.log:

1. Pick a path for our named pipe. I intend to keep my logs in /var/log/nginx, so I'll put my pipes there as well. The name is up to you; I append .fifo, and it's access.log, so mine will be at /var/log/nginx/access.log.fifo.
2. Delete the file if it exists.

3. Make a named pipe for the logfile:

   mkfifo /var/log/nginx/access.log.fifo
   

4. Configure nginx.conf to point the log at the pipe you just made:

   access_log /var/log/nginx/access.log.fifo;
   

5. Modify your init.d script to start the log rotator listening to the pipe before we start the server:

   LOGS="/var/log/nginx"
   pkill -f "/usr/sbin/cronolog --symlink $LOGS/access.log"
   ( cat $LOGS/access.log.fifo | /usr/sbin/cronolog --symlink $LOGS/access.log "$LOGS/%Y/%m/%d/access.log" ) &
   

   A similar commandline would be used for rotatelogs should you prefer it to cronolog -- see their docs for the syntax.

   If your distrobution has a start-stop-daemon, you should be using that instead, as it theoretically has whatever special knoweldge about your platform there is, and taking care of pkill for you. Simply wrap the command in a script, and pass it as --exec to start-stop-daemon in your init.d/nginx.

This is probably because even though you truncate the file, the process writing to the file will continue writing at whichever offset it were at last. So what's happening is that logrotate truncates the file, size is zero, process writes to the file again, continuing at the offset it left off, and you now have a file with NULL-bytes up to the point where you truncated it plus the new entries written to the log.

od -c after truncate + sudden growth, generated output along the lines of:

0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
33255657600  \0   C   K   B   -   s   e   r   v   e   r       [   h   t   t
33255657620 <more log output>

What this says is from offset 0 to 33255657600 your file consists of null bytes, and then some legible data. Getting to this state doesn't take the same amount of time it would take to actually write all those null-bytes. The ext{2,3,4} filesystems support something called sparse files, so if you seek past a region of a file that doesn't contain anything, that region will be assumed to contain null-bytes and won't take up space on disk. Those null bytes won't actually be written, just assumed to be there, hence the time it takes to go to 0 to 3.5GB don't take a lot of time to. (You can test the amount of time it takes by doing something like dd if=${HOME}/.bashrc of=largefile.bin seek=3432343264 bs=1, this should generate a file of over 3GB in a few milliseconds).

If you run ls -ls on your logfiles after they've been truncated and had a sudden growth again, it should now report a number at the beginning of the line which represents the actual size (in blocks occupied on disk), which probably is orders of magnitude smaller than the size reported by just ls -l.