Linux – How to continue redirecting stdout to a file after logrotate moves it

linuxlogrotateredirect

I have a simple script which outputs a bunch of logs to screen and I piped the STDOUT to a file to store the logs. Since this script is long running, I needed to rotate the log files so they are chucked into smaller more manageable ones.

The problem I faced was that once the logrotate moves the current log file into a new one, the newly created log file is not populated with the logs anymore. It seems that the once the original log file is removed, its file handler is lost and redirection won't work anymore.

I also found this post which had the same problem as me and claims that it can be fixed by using >> instead of > to redirect the output. I tested his solution but it didn't work for me. Does anyone have any idea how to keep the redirection work?

Best Answer

You should use the copytruncate directive in your logrotate config for this log file.

copytruncate Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place

Related Solutions

Logrotate Issue – Original File Size Restored After Rotation

This is probably because even though you truncate the file, the process writing to the file will continue writing at whichever offset it were at last. So what's happening is that logrotate truncates the file, size is zero, process writes to the file again, continuing at the offset it left off, and you now have a file with NULL-bytes up to the point where you truncated it plus the new entries written to the log.

od -c after truncate + sudden growth, generated output along the lines of:

0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
33255657600  \0   C   K   B   -   s   e   r   v   e   r       [   h   t   t
33255657620 <more log output>

What this says is from offset 0 to 33255657600 your file consists of null bytes, and then some legible data. Getting to this state doesn't take the same amount of time it would take to actually write all those null-bytes. The ext{2,3,4} filesystems support something called sparse files, so if you seek past a region of a file that doesn't contain anything, that region will be assumed to contain null-bytes and won't take up space on disk. Those null bytes won't actually be written, just assumed to be there, hence the time it takes to go to 0 to 3.5GB don't take a lot of time to. (You can test the amount of time it takes by doing something like dd if=${HOME}/.bashrc of=largefile.bin seek=3432343264 bs=1, this should generate a file of over 3GB in a few milliseconds).

If you run ls -ls on your logfiles after they've been truncated and had a sudden growth again, it should now report a number at the beginning of the line which represents the actual size (in blocks occupied on disk), which probably is orders of magnitude smaller than the size reported by just ls -l.

Where does logrotate save its own log

logrotate does not log anything by default. normally it should be in your cron somewhere, for instance:

$ grep -r -- 'logrotate.conf' /etc/cron*
/etc/cron.daily/logrotate:/usr/sbin/logrotate /etc/logrotate.conf

You can either run that manually to see what is wrong, or redirect the logrotate output to a file in the above cron to see what happened next day.

Likely somewhere the config is incorrect and caused the logrotate run to break.

Best Answer

Related Solutions

Logrotate Issue – Original File Size Restored After Rotation

Where does logrotate save its own log

Related Topic