Linux – How to avoid log files growing too big

Linux has a kernel ring buffer. You can use dmesg to display it.

Or here is a Linux kernel module that appears to do what you want.

What is emlog?

emlog is a Linux kernel module that makes it easy to access the most recent (and only the most recent) output from a process. It works just like "tail -f" on a log file, except that the storage required never grows. This can be useful in embedded systems where there isn't enough memory or disk space for keeping complete log files, but the most recent debugging messages are sometimes needed (e.g., after an error is observed).

The emlog kernel module implements simple character device driver. The driver acts like a named pipe that has a finite, circular buffer. The size of the buffer is easily configurable. As more data is written into the buffer, the oldest data is discarded. A process that reads from an emlog device will first read the existing buffer, then see new text as it's written, similar to monitoring a log file using "tail -f". (Non-blocking reads are also supported, if a process needs to get the current contents of the log without blocking to wait for new data.)

If your version of sed has the --in-place switch you could use that. It still does a cp and mv but it's behind the scenes so the script would look a little less messy depending on how you feel about sed syntax. Otherwise your tail (you missed one in your edit) and append method is the way I'd approach it.

Another approach using sed would be to keep your log upside down with the newest entry at the beginning.

sed -ni '1s/^/New Entry\n/;100q;p' logfile

That's all there is to it (other than getting "New Entry" to be what you want). That will keep a rolling 100 line log file in reverse chronological order. The advantage is that you don't have to know how many lines are in the file (tail takes care of this for you, but sed won't). This assumes to some extent that your log entries consist of a single line. If you're logging a significant amount of output from a script I think you're back to tail and append.

Yet another option which might work well for detailed multi-line output would be to let the filesystem do the work for you. Each log entry would be a separate file.

#!/bin/bash
# *** UNTESTED ***
logdir="/path/to/your/log/dir"

# run the script and log its output to a file named with the date and time
script > "$logdir/$(date "+%Y%m%dT%H%M%S").log"

# maintain the logs
# keep the last 100 days of daily logfiles
# (you could keep the last 100 hours of hourly logfiles, for example,
# by using -mmin and doing some math)

# the count is intended to keep old entries from being deleted
# leaving you with nothing if there are no new entries
count=$(find "$logdir" -maxdepth 1 -type f | wc -l)
if (( count > 100 )) 
then
    find "$logdir" -maxdepth 1 -type f -mtime +100 -delete
fi