Linux – logrotate does not compress /var/log/messages

linuxlog-fileslogginglogrotatesyslog

Over time I noticed some logs in /var/log such as auth, kern and messages were getting huge. I made logrotate entries for them:

$ cat /etc/logrotate.d/auth.log 
/var/log/kern.log {
    rotate 5
    daily
}
$ cat /etc/logrotate.d/kern.log 
/var/log/kern.log {
    rotate 5
    daily
}
$ cat /etc/logrotate.d/messages 
/var/log/messages {
    rotate 5
    daily
    postrotate
        /bin/killall -HUP syslogd
    endscript
}

also I have the compress option enabled:

$ grep compress /etc/logrotate.conf 
# uncomment this if you want your log files compressed
compress

This works great for auth.log, kern.log and others, meaning that each of those logs is gzipped and rotated, with the last 5 days of logs retained. /var/log/messages however is not being compressed, resulting in way more than 5 days of logs:

$ ls /var/log/messages*
/var/log/messages           /var/log/messages-20100213
/var/log/messages-20100201  /var/log/messages-20100214
/var/log/messages-20100202  /var/log/messages-20100215
/var/log/messages-20100203  /var/log/messages-20100216
/var/log/messages-20100204  /var/log/messages-20100217
/var/log/messages-20100205  /var/log/messages-20100218
/var/log/messages-20100206  /var/log/messages-20100219
/var/log/messages-20100207  /var/log/messages-20100220
/var/log/messages-20100208  /var/log/messages-20100221
/var/log/messages-20100209  /var/log/messages-20100222
/var/log/messages-20100210  /var/log/messages-20100223
/var/log/messages-20100211  /var/log/messages-20100224
/var/log/messages-20100212

As is explained in another logrotate question on ServerFault, the old logs are (most likely) not being removed because the file endings are different for each file. This appears to be because the files are not being gzipped.

What can I do to have /var/log/messages compressed and rotated with the last 5 days of logs retained just like all my other log files? What am I missing?

EDIT 1: additional information as requested in the first couple answers.

I'm running Gentoo Linux. My /etc/logrotate.conf file:

$ cat /etc/logrotate.conf 
# $Header: /var/cvsroot/gentoo-x86/app-admin/logrotate/files/logrotate.conf,v 1.3 2008/12/24 20:49:10 dang Exp $
#
# Logrotate default configuration file for Gentoo Linux
#
# See "man logrotate" for details
# rotate log files weekly
weekly
#daily
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
compress
# packages can drop log rotation information into this directory
include /etc/logrotate.d
notifempty
nomail
noolddir
# no packages own lastlog or wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
    rotate 1
}
/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

/etc/logrotate.d contains my custom config files as mentioned above along with configs for mysql, rsync, etc installed by those packages.

My root crontab is empty:

$ sudo crontab -l
no crontab for root

I checked all /etc/cron.{daily,hourly,monthly,weekly} for anything syslog related, and there's a script which rotates /var/log/syslog and /var/log/auth.log.

Next, I made a /var/log/messages-only logrotate config file as suggested by CarpeNoctem:

$ cat logrotate-messages 
weekly
rotate 4
create
dateext
compress
notifempty
nomail
noolddir
/var/log/messages {
    rotate 5
    daily
    postrotate
        /bin/killall -HUP syslogd
    endscript
}

Then I ran logrotate manually:

$ logrotate -d logrotate-messages -f
reading config file logrotate-messages
reading config info for /var/log/messages 

Handling 1 logs

rotating pattern: /var/log/messages  forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/messages
  log needs rotating
rotating log /var/log/messages, log->rotateCount is 5
dateext suffix '-20100224'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
renaming /var/log/messages to /var/log/messages-20100224
creating new /var/log/messages mode = 0644 uid = 0 gid = 0
running postrotate script
running script with arg /var/log/messages : "
        /bin/killall -HUP syslogd
"
compressing log with: /bin/gzip
$ which gzip
/bin/gzip
$ file /bin/gzip
/bin/gzip: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped

According to the log above, logrotate compressed the log with /bin/gzip, but I don't see a compressed messages file in /var/log. Also, globbing for old rotated files failed.

EDIT 2: adding debug output of logrotate run after appending a .gz suffix to old /var/log/message-* files.

We start out with:

$ ls /var/log/messages*
/var/log/messages              /var/log/messages-20100222.gz
/var/log/messages-20100219.gz  /var/log/messages-20100223.gz
/var/log/messages-20100220.gz  /var/log/messages-20100224.gz
/var/log/messages-20100221.gz

Then run logrotate with our custom config file:

$ logrotate -d logrotate-messages -f
reading config file logrotate-messages
reading config info for /var/log/messages 

Handling 1 logs

rotating pattern: /var/log/messages  forced from command line (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/messages
  log needs rotating
rotating log /var/log/messages, log->rotateCount is 5
dateext suffix '-20100224'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
removing /var/log/messages-20100219.gz
removing old log /var/log/messages-20100219.gz
destination /var/log/messages-20100224.gz already exists, skipping rotation

This time, logrotate's glob succeeds and finds the sixth compressed logfile, intending to remove it. The file isn't actually removed; I guess that's because we're running in debug mode.

I'm curious whether enabling the delaycompress option for /var/log/messages will help. I enabled it and will check the results the next morning.

Best Answer

Adding delaycompress to the configuration section for /var/log/messages solved the problem.

From man logrotate:

   delaycompress
          Postpone  compression of the previous log file to the next rota‐
          tion cycle.  This only has effect when used in combination  with
          compress.   It  can  be used when some program cannot be told to
          close its logfile and thus might continue writing to the  previ‐
          ous log file for some time.

I guess sysklogd, my syslog daemon, cannot be told to close its logfile, and thus this is necessary.

Interestingly, the original configuration I had (without the delaycompress directive), came straight out of man logrotate (except I changed weekly to daily):

   # sample logrotate configuration file
   compress

   /var/log/messages {
       rotate 5
       weekly
       postrotate
           /usr/bin/killall -HUP syslogd
       endscript
   }

Related Solutions

Cron – Logrotate ERROR: No file found for /var/log/apache/error.log

Ok, it seems that it has nothing to do with logrotate. I think the errors you see are from fail2ban which is looking for these files. Try tu run manually the postscript from fail2ban in /etc/logrotate.d/fail2ban (o something like that) and you will see the errors.

Linux – Logrotate Successful, original file goes back to original size

This is probably because even though you truncate the file, the process writing to the file will continue writing at whichever offset it were at last. So what's happening is that logrotate truncates the file, size is zero, process writes to the file again, continuing at the offset it left off, and you now have a file with NULL-bytes up to the point where you truncated it plus the new entries written to the log.

od -c after truncate + sudden growth, generated output along the lines of:

0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
33255657600  \0   C   K   B   -   s   e   r   v   e   r       [   h   t   t
33255657620 <more log output>

What this says is from offset 0 to 33255657600 your file consists of null bytes, and then some legible data. Getting to this state doesn't take the same amount of time it would take to actually write all those null-bytes. The ext{2,3,4} filesystems support something called sparse files, so if you seek past a region of a file that doesn't contain anything, that region will be assumed to contain null-bytes and won't take up space on disk. Those null bytes won't actually be written, just assumed to be there, hence the time it takes to go to 0 to 3.5GB don't take a lot of time to. (You can test the amount of time it takes by doing something like dd if=${HOME}/.bashrc of=largefile.bin seek=3432343264 bs=1, this should generate a file of over 3GB in a few milliseconds).

If you run ls -ls on your logfiles after they've been truncated and had a sudden growth again, it should now report a number at the beginning of the line which represents the actual size (in blocks occupied on disk), which probably is orders of magnitude smaller than the size reported by just ls -l.

Best Answer

Related Solutions

Cron – Logrotate ERROR: No file found for /var/log/apache*/*error.log

Linux – Logrotate Successful, original file goes back to original size

Related Topic

Cron – Logrotate ERROR: No file found for /var/log/apache/error.log