We rotate and compress our Apache logs each day but it's become apparent that this isn't frequently enough. An uncompressed log is about 6G, which is getting close to filling our log partition (yep, we'll make it bigger in the future!) as well as taking a lot of time and CPU to compress each day. We have to produce a gziped log for each day for our stats processing. Obviously we could move our logs to a partition with more space but I also want to spread the compression overhead throughout the day.
Using Apache's rotatelogs we can rotate and compress the log more often — hourly, say — but how can I concatenate all the hourly compressed logs into a running compressed log for the day, without decompressing the previous logs? I don't want to uncompress 24 hours' worth of data and recompress it because that has all the disadvantages of our current solution.
Gzip doesn't seem to offer any append or concatenate option but perhaps I've missed something obvious. This question suggests straight shell concatenation "works" in that the archive can be decompressed but that gzip -l doesn't work seems a bit dodgy.
Alternatively, perhaps this is still a bad way to do things. Other suggestions are welcome — our only constraints are our relatively small log partitions and the need to provide a daily compressed log.
Best Answer
The gzip man page should have what you want, but you can concatate them directly:
Compression is not as good as if it was just one file compressed, but you can recover with: