Unix Log Files – Best Way to Reduce Size of Active Log File

bashlinuxlog-filesunix

On a linux server, I want to reduce the size of a log file which is several GB big. Cutting off the top half, or maybe the first million lines would work.

Best Answer

This comes up in interviews quite often...

Are you looking to truncate the file without disrupting the processes? Is any of the information in the log file valuable? If so, I usually "zero" the file with a simple bash string.

: > /var/log/badlogfile

This comes up in situations where you may have an application that can't be restarted in a controlled manner. Let's say it's a financial trading application and the program can't be halted or restarted during the trading day. However, the log files are growing at some obscene rate due to an application bug. Truncating the log files using the method above or below can keep the system running.

Also see: http://www.cyberciti.biz/faq/truncate-large-text-file-in-unix-linux/

Related Solutions

Linux – filesystem for millions of small files

In terms of random seeks Reiser wins, followed by EXT4, followed by JFS. I'm not sure if this will correlate exactly to directory lookups, but it seems like it would be an indicator. You'll have to do your own tests for that specifically. EXT2 beats the pants off everything for file creation times, likely due to its lack of a journal, still EXT4 beats everything except Reiser which you may not want to use due to hans reiser's current status.

You might want to look into drives that support NCQ, and make sure your install is setup to use it. Under heavy seeking it should provide a speed boost.

Lastly, make sure your machine has a ton of ram. Since the files aren't often updated, linux will end up caching most of them to ram if it's got free space. If your usage patterns are right, this will give you a massive speed boost.

Best Way to Check if a Volume is Mounted in a Bash Script – Linux Bash

Avoid using /etc/mtab because it may be inconsistent.

Avoid piping mount because it needn't be that complicated.

Simply:

if grep -qs '/mnt/foo ' /proc/mounts; then
    echo "It's mounted."
else
    echo "It's not mounted."
fi

(The space after the /mnt/foo is to avoid matching e.g. /mnt/foo-bar.)

Best Answer

Related Solutions

Linux – filesystem for millions of small files

Best Way to Check if a Volume is Mounted in a Bash Script – Linux Bash

Related Topic