Ah, there's a flag for this.
instead of using tail -f /var/log/file
we should be using tail -F /var/log/file
tail -F
translates to tail --follow=name --retry
as in;
--follow=name
: follow the name of the file instead of the file descriptor
--retry
: if the file is inaccessible, try again later instead of dying
sed -n '10000000,10000020p' filename
You might be able to speed that up a little like this:
sed -n '10000000,10000020p; 10000021q' filename
In those commands, the option -n
causes sed
to "suppress automatic printing of pattern space". The p
command "print[s] the current pattern space" and the q
command "Immediately quit[s] the sed script without processing any more input..." The quotes are from the sed
man
page.
By the way, your command
tail -n 10000000 filename | head 10
starts at the ten millionth line from the end of the file, while your "middle" command would seem to start at the ten millionth from the beginning which would be equivalent to:
head -n 10000010 filename | tail 10
The problem is that for unsorted files with variable length lines any process is going to have to go through the file counting newlines. There's no way to shortcut that.
If, however, the file is sorted (a log file with timestamps, for example) or has fixed length lines, then you can seek into the file based on a byte position. In the log file example, you could do a binary search for a range of times as my Python script here* does. In the case of the fixed record length file, it's really easy. You just seek linelength * linecount
characters into the file.
* I keep meaning to post yet another update to that script. Maybe I'll get around to it one of these days.
Best Answer
tail -9999f
will do something close to what you want. Add more 9s if your file is bigger.Problems:
tail -f
will wait for a newline before printing anything out.tail
on Solaris (you didn't mention which Solaris but it probably doesn't matter) probably doesn't support that option. It may supporttail -n 9999 -f
. You may have to acquire the GNU version of tail.tail
won't know when you have really finished writing to the file so yourgzip
process will never finish either. I'm not sure what will happen when youctrl-c
to end thetail
process but it's likely that gzip will clean up after itself and remove the file it was working on.My suggestion would be to start your original program up and pipe the output to gzip like this:
That way, gunzip will wait if
my_program
is going slow but will still finish when the true end of the file is indicated bymy_program
finishing.You may need to rewrite your program to write to STDOUT rather than directly to a file.
Edit:
After a look at the man page, three of the issues above can be resolved. Using the
-c <bytes>
option instead of-n <lines>
mitigates problem 1. Using-n +0
or-c +0
mitigates problem 3. Using--pid=<PID>
will make tail terminate when the original program ( running as<PID>
) terminates which mitigates problem 4.