Ubuntu – tail-pipe-awk script, fine on Centos but weird on Ubuntu

awkbashcentostailUbuntu

I wrote a script on CentOS and now I'm trying to run it on Ubuntu and it is behaving unexpectedly. Here's something like the script, which I run under bash:

sudo tail -F /var/opt/my-application/log/my-application.log |
awk '
BEGIN {
    ORS=" "
}
{ if ($8 ~ /MATCH-TEXT/) {
            # do a whole bunch of stuff here
            # like look in /proc and calculate CPU and interface stats
    };
 };
} '

Previously when a line with MATCH-TEXT appeared in the $8 field of my log all of the "do a whole bunch of stuff" (the meat of the script) would happen and I would see the output. My application generates MATCH-TEXT lines about once per second (sometimes more, sometimes less) and so does the output of this script.

Now, however, when I start this script I get no output for a long time and then I get maybe 30 lines of output all at once. It seems like the script is queuing the log lines it receives, and then executing several times quickly because the calculations on the CPU and interface are right for "since a tiny fraction of a second ago" rather than "since a second ago".

Stranger still is that commenting out all of the meat of the awk command to replace it with a "print $0" (turning the whole script into an awkward grep command) produces the same result. So I'm confident it is the way the system is executing this script and not something in the script itself.

Meanwhile, in another window a plain "sudo tail -F my-application.log | grep MATCH-TEXT" gets output every second or so (just as expected).

Any ideas what's going on here? Any hints where I should start hunting for what's causing this queueing behavior?

Best Answer

If this is not due to a difference in tail, then I agree with Hauke Laging that it is probably a buffering problem. To flush the output buffer of awk, try adding fflush() after your print statements like I did in the following test code:

tail -F /var/log/apache2/access_log | awk '{ if ($8 ~ /MATCH-TEXT/) { print $0; fflush(); }}'
Related Topic