Apache log meaning of %T and %D

In addition to your test script try also running a benchmark application like ApacheBench (ab) or Siege. Also run this and your test script on the local server itself. If it is a router/network issue your tests on the local server should show no issue. You can play with the concurrency request of ab/siege to see if the issue happens more at a higher concurrency. Test it with a variety of files and types of files to see if that makes any difference.

The key to most troubleshooting is replicating the issue and then narrowing down the cause. Since you describe it as "random" I would try and focus on attempting to replicate the issue as reliably as possible. If you cannot replicate the issue then you won't truly know if you actually solve it or not in the end.

It may be in fact the case that "no-one on the internet has experienced this issue". There are virtually an infinite number of hardware/software/application configurations that make diagnosing some issues like this difficult. People also may use a different description for the same problem: what you call "random Apache lag" someone else may call a "remote Ethernet performance issue".

Running your script via cron but using logtail or logtail2 to read the file will avoid reading the whole file every minute. Logtail keeps track of where it last read to and jumps to that point the next time you use it.

If you want to act on new log lines immediately rather than waiting up to 59 seconds between cron invocations, you will have to use tail -f or some equivalent.

Janne's and Khaled's answers both look to solve this problem well.