I'm looking for a compression format that supports being tailed. Meaning you dont have to read the entire file to get the last X uncompressed bytes.
Is this possible with any of the formats like bzip2, xz, lzma, etc?
I once coded something using gzip that could do this. Basically on a really high level, what it did was cat multiple gzip blocks together, then I had a util that could seek backwards from the end of the file until when the last block started. These files were fully readable by the standard gzip utilities, but I'm hoping theres something a little more standardized available.
The ultimate purpose for this is for log files which I can write out compressed, and then be able to tail them (even when they havent been fully written; i.e. streaming) without having to wait for the whole thing to be read from disk or network.
Best Answer
gzip has an --rsyncable option which does essentially the same. The non-standard part would be the gzip-block-aware "ztail" utility, but it seems like you've dealt with that already.