Is it a good practice to use NTFS Compression on a IIS Log folders and files?
I was able to go down from 20GB to 7GB by doing this. The IIS logs are per day, and have an average size of 20MB but some extreme days have 200MB.
I'm wondering if IIS has to open the whole file in memory, forcing NTFS to unzip 20MB (or 200MB in extreme case) each time? Or is there some magic that allows IIS to append content? What's the system impact? Could it become a problem if we grow our traffic?
Should I split them per hour instead of per day?
Any official Microsoft paper on this? I could not locate one.
Best Answer
As Evan already gave a general answer, I like to address two of your sub-questions:
Does IIS flushes logs every X minutes?
http.sys, the kernel mode part of IIS is responsible for logging and it buffers the data in memory before writing it to the log files. I'm not certain but I don't think it does the flushing every x seconds, more likely after its buffer is getting full.
Does the whole file need to be read when adding a single line?
No, NTFS writes updates to a file into its own cache and then compresses and appends the data asynchronously to the file. Writing to a compressed file is not significantly slower than to an uncompressed file.
So there should be no problem with using NTFS compression on IIS log files.
Sources:
IIS 7 Resource Kit, Chapter 15:Logging - Microsoft Press 2008
Windows Internals 6th Edition Part2, Chapter 12: File Systems Microsoft Press 2012