Application Event Log keeps getting corrupted

eventviewerwindows-server-2008-r2

I recently asked about repairing a corrupt event log, because it seemed to be a one-off event. The event log has since exhibited the same behavior 3 times. We have been trying to find patterns, but so far we have found nothing. The server runs several ASP.NET applications and three scheduled tasks written in .NET. The last modified date of the event log once happened to be the same time as one of the scheduled tasks, but the others have not been.

Any suggestions of where to look next or a way we can get any information out of a corrupt evtx file?

The server is running critical e-commerce applications, so we want to keep the number of restarts required to a minimum.

Edit: I ran DUMPEL and got very strange results.

1/9/2012    4:14:05 PM  1   100 1000    Application Error       N/A SERVERNAME  Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8  Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58  Exception code: 0xc0000374  Fault offset: 0x000ce653  Faulting process id: 0x1070  Faulting application start time: 0x01cccf1386d30991  Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe  Faulting module path: C:\Windows\SysWOW64\ntdll.dll  Report Id: dbf4f691-3b06-11e1-9025-005056a602e6  
1/9/2012    4:14:07 PM  4   0   1001    Windows Error Reporting N/A SERVERNAME  Fault bucket , type 0  Event Name: APPCRASH  Response: Not available  Cab Id: 0    Problem signature:  P1: w3wp.exe  P2: 7.5.7601.17514  P3: 4ce7a5f8  P4: StackHash_79d9  P5: 6.1.7601.17514  P6: 4ce7ba58  P7: c0000374  P8: 000ce653  P9:   P10:     Attached files:  C:\Windows\Temp\WER975.tmp.appcompat.txt  C:\Windows\Temp\WERA03.tmp.WERInternalMetadata.xml  C:\Windows\Temp\WERA13.tmp.hdmp  C:\Windows\Temp\WERD21.tmp.mdmp    These files may be available here:  C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_cd7d09dfc84119d82a2ac6a789038bd5661acfb_cab_128f0e67    Analysis symbol:   Rechecking for solution: 0  Report Id: dbf4f691-3b06-11e1-9025-005056a602e6  Report Status: 4  
1/9/2012    4:14:07 PM  4   0   1001    Windows Error Reporting N/A SERVERNAME  Fault bucket , type 0  Event Name: APPCRASH  Response: Not available  Cab Id: 0    Problem signature:  P1: w3wp.exe  P2: 7.5.7601.17514  P3: 4ce7a5f8  P4: StackHash_79d9  P5: 6.1.7601.17514  P6: 4ce7ba58  P7: c0000374  P8: 000ce653  P9:   P10:     Attached files:  C:\Windows\Temp\WER975.tmp.appcompat.txt  C:\Windows\Temp\WERA03.tmp.WERInternalMetadata.xml  C:\Windows\Temp\WERA13.tmp.hdmp  C:\Windows\Temp\WERD21.tmp.mdmp    These files may be available here:  C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_cd7d09dfc84119d82a2ac6a789038bd5661acfb_cab_128f0e67    Analysis symbol:   Rechecking for solution: 0  Report Id: dbf4f691-3b06-11e1-9025-005056a602e6  Report Status: 0  
1/9/2012    4:14:12 PM  1   100 1000    Application Error       N/A SERVERNAME  Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8  Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58  Exception code: 0xc0000374  Fault offset: 0x000ce653  Faulting process id: 0x16ac  Faulting application start time: 0x01cccf139f475c0c  Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe  Faulting module path: C:\Windows\SysWOW64\ntdll.dll  Report Id: e03bae70-3b06-11e1-9025-005056a602e6  
1/9/2012    4:14:16 PM  4   0   1001    Windows Error Reporting N/A SERVERNAME  Fault bucket , type 0  Event Name: APPCRASH  Response: Not available  Cab Id: 0    Problem signature:  P1: w3wp.exe  P2: 7.5.7601.17514  P3: 4ce7a5f8  P4: StackHash_9c6c  P5: 6.1.7601.17514  P6: 4ce7ba58  P7: c0000374  P8: 000ce653  P9:   P10:     Attached files:  C:\Windows\Temp\WER2579.tmp.appcompat.txt  C:\Windows\Temp\WER25F7.tmp.WERInternalMetadata.xml  C:\Windows\Temp\WER25F8.tmp.hdmp  C:\Windows\Temp\WER28F6.tmp.mdmp    These files may be available here:  C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_c49a67649524ad11b64bbf809211bc5ba742a3d6_cab_0b63321b    Analysis symbol:   Rechecking for solution: 0  Report Id: e03bae70-3b06-11e1-9025-005056a602e6  Report Status: 4  
1/9/2012    4:14:16 PM  4   0   1001    Windows Error Reporting N/A SERVERNAME  Fault bucket , type 0  Event Name: APPCRASH  Response: Not available  Cab Id: 0    Problem signature:  P1: w3wp.exe  P2: 7.5.7601.17514  P3: 4ce7a5f8  P4: StackHash_9c6c  P5: 6.1.7601.17514  P6: 4ce7ba58  P7: c0000374  P8: 000ce653  P9:   P10:     Attached files:  C:\Windows\Temp\WER2579.tmp.appcompat.txt  C:\Windows\Temp\WER25F7.tmp.WERInternalMetadata.xml  C:\Windows\Temp\WER25F8.tmp.hdmp  C:\Windows\Temp\WER28F6.tmp.mdmp    These files may be available here:  C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_c49a67649524ad11b64bbf809211bc5ba742a3d6_cab_0b63321b    Analysis symbol:   Rechecking for solution: 0  Report Id: e03bae70-3b06-11e1-9025-005056a602e6  Report Status: 0  
1/9/2012    4:14:21 PM  1   100 1000    Application Error       N/A SERVERNAME  Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8  Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58  Exception code: 0xc0000374  Fault offset: 0x000ce653  Faulting process id: 0x17f8  Faulting application start time: 0x01cccf13a4ba5126  Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe  Faulting module path: C:\Windows\SysWOW64\ntdll.dll  Report Id: e57a0a85-3b06-11e1-9025-005056a602e6  
1/9/2012    4:14:21 PM  4   0   1001    Windows Error Reporting N/A SERVERNAME  Fault bucket , type 0  Event Name: APPCRASH  Response: Not available  Cab Id: 0    Problem signature:  P1: w3wp.exe  P2: 7.5.7601.17514  P3: 4ce7a5f8  P4: StackHash_9c6c  P5: 6.1.7601.17514  P6: 4ce7ba58  P7: c0000374  P8: 000ce653  P9:   P10:     Attached files:    These files may be available here:  C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_c49a67649524ad11b64bbf809211bc5ba742a3d6_1cfb4872    Analysis symbol:   Rechecking for solution: 0  Report Id: e57a0a85-3b06-11e1-9025-005056a602e6  Report Status: 4  
1/9/2012    4:14:21 PM  4   0   1001    Windows Error Reporting N/A SERVERNAME  Fault bucket , type 0  Event Name: APPCRASH  Response: Not available  Cab Id: 0    Problem signature:  P1: w3wp.exe  P2: 7.5.7601.17514  P3: 4ce7a5f8  P4: StackHash_9c6c  P5: 6.1.7601.17514  P6: 4ce7ba58  P7: c0000374  P8: 000ce653  P9:   P10:     Attached files:    These files may be available here:  C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_c49a67649524ad11b64bbf809211bc5ba742a3d6_1cfb4872    Analysis symbol:   Rechecking for solution: 0  Report Id: e57a0a85-3b06-11e1-9025-005056a602e6  Report Status: 0  

None of the files referenced actually exist (not even in WER ReportArchive). These should not be the only events mentioned. The log file has been cleared twice since January 9, so those events should not even be listed at all.

Update (2016-06-14):
We don't have this server anymore and therefore can no longer test proposed solutions. We never found out what was wrong, but we moved all our services onto new servers since then.

Best Answer

Surprised this hasn't been mentioned before; Have you verified the filesystem? If it's a local disk, and you can suck-up the downtime, flag the volume for a chkdsk and reboot. Do a surface scan if at all possible.

Note that this will be very time-consuming. Especially on a large (+50gb) volume. Shoot for a weekend if at all possible.