A team member informed me that one of our Windows Server 2008 (not 2008 R2) based MSSQL Servers had begun to generate CAPI2 Event ID 513 Errors in the Application Event Log:
Application\CAPI2
Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.
Details:
AddCoreCsiFiles: BeginFileEnumeration() failed.
System Error: Access is denied.
A little PowerShell reveals that the issue started back on 08/06/14 and primarily seems to occur after 22:00 on a daily basis:
PS C:\Users\Administrator> Get-EventLog -LogName Application | ? { $_.EventID -like "513" -and $_.Source -like "Microsoft-Windows-CAPI2" } | Select -Property TimeGenerated
TimeGenerated
-------------
8/18/2014 10:41:32 AM
8/18/2014 10:25:17 AM
8/18/2014 10:15:20 AM
8/17/2014 10:55:41 PM
8/17/2014 10:55:27 PM
8/17/2014 10:55:26 PM
8/16/2014 10:49:44 PM
8/16/2014 10:49:28 PM
8/16/2014 10:49:28 PM
8/15/2014 10:52:11 PM
8/15/2014 10:51:58 PM
8/15/2014 10:51:57 PM
8/15/2014 1:03:06 AM
8/15/2014 1:02:45 AM
8/15/2014 1:02:45 AM
8/13/2014 10:58:49 PM
8/13/2014 10:58:32 PM
8/13/2014 10:58:31 PM
8/12/2014 10:57:09 PM
8/12/2014 10:56:56 PM
8/12/2014 10:56:56 PM
8/11/2014 10:56:13 PM
8/11/2014 10:55:56 PM
8/11/2014 10:55:55 PM
8/10/2014 10:50:15 PM
8/10/2014 10:50:04 PM
8/10/2014 10:50:03 PM
8/10/2014 7:12:09 AM
8/10/2014 7:11:52 AM
8/10/2014 7:11:51 AM
8/8/2014 10:57:00 PM
8/8/2014 10:56:44 PM
8/8/2014 10:56:43 PM
8/6/2014 9:47:26 PM
8/6/2014 9:47:03 PM
8/6/2014 9:47:02 PM
8/6/2014 10:48:33 AM
Curious no? I wonder what the System Writer Object is used for? Shadow Copy! Oh duh! I started doing VSS-based Application Aware backups of this virtual machine using Veeam this month. Naturally the backup process starts at 22:00 which explains the repeated frequency instead of the the error just occurring at "random" times.
Interestingly enough Veeam did not register this as a failed backup attempt which makes me wonder if the Restore Points are actually transaction consistent. Regardless, I did a quick search through the Veeam Backup Logs and did not find anything obviously wrong but it is probably worth reviewing them closer and confirming that a recovery from those Restore Points is transaction consistent.
The Event ID 513 TechNet reference recommended Resolution indicates a NTFS permission issue could be at fault, however the C:\Windows\Registration
COM+ registration folder has the appropriate permissions.
Ideas?
Best Answer
Mathias R. Jessen got me pointed in the right direction, the ever famous WinSxS folder. However I did not see any of the VSS errors in the Event Log which made me a little hesitant to just nuke all the NTFS permissions lest I break something else.
Lesson 1: Read
I went back and read Event ID 513 TechNet reference again and noted that under the Verify section it was recommended that I check to see the
System Writer
was available as VSS writer usingvssadmin list writers
and sure enough it was NOT. Lesson Learned #1: Read the whole KB/TechNet/BlogLesson 2: Reproduce
Doing a bit more researching I came across Missing System Writer Case Explained which seemed to indicate the issue was originating with Cryptographic Services. I found that I could reproduce by
CAPI2
error at will by stopping and starting theCryptSVC
service. Lesson Learned #2: Try to figure out a way to reproduce your error at will.Using ProcMon
At this point, I pretty much followed the post's instructions. I located the PID of which instance of
svchost
was wrappingCryptSVC
using Task Manager. You could alternatively forceCryptSVC
to run as its own process using sc config if you can reboot the server in question. Depending on how deep you get into ProcMon it is worth isolating services under a single PID just to cut down the amount of events you have to sort through.From here it is back to good old ProcMon. Setup a filter to exclude all PIDs that are not the one used by the
svchost
process that is wrappingCryptSVC
:Lesson 3: Love ProcMon
I applied my trusty first pass filter which is to exclude all events that have the results of
SUCCESS
. This reduced the events from 31,118 to a much more manageable 139 and at the bottom I found theACCESS DENIED
event I was looking for, not surprisingly in theWinSxS
folder (C:\Windows\winsxs\FileMaps\$$.cdf-ms
). Lesson Learned #3: Learn to use and love ProcMonLesson 4: Verify
Now what? KB2009272 that Mathias linked has the solution but now I know why. Lesson Learned #4: Don't guess, know!
Lesson 5: Start small
The resolution is exactly as explained in KB2009272. Take ownership and reset the permissions of the
FileMaps
folder and then restartCryptSVC
:and... we have lift off! The
System Writer
is now available as a VSS writer. No need to change the permissions for thePendingRename
folder. Lesson Learned #5: Start with the smallest changes and work your way out to changes that effect more things.Restarting
CryptSVC
at will no longer produced theCAPI2
error and after a day or two of monitoring it looks resolved.