Try running the DOS command "vssadmin list writers" and "vss admin list providers". Capture the output of these command, as they will help in debugging the issue. If a particular writer reports any unexpected state, search the web for conditions that may be causing this issue.
A quick way to reset the writers is to reboot the server.
If VSS errors continue to persist, try re-registering VSS. Follow the steps outlined in MS KB article 940184.
Doublecheck for other VSS providers, i.e. any backup software that installs any sort of Open File agent (examples would be St Bernard, older versions of Backup Exec Open File Option, etc). These can be shown by typing "vssadmin list providers" at a command prompt. Providers from uninstalled software can be deleted by removing the offending CLSID of the VSS provider and rebooting the machine.
Is the machine a 64-bit machine? Are you getting "A function call was invalid because of the state of either the backup extensions or the coordinator. For example calling AddToSnapshot set prior to calling StartSnapshotSet." ? If so, try deleting the Subscription key per MS KB 940184
1. Click Start, click Run, type Regedit, and then click OK.
2. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EventSystem\{26c409cc-ae86-11d1-b616-00805fc79216\Subscriptions
3. On the Edit menu, click Delete, and then click Yes to confirm that you want to delete the subkey.
4. Exit Registry Editor.
5. Reboot the machine
Next you can try running a utility called vshadow. It's part of the windows SDK.
To do a single manual snapshot, run
vshadow [drive letter]
which would take and immediately destroy a VSS snapshot of C: and D:. If that works, VSS is probably OK; if not it's a VSS problem.
All of this is where I would start to troubleshoot VSS problems.
As with anything, be extremely careful when modifying the registry
Best Answer
A Volume Shadow Copy is basically a way of ensuring that your backup image is consistent and any files that are locked (for example a spreadsheet you have open in Excel) can still be backed up.
The VSS writer can fail in many ways and the error message you receive usually gives you a fair clue as to what the problem might be. In your case, it seems the system waited an unusually long amount of time for Windows to create a consistent snapshot of the filesystem and aborted with this error because it took longer that it should.
Because all I/O operations are suspended while the shadow copy is created, the system is actually only given 10 seconds to create a consistent snapshot or the shadow copy creation will fail.
Firstly I'd simply try the backup again. I've personally seen it throw this error before and literally 30 seconds later when I tried again it was fine.
Perhaps the system was simply overloaded and the VSS snapshot timed out because the system couldn't keep up with the workload. Maybe there is a hardware problem that is causing this error (probably hard disks if anything), however I'd expect other things to be going wrong than just this.
Your Windows Event Log may contain some extra details relating to the problem, and that would be my next port of call. Check any events around the time the backup started and see if any are relevant.
Bonus reading: This blog post from someone on the file server team at Microsoft is quite interesting and explains the whole process and history quite well.