Windows 2008 R2 – shadow copy management and retention

vsswindows-server-2008-r2

This has bothered me over the years and I'm dealing with it again now. I've never quite understood the full black magic that is Shadow Copies/VSS. I get it to the extent of what it does and how to initially set it up and set the schedules on the volumes. I also understand its role in backups.

Scenario

To simplify it, I'll stick with a single volume in my environment. I have the following schedules enabled:

1 - M-F 7am starting on 4/14/13
2 - M-F 12pm starting on 4/14/13
3 - 1st day of the month at 9am starting on 6/11/13

The Use limit for that volume's shadow copies is 25,599MB (25GB).

The volume is 250GB and has 44GB of free space.

Going backwards from today I show all the correct times/schedules for shadow copies back until 7/29/13. It doesn't go back further, and there isn't a 7/1/13 monthly copy.

I'm pretty sure I remember reading there is a 64 copy limit before the oldest gets overwritten. This makes sense since I count exactly 64 copies currently in the list.

My Questions

Is there a way to change this 64 copy limitation?
Can I prevent monthly copies from being overwritten somehow and let it overwrite just the "dailies" instead?
Does MS or others provide a way to manage shadow copies in a more granular fashion than the basic scheduling on the volume, etc. that I've already done above? ie. the ability to tell it "keep only the previous 4 weeks of this schedule" or similar.

My goal is to get to a point where i can have 12 "monthlies" and the previous month twice "dailies" (for a total roughly of 72 copies it would seem).

I have "real backups" using Commvault Simpana but it's so much easier to deal with a restore from shadow copies for simple file shares.

Best Answer

Yes you can change the limit up to 512 shadow copies per volume, by using the MaxShadowCopies registry key. You will probably also need to adjust the amount of storage space VSS is allowed to use with vssadmin.exe Resize ShadowStorage. (You may configure Shadow Copies for a volume to back up to a separate volume, such as one with more space. Again, vssadmin.exe is your friend here.)
No.
No.

VSS on its own works well, but it's pretty basic. Most good enterprise backup solutions make use of VSS on Windows machines. But to get those really granular and fine-tuned retention policies, you'll need to use a more thorough backup solution.

Related Solutions

Volume Shadow Copy – How to Use VSS for Backups on Windows Server 2003

So in the spirit of reinventing the wheel, I present to you Tomalak's excellent script (see above) but completely rewritten in Powershell!!! The main reason I did this was to evangelise the awesome powers of Powershell, but also because I despise vbscript with my entire being.

It's mostly feature to feature identical, but I did implement some things a little differently for various reasons. The debugging output is definitely more verbose.

One very important thing to note is that this version detects the OS version and bitness and calls the appropriate version of vshadow.exe. I've included a chart below to show which versions of vshadow.exe to use, where to get them, and what to name them.

Here's the usage info:

VssSnapshot.ps1

Description:
  Create, mount or delete a Volume Shadow Copy Service (VSS) Shadow Copy (snapshot)

Usage:
  VssSnapshot.ps1 Create -Target <Path> -Volume <Volume> [-Debug]
  VssSnapshot.ps1 Delete -Target <Path> [-Debug]

Paremeters:
  Create  - Create a snapshot for the specified volume and mount it at the specified target
  Delete  - Unmount and delete the snapshot mounted at the specified target
  -Target - The path (quoted string) of the snapshot mount point
  -Volume - The volume (drive letter) to snapshot
  -Debug  - Enable debug output (optional)

Examples:
  VssSnapshot.ps1 Create -Target D:\Backup\DriveC -Volume C
  - Create a snapshot of volume C and mount it at "D:\Backup\DriveC"

  VssSnapshot.ps1 Delete -Target D:\Backup\DriveC
  - Unmount and delete a snapshot mounted at "D:\Backup\DriveC"

Advanced:
  VssSnapshot.ps1 create -t "c:\vss mount\c" -v C -d
  - Create a snapshot of volume C and mount it at "C:\Vss Mount\C"
  - example mounts snapshot on source volume (C: --> C:)
  - example uses shortform parameter names
  - example uses quoted paths with whitespace
  - example includes debug output

Here's the script:

# VssSnapshot.ps1
# http://serverfault.com/questions/119120/how-to-use-a-volume-shadow-copy-to-make-backups/119592#119592

Param ([String]$Action, [String]$Target, [String]$Volume, [Switch]$Debug)
$ScriptCommandLine = $MyInvocation.Line
$vshadowPath = "."

# Functions
Function Check-Environment {
  Write-Dbg "Checking environment..."

  $UsageMsg = @'
VssSnapshot

Description:
  Create, mount or delete a Volume Shadow Copy Service (VSS) Shadow Copy (snapshot)

Usage:
  VssSnapshot.ps1 Create -Target <Path> -Volume <Volume> [-Debug]
  VssSnapshot.ps1 Delete -Target <Path> [-Debug]

Paremeters:
  Create  - Create a snapshot for the specified volume and mount it at the specified target
  Delete  - Unmount and delete the snapshot mounted at the specified target
  -Target - The path (quoted string) of the snapshot mount point
  -Volume - The volume (drive letter) to snapshot
  -Debug  - Enable debug output (optional)

Examples:
  VssSnapshot.ps1 Create -Target D:\Backup\DriveC -Volume C
  - Create a snapshot of volume C and mount it at "D:\Backup\DriveC"

  VssSnapshot.ps1 Delete -Target D:\Backup\DriveC
  - Unmount and delete a snapshot mounted at "D:\Backup\DriveC"

Advanced:
  VssSnapshot.ps1 create -t "c:\vss mount\c" -v C -d
  - Create a snapshot of volume C and mount it at "C:\Vss Mount\C"
  - example mounts snapshot on source volume (C: --> C:)
  - example uses shortform parameter names
  - example uses quoted paths with whitespace
  - example includes debug output
'@

  If ($Action -eq "Create" -And ($Target -And $Volume)) {
    $Script:Volume = (Get-PSDrive | Where-Object {$_.Name -eq ($Volume).Substring(0,1)}).Root
    If ($Volume -ne "") {
      Write-Dbg "Verified volume: $Volume"
    } Else {
      Write-Dbg "Cannot find the specified volume"
      Exit-Script "Cannot find the specified volume"
    }
    Write-Dbg "Argument check passed"
  } ElseIf ($Action -eq "Delete" -And $Target ) {
    Write-Dbg "Argument check passed"
  } Else {
    Write-Dbg "Invalid arguments: $ScriptCommandLine"
    Exit-Script "Invalid arguments`n`n$UsageMsg"
  }


  $WinVer = ((Get-WmiObject Win32_OperatingSystem).Version).Substring(0,3)
    Switch ($WinVer) {
    "5.2" {
      $vshadowExe = "vshadow_2003"
      $WinBit = ((Get-WmiObject Win32_Processor)[0]).AddressWidth
    }
    "6.0" {
      $vshadowExe = "vshadow_2008"
      $WinBit = (Get-WmiObject Win32_OperatingSystem).OSArchitecture
    }
    "6.1" {
      $vshadowExe = "vshadow_2008R2"
      $WinBit = (Get-WmiObject Win32_OperatingSystem).OSArchitecture
    }
    Default {
      Write-Dbg "Unable to determine OS version"
      Exit-Script "Unable to determine OS version"
    }
  }

  Switch ($WinBit) {
    {($_ -eq "32") -or ($_ -eq "32-bit")} {$vshadowExe += "_x86.exe"}
    {($_ -eq "64") -or ($_ -eq "64-bit")} {$vshadowExe += "_x64.exe"}
    Default {
      Write-Dbg "Unable to determine OS bitness"
      Exit-Script "Unable to determine OS bitness"
    }
  }

  $Script:vshadowExePath = Join-Path $vshadowPath $vshadowExe
  If (Test-Path $vshadowExePath) {
    Write-Dbg "Verified vshadow.exe: $vshadowExePath"
  } Else {
    Write-Dbg "Cannot find vshadow.exe: $vshadowExePath"
    Exit-Script "Cannot find vshadow.exe"
  }

  Write-Dbg "Environment ready"
}

Function Prepare-Target {
  Write-Log "Preparing target..."
  Write-Dbg "Preparing target $Target"


  If (!(Test-Path (Split-Path $Target -Parent))) {
  Write-Dbg "Target parent does not exist"
  Exit-Script "Invalid target $Target"
  }
  If ((Test-Path $Target)) {
    Write-Dbg "Target already exists"
    If (@(Get-ChildItem $Target).Count -eq 0) {
      Write-Dbg "Target is empty"
    } Else {
      Write-Dbg "Target is not empty"
      Exit-Script "Target contains files/folders"
    }
  } Else {
    Write-Dbg "Target does not exist. Prompting user..."
    $PromptYes = New-Object System.Management.Automation.Host.ChoiceDescription "&Yes", "Create target folder"
    $PromptNo = New-Object System.Management.Automation.Host.ChoiceDescription "&No", "Do not create target folder"
    $PromptOptions = [System.Management.Automation.Host.ChoiceDescription[]]($PromptYes, $PromptNo)
    $PromptResult = $Host.UI.PromptForChoice("Create folder", "The target folder `"$target`" does not exist.`nWould you like to create the folder?", $PromptOptions, 0) 
    Switch ($PromptResult) {
      0 {
        Write-Dbg "User Accepted. Creating target..."
        $Null = New-Item -Path (Split-Path $Target -Parent) -Name (Split-Path $Target -Leaf) -ItemType "Directory"
      }
      1 {
        Write-Dbg "User declined. Exiting..."
        Exit-Script "Target does not exist"
      }
    }
  }
  Write-Log "Target ""$Target"" ready"
  Write-Dbg """$Target"" ready"
}

Function Create-Snapshot {
  Write-Log "Creating snapshot..."
  Write-Dbg "Creating snapshot of $Volume"
  $Cmd = "$vshadowExePath -p $Volume"
  $CmdResult = Run-Command $Cmd -AsString

  Write-Dbg "Snapshot created successfully"

  $SnapshotID = $CmdResult -Match 'SNAPSHOT ID = (\{[^}]{36}\})'
  If ($SnapshotID) {
    $SnapshotID = $Matches[1]
    Write-Dbg "SnapshotID: $SnapshotID"
    Write-Log "Snapshot $SnapshotID created"
  } Else {
    Write-Dbg "Unable to determine SnapshotID"
    Exit-Script "Unable to determine SnapshotID"
  }

  Return $SnapshotID
}

Function Mount-Snapshot ($SnapshotID) {
  Write-Log "Mounting snapshot..."
  Write-Dbg "Mounting $SnapshotID at ""$Target"""

  $Cmd = "$vshadowExePath `"-el=$SnapshotId,$Target`"" #Must use escaped quotes because Invoke-Expression gets all weird about curly braces
  $CmdResult = Run-Command $Cmd

  Write-Log "Snapshot $SnapshotID mounted at target ""$Target"""
  Write-Dbg "$SnapshotID mounted at ""$Target"""
}

Function Delete-Snapshot {
  Write-Log "Deleting snapshot..."
  Write-Dbg "Deleting snapshot at target ""$Target"""

  $SnapshotID = Get-SnapshotIdbyTarget

  $Cmd = "$vshadowExePath `"-ds=$SnapshotId`""
  $CmdResult = Run-Command $Cmd

  Write-Log "Snapshot $SnapshotID deleted at target ""$Target"""
  Write-Dbg "$SnapshotID deleted at ""$Target"""
}

Function Get-SnapshotIdbyTarget {
  Write-Dbg "Finding SnapshotID for $Target"

  $Cmd = "$vshadowExePath -q"
  $CmdResult = Run-Command $Cmd -AsString

  $TargetRegEx = '(?i)' + $Target.Replace('\','\\') + '\\?\r'
  $Snapshots = ($CmdResult.Split('*')) -Match $TargetRegEx | Out-String

  If ($Snapshots) {
    $Null = $Snapshots -Match '(\{[^}]{36}\})'
    $SnapshotID = $Matches[0]
  } Else {
    Write-Dbg "Unable to determine SnapshotID for target $Target"
    Exit-Script "Unable to determine SnapshotID"
  }  

  Write-Dbg "SnapshotID: $SnapshotID"

  Return $SnapshotID
}

Function Run-Command ([String]$Cmd, [Switch]$AsString=$False, [Switch]$AsArray=$False) {
  Write-Dbg "Running: $Cmd"

  $CmdOutputArray = Invoke-Expression $Cmd
  $CmdOutputString = $CmdOutputArray | Out-String
  $CmdErrorCode = $LASTEXITCODE

  If ($CmdErrorCode -eq 0 ) {
    Write-Dbg "Command successful. Exit code: $CmdErrorCode"
    Write-Dbg $CmdOutputString
  } Else {
    Write-Dbg "Command failed. Exit code: $CmdErrorCode"
    Write-Dbg $CmdOutputString
    Exit-Script "Command failed. Exit code: $CmdErrorCode"
  }

  If (!($AsString -or $AsArray)) {
    Return $CmdErrorCode
  } ElseIf ($AsString) {
    Return $CmdOutputString
  } ElseIf ($AsArray) {
    Return $CmdOutputArray
  }
}

Function Write-Msg ([String]$Message) {
  If ($Message -ne "") {
    Write-Host $Message
  }
}

Function Write-Log ([String]$Message) {
  Write-Msg "[$(Get-Date -Format G)] $Message"
}

Function Write-Dbg ([String]$Message) {
  If ($Debug) {
    Write-Msg ("-" * 80)
    Write-Msg "[DEBUG] $Message"
    Write-Msg ("-" * 80)
  }
}

Function Exit-Script ([String]$Message) {
  If ($Message -ne "") {
    Write-Msg "`n[FATAL ERROR] $Message`n"
  }
  Exit 1
}

# Main
Write-Log "VssSnapshot started"
Check-Environment

Switch ($Action) {
  "Create" {
    Prepare-Target
    $SnapshotID = Create-Snapshot
    Mount-Snapshot $SnapshotID
  }
  "Delete" {
    Delete-Snapshot
  }
}

Write-Log "VssSnapshot finished"

Here are the vshadow.exe versions to use:

Windows 2003/2003R2
- Volume Shadow Copy Service SDK 7.2
- x86: C:\Program Files\Microsoft\VSSSDK72\TestApps\vshadow\bin\release-server\vshadow.exe
  - Rename to: vshadow_2003_x86.exe
- x64: I have not been able to locate an x64 version of vshadow.exe for Windows 2003 x64
Windows 2008
- Windows SDK for Windows Server 2008 and .NET Framework 3.5
- x86: C:\Program Files\Microsoft SDKs\Windows\v6.1\Bin\vsstools\vshadow.exe
  - Rename to: vshadow_2008_x86.exe
- x64: C:\Program Files\Microsoft SDKs\Windows\v6.1\Bin\x64\vsstools\vshadow.exe
  - Rename to: vshadow_2008_x64.exe
Windows 2008R2
- Microsoft Windows SDK for Windows 7 and .NET Framework 4
- x86: C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\vsstools\vshadow.exe
  - Rename to: vshadow_2008R2_x86.exe
- x64: C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\x64\vsstools\vshadow.exe
  - Rename to: vshadow_2008R2_x64.exe

Hard volume size limitation for shadow copies w/ 2k3 x86

This Technet article on VSS scalability indicates that the issue probably relates to exhaustion of the nonpaged memory pool which is limited to 256MB on Windows 2003 32bit (x86). VSS uses more resources during snapshot creation and the article specifically states that a 5TB snapshot would require 200MB for that snapshot alone. Depending on what else is making demands on the non-paged pool it seems likely that you are hitting that limit.

For what it's worth 64bit systems have a much larger nonpaged pool and would not suffer in the same way.

The "not enough storage" error generally indicates exhaustion of system resources like the non-paged pool or heap exhaustion in my experience and does not typically refer to either disk or general memory.