Expanding 16 TB volume using Windows dynamic disks

The two main candidates are hardware fault, and driver errors. A list of the STOP errors that occur over time is essential to establish a pattern.

If there are multiple different "blue screen" fault codes, chances are that it is a hardware error and you should probably skip to step 3 and start replacing gear. For a single error, there are three main methods for trying to resolve/diagnose further:

1. Upgrade Drivers: Find and install all the latest drivers for the main system devices
2. "Spray and Pray" with hotfixes: Search TechNet for all hotfixes that reference the particular STOP error (e.g. 0x3b or 0x0000003b), and install
3. General kernel debug dumps and use the kernel debug tools (Debugging Tools for Windows - Native x6) to try and locate the exact culprit
4. Change hardware: start swapping the suspect components

Hardware faults are messy to accurately diagnose, particularly on a new server - it could be just about any component. I would suggest testing with the following in mind:

• If the controller was in use in the old server it's probably OK and just a driver update might solve the problem. Unless it was a power issue that caused the old server to fail...
• Faulty RAM would easily be the most common fault I've seen that causes STOP errors. Try a complete replacement, or test one stick at a time if possible.
• Don't discount a faulty mainboard/motherboard, they do happen
• Inadequate power supplies can sometimes cause irregular faults when too many disks are active

When you are backing up a volume like that you are going to be seriously exercising the underlying storage. When you start reading in those millions of small files scattered around the filesystem the limiting factor is going to be the random read IOPS that the underlying disks on your SAN can deliver. The SAN itself might not be stressed at all but the volume you are reading from will be taking a hit and any other process that tries to do anything else at the same time will suffer unless you throttle back the backup activity.

The thing to look at is the queue depth on that volume. If it's peaking significantly higher than the number of disks that back up the volume then you are hitting the IOPS limits. Perfmon will give you an idea but the best data will come from the Storage Array's own analytics if it's possible to get those. I seriously doubt that your problem is anything to do with file locking. The Storage folks need to look at the IOPS on the disks in the RAID pack that your volume is carved up from, I suspect that those disks are hitting over ~ 150 IOPs each (higher if they are 15k, lower if they are 7.2k). If you have a 6 disk RAID 10 group hosting that volume then it will max out at a rate not much better than 10Meg/sec if it is genuinely backing up millions of 10k files and very little that is much bigger.

NtfsDisableLastAccessUpdate will help in your case - it drops a set of IOPS from each file activity and in particular it avoids a couple of extra reads and writes associated with each file. Given that you have millions and that your files or so small there should be a significant win, it may be as much as a 50% win. That said the most likely effect you will see is that your backup will speed up, but still run into an IOPs limit.

You should also consider aligning the disk partition if that hasn't already been done. In a case like this (lots of small reads) it isn't as big a win as it can be for other IO patterns, probably about 10% assuming your RAID stripe size is 128k and your average read is around 10k, but it might be worth the effort. It will require backing up the whole volume, repartitioning and reformatting it and then restoring the data so its not a trivial exercise.