Setting recursive permissions: how to increase performance

We have a huge directory tree, many levels deep and with a (very) large number of small files at each level.

From time to time we have large file data changes where parts of the tree are replaced and the permissions for these parts will need to be reset. It is not possible to foresee which parts will change next time.

The files currently reside on a Windows NTFS partition.

Resetting permissions needs to be done recursively from the root. This takes the better part of a working day, where the actual requirement is near instant change (or the business suffers).

I have tried GUI. I have tried robocopy. I have tried powershell. I have tried a Go library (Wrapper for the Windows API), as Go has a reputation for being fast, but it turned out little was gained.

I have contemplated letting applications work through symlinks, where the symlinks have the restrictive permissions and the data has (very) permissive permissions. But the root problem would remain: after a data replacement we would still need to trawl through the tree and set these permissive permissions.

Setting group acl:s is not a solution, we use that already. The replacement data has a different set of permissions and these need to be replaced.

Windows is not a requirement, we also run Linux. If yet other OS platforms could solve the task of setting massive file permissions and making the files accessible through common file sharing protocols (http, smb et c), such an OS could be considered.

So my question is: Using Windows NTFS as a baseline, are there file system/OS combinations which provide significantly faster recursive file system operations (setting permissions), whilst still serving the files through a common file sharing protocol?

Procedural suggestions are also welcome – have I and my colleagues just missed some obviously more simple solution than replacing the file system or OS?

EDIT based on constructive comments: yes we have a dev team inhouse and can potentially leverage anything between sysadmin hacks (me) to well designed code (devs).

EDIT2 Answering questions from @GregAskew (approximations as on holiday until monday)

1. How many ACE's are in the ACL?

   • About 8 ACE’s per ACL.

2. Is the file system optimized for performance (short file names disabled, last access time disabled)?

   • No, I was not aware of these optimizations and will try them out.

3. What are the maximum number of files in a directory>

   • Will have to measure.

4. What is the state of file system fragmentation and directory index fragmentation before setting the ACL?

   • Unknown, will investigate.

5. What is the allocation unit size of the volume? What operating system version is hosting the volume?

   • Currently a VMware vSAN hosting a Windowws 2008 R2 Std, upgrade imminent to Win2016 Std.

6. Are you setting the permissions locally (based on another comment, it seems like you are doing this over the network)?

   • We are setting the permissions locally in the VM, but then let them DFS-replicate for redundancy (excruciating and this will be redesigned). We have full control over design, it is just the implementation of the initial file delivery which is out of our control (but based on a comment we may try to change this). The question is just about changing the local file permissions (I realize the underlying SAN is network connected but will gladly take constructive suggestions along those lines all the same).