none
Fastest way to change file permissions on 2 million files RRS feed

  • Question

  • I need to grant access to approximately 2 million files that are on a share for a new User group.  The security is inherited from a parent directory.  There are thousands of sub folders, all inheriting permissions from their parent folder.  I've thought of adjusting the parent folder's permissions, check replace children, and go.... Then the window pops up and I can see it go, file by file changing permissions.

    In order for the permission change to be completed over a weekend (48 hours) it would have to be changing 11 files a second.  It's not, I can read (short file names) the file names that they are changing, while fairly fast it isn't 11 files a second fast.  More along the lines of 2 or 3 at best.

    Worse yet, there are some ownership issues on some of the files.  These files are image files, so there are hundreds of thumbs.db files, that for some reason the permission change gets hung up on.  I've deleted them after doing a search and delete.  However there are still other files within that get hung up, forcing me to monitor the file change process, returning every hour or so.  If it gets hung up, the whole process waits for me to click continue.

    Is there a tool out there that would allow a faster change of permissions?  This isn't the only system where we have this many files.  I'd like to just change the entire design, creating a Group for each permission type so I can just assign AD Groups to the local Groups to grant the appropriate rights, but even that will require changing all the permissions.

    Saturday, August 17, 2019 8:50 PM

All replies

  • Hi,

    my favourite would be SetACL by Helge Klein.

    From Microsoft, there is icacls and calcs.

    If the performance of the storage subsystem is sufficient, you could further speed this up by processing multiple folder trees in parallel by either of these tools.


    Evgenij Smirnov

    http://evgenij.smirnov.de

    Sunday, August 18, 2019 8:40 AM
  • Hello JamesA-CHI,

    I don't know the answer to your question but it reminded me that this is a topic that perhaps needs some more investigation and public documentation.

    The documentation for the "obsolete" API routine SetFileSecurity says:

    • Note that security applied to a directory is not inherited by its children.

    There is no discussion about inheritable ACEs in the security applied to a directory. The implication that I draw from this is that one would have to call SetFileSecurity individually for each object in the directory tree.

    SetFileSecurity was obsoleted by SetNamedSecurityInfo; the documentation for SetNamedSecurityInfo says:

    • If you are setting the discretionary access control list (DACL) or any elements in the system access control list (SACL) of an object, the system automatically propagates any inheritable access control entries (ACEs) to existing child objects, according to the rules of inheritance.

    The implication that I draw from this is that one just needs to call SetNamedSecurityInfo once to update a whole directory tree "according to the rules of inheritance" (there might be entries in the directory tree that have opted out of inheritance by setting SE_DACL_PROTECTED, for example). There is no callback functionality in the API, so one does not know (and one can not show in a UI) which files are being updated.

    NTFS stores the security information for most files in \$Secure and the NTFS file headers just contain an index into this information; many/most files share the same information (i.e. have the same security index). I would imagine that this structure could be used to (relatively) quickly apply a change of inherited security to a directory tree (although it would not be trivial and one would still have to update the security index in potentially millions of file headers).

    Implicit in these considerations is that "effective" security is computed for each object and stored with the object when security changes are made, rather than dynamically computed (by examining the object and all of its ancestors) whenever access to a security descriptor is required.

    Once one knows what has to be done at the lowest level to achieve one's goal for the permission change, one is in a better position to know which tool does that best.

    Gary


    Sunday, August 18, 2019 10:26 AM
  • Hello JamesA-CHI,

    I looked in more detail at what actually happens when using the various APIs. When using the current APIs (e.g. SetNamedSecurityInfo), the call stack at the time that the NTFS driver gets involved looks like:

    ntoskrnl!IofCallDriver
    ntoskrnl!IopGetSetSecurityObject
    ntoskrnl!ObSetSecurityObjectByPointer
    ntoskrnl!NtSetSecurityObject
    ntoskrnl!KiSystemServiceCopyEnd
    ntdll!NtSetSecurityObject
    KERNELBASE!SetKernelObjectSecurity
    ntmarta!MartaSetFileRights
    ntmarta!MartaUpdateTree
    ntmarta!MartaUpdateTree
    ntmarta!MartaManualPropagation
    ntmarta!AccRewriteSetNamedRights
    ntmarta!SetNamedSecurityInfoW

    So the user-mode DLL "NT MARTA" actually does "manually" propagate security changes to a tree, whereas the old APIs just change the security for one object (without propagating inheritable changes).

    I used a very small tree for my tests and the timing results look like this:

    The "Duration" column shows the count of 100 nanosecond intervals, so a duration of 971 means 97.1 microseconds. The actual time between issuing successive SetSecurity I/Os is 464 microseconds in the first case and similar values in the remaining cases. 

    This trace extract shows the I/O requests needed for each object:

    Broadly speaking, it looks as though it takes 500 microseconds to update each child, 250 microseconds of which is spent in NTFS code. A very unreliable extrapolation might suggest that 2000 objects could be updated per second; extrapolating even more wildly, your 2 million files might just take 1000 seconds (about 15 minutes).

    There is no reason why using an old API (or using a new API but disabling automatic propagation) and performing the propagation explicitly in customer code should not give similar results, but many tools seem to give user interface "progress" feedback when performing such operations and it is perhaps this which slows them down.

    Gary


    Sunday, August 18, 2019 4:35 PM
  • Hi,

     

    Just want to confirm the current situations.

     

    Please feel free to let us know if you need further assistance.

     

    Best Regards,

    Fan


    Please remember to mark the replies as an answers if they help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, August 26, 2019 8:18 AM
  • The images you provided in your post don't appear to match what you're describing. Could you verify your intended post? The first screenshot is of you creating a virtual server and the second is a Microsoft Visual Studio error dialog box.

    I don't have a load of programming experience, could you break down which API you're recommending to be used? I'll be going to our programming department to see what they can cook up.

    If I'm reading your post correctly, your testing shows that using the SetNamedSecurityInfo API call can result in changing the permission on 2 million file/folder objects within about 15 minutes?

    PS: I've been continuing to try and change the permissions the standard way, but keep encountering issues with access to the files, session being disconnected.

    A 15 minute fix would be a HUGE win for me!  Please provide more detail so I can go to my programming team.

    Monday, August 26, 2019 4:06 PM
  • I've been trying to reply to this thread several times over the past week, but kept running into errors from the website.  Last week, there was a day that I couldn't even access the question.  Very weird.
    Monday, August 26, 2019 4:07 PM
  • Hello JamesA-CHI,

    Those are not the images that I posted (they don't belong to me). I also noticed that this thread disappeared from my list of "alerts" a day or two after I contributed to the thread.

    Your question prompted me to investigate how security propagation actually works. It was indeed possible to just call one API routine once (e.g. TreeSetNamedSecurityInfo (has more options) or SetNamedSecurityInfo (has fewer options)) and have the propagation performed by the system.

    I timed an absolutely trivial program (since deleted) that just called SetNamedSecurityInfo and used Event Tracing for Windows to see exactly what was happening with sub-microsecond timing resolution. This was performed on a PC (not a server) with SDD storage. The original images have been deleted too, but here is another related image:

    Gary

    Monday, August 26, 2019 4:43 PM
  • Monday, August 26, 2019 10:55 PM
  • Hello JamesA-CHI,

    Yes, that is the API. I wrote a short program using SetNamedSecurityInfo and analysed that and Windows Explorer (which uses TreeSetNamedSecurityInfo).

    What I learnt from this exercise was:

    • There is no support for propagating inheritance in the NTFS driver - the functionality is implemented in a user mode library.
    • The "modern" APIs (those that automatically propagate ACLs) are efficient but propagate ACLs synchronously/sequentially. Inheritance can be complex and the Microsoft implementation is most likely to implement it correctly.

    Other observations are:

    • CPU, I/O subsystem and storage media will obviously have an impact on any absolute estimate of time required for an operation.
    • Any file system filter driver present (such as Data Loss Prevention, Anti-Virus, etc.) could slow down the propagation of ACLs.
    • There is probably scope for a faster implementation of propagation using asynchronous I/O or multi-threading, but the complexity of ensuring a correct implementation should not be underestimated.
    • Event Tracing for Windows (ETW) is a good mechanism to observe, compare and contrast the behaviour of competing implementations of ACL propagation in different tools.

    Gary

    Tuesday, August 27, 2019 8:45 AM