locked
DFS Replication Issues - Not replicating RRS feed

  • Question

  • Hello Microsoft Experts, 

    We recently had issues that started this week about files not replicating. Honestly, we're not sure when it actually started but was only known Monday this week.

    We are using Windows Server 2012 r2 for both sites' File Servers.

    Our files are not replicating and upon checking event viewer, we found staging errors. What we did was to increase our staging quota from the default 4096 MB to 50GB, 70GB, and one replicated folder with 100GB. This resolved the errors about staging quota. 

    Unfortunately, we're still not getting any files replicated. 

    Checking the logs again, we're getting a lot of Event 4412. We're now getting thousands of Event ID 4412 for 3 days (at least from the days we noticed the event ID). Is it cross checking the files in both sites?

    Running dfsrdiag backlog shows hundreds of thousands of files in the backlog for several folders. 

    Is there nothing else we can do but to wait until they finish? Any way we can set prioritization to these folders with hundreds of thousands of files in the backlog?

    Friday, August 2, 2019 2:16 AM

Answers

  • Auditing settings on object were changed.
    Subject:
     Security ID:  domain\Service.Audit
     Account Name:  Service.Audit
     Account Domain:  domain
     Logon ID:  
    Object:
     Object Server: Security
     Object Type: File
     Object Name: F:\Data\....
     Handle ID: 
    Process Information:
     Process ID: 0x4
     Process Name: 

    Auditing Settings:
     Original Security Descriptor: S:AI(AU;IDSA;CC;;;WD)(AU;IDSA;DCLCRPDTCRSDWDWO;;;WD)(AU;IDFA;CCDCLCRPDTCRSDWDWO;;;WD)
     New Security Descriptor:  S:ARAI(AU;IDSA;CC;;;WD)(AU;IDSA;DCLCRPDTSDWDWO;;;WD)(AU;IDFA;CCDCLCRPDTSDWDWO;;;WD)

    As it looks like, some permissions have been modified on "F:\Data\...." by this auditing software. I've bolded the ones that have been modified, the S: means that SACL (Security Access Control List) entries have been modified.

    You can view what these security descriptors mean and what permissions they are over here:
    4907(S): Auditing settings on object were changed.

    --------------------------------------------------------------------------------------

    S:AI has been modified to: S:ARAI (added "AR")

    • "AR" - SDDL_AUTO_INHERIT_REQ, Child objects inherit permissions from this object.

    DCLCRPDTCRSDWDWO has been modified to: DCLCRPDTSDWDWO (missing "CR")

    • "CR" All Extended Rights

    DCLCRPDTSDWDWO has been modified to: CCDCLCRPDTSDWDWO (added "CR")

    • "CR" All Extended Rights

    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, August 6, 2019 9:40 AM

All replies

  • Hi

    4412 events indicate that there has been a conflict between two files/directory between the servers involved in DFS replication and one needs to be removed.

    For how to resolve event 4412, you can refer to the following link:

    https://social.technet.microsoft.com/Forums/en-US/dd8a3e81-b4ac-4142-a1bc-34f9cfe90151/problem-with-dfsr-event-4412?forum=winservergen

    Best Regards,

    Fan



    Please remember to mark the replies as an answers if they help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, August 5, 2019 3:29 AM
  • Hello,

    I'm sure this would resolve the issue with the error. But what we did was to just let the replication run one at a time for the replicated folder. This resolved the issue with the replication -- but only for 2 days.

    Now the problem keeps coming back. DFS keeps on generating backlogs millions of backlogs for the server. 

    Can you think of any reason why it would do so? 

    Tuesday, August 6, 2019 3:53 AM
  • Hello Jan,

    Has there been any changes recently that may have caused replication issues? Has the DFS Replication service been stopped/restarted for some reason? or has the servers been restarted?

    Are you using a simple "Server A" --> "Server B" DFS replication topology?

    The event 4412 can actually be quite common, especially in scenarios where users are constantly accessing/modifying data.

    These events can also be triggered by one of the following:

    • An anti-virus software which is handling the files inside the replicated folder or making changes to any attributes
    • A backup software that changes/holds a handle to the file.
    • Any program that changes/locks the files/folders.

    Could you share a DFS Replication diagnostics (Health report) to us?

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, August 6, 2019 7:01 AM
  • Hello Jan,

    Has there been any changes recently that may have caused replication issues? Has the DFS Replication service been stopped/restarted for some reason? or has the servers been restarted?

    Are you using a simple "Server A" --> "Server B" DFS replication topology?

    The event 4412 can actually be quite common, especially in scenarios where users are constantly accessing/modifying data.

    These events can also be triggered by one of the following:

    • An anti-virus software which is handling the files inside the replicated folder or making changes to any attributes
    • A backup software that changes/holds a handle to the file.
    • Any program that changes/locks the files/folders.

    Could you share a DFS Replication diagnostics (Health report) to us?

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Hello Leon,

    I have just found security event logs of event ID 4907 in our server. And now thinking about it, we may be the reason why this happened to the DFS. We have an auditing software that we recently updated and it matched the timing when the DFS issue happened. It's just so weird that the auditing software would change something that caused this.

    Auditing settings on object were changed.
    Subject:
     Security ID:  domain\Service.Audit
     Account Name:  Service.Audit
     Account Domain:  domain
     Logon ID:  
    Object:
     Object Server: Security
     Object Type: File
     Object Name: F:\Data\....
     Handle ID: 
    Process Information:
     Process ID: 0x4
     Process Name: 

    Auditing Settings:
     Original Security Descriptor: S:AI(AU;IDSA;CC;;;WD)(AU;IDSA;DCLCRPDTCRSDWDWO;;;WD)(AU;IDFA;CCDCLCRPDTCRSDWDWO;;;WD)
     New Security Descriptor:  S:ARAI(AU;IDSA;CC;;;WD)(AU;IDSA;DCLCRPDTSDWDWO;;;WD)(AU;IDFA;CCDCLCRPDTSDWDWO;;;WD)




    • Edited by Jan Forbile Tuesday, August 6, 2019 9:06 AM
    Tuesday, August 6, 2019 9:03 AM
  • Auditing settings on object were changed.
    Subject:
     Security ID:  domain\Service.Audit
     Account Name:  Service.Audit
     Account Domain:  domain
     Logon ID:  
    Object:
     Object Server: Security
     Object Type: File
     Object Name: F:\Data\....
     Handle ID: 
    Process Information:
     Process ID: 0x4
     Process Name: 

    Auditing Settings:
     Original Security Descriptor: S:AI(AU;IDSA;CC;;;WD)(AU;IDSA;DCLCRPDTCRSDWDWO;;;WD)(AU;IDFA;CCDCLCRPDTCRSDWDWO;;;WD)
     New Security Descriptor:  S:ARAI(AU;IDSA;CC;;;WD)(AU;IDSA;DCLCRPDTSDWDWO;;;WD)(AU;IDFA;CCDCLCRPDTSDWDWO;;;WD)

    As it looks like, some permissions have been modified on "F:\Data\...." by this auditing software. I've bolded the ones that have been modified, the S: means that SACL (Security Access Control List) entries have been modified.

    You can view what these security descriptors mean and what permissions they are over here:
    4907(S): Auditing settings on object were changed.

    --------------------------------------------------------------------------------------

    S:AI has been modified to: S:ARAI (added "AR")

    • "AR" - SDDL_AUTO_INHERIT_REQ, Child objects inherit permissions from this object.

    DCLCRPDTCRSDWDWO has been modified to: DCLCRPDTSDWDWO (missing "CR")

    • "CR" All Extended Rights

    DCLCRPDTSDWDWO has been modified to: CCDCLCRPDTSDWDWO (added "CR")

    • "CR" All Extended Rights

    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, August 6, 2019 9:40 AM
  • Hello,

    Thank you for pointing this out. 

    Any ideas why changing permission would cause this problem? We didn't have any issues like these before and changing permissions are normal in our business. The problem we're facing is that when we change permission to any shared folders, these will create hundreds of thousands of backlogs file count which would stop our replication from working. 

    Wednesday, September 4, 2019 2:23 AM
  • Changing permissions might cause the DFS Replication to become slower, if a lot has changed then there's a lot to replicate (since DFS Replication replicates all changes).

    Does DFS Replication replicate updated permissions on a file or folder?

    Yes. DFS Replication replicates permission changes for files and folders. Only the part of the file associated with the Access Control List (ACL) is replicated, although DFS Replication must still read the entire file into the staging area.

    Note:
    Changing ACLs on a large number of files can have an impact on replication performance. However, when using RDC, the amount of data transferred is proportionate to the size of the ACLs, not the size of the entire file. The amount of disk traffic is still proportional to the size of the files because the files must be read to and from the staging folder.

    Reference:
    https://docs.microsoft.com/en-us/windows-server/storage/dfs-replication/dfsr-faq#does-dfs-replication-replicate-updated-permissions-on-a-file-or-folder


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, September 4, 2019 7:09 AM