none
Issue in the AD replication after VM restore RRS feed

  • Question

  • Hi,

    This is my first Thread in this Forum.

    We have a problem in our AD. Let me first give a glimpses of the environment.

    2 no. of DC in the forest. It replicated with each other .

    Both the DC are VM on VMware ESXi VSphare host.

     previously there was a DNS issue...every entry got removed from both the DC. Than we restored the primary from 2 months old backed up VM done by HP Dataprotector.

    We again synced the DNS from secondary to primary DC. Its was fine..

    After some time the replication got stopped.

    we digonised and found that the below event ..

    *************************************

    Log Name:      Directory Service
    Source:        Microsoft-Windows-ActiveDirectory_DomainService
    Date:          3/12/2015 11:14:23 AM
    Event ID:      1988
    Task Category: Replication
    Level:         Error
    Keywords:      Classic
    User:          ANONYMOUS LOGON
    Computer:      xxx-Primary.xxxxxxx.net
    Description:
    Active Directory Domain Services Replication encountered the existence of objects in the following partition that have been deleted from the local domain controllers (DCs) Active Directory Domain Services database.  Not all direct or transitive replication partners replicated in the deletion before the tombstone lifetime number of days passed.  Objects that have been deleted and garbage collected from an Active Directory Domain Services partition but still exist in the writable partitions of other DCs in the same domain, or read-only partitions of global catalog servers in other domains in the forest are known as "lingering objects".
     
     
    Source domain controller:
    52c29991-6165-4537-ac08-2726ce4e0dec._msdcs.xxxxx.net
    Object:
    DC=DR1380056,DC=xxxxxxx.net,CN=MicrosoftDNS,DC=DomainDnsZones,DC=xxxxxx,DC=net
    Object GUID:
    dc448323-f444-4d52-acc4-cf2ea252c7fe  This event is being logged because the source DC contains a lingering object which does not exist on the local DCs Active Directory Domain Services database.  This replication attempt has been blocked.
     
     The best solution to this problem is to identify and remove all lingering objects in the forest.

    *****************************************

    Than I used the registry edit and made the replication in loose mode...less secure replication..

    HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
    Strict Replication Consistency = 0

    Its was fine...for some time..

    Our sys admin have reseted some user's password and a security update...

    Again the same problem identified..

    I have done lost of thing check the Disable the replication using the repadmin command.

    nothing worked..

    Inbetween going for any changed in the registery ..I have done a VM snapshot on both the VM...

    Now, I have restore the VM from snapshot of VM done in VMware...

    now , the following issue have arised..

    1. I am opening the DNS/ad users and computers/other  console from secondary DC the primary DC was not added..giving a stop icon but canable to see the secondary DC.

     2. I have executed the "repadmin /showrepl  /verbose /all /intersite " below is the output

    *********************************

    R
    epadmin: running command /showrepl against full DC DC1 .Domainxxx .net
    Default-First-Site-Name\DC1 
    DSA Options: IS_GC 
    Site Options: (none)
    DSA object GUID: 202bc518-9cac-4cc3-8743-9394abe42dfe
    DSA invocationID: c32157a2-2130-4200-9975-9c18d0823cf3
    ==== INBOUND NEIGHBORS ======================================
    ==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============
    CN=Configuration,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC2via RPC
            DSA object GUID: 52c29991-6165-4537-ac08-2726ce4e0dec
            Address: 52c29991-6165-4537-ac08-2726ce4e0dec._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-18 08:33:56 was successful.
     
    CN=Schema,CN=Configuration,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC2via RPC
            DSA object GUID: 52c29991-6165-4537-ac08-2726ce4e0dec
            Address: 52c29991-6165-4537-ac08-2726ce4e0dec._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-04 17:21:31 was successful.
     
    DC=DomainDnsZones,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC2via RPC
            DSA object GUID: 52c29991-6165-4537-ac08-2726ce4e0dec
            Address: 52c29991-6165-4537-ac08-2726ce4e0dec._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-18 15:48:32 was successful.
     
    DC=ForestDnsZones,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC2 via RPC
            DSA object GUID: 52c29991-6165-4537-ac08-2726ce4e0dec
            Address: 52c29991-6165-4537-ac08-2726ce4e0dec._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-17 20:45:39 was successful.
     ==== KCC CONNECTION OBJECTS ============================================
    Connection --
        Connection name : DC2 
        Server DNS name : DC1 .Domainxxx .net
        Server DN  name : CN=NTDS Settings,CN=DC1 ,CN=Servers,CN=Default-First-Site-

    Name,CN=Sites,CN=Configuration,DC=Domainxxx ,DC=net
            Source: Default-First-Site-Name\DC2 
    ******* 157 CONSECUTIVE FAILURES since 2015-03-08 19:17:35
    Last error: 8606 (0x219e):
                Insufficient attributes were given to create an object. This object

    may not exist because it may have been deleted and already garbage collected.
            TransportType: intrasite RPC
            ReplicatesNC: DC=DomainDnsZones,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: CN=Schema,CN=Configuration,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: CN=Configuration,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: DC=ForestDnsZones,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            enabledConnection: TRUE
            whenChanged: 20150317153841.0Z
            whenCreated: 20150310120919.0Z
            Schedule:
            day: 0123456789ab0123456789ab
            Sun: ffffffffffffffffffffffff
            Mon: ffffffffffffffffffffffff
            Tue: ffffffffffffffffffffffff
            Wed: ffffffffffffffffffffffff
            Thu: ffffffffffffffffffffffff
            Fri: ffffffffffffffffffffffff
            Sat: ffffffffffffffffffffffff
    1 connections found.
    Partition Replication Schedule Loading:
         
          00      01      02      03      04      05      06      07      08      09 

        10      11
         
     0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3

    0 1 2 3 0 1 2 3
            Sun:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Sun:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Mon:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Mon:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Tue:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Tue:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Wed:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Wed:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Thu:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Thu:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Fri:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Fri:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Sat:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
            Sat:

    050505050505050505050505050505050505050505050505050505050505050505050505050505050

    505050505050505
     
    Repadmin: running command /showrepl against full DC DC2 .Domainxxx .net
    Default-First-Site-Name\DC2 
    DSA Options: IS_GC 
    Site Options: (none)
    DSA object GUID: 52c29991-6165-4537-ac08-2726ce4e0dec
    DSA invocationID: 5e360a54-dd2a-4a31-be22-f1307d332269
     
    ==== INBOUND NEIGHBORS ======================================
     
    ==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============
     
    DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC1  via RPC
            DSA object GUID: 202bc518-9cac-4cc3-8743-9394abe42dfe
            Address: 202bc518-9cac-4cc3-8743-9394abe42dfe._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-18 15:26:03 failed, result -2146893022

    (0x80090322):
                The target principal name is incorrect.
            9 consecutive failure(s).
            Last success @ 2015-03-18 06:25:10.
     
    CN=Configuration,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC1  via RPC
            DSA object GUID: 202bc518-9cac-4cc3-8743-9394abe42dfe
            Address: 202bc518-9cac-4cc3-8743-9394abe42dfe._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-18 08:28:51 failed, result -2146893022

    (0x80090322):
                The target principal name is incorrect.
            1 consecutive failure(s).
            Last success @ 2015-03-18 07:48:41.
     
    CN=Schema,CN=Configuration,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC1  via RPC
            DSA object GUID: 202bc518-9cac-4cc3-8743-9394abe42dfe
            Address: 202bc518-9cac-4cc3-8743-9394abe42dfe._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-04 17:21:06 was successful.
     
    DC=DomainDnsZones,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC1  via RPC
            DSA object GUID: 202bc518-9cac-4cc3-8743-9394abe42dfe
            Address: 202bc518-9cac-4cc3-8743-9394abe42dfe._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-18 15:36:48 failed, result -2146893022

    (0x80090322):
                The target principal name is incorrect.
            8 consecutive failure(s).
            Last success @ 2015-03-18 02:54:00.
     
    DC=ForestDnsZones,DC=Domainxxx ,DC=net
        Default-First-Site-Name\DC1  via RPC
            DSA object GUID: 202bc518-9cac-4cc3-8743-9394abe42dfe
            Address: 202bc518-9cac-4cc3-8743-9394abe42dfe._msdcs.Domainxxx .net
            WRITEABLE
            Last attempt @ 2015-03-17 20:31:22 was successful.
     
    ==== KCC CONNECTION OBJECTS ============================================
    Connection --
        Connection name : 21b8891a-2a65-4487-bbcb-4ddacc0d3e35
        Server DNS name : DC2 .Domainxxx .net
        Server DN  name : CN=NTDS Settings,CN=DC2 ,CN=Servers,CN=Default-First-Site-

    Name,CN=Sites,CN=Configuration,DC=Domainxxx ,DC=net
            Source: Default-First-Site-Name\DC1 
    ******* 32694 CONSECUTIVE FAILURES since 2015-03-07 13:29:06
    Last error: 8442 (0x20fa):
                The replication system encountered an internal error.
            TransportType: intrasite RPC
            options:  isGenerated
            ReplicatesNC: DC=DomainDnsZones,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: CN=Schema,CN=Configuration,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: CN=Configuration,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            ReplicatesNC: DC=ForestDnsZones,DC=Domainxxx ,DC=net
            Reason:  StaleServersTopology
                    Replica link has been added.
            enabledConnection: TRUE
            whenChanged: 20150318025836.0Z
            whenCreated: 20150221004957.0Z
            Schedule:
            day: 0123456789ab0123456789ab
            Sun: 111111111111111111111111
            Mon: 111111111111111111111111
            Tue: 111111111111111111111111
            Wed: 111111111111111111111111
            Thu: 111111111111111111111111
            Fri: 111111111111111111111111
            Sat: 111111111111111111111111
    1 connections found.
    Partition Replication Schedule Loading:
         
          00      01      02      03      04      05      06      07      08      09 

        10      11
         
     0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3

    0 1 2 3 0 1 2 3
            Sun:

    050000000500000005000000050000000500000005000000050000000500000005000000050000000

    500000005000000
            Sun:

    050000000500000005000000050000000500000005000000050000000500000005000000050000000

    500000005000000
            Mon:

    050000000500000005000000050000000500000005000000050000000500000005000000050000000

    500000005000000
            Mon:

    050000000500000005000000050000000500000005000000050000000500000005000000050000000

    500000005000000

    ***************************************************

    I have also done the DCdiag...as its bit big thus not posted the same...

    Wednesday, March 18, 2015 11:45 AM

Answers

  • Hi,

    Lingering objects can occur if a domain controller does not replicate for an interval of time that is longer than the tombstone lifetime (TSL). The domain controller then reconnects to the replication topology. Objects that are deleted from the Active Directory directory service when the domain controller is offline can remain on the domain controller as lingering objects.

    You could refer to the MS article to remove the Lingering objects:

    https://support.microsoft.com/en-us/kb/910205

    Regards.


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com

    Monday, April 13, 2015 1:41 AM
    Moderator

All replies

  • Hello,

    DNS records don't delete all atuomatically out of th eblue. SO there MUST have happened something before this starts.

    "Than we restored the primary from 2 months old backed up VM done by HP Dataprotector."

    Is that an AD aware restore from HP, so there is NOT the VM just restored in its old state? Then you are in an USN rollback scenario.

    Please see https://support.microsoft.com/en-us/kb/870695 and https://technet.microsoft.com/de-de/library/cc949134%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396 about lingering objects.


    Best regards

    Meinolf Weber

    MVP, MCP, MCTS

    Microsoft MVP - Directory Services

    My Blog: http://blogs.msmvps.com/MWeber

    Disclaimer: This posting is provided AS IS with no warranties or guarantees and confers no rights.

    Twitter:  

    Wednesday, March 18, 2015 12:54 PM
  • I would agree with Meinolf. You will need to make sure that your backup solution is AD aware.

    Also, your backup is quite old so better to increase the frequency of backups in the future.

    I would recommend that you proceed that way:

    • Run dcpromo /forceremoval on the faulty DC to force its demotion
    • Run netdom query fsmo to check if the faulty DC was an FSMO holder or not. If yes, seize the FSMO roles on another DC: https://support.microsoft.com/en-us/kb/255504?wa=wsignin1.0
    • Do a metadata cleanup: https://technet.microsoft.com/en-us/library/cc816907%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396
    • Promote again the faulty DC

    This posting is provided AS IS with no warranties or guarantees , and confers no rights.

    Ahmed MALEK

    My Website Link

    My Linkedin Profile

    My MVP Profile

    Wednesday, March 18, 2015 1:46 PM
  • Hi,

    In addition to others.

    For the event id 1988,this issue occurs if the source domain controller has outdated objects that have been out of replication for more than one tombstone lifetime. The source domain controller is identified in the event message.   These outdated objects are referred to as lingering objects. A domain   controller that was offline for longer than the value of the tombstone lifetime   setting may contain objects that have been deleted on other domain controllers   or global catalog servers.  Additionally, tombstones for these objects may no longer exist. When you bring  the outdated domain controller back online, it cannot be notified of the object  deletions.

    For more detailed information, you could refer to:

    https://support.microsoft.com/en-us/kb/870695?wa=wsignin1.0

    Regards.


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com

    Friday, March 20, 2015 7:48 AM
    Moderator
  • Hi,

    I am very much new to 2012 AD so..have some queries..

    1. If I delete the lingering object will that effect my user login or other user which will be get deleted...as lingering object.

    2. How will I understand which DC is having  problem ...as of now when I login to Primary DC its shows both the Primary and secondary objects like...DNS, USer and Computers, etc. But when I login to secondary DC it only show the secondary objects (DNS record, AD users and computers, ect) but no primary information ...

    It says that it can not add the primary server ...which I guess that it unableto connect the primary server from the secondary servers...

    Friday, March 20, 2015 10:25 AM
  • Hi,

    As of now we have created another DC in the forest..

    When creating this new DC there was an error that ...

    When executing the DCpromo...the following error were given..

    "RID master is offline..."

    Than we have transferred the following FSMO role from the Primary DC (root Domain) to the secondary DC using the "NTDSutil" command line utility.

    1. Schema Master role

    2. DNS role

    3. Relative ID (RID) master role.

    4. PCD role.

    After this transfer to the secondary DC we can able to create the 3rd DC and sync all the object of the forest.

    Now , again we need to loose the replication as we are still having the lingering object..

    we have loosen on primary DC and Secondary ..but not changed in the 3rd DC..

    What next...

    Now we are looking to transfer every thing from the primary to the secondary DC...and make the secondary as root domain...

    Please, help me to do the same..

    Monday, March 23, 2015 7:34 AM
  • Hi,

    Lingering objects can occur if a domain controller does not replicate for an interval of time that is longer than the tombstone lifetime (TSL). The domain controller then reconnects to the replication topology. Objects that are deleted from the Active Directory directory service when the domain controller is offline can remain on the domain controller as lingering objects.

    You could refer to the MS article to remove the Lingering objects:

    https://support.microsoft.com/en-us/kb/910205

    Regards.


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com

    Monday, April 13, 2015 1:41 AM
    Moderator