none
CrossRef ncName Object Conflict - Configuration Partition

    Question

  • Hi All

    Oh Wow, I don't even know where to start, but I'll try to explain my predicament.


    I have a forest root called holdingco.int . Then I have a subdomain called sap.holdingco.int . I then have separate trees called ccs.africa and ccs.asia (this domain is to be decommissioned though, so not included in this example ) Here is my logical structure:



    Indside the sap.holdingco.int domain, I have a conflicting crossref object because of a botched DC demotion about 5 years ago ( before I started working here ). Someone apparently ticked the box "This is the last DC in the domain" and realised their mistake and stopped the demotion before it completed....or so the story goes. The domain functions OK at the moment, and there is only ONE Global Catalog in this domain. There are other GC's from other domains in the same site, so no train smash ..yet

    They now want me to try and fix the conflicting crossref object because I cannot bring any new GC's into sap.holdingco.int due to the broken crossref. I did a Semantec Database Analysis with a go fix and this converted the mangled crossref back to the way it should be, but it seems that the conflict comes back as soon as the Infrastructure master updates itself.

    I have cloned the holdingco.int and sap.holdingco.int domain controllers to an isolated environment on Vmware to do some tests, and what I have basically done was to disable replication between all DC's in sap, then did the fixup, restarted and kept replication disabled ( both ways ). After a few minutes, the Infrastructure master shows the ccs.africa ncName as conflicting, however this ncName is still intact on the other DC's, Proving that the source of the conflict is coming from the Infrastructure Master.

    Even if I move the IM FSMO role to another DC and do the fix all over again, the DC with the IM FSMO role always conflicts first, and this then gets replicated out to the other DC's in sap.holdingco.int.

    Note, neither ccs.africa, nor holdingco.int has this conflict on the partitions node if I look at it through adsiedit.

    I assume there are still references in sap.holdingco.int somewhere that causes the conflict, but I just cannot see how this can be fixed. I have NO healthy sap.holdingco.int backup to fall back on, so I cannot even do an authoritative restore.


    I'm taking a long shot here, and I hope someone may be able to help me before I will be forced to log a ticket with MS. Here is a screenshot of where the issue lies : 



    Thanks in advance for your help.

    Pieter


    Tuesday, June 18, 2013 1:53 PM

Answers

  • What if you perform the auth restore that you did in SAP on a root DC on the healthy cross-ref, but use the following command instead:
    restore object CN=CCS,DC=... verinc 100002

    Just to be clear:
    un-managled cross-ref = health cross-ref without CNF:<GUID> name
    managed cross-ref = bad cross-ref with CNF:<GUID> name

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Thursday, June 27, 2013 12:00 PM

All replies

  • Do you have other DCs in the domain with the IM (except for the DC that being the IM)? What if you just shutdown this role and size the IM over to another DC?

    It will be hard to troubleshooting this without knowing the cross-references (ref-counts) to the cross-ref at a NTDS.dit database level, so if the suggestion above dosen't work I actuelly suggest you to open up a case with Microsoft Product Support Service.

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    • Proposed as answer by 朱鸿文 Wednesday, June 19, 2013 1:58 AM
    Tuesday, June 18, 2013 2:41 PM
  • Hi Christoffer

    Thanks for the reply.

    I actually read your blog earlier today where you explained a similar scenario, but not exactly like mine.

    When you say another DC's with the IM role, do you mean have I got another DC in the sap.holdingco.int domain or in one of the other domains that can hold the IM role ?

    Each domain has an IM, so I guess I don't understand what you mean. Like I said in my scenario, I have transfered the IM to other DC's inside the SAP domain after fixing the conflict with SEMANTEC DB fixup , but as soon as I enable replication after a reboot, the conflict then starts happening on the new IM first, then replicates out to the other "healty" DC's until all 3 of them have conflicting crossRef Objects in their config partitions.

    Thanks for the help

    Tuesday, June 18, 2013 2:57 PM
  • Okey - I see.

    Can you post the results of repadmin /showobj <DSA> <CNF-CROSSREF> as well repadmin /showobj <FIXED-CROSSREF>?

    I'm intressted in knowing the USNs here, that determines the object that is going to be replicated first.


    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog


    Tuesday, June 18, 2013 3:48 PM
  • Hi Chris

    I had to wipe my LAB coz the reverting of snapshots broke my DC's completely. I will clone them from scratch and do the tests and post the results here soonest

    Thanks very much for your help thus far

    Much Appreciated.

    Wednesday, June 19, 2013 7:44 AM
  • Hi Chris

    I ran the fix on my Infrastructure Master DC on the SAP domain from scratch after cloning all DC's from the Holdingco and SAP domains to recreate my environment to some extent. It appears that the USN stays the same on the Infra Master from the looks of things, unless I'm doing something wrong :

    So, on the Infra Master FSMO holder on the SAP Domain, I disabled replication in and out, then I ran repadmin /showobjmeta <Infra master DC> <Crossref DN> before the fix and saved the output, restarted, logged into DS Repair mode and ran the SEM d a go fix (see the output in fig 1) . I then restarted again, and then ran the repadmin command again (specifying the Infra Master DC each time ) - output can be seen in fig 2 before and after the fix was run :

    Fig 1

    Fig 2

    Thanks !


    Wednesday, June 26, 2013 8:14 AM
  • Can you try update something on the object now, so that the USN bumps up - for example you can set the 'adminDescription' attribute on the cross-Ref to some value, then let the IM replicate with your other DC(s) in the lab.


    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 9:14 AM
  • If I change the Description, it takes, but as soon as I hit apply and go back to check if its there, the field goes back to <Not Set>

    So the change does not stick ? Any idea why ? I'm making the change as the Enterprise Admin and that has Full control on the object...doesn't make sense.

    Thanks

    Wednesday, June 26, 2013 10:20 AM
  • Dose the change "rollback" or revert even before you have enabled replication to/from the infrastructure master?

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 11:02 AM
  • I'm trying to make the change while replication is still disabled.

    Even with FULL control on this crossref object now - I still cannot make a change - it just reverts back to <not set>. It appears to only happen on the crossref objects (maybe by design ? )

    Thanks

    Wednesday, June 26, 2013 11:17 AM
  • very strange, have you disabled both inbound and outbund replication? as well make sure that you're connected to the right DC while making the change?
    repadmin /options <DC NAME> +DISABLE_OUTBOUND_REPL
    repadmin /options <DC NAME> +DISABLE_INBOUND_REPL


    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 11:23 AM
  • Yes sir !

    The highlighted DC is our IM for the SAP domain, and the replication is still disabled as you can see :

     

    Wednesday, June 26, 2013 1:06 PM
  • I can confirm the same behaivor with the 'description' attribute, try set the value of the 'adminDescription' attribute specifically instead.

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 1:15 PM
  • Hey Chris

    It doesn't matter which one I try to set, the saved values do not stick. It reverts back to the way it was before i.e. <not set>.

    Are these Crossref objects not protected from editing somehow ? Strange thing is : I can set values on the parent folder ( CN=Partitions ) and those do stick, and even setting permissions on this level are propagated down to the crossref object itself.

    However, on the crossref object, even with FULL control access, I cannot change a thing. Attribute or security related changes, it just discards any change made.

    Thanks very much for your help on this !



    Wednesday, June 26, 2013 1:27 PM
  • Pieter,

    See if this helps. I had to do this for someone last year. I documented everything I did. It may be easiest to delete the CrossRef and recreate it. Keep in mind, you may have to remove the zone from AD integrated first, perform the deletion, allow it to auto-recreate, then put the zone back in. Look at the steps to see what I mean.

    Steps taken to resolve an issue with corrupted application partitions, specifically, DNS partitions and their CrossRef(erence) objects in the AD Configuration Container
    Published by acefekay on Jun 20, 2012 at 11:32 PM
    http://msmvps.com/blogs/acefekay/archive/2012/06/20/steps-taken-to-resolve-an-issue-with-corrupted-application-partitions-specifically-dns-partitions-and-their-crossref-erence-objects-in-the-ad-configuration-container.aspx


    Ace Fekay
    MVP, MCT, MCITP/EA, MCTS Windows 2008/R2 & Exchange 2007, Exchange 2010 EA, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Technical Blogs & Videos: http://www.delawarecountycomputerconsulting.com/

    This post is provided AS-IS with no warranties or guarantees and confers no rights.

    FaceBook Twitter LinkedIn

    Wednesday, June 26, 2013 1:29 PM
  • Hi Ace

    I appreciate your help, but the issue here is not a DNS partition, it's the actual crossref object in the configuration partition. 

    I will not be able to delete the crossref object anyway, as this is protected by the system by the looks of it.

    This is what I get if I try to delete it :



    Wednesday, June 26, 2013 1:39 PM
  • Are you logged on as an Enterprise Admin? I can set the value on my domain cross-ref objects fine - strange

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 1:46 PM
  • Are you changing it on your forest root domain config partition ?

    I am trying to make the change on SAP DC's config partition ( the Infra master  ) ( SAP is a subdomain of my forest root domain ) - please refer to the image at the start of this thread.

    Thanks

    I can change the values on my forest root domain's crossref object ( is that where i'm suppose to make the change so it can replicate down to the SAP DC ? ) 

    Just a bit of confusion i think

    Wednesday, June 26, 2013 1:50 PM
  • Yes - but AD seems to generate a refferal to the domain naming master, But we can do this without the Domain Naming Master by performing a auth restore of the cross-ref that will bump up the USN as well - do you know how to do a such? (if not I will guide you) 


    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog



    Wednesday, June 26, 2013 2:08 PM
  • My Domain Naming Master is in the LAB. It is a forest root DC.

    Can I restore only the config partition with ntdsutil ? Your help will be appreciated.

    Wednesday, June 26, 2013 2:22 PM
  • My example was based on corrupted zone partitions AND a corrupted CrossRef. The steps are still the same to delete and recreate the CrossRef. As for the deletion not allowed, do you have 2008 and newer DCs in the mix? If yes, it may be possible there's a "protect from accidental deletion" set. Look in the properties for that checkbox.

    Ace Fekay
    MVP, MCT, MCITP/EA, MCTS Windows 2008/R2 & Exchange 2007, Exchange 2010 EA, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Technical Blogs & Videos: http://www.delawarecountycomputerconsulting.com/

    This post is provided AS-IS with no warranties or guarantees and confers no rights.

    FaceBook Twitter LinkedIn

    Wednesday, June 26, 2013 2:40 PM
  • In this case unfortunately, the Domain with the issue only has 2003 DC's. I think next best step is an authoritative restore of either that crossref object, or the Configuration partition from one of my forest root DC's
    Wednesday, June 26, 2013 2:49 PM
  • You should do a restore of only the "fixed" cross-ref object on the IM (before it replicates with anyone else, after that you have run semantic database analyses with fix go).

    We're actuelly not really "restoring anything here" the only thing we want to do is to get a higher USN on the fixed cross-ref object so it replicates before the mangled one.

      • Ensure the DC has the following hotfix installed:
        http://support.microsoft.com/kb/943576
      • Set the Directory Services Restore Mode (DSRM) password on the recovery DC if you do not know the password. See KB article 322672 for more information.
      • Restart the domain controller and boot to DSRM.  You can set the boot.ini to boot directly to DSRM with MSCONFIG on the boot.ini tab using /safeboot disrepair, otherwise press the F8 key when you see the Boot menu
      • If you used the F8 method to enter the boot menu select Directory Services Restore Mode, and then press ENTER.

      • Restore the fixed-cross ref:
      1. At a command prompt, type ntdsutil, and then press ENTER.
      2. Type authoritative restore, and then press ENTER.
      3. Type the following command, and then press ENTER:

        restore object CN=cross-ref,CN=Partitions,DC=foo

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 3:02 PM
  • I've added the steps above on how to do the auth resstore of the cross-ref, if this dosen't (e.g the change is not accepted) you need a low-level database tool that can change the USN - in that case I will suggest you to open up a support case with Microsoft Product Support (PSS)

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Wednesday, June 26, 2013 3:04 PM
  • It seems to have worked for the SAP DC's. The USN was bumped up and everything, but now the forest root DC's do not want to replicate the new object in because it thinks there is a collision in object names :

    EDIT: I just checked again, and the CNF is back :


    I'm starting to think that this issue is bigger than me, and I may need help from MS..


    Thursday, June 27, 2013 6:53 AM
  • Can you post a repadmin /showreps output? Is the name collision only happening on root DCs?

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Thursday, June 27, 2013 8:30 AM
  • The image below shows repadmin output from a root DC ( HOLDC03 ). 

    It's trying to pull changes from SAPDC02 ( which is where I did the fix and did the auth restore on ). After the restore I only enabled OUTBOUND replication on SAPDC02 for the time being to see if the other DC's would pull the new crossref object, but the root DC's seems to have a problem with it.



    Thursday, June 27, 2013 9:02 AM
  • Just so we're on the same page, has the cross-ref dissapeard on all SAP DCs? Dose the root DCs has both a mangled and none-mangled CCS cross-ref now?


    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Thursday, June 27, 2013 9:44 AM
  • Hi

    On the SAP DC's - the Mangled cross-ref is back on all of them, but on the root DC's the cross-ref's are still OK, because they seems to have refused to replicate in the restored crossref object i restored earlier on the SAP Infra Master.

    Thursday, June 27, 2013 10:11 AM
  • Can you attach a print screen of the Partitions container from DCs in both domains, as well repadmin /showobjmeta from a DC in each domain for both cross-refs (e.g both the managled and unmanagled)

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Thursday, June 27, 2013 10:26 AM
  • Partitions folder :

    I notice the DNS crossref is also now mangled on the SAP DC's
    Thursday, June 27, 2013 10:58 AM
  • ROOT Domain above:

    SAP Domain Below:

    There are NO unmangled crossref objects in SAP, so I cannot pull the objmeta on that.
    Thursday, June 27, 2013 11:04 AM
  • What if you perform the auth restore that you did in SAP on a root DC on the healthy cross-ref, but use the following command instead:
    restore object CN=CCS,DC=... verinc 100002

    Just to be clear:
    un-managled cross-ref = health cross-ref without CNF:<GUID> name
    managed cross-ref = bad cross-ref with CNF:<GUID> name

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Thursday, June 27, 2013 12:00 PM
  • OK - I'll give that a go and let you know !

    Thx

    Thursday, June 27, 2013 12:06 PM
  • Hi Chris

    I don't want to get excited too early, but I think we cracked this one ! Your last suggestion to restore the object on the root DC seems to have done the trick !

    I'm going to leave it over the weekend and see if the mangled object returns, but so far so good ! If it doesn't return, I will mark your last suggestion as the answer !

    Thank you very very much ! I really appreciate all your help !

    Friday, June 28, 2013 10:02 AM
  • Hey Chris !

    Success !

    Thanks so much for all your help and suggestions! Problem seems to have been sorted. I have marked your last suggestion as the answer !

    Cheers !

    Monday, July 1, 2013 7:23 AM
  • I'm glad that this is now sorted out, and that I could help :)

    Enfo Zipper
    Christoffer Andersson – Principal Advisor
    http://blogs.chrisse.se - Directory Services Blog

    Monday, July 1, 2013 7:30 AM