none
How to remove spoilt AD?

    Question

  • Dear Server Expert,

    Our AD infrastructure details are as below:

    1x Primary DC (On Premise) -> "DC01" (Windows Server 2008 R2)

    1x Secondary DC (On premise) for backup -> "DC02" (writeable DC) (Windows Server 2008 R2)

    1x Secondary DC (Remote Site) for remote site servers authentication -> "RDC01" (writeable DC) (Windows Server 2008 R2)

    1x Secondary DC (Remote Site) for remote site servers authentication -> "RDC02" (writeable DC) (Windows Server 2008 R2)

    1x Secondary DC (Cloud) for cloud server authentication -) "CDC01" (Writable DC) (Windows Server 2012 Standard)

    Issues:

    The Cloud DC had actually being disjoined from domain by ex-admin staff and currently we had actually shutted down the whole Cloud DC half year ago. we don't know what happen and I guessed it could be forcefully disjoined leaving a lot of rubbish configurations inside due to improper demotion of AD. Currently, we would like to destroy the spoilt AD VM in the cloud and at the same time clean up the those messy setting in our Primary AD.

    do you guys have any idea what is the best recommended way of doing this? will that be high risk of doing so called "metadata" clean up from what I read from the forum?

    Thanks

    Appreciate your help

    Sunday, December 11, 2016 7:03 AM

All replies

  • HI Henry,

    since the cloud DC is already removed or shutdown or in other words "not alive" anymore then i'd recommend doing metadata cleanup from command line as it's easy, fast and very reliable tool to "clean" your AD database from that Cloud DC..

    there is a step by step in following article and really don't worry using metadata cleanup as it's really very easy and i've done it like 100 times so far.

    https://support.microsoft.com/en-us/kb/216498

    let me know if you need any help

    Mahmoud

    Microsoft CTS


    Thanks Mahmoud

    • Proposed as answer by mahelsay Sunday, December 11, 2016 7:22 AM
    Sunday, December 11, 2016 7:22 AM
  • HI Henry,

    since the cloud DC is already removed or shutdown or in other words "not alive" anymore then i'd recommend doing metadata cleanup from command line as it's easy, fast and very reliable tool to "clean" your AD database from that Cloud DC..

    there is a step by step in following article and really don't worry using metadata cleanup as it's really very easy and i've done it like 100 times so far.

    https://support.microsoft.com/en-us/kb/216498

    let me know if you need any help

    Mahmoud

    Microsoft CTS


    Thanks Mahmoud

    Hi Mahelsay,

    Thanks for the article. I think it is quite complicated, isn't it? is there any GUI ways of cleaning metadata? btw, will cleaning metadata is a risky work that will affect Primary AD since you are already experience with 100 times so far?

    Another question that I have is another DC that is on another cloud has down for about 3 months (being shutted down but not spoilt and I have turn it back on. since it has down for 3 months and now I bring it on, will there be any replication issue? I checked our tombstone lifetime for the on premise and Cloud AD is actually 180 days.

    Please help to advise.

    many thanks

    Regards,

    H

    Thanks

    Sunday, December 11, 2016 8:11 AM
  • Hi Henry,

    unfortunately metadata cleanup is only through cmd as it's actually using the ntdsutility ..

    doing this metadata cleanup will not affect any other DC. think of it like deleting a folder in windows by simple > right click folder > delete however in your case it's just a way to make sure all of this folder tree and subtree and branches are not only removed but also to be replicated to the DC partners that this DC has been deleted. so basically you are deleting all of its entries from whole forest.

    actually the procedure is not complicated at all if you just visualize the idea behind each step and you will find it making sense. starting by connections: to connect to certain DC to issue the commands from and then listing all domain, all sites and all servers so that you eventually delete selected server... 

    for the other DC you mentioned since you brought it up again then you may want to simply force replication and the difference between AD USN number versions will do all the work..

    hope this helps and please mark reply as helpful if you find it helpful


    Thanks Mahmoud

    • Proposed as answer by mahelsay Monday, December 12, 2016 4:53 AM
    Sunday, December 11, 2016 9:14 AM
  • Hi Henry,

    unfortunately metadata cleanup is only through cmd as it's actually using the ntdsutility ..

    doing this metadata cleanup will not affect any other DC. think of it like deleting a folder in windows by simple > right click folder > delete however in your case it's just a way to make sure all of this folder tree and subtree and branches are not only removed but also to be replicated to the DC partners that this DC has been deleted. so basically you are deleting all of its entries from whole forest.

    actually the procedure is not complicated at all if you just visualize the idea behind each step and you will find it making sense. starting by connections: to connect to certain DC to issue the commands from and then listing all domain, all sites and all servers so that you eventually delete selected server... 

    for the other DC you mentioned since you brought it up again then you may want to simply force replication and the difference between AD USN number versions will do all the work..

    hope this helps and please mark reply as helpful if you find it helpful


    Thanks Mahmoud

    Hi Mahelsay,

    I found one article that doing the metadata clean up is getting easier in windows 2008. please have a look on the article from Microsoft below. do you think make sense and will it different with your technique?

    https://technet.microsoft.com/en-us/library/cc816907(v=ws.10).aspx

    Another question pertaining to the Cloud AD. I have a concern. if let's say for instance the tombstone is 60 days and now has already expired. what will happen if i turn on the Cloud DC since the Cloud DC don't know that the modified or deleted objects has already remove from the Primary AD due to the expired tombstone? what will happen when the replication occured? will it cause bad issue that affect Primary AD? i am worry that the Cloud AD will replicate back to the Primary AD and cause corruption due to the Primary AD will think that those objects has already tombstones and Cloud AD replicate it back.

    Please advise. Appreciate your sharing knowledge on this.

    Thanks

    Sunday, December 11, 2016 4:10 PM
  • Hi Henry, Please look carefully for this : You will run into problems only if the period when the DC was down "exceeds" the tombstone value configured in your environment. This could cause a lingering objects being kept in the problematic DC after you bring it up and running again. Following is a nice short article that explains it and explains 3 options you have. In addition to the 3 options I want to add another option which is to manually make AD authoritative restore in order to force replication .. Please read this article : https://pmeijden.wordpress.com/2011/01/12/domain-replication-has-exceeded-the-tombstone-lifetime/ Thanks

    Thanks Mahmoud

    • Proposed as answer by mahelsay Monday, December 12, 2016 4:53 AM
    Sunday, December 11, 2016 4:40 PM
  • Hi Henry, Please look carefully for this : You will run into problems only if the period when the DC was down "exceeds" the tombstone value configured in your environment. This could cause a lingering objects being kept in the problematic DC after you bring it up and running again. Following is a nice short article that explains it and explains 3 options you have. In addition to the 3 options I want to add another option which is to manually make AD authoritative restore in order to force replication .. Please read this article : https://pmeijden.wordpress.com/2011/01/12/domain-replication-has-exceeded-the-tombstone-lifetime/ Thanks

    Thanks Mahmoud

    HI Mahmoud,

    thanks again for the useful article. yes, i have read the article. it is indeed very good article for the AD that has expired tombstone.

    But now, since i have checked that my AD tombstone is being set to 180, which is 6 months and my server is only down for 3 months, My cloud AD should be still working right? is there anyway that we can ensure whether the cloud AD downtime has passed the tombstone?

    Now, though the Cloud AD server can be up and working perfectly,i shutted down again to ensure that replication won't happen.

    Appreciate your advise again. I would like to know the way of how we can check whether the AD has passed the tombstone period?

    Many thanks

    Regards,

    H

    Sunday, December 11, 2016 5:25 PM
  • hi Henry,

    the configured tombstone value can be checked from adsiedit.msc.  also if it happen and DC pass the configured tombstone value you will find an event thrown in event log stating that replication has been not running for period exceeding tombstone value.

    from what i understood so far now since your DC was kept shutdown for period less than tombstone then all what you have to do is to get it up and running again and manually force replication and monitor for 2 hours as it's a sufficient time to get a picture of what is happening specially if you just monitor the event log for first 1 hour and you will get to know how it goes...

    i hope this helps and please mark it as helpful or answered and that would be much appreciated

    Thanks


    Thanks Mahmoud

    • Proposed as answer by mahelsay Monday, December 12, 2016 4:53 AM
    Monday, December 12, 2016 4:51 AM
  • hi Henry,

    the configured tombstone value can be checked from adsiedit.msc.  also if it happen and DC pass the configured tombstone value you will find an event thrown in event log stating that replication has been not running for period exceeding tombstone value.

    from what i understood so far now since your DC was kept shutdown for period less than tombstone then all what you have to do is to get it up and running again and manually force replication and monitor for 2 hours as it's a sufficient time to get a picture of what is happening specially if you just monitor the event log for first 1 hour and you will get to know how it goes...

    i hope this helps and please mark it as helpful or answered and that would be much appreciated

    Thanks


    Thanks Mahmoud

    Hi Mahmoud,

    thanks all along for your patient guidance.

    I have use ADSIEDIT to check the tombstone of my Primary AD and the result is 180, which is 6 months? please find below picture attached:

    I could bring the Cloud AD online, but I just worry because we don't know when the Cloud AD has started to down and I am worry that when the replication happen it will spoilt the Primary AD due to some lingering objects that being replicated back from Cloud AD to the Primary AD. If it is within the tombstone then it is good, but what happen if it has passed the tombstone.

    Now, do you have any suggestion on how to check whether the Cloud AD has already passed over the tombstone period? just to ensure that I didn't calculate wrongly (though after i checked the log from the Cloud AD, the log is still updated till September 15 and after that there is no more log, meaning during that time the Cloud AD might has already shutted down due to certain reason). I did some check that object like users created after 14 September is already replicated to the Cloud AD, so I think it is quite sure that the Cloud AD might has already shutted down after that.  

    Or the only way is to do manual Replication from Primary AD to check?

    Thanks

    Regards,

    Monday, December 12, 2016 12:14 PM
  • Hi Henry,

    to get detailed replication status with timing like last success for each DC in your whole forest you can run following simple command and export it to CSV file:  repadmin /showrepl * /csv > C:\logs\repl.csv

    it will give you a very nice excel sheet with columns as source, destination, ....etc that you can simply filter it and get an idea on last time replication was successful to the cloud DC which could be considered as starting point that DC went down.

    hope this is what you are looking for

    please mark it as helpful or answered 


    Thanks Mahmoud

    • Proposed as answer by mahelsay Tuesday, December 13, 2016 6:43 AM
    Monday, December 12, 2016 4:55 PM
  • Hi Henry,

    to get detailed replication status with timing like last success for each DC in your whole forest you can run following simple command and export it to CSV file:  repadmin /showrepl * /csv > C:\logs\repl.csv

    it will give you a very nice excel sheet with columns as source, destination, ....etc that you can simply filter it and get an idea on last time replication was successful to the cloud DC which could be considered as starting point that DC went down.

    hope this is what you are looking for

    please mark it as helpful or answered 


    Thanks Mahmoud

    Hi Mahmoud,

    thanks for the valuable advise again.

    questions:

    1. to get the result of the repadmin /showrepl, does the Cloud Ad need to be online?

    2. I run the command without logging to excel and give me the result below:

    does it mean that it has down since 1 september 2016 and currently since the tombstone is 180 days, meaning it is still 3 months to go?if yes, then i think i could try to start my Cloud DC and let the replication to happen since the server is still able to up and running perfectly. Just we don't know whether the replication is working properly or not.

    Many Thanks

    Regards,

    H

    Tuesday, December 13, 2016 2:44 AM
  • Hi Henry,

    Glad to hear again from you.

    no the cloud DC doesn't have to be online to run the command.

    also the excel sheet is way better than the command line output to avoid any confusion.

    based on what you are saying YES it seems it's safe to get the cloud DC up and running..

    bottom line: you can get the cloud DC up and running again and all what you need to do is to just force replication and once replication attempt Is finished it will throw event status in event viewer and usually these directory service and replication events are very rich in details it contain that will point direct to the issue IF ANY or will state that the operation was successful

    hope this helps and glad to assist anytime

    just i'd appreciate if you mark the post as helpful or answered if you find it useful to you

    Thank You


    Thanks Mahmoud

    • Proposed as answer by mahelsay Tuesday, December 13, 2016 6:43 AM
    Tuesday, December 13, 2016 4:24 AM
  • Hi Henry,

    Glad to hear again from you.

    no the cloud DC doesn't have to be online to run the command.

    also the excel sheet is way better than the command line output to avoid any confusion.

    based on what you are saying YES it seems it's safe to get the cloud DC up and running..

    bottom line: you can get the cloud DC up and running again and all what you need to do is to just force replication and once replication attempt Is finished it will throw event status in event viewer and usually these directory service and replication events are very rich in details it contain that will point direct to the issue IF ANY or will state that the operation was successful

    hope this helps and glad to assist anytime

    just i'd appreciate if you mark the post as helpful or answered if you find it useful to you

    Thank You


    Thanks Mahmoud

    Hi Mahmoud,

    Thanks for the professional advise again.

    OK,  just to be confirm, i would like to try on the manual replication, because i am not sure how to check when the AD will sync daily. is there any location where we can check?

    If for instance, our calculation is wrong and the tombstone is expired, and at the same time we do manual replication, will it cause any bad effect on the Primary AD? or the system will just throw an error message?

    May i know how to do a manual replication from Primary to Cloud AD ONLY, because this is to ensure that the Cloud AD won't sync back to Primary when it is up.

    Thanks

    Tuesday, December 13, 2016 5:33 AM
  • Hi Henry,

    after you bring the cloud DC up and running you can manually force replication using following command:

    repadmin /syncall <DomainControllerName> /e /d /A /P /q

    it's mentioned in following article :https://technet.microsoft.com/en-us/library/cc816915(v=ws.10).aspx

    also if the tombstone value was exceeded nothing serious will affect your primary DC and all what could happen is for the cloud DC to become "confused" and have many lingering objects that you can also clean manually..

    bottom line is :

    for replication to happen there should be difference in revision number between the 2 DCs so the higher the number the DC has it will replicate the changes as a "source"

    so at the end of the day since the cloud DC was shutdown for long time then for sure he will not replicate as a "source" because simply this DC missed a huge period of object changes since last successful replication!

    hope that makes sense


    Thanks Mahmoud

    • Proposed as answer by mahelsay Tuesday, December 13, 2016 6:43 AM
    Tuesday, December 13, 2016 6:43 AM
  • Hi Henry,

    after you bring the cloud DC up and running you can manually force replication using following command:

    repadmin /syncall <DomainControllerName> /e /d /A /P /q

    it's mentioned in following article :https://technet.microsoft.com/en-us/library/cc816915(v=ws.10).aspx

    also if the tombstone value was exceeded nothing serious will affect your primary DC and all what could happen is for the cloud DC to become "confused" and have many lingering objects that you can also clean manually..

    bottom line is :

    for replication to happen there should be difference in revision number between the 2 DCs so the higher the number the DC has it will replicate the changes as a "source"

    so at the end of the day since the cloud DC was shutdown for long time then for sure he will not replicate as a "source" because simply this DC missed a huge period of object changes since last successful replication!

    hope that makes sense


    Thanks Mahmoud

    Hi Mahmoud,

    thanks again and from your explanation, it looks like quite safe the bring the CLoud DC online since i know that this VM is still able to run.

    i am sorry as i m not good in using the command. I am thinking to use the Active Directory of Sites and Services for the manual replication since we are using Windows Server 2008  R2. is it giving the same effect?

    Is there any way that we could check every how long the replication will occur in AD?

    thanks.

    Regards,

    H

    Tuesday, December 13, 2016 8:59 AM
  • I find another informative article which summarize step-wise instructions to perform metadata cleanup in active directory - https://community.spiceworks.com/how_to/132621-how-to-perform-metadata-cleanup-in-active-directory
    Tuesday, December 13, 2016 10:15 AM
  • I find another informative article which summarize step-wise instructions to perform metadata cleanup in active directory - https://community.spiceworks.com/how_to/132621-how-to-perform-metadata-cleanup-in-active-directory

    Hi Andres,

    Thanks for the article and indeed it is very interesting which just the same as below article from Microsoft, which is clearing MetaData by using the GUI, which is only work for Windows 2008 and above and it is written that it could clear the metadata automatically, which is very fantastic. 

    https://technet.microsoft.com/en-us/library/cc816907(v=ws.10).aspx

    So there might no need to use NTDSUTIL.EXE which is always use for Windows Server 2003 SP1 and below. However i am really not sure whether such a simple way will work but of course it is better to try out with simple thing first rather than going straight to the complicated one rights.

    However, did you ever try to experiment of whether this simple method of using GUI is enough to clear every metadata?

    Thanks


    Tuesday, December 13, 2016 1:08 PM
  • Hi,

    According to my experience and the share from others, the method of GUI could clean the metadata, and you could also have a try. But I am used to use Ntdsutil.exe method, as it could be also used in the scenario when an unsuccessful domain controller demotion happen: https://support.microsoft.com/en-sg/kb/216498

    Best regards,

    Wendy


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, December 16, 2016 2:01 AM
    Moderator
  • hi Henry,

    i'm back..

    i second what Wendy said. command line is a way to go to avoid any possible unsuccessful demotion. it's not a matter of doing it the easy way but a successful way from first time

    Thanks


    Thanks Mahmoud

    Friday, December 16, 2016 4:27 AM