locked
Time service reset to ten years ago on PDC RRS feed

  • Question

  • Time service reset to ten years ago on PDC today, which causes many network and replication issue. Is there public statement to describe this issue for the Microsoft's time provider? I notice it's not just me to encounter this issue.
    Tuesday, November 20, 2012 3:14 AM

Answers

All replies

  • Same here. PDCe got sent back in time to 11/19/2000 by the USNO time source. AD tombstone period issues (of course). Other DCs remained at correct time so they aren't replicating with the PDCe anymore (eventID 2042). I can't even move the FSMO roles to another DC and demote this one. I need to do the full sing and dance: yank the server out, clean up AD, seize FSMO, promote another DC. We are a good size shop and there's a lot on the line. No way I'm doing this on my own. I put in a call to PSS almost 3 hours ago. The level 1 guy told me I'd get a call back from an engineer within 2 hours. Well two hours have come and gone. Now what?
    Tuesday, November 20, 2012 4:10 AM
  • Not just you? Do you have links on what you've read (assuming you've read it somewhere on the internet)?

    Without knowing any specifics about your environment, confguration, event log errors, number of domains, etc, since none was offered, what I can offer is if you see a 10 year shift in time is usually a bad battery, unless there is some sort of corruption in the time service (albeit a temporal distortion).

    Virtualizing Domain Controllers and the Windows Time Service
    Published by acefekay on Aug 23, 2011 at 1:15 AM
    http://msmvps.com/blogs/acefekay/archive/2011/08/23/virtualizing-domain-controllers-and-the-windows-time-service.aspx

    .

    Is the PDC in a virtual environment? If so, you must disable the VM host time service, and this is no matter what VM hosts you are using, such as whether VMWare, VirtualBox, HyperV, etc. Then configure the time service on the PDC.

    I assume that the older operating systems, including whether the are the PDC Emulator, other DCs and client machines, that they've had the DST hotfix installed.

    It also depends on how you configured the PDC Emulator. As you know, the PDCe in the forest root is the top of the time hierarchy in an AD forest. It's recommended that the PDCe is configured to sync time with an outside source. A few liinks below offer methods to configure the time service. There is a Microsoft Mr FixIt tool in KB816042 (link below) to fix the time service on the DCs.

    Configuring the Windows Time Service for Windows 2000, 2003, 2008 and newer, explanation of the time service hierarchy, and more
    Published by Ace Fekay, MCT, MVP DS on Sep 18, 2009 at 8:14 PM  3050  1 
    http://msmvps.com/blogs/acefekay/archive/2009/09/18/configuring-the-windows-time-service-for-windows-server.aspx

    Tehcnet thread: "Time Sync best practices," by Halo-NEXT,
    http://social.technet.microsoft.com/Forums/en-US/winserverDS/thread/043b1ebe-e7bc-40ca-91e0-174a6854808e/

    Good discussion on the default Time service in a forest
    Technet Thread: "Time Sync best practices" by Halo-NEXT 5/21/2012
    http://social.technet.microsoft.com/Forums/en-US/winserverDS/thread/043b1ebe-e7bc-40ca-91e0-174a6854808e/

    How to configure an authoritative time server in Windows Server
    http://support.microsoft.com/kb/816042

    .

    If there is any corruption in the Time Service, to reset the time service, here aer a few steps, and note, it is not necessary to make any changes in the registry, since I've found many that have, have corrupted it. It's really easier to simply use the W32time command. So if you've experimented changing time settings to unknowlingly avert default behavior, you can set the time settings back to default:

    1. On the DC that you're experiencing issues with, run the following in a command prompt:
     net stop w32time
     w32tm /unregister
     w32tm /register
     net start w32time
     
    2. On the Server in question, run the following in a command prompt:
     "net time /setsntp: " (without the quotes, but I put that in there to signify the blank space prior to the closing quote)
     [This tells the client (whether a DC or workstation) to delete the current registry settings for time and use default settings.]
     
     Restart the time service:
     Net stop w32time && net start w32time
     
    3. On the PDC Emulator run the following in a command prompt:
     W32tm /config /manualpeerlist:time.windows.com /syncfromflags:manual /reliable:yes /update
     W32tm /resync /rediscover
     
     Restart the time service:
     Net stop w32time && net start w32time
     
    4. On each DC that are not holding the PDC Emulator role, run the following in a command prompt:
     w32tm /config /syncfromflags:domhier /update
     W32tm /resync /rediscover

     Restart the time service:
     Net stop w32time && net start w32time
     
    5. This will take out any errors in the Event Viewer, if there are any.

    .


    Ace Fekay
    MVP, MCT, MCITP/EA, MCTS Windows 2008/R2 & Exchange 2007, Exchange 2010 EA, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Technical Blogs & Videos: http://www.delawarecountycomputerconsulting.com/

    This post is provided AS-IS with no warranties or guarantees and confers no rights.

    FaceBookTwitterLinkedIn


    • Edited by Ace Fekay [MCT] Tuesday, November 20, 2012 4:16 AM
    • Proposed as answer by VenkatSP Tuesday, November 20, 2012 4:22 AM
    Tuesday, November 20, 2012 4:15 AM
  • I totally agree with Ace. He gave you all the possible options to fix Time Sync issue that you are referring to. Hope you will fix the issue.
    • Edited by VenkatSP Tuesday, November 20, 2012 5:07 AM
    Tuesday, November 20, 2012 4:24 AM
  • Please read again. Carefully.

    External time source. NTP server. Run by US Naval observatory. Sent a wrong time synch to anyone who used them. Rolled back our PDCe to 11/19/2000, and then back to present when the issue on their end was corrected. I already e-mailed US Navy about this and, I kid you not, within 5 minutes someone from the Navy replied and apologized. Meanwhile I'm waiting 3+ hours for a callback from PSS. The SLA is 2 hours.

    This isn't about time sync any more. This is about AD tombstone period. I've got a lot of work to do to clean up the AD.

    Tuesday, November 20, 2012 4:38 AM
  • Sir, that's ture. Actually, we needn't clean up the domain controller, just need to modify one registry to force replicaion. Because there is no lingering object.

    Tuesday, November 20, 2012 5:05 AM
  • Could you please forward the apologized email to me? So I can give my boss an explanation. Thanks, My box is tommy2010@foxmail.com.Thanks again,

    Tuesday, November 20, 2012 5:07 AM
  • HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and Corrupt Partner

    That? On all DCs. And then leave it there indefinitely? Seems like a quick patch rather than a proper fix.

    I was going to implement that reg key and then move FSMOs and demote this DC. That was my plan. But I wanted PSS on the horn. This is too big to screw up.

    If that doesn't work for whatever reason, then we are looking at shutting down this DC, cleaning up AD metadata and seizing FSMO roles.

    Tuesday, November 20, 2012 5:08 AM
  • Yes, Tommy is right. If you want us to help you really then I would suggest you to tell us about your Infrastucture. Like howmany DCs and how's the replication and DCDiag log and IPConfig /all of all the DCs. If not just run this command on all the DCs and let us know if you find any errors. repadmin /replsum
    Tuesday, November 20, 2012 5:12 AM
  • HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and Corrupt Partner

    That? On all DCs. And then leave it there indefinitely? Seems like a quick patch rather than a proper fix.

    I was going to implement that reg key and then move FSMOs and demote this DC. That was my plan. But I wanted PSS on the horn. This is too big to screw up.

    If that doesn't work for whatever reason, then we are looking at shutting down this DC, cleaning up AD metadata and seizing FSMO roles.

    That should work. That resets replication with a DC passed the tombstone.

    If you need specific step by step for a metadata cleanup, I hope this helps:

    Complete Step by Step Guideline to Remove an Orphaned Domain controller (including seizing FSMOs, running a metadata cleanup, cleanup DNS (Nameserver tab), AD Sites (old DC references), transfer or fix time settings, WINS settings, etc.
    Published by Ace Fekay, MCT, MVP DS on Oct 5, 2010 at 12:14 AM
    http://msmvps.com/blogs/acefekay/archive/2010/10/05/complete-step-by-step-to-remove-an-orphaned-domain-controller.aspx


    Ace Fekay
    MVP, MCT, MCITP/EA, MCTS Windows 2008/R2 & Exchange 2007, Exchange 2010 EA, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Technical Blogs & Videos: http://www.delawarecountycomputerconsulting.com/

    This post is provided AS-IS with no warranties or guarantees and confers no rights.

    FaceBook Twitter LinkedIn

    Tuesday, November 20, 2012 5:12 AM
    • Marked as answer by Tommy T11 Tuesday, November 20, 2012 5:45 AM
    Tuesday, November 20, 2012 5:45 AM
  • Thanks. I fixed it on my own. PSS did not call back.
    Tuesday, November 20, 2012 7:13 AM
  • Hi ITPeonLevel1,

    Could you post the email response you received??

    I think a lot of people will be looking for some "proof" of the root cause of this issue today.

     

    Thanks

    Tuesday, November 20, 2012 3:02 PM
  • Hello,

    This has caused major issues in 102 servers that we maintain. Is there a possibility you could e-mail me the letter you had received or may you provide me with the contact info of the people you called/emailed to receive this information. This is very important and i would really appreciate it.

    Thank you.

    Tuesday, November 20, 2012 4:45 PM
  • Subject: FW: Question from NOP: NTP 2000 problem

     

    On Monday, November 19, USNO (United States Navy Observatory) made what was expected to be a routine upgrade, but unfortunately for about 50 minutes between 21:07 UTC and 21:59 UTC (16:07-16:59 EST) the server gave out the year as 2000 instead of 2012. We have fixed the failure mode, and this it should never happen again.

     

    We deeply regret the disruptions this has caused. We also recommend users configure their software to get time from three different sources at a minimum. Most software should be able to use the redundant data to identify and ignore bad time sources. You can find a list of other time sources on this web page.

     

    ________________________________

    From:

    Sent: Tuesday, November 20, 2012 9:37 AM

    Subject: Question from NOP: NTP

     

    FROM: Nick Squires

     

    My Active Directory Server reset to the year 2000 yesterday causing complete company network outage for over 30 minutes until we discovered the issue. We were synchronizing with your NTP. What happened? Has the problem been resolved? Please let me know so I can inform my CEO why this issue occurred. Thanks. -Nick

    Tuesday, November 20, 2012 7:10 PM
  • Another resource that might help:

    "Windows Server 2008 and Windows Server 2008 R2 domain controllers added time-rollback protection to help prevent domain controllers from adopting bad time. We recommend that you add time-rollback protection on Windows Server 2003 domain controllers and Windows Server 2008 and Windows Server 2008 R2 Hyper-V hosts by using Group Policy, making sure that you have the policy detail fixes in place before you do. For more information, see article 884776 in the Microsoft Knowledge Base (http://go.microsoft.com/fwlink/?LinkId=178255)."

    that is quoted from http://technet.microsoft.com/en-us/library/upgrade-domain-controllers-to-windows-server-2008-r2(WS.10).aspx

    Justin [MSFT] 

    Tuesday, November 20, 2012 8:35 PM
  • Hi ITPeonLevel1, We were affected by the same issue. I have been waiting for MS to call back now for two hours, myself. In the mean time I went through the steps in the Technet Blog post that Tommy T11 refers to above,  which seemed promising -  setting reg entry: Allow Replication With Divergent and Corrupt Partner to 1 and forcing replication. But we are still having issues. None of the DC's will replicate with the PDC emulater. If I don't hear back from MS soon I'll be forced to resort to demoting, cleaning up metadata, etc.. Can you please let me know what worked for you? I'd sincerely appreciate it. Thanks very much.
    Wednesday, November 21, 2012 5:32 AM
  • Hello,

    We had exactly the same issue and more than 40 Dc were impacted ...

    After 1 week of work, everything is back to normal ...

    I want to prevent this kind of problem for the futur.

    I read KB 884776 and I think it's work if you have some trouble with a hardware failure but if the NTP server send something wrong I think that does not prevent ....

    Is there something to prevent this (mulitple NTP servers ) ? what is the process if you have a list of NTP servers (example 2/3 have the time but not the third) ?

    Thanks for your help ...

     

    Monday, November 26, 2012 2:30 PM