none
Can't Logon

    Question

  • Last night all of the computers in the domain underwent a Windows Update and most of them rebooted. Now no one can logon using their domain ID - "The user name or password is incorrect". On every domain controller I can't even logon locally as Administrator - same message. I've disconnected the PDC server from the network by removing the ethernet cable, booted from the installation DVD, Repaired the computer by restoring a System Image stored on a local drive twice, selecting different images. I still can't logon.

    I'm totally confused. If I restore an image from the computer which was made a time when everything was working, how is it possible that no one can logon to this restored computer?

    And I'm desperate - can anyone help me?

    Tom Mason


    Tomás Mason
    Wednesday, July 13, 2011 11:57 PM

Answers

  • Ace,

    I blundered into the solution of this problem.

    I was trying to use Server Manager, from a working server, to connect to the locked physical DCs. It could not connect, and the error message mentioned, as one of the possible causes, kerberos. Ah Ha! I knew I can use Computer Management to connect to these DCs, so I fired it up, selected Connect to another computer, selected Services, and saw that the Kerberos Key service was running. On a whim, I restarted it, and, Voila!, I can now logon to theis DC. Repeat with other DC.

    I rebooted the DC and I can still logon.

    Now the question is: how did Kerberos get stuck and remain stuck through multiple reboots?

    Thanks again,

    Tom


    Tomás Mason
    • Marked as answer by Tomás Mason Thursday, July 14, 2011 10:46 PM
    Thursday, July 14, 2011 10:46 PM

All replies

  • Just an FYI, the use of "images" with domain controllers is not supported and should never be used. Numerous issues will result to the point where services and functionality will fail. You should only use proper backups of the System State data and the whole system & boot drives.

    How many DCs do you have? It's not possible to logon locally to a DC, unless you are referring to the DSRM (directory services restore mode)?

    What DNS addresses are set on the DCs and the client machines? The internal DNS server(s) only, correct? Or are there any ISP DNS addresses or the router being used as a DNS address?

    How old was the image you restored it from? If you restored it to an older image say possibly at least a week old, then the computers may lose connectivity due to the computer account's different secure channel passwords.

    Were you able to logon locally on any of the client machines (not the DCs) before the image restore? If so, were you able to see any event log errors on the local machines?

    Ace


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Thursday, July 14, 2011 3:07 AM
  • Ace,

    Thanks for the reply.

    * I believe I am restoring from the system state backup.

    * We have three DCs, but only two were online last night. The VM DC was in the process of being moved from one Hyper-V host to another. It is currently in limbo be cause of the inability to logon to anything.

    * Our computers are only aware of internal DNS servers.

    * I restored the System State/Image from two days ago and five days ago. Neither allows a logon.

    * I cannot logon to any machine with my domain ID. I have found one client work station which allows me to logon locally as Administrator, but only one. There are no errors in the Security Log and I don't see anything relevent in the System Log.

    Tom


    Tomás Mason
    Thursday, July 14, 2011 4:28 AM
  • Ace is on spot, DC's are not used to be configured from either cloning/snapshots/images, else you will see issues like USN roll back, name resolution, replication failures etc. You should have used healthy system state backup to restore the DC. By just using the image & if that images has passed tombstone period, then you have made the problem worse.

    Please provide the info ACE has requested like do you have other DC's in the domain, if yes, you can demote this DC & re-promote it. Is all your DC is also a DNS/GC server, if not making them is a good practice, also configure local DNS in each clients machine as primary & alternate DNS servers.

    Things to consider when you host Active Directory domain controllers in virtual hosting environments

    http://support.microsoft.com/kb/888794%C2%A0

     

    Regards  


    MVP-Directory Services

    Awinish Vishwakarma| CHECK MY BLOG

    Disclaimer: This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Thursday, July 14, 2011 4:35 AM
  • Ace,

    Thanks for the reply.

    * I believe I am restoring from the system state backup.

    * We have three DCs, but only two were online last night. The VM DC was in the process of being moved from one Hyper-V host to another. It is currently in limbo be cause of the inability to logon to anything.

    * Our computers are only aware of internal DNS servers.

     

    Two DCs out of three online? How long was the third one offline?

    • Which one is the PDC Emulator? (It MUST be the physical machine)
    • Did you disable Time sync from the VM host? (this is a must)
    • Are all three DCs GCs? (they should be)
    • Are all three DNS servers? If so, which DNS are the clients pointing to as the first entry? The one offline?

     

     

     

    * I restored the System State/Image from two days ago and five days ago. Neither allows a logon.

     

    On to which DC di you restore it to? You really can't do that. Things change too much on a DC to use snapshots like that, which I mentioned and Awinish concurred, will cause numerous issues.

     

    * I cannot logon to any machine with my domain ID. I have found one client work station which allows me to logon locally as Administrator, but only one.

     

    You should be able to logon to any machine locally with the local Administrator account and password. If only one, then it indicates a possible forgotten password issue?

     

    * There are no errors in the Security Log and I don't see anything relevent in the System Log.

     

    We'll need the errors in the NTFRS, File Replication Service, and other AD related logs.

     

    Tom


    Tomás Mason

     

    If you shutdown the two and only bring up the one you took offline, set the DNS address to itself, and the clients to use it for DNS (assuming it's a DNS server), then try to login. If you can logon to the DC, but not from the clients, well, at least you have a DC up and running.

    If it works, then we'll need to stick to this as the good GC. The other two will be useless at this point, if I understand the whole scenario. If this is the case, then you'll have to make this guy a GC if it's not already, seize all FSMO to it, metadata cleanup on the other two, etc.  You may have to disjoin/rejoin the clients if they don't connect due to the computer secure channel password.

    I know it sounds far fetched, but if that works, well, your options are limited, but at least you have an option to recover your AD infrastructure.

    Ace

     


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Thursday, July 14, 2011 5:09 AM
  • Ok, now I'm confused. The only way I can do the restore is by booting from the installation CD, selecting Repair my System, select System Image Recovery, choose the image, and let it run. Since our backup schedule is set up to back up the System State and the C: drive only, what am I restoring if not a " healthy system state backup "?

    Tom

     


    Tomás Mason
    Thursday, July 14, 2011 5:14 AM
  • I think if you are doing this part right, then there is still more to this when it comes to DCs. Check this out and what can occur:

    Restore Domain Controllers
    http://technet.microsoft.com/en-us/library/cc526503.aspx

     


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Thursday, July 14, 2011 5:26 AM
  • Two DCs out of three online? How long was the third one offline?

    • Which one is the PDC Emulator? (It MUST be the physical machine)
    • Did you disable Time sync from the VM host? (this is a must)
    • Are all three DCs GCs? (they should be)
    • Are all three DNS servers? If so, which DNS are the clients pointing to as the first entry? The one offline?

    * Three DCs, A, B, & C - A and B are physical, C is virtual. A has the PDC role.

    * No - didn't know I should.

    * Yes.

    * Yes. No.

    You should be able to logon to any machine locally with the local Administrator account and password. If only one, then it indicates a possible forgotten password issue?

    * I cannot logon to A, B, or C with my domain ID or with the local Administrator account and password. I have not forgotten the password.

    We'll need the errors in the NTFRS, File Replication Service, and other AD related logs.

    * Hard to do since I can't logon.

    If you shutdown the two and only bring up the one you took offline, set the DNS address to itself, and the clients to use it for DNS (assuming it's a DNS server), then try to login. If you can logon to the DC, but not from the clients, well, at least you have a DC up and running.

    If it works, then we'll need to stick to this as the good GC. The other two will be useless at this point, if I understand the whole scenario. If this is the case, then you'll have to make this guy a GC if it's not already, seize all FSMO to it, metadata cleanup on the other two, etc. You may have to disjoin/rejoin the clients if they don't connect due to the computer secure channel password.

    I know it sounds far fetched, but if that works, well, your options are limited, but at least you have an option to recover your AD infrastructure.

    * See above answer.

    Thanks for your help.

    I've since discovered that I can logon to our old Hyper-V Server 2008 R2 which is the computer that used to host DC C. I'm in the process of copying C to the new Hyper-V host now. I'll let you know what happens when I start it up.

    Tom


    Tomás Mason
    Thursday, July 14, 2011 7:47 AM
  • This quote from the article expresses my operative condition -

    "The only time you should use domain controller backup images is when the failure has resulted in loss of all the domain controllers in the infrastructure."

    which I have.

    Tom


    Tomás Mason
    Thursday, July 14, 2011 7:50 AM
  • When a server becomes a DC, you don't see local accounts any more, its domain account. You should be using DSRM mode password, if you forgot it, reset it following below article.

    http://technet.microsoft.com/en-us/library/cc754363%28WS.10%29.aspx

    http://www.petri.co.il/change_recovery_console_password.htm

    You can kick out problem DC & promote it back.

     

    Regards


    Awinish Vishwakarma 

    MVP-Directory Services

    MY BLOG: http://awinish.wordpress.com

    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Thursday, July 14, 2011 10:52 AM
  • I'm happy to hear you are able to logon to the "C" DC. At least we have a starting point to recover the infrastructure. I'm not sure what occured that caused the whole thing. The questions I've asked were based on trying to ferret out additional info to get some sort of starting point to help recover from this issue. When I first read the original post, it was a little confusing and didn't provide specifics (config info, # of DCs, etc) that we could use to diagnose or come up with a plan to fix it.

    Are you able to logon from a workstation with this DC up? As long as it's a GC, and DNS is running on it, the DC is pointing to itself for DNS, and the workstations are pointing only to it for DNS, and you can logon, then we have a solid basis to rebuild.

    Therefore from what I suggest now as I mentioned previously, as well as Awinish, that we can keep A & B shutdown, seize all the FSMO roles to C, then rebuild A & B from scratch, and re-promote them into the domain.

    For future VM considerations, Awinish posted a good link on the guidelines. I would like to summarize:

    1. Make one DC physical. Make that one the PDC Emulator, Domain Name Master and the Schema Master.
    2. Disable Time Sync on the VM host.
    3. You can add as many virtual replica DCs as the VM host can support.

    Let us know if you can logon with a client workstations.

    I think we're getting closer to resolving this! :-)

    Ace

     

     

     

    As a reference, the following are my notes on viritualizing DCs. It seems you have most of it covered. I'm just posting it more so to benefit others that may read this thread that may have a similar issue. I added the link Awinish provided into my own notes, but not below (since it's already posted):

     

    ==================================================================
    Domain Controllers HyperV and virtualization, and the Time Service

    Regarding DC virtualization, please adhere to the following best practices:

        1) Do not use imaging software to take an image of the DC.
        2) Do not take or apply snapshots of the DC.
        3) Do not shut the Virtual Machine down and simply copy the virtual disk as a backup.
        4) If you have the ability to “discard changes” as you do if you are running “Virtual Server 2005 R2”, do not enable this type of setting on a DC Virtual Machine.
        5) Use NTBACKUP.EXE, WBADMIN.EXE, or any third party software that is available as long as it is certified to be AD-compatible to take system state backups.
        6) Only restore a system state to the DC or restore a full backup.
        7) Make at least one DC, the PDC Emulator, a physical DC. The PDC is the default time service in the hierarchy and should not be virtualized.

    For more information, please refer to:

    DC’s and VM’s – Avoiding the Do-Over
    http://blogs.technet.com/b/askds/archive/2009/06/05/dc-s-and-vm-s-avoiding-the-do-over.aspx 

    In addition, basically, running Domain Controllers in virtual machines requires special considerations (Time synch configuration included). I recommend reading the articles below. You will also want one Physical DC in the environment, but you can have the remaining DCs virtualized. It's recommended to have the PDC as the physical DC.

    Running Domain Controllers in Hyper-V
    http://technet.microsoft.com/en-us/library/virtual_active_directory_domain_controller_virtualization_hyperv(WS.10).aspx

    Deployment Considerations for Virtualized Domain Controllers
    http://technet.microsoft.com/en-us/library/dd348449(WS.10).aspx

     

    Time service and DCs

    For virtual machines that are configured as domain controllers, disable time synchronization with the host through Integration Services. Instead, accept the default Windows Time service (W32time) domain hierarchy time synchronization.

    Host time synchronization makes it possible for guest operating systems to synchronize their system clocks with the system clock of the host operating system. Because domain controllers have their own time synchronization mechanism, host time synchronization must be disabled on virtual machines that are configured as domain controllers. If domain controllers synchronize time from their own source and also synchronize time from the host, the domain controller time can change frequently. Because many domain controller tasks are tied to the system time, a jump in the system time could cause lingering objects to be left in the directory and replication to be stopped.

    W32Time, Windows Time, should run as LocalService in 2K8 R2 Domain Controllers. You can see the account used in Services.msc -> Windows Time -> Properties.

    You can disable host time synchronization in the virtual machine settings in the Integration Services section of the Hyper-V Manager by clearing the Time Synchronization check box.

    Configuring the Windows Time Service for Windows 2000, 2003, 2008 and newer, explanation of the time service hierarchy, and more
    Published by Ace Fekay, MCT, MVP DS on Sep 18, 2009 at 8:14 PM  3050  1 
    http://msmvps.com/blogs/acefekay/archive/2009/09/18/configuring-the-windows-time-service-for-windows-server.aspx
    ==================================================================

     

     


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Thursday, July 14, 2011 1:00 PM
  • Ace,

    Thank you very much for your very informative reply.

    Here's where I am now:

    I can logon with my domain ID to the virtual DC. On it, I have seized all FSMO roles (I know this is not desirable and will redistribute them when/if I get the physical DCs back to normal).

    I can logon to all non-DC servers. I can logon to most workstations with my domain ID. However, I can't logon to my personal workstation with my domain ID - "no logon servers are availble to service this request". I have to logon locally. From it I can ping by name all three DCs.

    I have disabled host time synchronization on the virtual DC.

    As far as I can tell, the physical DCs are working, providing all the services set up on them - DHCP, DNS, DFS, et cetera.

    So the problem now is narrowed down to one item - I can't logon to the physical DCs - User name or password is invalid - AND I have not forgotton the password!

    Can I remotely forceably demote these physical DCs?

    Tom


    Tomás Mason
    Thursday, July 14, 2011 9:26 PM
  • Hi Tom,

    If you've already seized the roles, the other DCs are useless. Remotely force demote? Nah, just reinstall them. It wouldn't matter to logon or not at this point. You're best best is to simply reinstall them from scratch from the ground up.

    Check the DNS address on your personal workstation making sure it points to the current DC that's running. You may have to logon locally. If you forget the local password, there are other methods to get the local password.

     

    For the old DCs, and just to give you a guideline with what to do next (you've already did some of the steps), here's my blog explaining it.

    Complete Step by Step Guideline to Remove an Orphaned Domain controller
    Published by Ace Fekay, MCT, MVP DS on Oct 5, 2010 at 12:14 AM
    http://msmvps.com/blogs/acefekay/archive/2010/10/05/complete-step-by-step-to-remove-an-orphaned-domain-controller.aspx

    Ace

     


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Thursday, July 14, 2011 9:59 PM
  • Ace,

    I blundered into the solution of this problem.

    I was trying to use Server Manager, from a working server, to connect to the locked physical DCs. It could not connect, and the error message mentioned, as one of the possible causes, kerberos. Ah Ha! I knew I can use Computer Management to connect to these DCs, so I fired it up, selected Connect to another computer, selected Services, and saw that the Kerberos Key service was running. On a whim, I restarted it, and, Voila!, I can now logon to theis DC. Repeat with other DC.

    I rebooted the DC and I can still logon.

    Now the question is: how did Kerberos get stuck and remain stuck through multiple reboots?

    Thanks again,

    Tom


    Tomás Mason
    • Marked as answer by Tomás Mason Thursday, July 14, 2011 10:46 PM
    Thursday, July 14, 2011 10:46 PM
  • Probably due to the Time service and using the VM host. Kerberos is time sensitive. If the clocks are off more than 5 min, Kerberos auth fails, and you can't logon, and the DC will not communicate.

    Now you've seized a role or two. That can have an affect. Which roles did you seize? This will have an impact if you were to keep a DC that had a role seized from it. It depends on which role.

    Ace


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Friday, July 15, 2011 12:45 AM
  • Ace,

     

    Since the virtual DC was the only one working, I seized all of the FSMO roles. I'm in the process of moving the roles back to the physical DCs now. Do you have any advice on which roles to seperate, i.e., put them on different servers?

    tom


    Tomás Mason
    Friday, July 15, 2011 3:28 AM
  • Tom, it depends on which Role was on the original that was seized. Here's a pic that explains which seized roles would prevent the abiilty to bring a specific DC back online, and which ones can't. More specifics below the image:

     

    Active Directory - FSMO - Seize Role - MOC6425C p12-50

    ======
    Returning a Role to Its Original Holder

    The following was quoted from MOC (Microsoft Official Curriculum) Course# 6425C, DMOC page 12-52:

    Seizing FSMO Roles Ref Slide:
    https://skydrive.live.com/?cid=0c7b9fd0852378b8&sc=photos&uc=1&id=C7B9FD0852378B8%21421#cid=0C7B9FD0852378B8&id=C7B9FD0852378B8%21476&sc=photos

    "If, however, a role has been seized and the former master is able to be brought back online, you must be very careful. The PDC emulator and infrastructure master are the only operations master roles that can be transferred back to the original master after having been seized.

    Note Do not return a seized schema, domain naming, or RID master to service. After seizing the schema, domain naming, or RID roles, you must completely decommission the original domain controller.
     
    If you have seized the schema, domain naming, or RID roles to another domain controller, you must not bring the original domain controller back online without first completely decommissioning the domain controller. That means you must keep the original role holder physically disconnected from the network, and you must remove AD DS by using the dcpromo /forceremoval command. You must also clean the metadata for that domain controller as described at http://go.microsoft.com/fwlink/?LinkId=80481."

    Ref:

    Course 6425C: Configuring and Troubleshooting Windows Server 2008 Active Directory Domain Services
    http://www.microsoft.com/learning/en/us/Course.aspx?ID=6425C&Locale=en-us

    6425C - Configuring and Troubleshooting Windows Server® 2008 Active Directory® Domain Services - Companion Content [PDF]
    Mod 12: Administering AD DS Domain Controllers
    http://download.microsoft.com/download/6/F/2/6F223B30-00CD-4C44-B024-0C7A4AEFAB33/6425C-ENU-Companion.zip

    Responding to operations master failures
    http://technet.microsoft.com/en-us/library/cc737648(WS.10).aspx

    What to do with FSMO roles...
    http://blogs.technet.com/b/bpuhl/archive/2005/12/07/415761.aspx

     

    Ace


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Friday, July 15, 2011 4:49 AM
  • Just to add what I think happened, is during a reboot when the updates were installed, the guests may have taken the time synch from the VM Host, which then set the time to something beyond the 5 minute skew that may have caused this whole mess.

    Ace


    Ace Fekay
    MVP, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007 & Exchange 2010, Exchange 2010 Enterprise Administrator, MCSE & MCSA 2003/2000, MCSA Messaging 2003
    Microsoft Certified Trainer
    Microsoft MVP - Directory Services
    Complete List of Technical Blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

    This posting is provided AS-IS with no warranties or guarantees and confers no rights.

    Friday, July 15, 2011 3:51 PM