none
Windows Server 2012 R2 Domain Controller Hang and "Please wait for the System Event Notification Service" Message RRS feed

  • Question

  • The problem we are experiencing is intermittent. At some point overnight (typically, but there are no error messages logged to pin to a specific time) the domain controller DRDC02 began to refuse to launch new programs. All DCs are seeing this issue. All DCs are VMs. DRDC02 is just the most recent case.

    If we connect to the VM desktop using vSphere remote console or using MSTSC.EXE over remote desktop protocol, we can sign in to the desktop of the hung DRDC02:
     1. I can launch notepad, calculator, file explorer and other simple programs.
     2. I cannot launch ServerManager.exe, eventvwr.msc or other MMC management tools.
     3. Ctrl-Alt-Del (Ctrl-Alt-End) works, but cannot launch Task Manager from the resulting splash screen.
     4. Programs>Run CMD opens a command window, but cannot perform commands in that command window, or if I can (NSLOOKUP, for instance) the command never ends and returns to the C:\> command prompt. I am then unable to close the command window.
     5. I can perform remote WMI operations (example, connect to DRDC02 to run eventvwr.msc remotely), or perform a remote restart ("SHUTDOWN -R -M \\DRDC02").
     6. I can perform remote SMB operations (example, connect to \\DRDC02\SYSVOL share).
     7. If I manually run an incremental backup of the DRDC02 client using our backup product
    a. Active Directory agent will run a backup and complete normally
    b. Windows File System agent will hang
    c. Although the server desktop was somewhat responsive up to this point, when the backup job stalled out, at this point the desktop became completely unresponsive as well.
     8. When I restart DRDC02 the system hangs during progress to shutdown with message “Please wait for the System Event Notification Service” this takes about 15 minutes to self-resolve and the server restarts. 
    a. Why is SENS not ending? 
      b. Is it possible SENS is the cause of the problems?
    9. Searching using Google for these symptoms, the only recommendation I find is to run SFC /SCANNOW
      a. SFC /SCANNOW results are
      b. Verification 100% complete. Windows Resource Protection did not find any integrity violations.
     

    Question: Need a fix to prevent the domain controllers from hanging/unresponsive.


    Friday, July 10, 2020 6:58 PM

All replies

  • Hello Geo,

    Does this only happen at night? If you restart the server and everything works after that, the problem will not reoccur intermittently during the day?

    Are there any scheduled backup jobs, system updates, etc. that occur during the night that may cause this problem?


    Miguel Fra
    Falcon IT Services
    https://www.falconitservices.com

     

    Friday, July 10, 2020 7:05 PM
  • What programs are you asking about? Generally is recommended to have domain controllers on dedicated windows instance and programs on a separate application server instance. As to AD issues I'd check health is 100% (dcdiag / repadmin tools) and system event log is free of errors. (please do not post log files in forums)

     

     



    Regards, Dave Patrick ....
    Microsoft Certified Professional
    Microsoft MVP [Windows Server] Datacenter Management

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

    Friday, July 10, 2020 7:06 PM
  • Hi,

     

    Just checking in to see if the information provided was helpful. Please let us know if you would like further assistance.

     

    Best Regards,

    Vicky


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, July 13, 2020 2:50 AM
  • It does not always happen at night, it does not appear to be related to scheduled jobs, updates, backup jobs, etc. We have been unable to correlate to a change in our environment. Backups, etc., have been running for a long time prior to this symptom with no issues. 
    Monday, July 13, 2020 2:41 PM
  • These domain controllers only run AD, DNS (and in a couple of cases DHCP). By "programs" I merely was describing the symptoms noted while connected as a console user or RDP user. I mentioned the programs in the problem original description above.

    Likewise, while troubleshooting we ran DCDIAG to confirm the problem hasn't introduced problems into AD. The DCDIAG runs clean.  As I mentioned in my original description, I have searched for other clues, but there are no messages in any of the event logs.

    Monday, July 13, 2020 2:45 PM
  • No, not really helpful so far, but I appreciate any thoughts and suggestions. 
    Monday, July 13, 2020 2:46 PM
  • Barring any new clues the simplest / safest / cleanest method may be to stand up a new fully patched  one for a test.

     

     



    Regards, Dave Patrick ....
    Microsoft Certified Professional
    Microsoft MVP [Windows Server] Datacenter Management

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

    Monday, July 13, 2020 2:48 PM
  • We are contemplating installing all new Win2019 DC servers. We already have 1 Win2019 which is asymptomatic, so just a painful process to carefully perform the migrations.
    Monday, July 13, 2020 8:46 PM
  • Not sure why its painful, it should be a very fast and rather simple process to execute.

     

     



    Regards, Dave Patrick ....
    Microsoft Certified Professional
    Microsoft MVP [Windows Server] Datacenter Management

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

    Monday, July 13, 2020 8:48 PM
  • Not sure why its painful...

    I've seen this before... A co-worker has excruciating pain while typing "DCPROMO". Turned out he had a broken finger.

    Miguel Fra
    Falcon IT Services
    https://www.falconitservices.com

     

    Monday, July 13, 2020 10:02 PM
  • Hi,

     

    Just checking in to see if the information provided was helpful. Please let us know if you would like further assistance.

     

    Best Regards,

    Vicky


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, July 15, 2020 9:10 AM
  • Hi,

     

    Just want to confirm the current situations.

     

    Please feel free to let us know if you need further assistance.

     

    Best Regards,

    Vicky


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, July 17, 2020 8:23 AM
  • Vicky: Still working on this issue. Some research has uncovered a possible conflict between the VMware filter driver vnetflt.sys (vShield) and our antivirus product, which was updated about the time this symptom began. I am testing a potential solution, disabling vnetflt.sys. I'll update this thread in a week or so if the solution turns out to be successful. If the filter driver turns out to be a red herring, I will accelerate replacement of the Win2012r2 domain controllers with Win2019 server editions, carefully migrating the application dependencies. 

    George

    P.S. As to the snarky comments from consultants who obviously don't work in the real world of competing priorities, limited staff resources, budgets, change management controls, application integration, etc., your comments are not really appreciated. If you have a solution, please post. If you think you are funny, there are other websites you can post your humor to. Thanks.

    Saturday, July 18, 2020 7:59 PM
  • Still working on this issue in my spare time.

    I believe the correlation exists with the introduction of Cylance Optics and the beginning of the symptom. I have not opened a case with Cylance technical support yet because I am trying to narrow the problem symptoms and specific cause/effect.

    It appears that the cause is a System State Backup (which can be triggered either using Microsoft Backup or using CommVault backup). Of course, it is recommended that Domain Controllers get a System State Backup to collect all of the Active Directory components.

    The above old articles recommend updating VMware Tools and performing a Custom install type. The previous VMware Tools install was a Complete install type. With Custom, I did not install AppDefense and NSX filter drivers.

    I am now letting the dust settle to see if we get better stability.

    Monday, August 3, 2020 7:00 PM