none
3rd Party Application Conflicting with Group Policy

    Question

  • OS: Microsoft Server 2012 R2
    Domain Environment: 2012 R2 Functional Level
    Test Server: Virtual Machine running on Hyper-V

    We have been struggling to solve a problem that a 3rd party tool has been causing in many of our domain environments for the last 6 months, and I am hoping there are some GP experts here that can help us to improve our debugging to flush out the root cause. 

    Overview: We are a software company, and we are using another company's application for our reporting module. This 3rd party company's tool (Pentaho) is utilizing PostgreSQL and Tomcat Apache - both are managed via a Windows Service we created. The application works well, but we have seen that when installed on domain joined machines with GPO's applied, there is a conflict with Group Policy client which causes major delays during reboots and problems running gpupdate/rsop.msc while the PostgresQL and Tomcat Apache services are actively running.

    Behavior: If we have the PostgreSQL/Tomcat services running, we find that a reboot will cause a delay of upwards of one hour, and running gpupdate /force will hang indefinitely. If gpupdate /force is run while the PostgreSQL/Tomcat services are running, it puts gpclient into a bad state, requiring a reboot to resolve. Simply disabling the services and rebooting brings the system back into a healthy state and allows group policy to operate normally until we re-enable the PostgreSQL/Tomcat services. 

    We have enabled all available debug logging in Group Policy, PostgreSQL, and Tomcat, performed xBootMgr traces, performed Process Monitor analysis, and Packet Captures, but we have been unable to pinpoint the cause of the conflict with GroupPolicy. We have also opened tickets with all other involved vendors to see if we can solve the problem from their side, but I would like to see if we can get a Group Policy expert to review our gpsvc logs to see if anything is obvious, or see if there is anything else we can enable to get more details in regards to what is causing this.

    I have collected a series of logs and network captures - descriptions and links below:

    1. Normal login with PostgreSQL/Tomcat fully stopped/disabled: Group Policy processes normally with no extended delays - https://www.dropbox.com/s/0yrkcky34pdnljb/normal_gp.txt?dl=0

    2. Normal login with PostgreSQL/Tomcat running: Group Policy completely hangs for multiple minutes at various points with no explanation, and eventually completes - https://www.dropbox.com/s/kzzjewmuj5ga9y5/essence_gp.log?dl=0

    3. 'gpupdate /force' run after login with PostgreSQL/Tomcat services enabled: https://www.dropbox.com/s/n4sobuaabs1f1li/gpupdate_fail.log?dl=0

    4. Packet capture while the gpupdate /force from above was running - https://www.dropbox.com/s/xa1032bcgq9bmib/gpupdate_fail_trace.pcapng?dl=0

    Is there anything obvious in these logs/captures that I am missing? Is there any additional debugging/tracing that we can enable to get further details about what is causing gpsvc to fail while PostgreSQL/Tomcat services are running?

    Please let me know if there is any additional information that I can provide.

    Nick






    • Edited by Nick Childs Saturday, November 07, 2015 1:13 AM
    Saturday, November 07, 2015 12:53 AM

All replies

  • Hi just quick question.

    What if you make the set the PostgreSQL/Tomcat services to delay start or set to manual start with a .bat file after log on ?

    Saturday, November 07, 2015 7:23 PM
  • I appreciate the reply, Muhammad.

    Unfortunately, setting PostgreSQL/Tomcat to Manual or Delayed start is not a solution. Using the 'Delayed Start' feature does not change the behavior at all - the services eventually start and cause Group Policy to go into a bad state. Setting the service to start "Manually" does help a bit during initial boot/reboot IF the services were not already running. However, as soon as the service is running, any Group Policy operation puts Group Policy into a bad state. If a reboot is performed after the services have been "Manually" started (still running), we will experience a 1 hour+ reboot delay unless we force kill Group Policy remotely, disable PostgreSQL/Tomcat, and perform a clean reboot with the services stopped.

    This behavior does not seem to have anything to do with the log in process. Whenever the both the PostgreSQL/Tomcat services are running, we are unable to perform group policy operations as expected.

    Please let me know if you have additional questions.

    Nick

    Tuesday, November 10, 2015 3:50 AM
  • Bump - this is still a problem. Please advise.
    Tuesday, November 17, 2015 5:54 PM
  • I have noticed that while these PostgreSQL/Tomcat services are running, and a gpupdate is initiated, the processing hangs at the following point - once for User Policies and once for Computer Policies: 

    GPSVC(340.5244) 125429376 WaitForGroupPolicySessionThreadsToTerminate() checked.
    GPSVC(340.5244) 125429376 CGPApplicationServiceRefreshEvent Wait to Make sure the service is completely initialized.
    GPSVC(340.5244) 125429377 CGroupPolicySessionRefreshGroupPolicyForPrincipal Beginning WaitForSingleObject.
    GPSVC(340.9e0) 130325741 ProcessGroupPolicyCompletedExInternal Entering. Extension = {B087BE9D-ED37-454F-AF9C-04291E351182}, dwStatus = 0x0

    How can I determine what thread it is waiting for or why this is happening? 

    Tuesday, November 24, 2015 9:17 PM
  • The following gplog shows a successful processing (postgresql and tomcat disabled) - completes in about 11 seconds: https://www.dropbox.com/s/anwxutdoxepkjun/gpsvc_clean.log?dl=0

    The following gplog shows a failed processing (postgresql and tomcat services running and working) - does not complete; takes 20 minutes: https://www.dropbox.com/s/wpvhm1n4y4ibxwj/gpsvc.log?dl=0

    How can we determine what gpupdate is waiting for and why?

    Tuesday, November 24, 2015 10:08 PM
  • Have you enabled Debug logging for netlogon?  Sorry, but I cannot get to dropbox from where I am to see if this is included.

    You can enable debug to follow some GPO processing.  The below link also gives other log files to be looked at.

    https://technet.microsoft.com/en-us/library/cc775423(v=ws.10).aspx

    Another tool is Windows Performance Toolkit.

    This link is a good write up to see GPO processing using WPT.

    https://4sysops.com/archives/troubleshoot-slow-group-policy-processing/

    Wednesday, November 25, 2015 5:23 PM
  • Thank you for the reply. I have not enabled netlogon debugging up to this point, only the client-side group policy debugging, which is attached via dropbox. I can try to enable additional logging, and re-test. Is there a particular log that would show why we are getting stuck on WaitForSingleObject, or do I need to enable all of them?  

    I have used Windows Performance Toolkit to collect many xbootmgr traces, but similar to the gpsvc logs, it just indicates that the gp processing is hanging, with no indication of why.

    Wednesday, November 25, 2015 7:03 PM