none
Sysmon 10.X gets W3Clogging with advanced loggning to fail on start if sysmon is running. RRS feed

  • Question

  • After we have upgraded to sysmon 10.X on servers running Windows 2012, 2016 and the advanced loggning enabled in the IIS. The W3CLogging service is not able to start if sysmon is started before the W3CLogging service. If sysmon is set to "Delayed start" in the service, it all works. If the event ID 22 (DNS) in sysmon is disabled:

    <!--SYSMON EVENT ID 22 : DNSQuery [DNSQuery]-->
            <!--EVENT 22: "DNSQuery"-->

            <!--DATA: RuleName, UtcTime, ProcessGuid, ProcessId, QueryName, QueryStatus, QueryResults, Image-->
            <RuleGroup name="" groupRelation="or">
                <DnsQuery onmatch="include">
                </DnsQuery>
            </RuleGroup>

    Then it also works.The issues is seen with sysmon version 10.0 and 10.41, I havnt tried the versions in between.


    • Edited by Lars_G Wednesday, November 27, 2019 10:12 AM
    Wednesday, November 27, 2019 9:40 AM

Answers

  • Thanks Lars_G for reporting this and to Mario for all his efforts.

    Looks like w3logsvc and Sysmon DNS reporting originally used the same sample code as they were both sharing the same session GUID which caused the other to fail! Although starting w3logsvc first appears to resolve the issue you will observe that the Sysmon DNS monitor fails silently. Thus while Sysmon continues to run, it will not log any DNS activity.

    I have updated the session GUID and confirmed that this resolves the issue and have also raised this with the team responsible for w3logsvc so that they too can address this in their code. We were due to publish a Sysmon update soon so once Mark R. has approved the change I will look at getting this updated.

    MarkC(MSFT)

    • Marked as answer by Lars_G Friday, December 6, 2019 9:31 AM
    Thursday, December 5, 2019 11:07 AM

All replies

  • Interesting..

    C:\Windows\system32>net start w3logsvc
    Errore di sistema 1068.

    Avvio del gruppo o del servizio di dipendenza non riuscito.

      ERROR_SERVICE_DEPENDENCY_FAIL                                  winerror.h
    # The dependency service or group failed to start.

    Dependency are set to: netprofm,netman,dcomlaunch and also to HTTP

    Procmon shows that it found 3 dependency, but not HTTP. WHen services try to look for HTTP, it fails and then net1. load helpmsg to load the error message.

    The reference to HTTP which is a driver and not a service seems strange..

    But it doesn't even try to start the dependencies.. seems the the missing key for HTTP is the cause of the problem..

    Uninstalled sysmon and had the same problem.. tried to add the missing HTTP key, same problem again..

    something is missing..

    Thanks
    -mario

    Wednesday, November 27, 2019 12:17 PM
  • Just installed a clean machine windows server 2016. Installed IIS and tried to sdtat the service W3LOGSVC.

    Everyhting worked well..

    Installed sysmon and started it. default config.

    Everything fine.. above you can see sysmon colecting info on Net1.exe which is starting the service..

    and here you can see the Service is starting and sysmon is not interfering in any way.

    As you can see, the service is loaded inside an already started svchost.exe

    so, the only thing that comes to my mind is that for any reason on your server the svchost.exe that should host the W3logsvc is not yet started, and while starting it sysmon may at a certain point interfere with the loading of the exe.. but you should get a crash, a wer report, or a dump or something like that..

    Can you take a Procmon trace while loading the service and share it? If you already get a wer report, you can get a dump also with wer following these instructions:

     

    https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

    HTH
    -mario

    Friday, November 29, 2019 2:04 PM
  • Hi Mario

    Thanks for your post. If you use the default sysmon configuration, it only takes proces start and termination (event ID 1 and 5). You need to load a configuration file with dns query enabled (event ID 22).

    Could you confirm this for me?I did the same to rule out the sysmon standard configuration and if it was anything in my configuration that did it, but that I have ruled out due to i works if DNS query (event ID 22) is set to not log.



    • Edited by Lars_G Monday, December 2, 2019 10:03 PM
    Monday, December 2, 2019 8:18 PM
  • Hi Lars,

    I loaded an xml version which is using also the DNS config.

    And with this version, I was able to reproduce your problem, probably. Please confirm that you see the same things I noticed.

    First of all the returned error when I start the service using Net Start:

    In the event Log I can found more information:

    So the error is 800700B7, which means

    I was under the impression that the file should be one of these:

    But I now suspect that the problem may be more down in the kernel..

    Both sysmon and W3logsvc will use an etl trace file in kernel to record data.. may be there is a conflict with the name in the kernel.. So I played a little with Process Explorer..

    When you start sysmon, these two trace file are created.

    EtwRTSYSMON TRACE.etl and EtwRTMy Event Trace Session.etl.

    If you stop sysmon and start w3logsvc, it starts correctly and create this log file:

    

    EtwRTHTTPSYS-IISEvent Trace Session.etl.

    Now, having a look at those file names, 
    EtwRTMy Event Trace Session.etl
    EtwRTHTTPSYS-IISEvent Trace Session.etl

    They share Event Trace Session.etl..

    I don't know how these two can be treated as equal.. but the error returned its clear..

    If I disable DNS logging, using another xml config, I get both the w3logsvc and sysmon started together, because the DNS trace file is not loaded..

    SO, probably Mark has to look into this because the name of tee etl file conflict with that of w3logsvc.. May be ha can call it "DNS logger" ro anything else that doesn't end in "Event Trace Session"...

    Will send this to SYSSITE@mcirosoft.com and see what Mark says..

    Thanks
    -mario

    Tuesday, December 3, 2019 10:37 PM
  • Hi Mario

    I see the same event 6001, so that match. Thanks, you have done a very good job!! It should be fairly easy to fix in the code, I hope :)

    One odd thing is that if you start sysmon delayed, so it startes after W3CLogging then it works..

    Thanks

    Lars


    • Edited by Lars_G Wednesday, December 4, 2019 8:32 AM
    Wednesday, December 4, 2019 8:28 AM
  • It's still weird, because the filename are not exactly equal..

    They satrt te same way "EtwRT" and finish almost the same "Event Trace Session.etl"

    but they are not exactly equal, so may be the reason is still to be found.. don't know if down in the kernel the rule are strictier and the space do not count and it's enough to be "similar" to cause an erro of duplicate name..

    But I cannot think to anything else right now..

    We will see. MarkC will have a look at it and will let us know.

    Thanks!

    -mario

    Wednesday, December 4, 2019 11:28 AM
  • Hi Mario

    just to let you know that I have been able to reproduce the issue and confirmed that modifying the name does not resolve it. Will let you know once I have identified the cause

    Regards

    MarkC(MSFT)

    Wednesday, December 4, 2019 6:17 PM
  • Sounds good!
    Wednesday, December 4, 2019 8:44 PM
  • Thanks Lars_G for reporting this and to Mario for all his efforts.

    Looks like w3logsvc and Sysmon DNS reporting originally used the same sample code as they were both sharing the same session GUID which caused the other to fail! Although starting w3logsvc first appears to resolve the issue you will observe that the Sysmon DNS monitor fails silently. Thus while Sysmon continues to run, it will not log any DNS activity.

    I have updated the session GUID and confirmed that this resolves the issue and have also raised this with the team responsible for w3logsvc so that they too can address this in their code. We were due to publish a Sysmon update soon so once Mark R. has approved the change I will look at getting this updated.

    MarkC(MSFT)

    • Marked as answer by Lars_G Friday, December 6, 2019 9:31 AM
    Thursday, December 5, 2019 11:07 AM
  • Hi Mark

    Thanks for the update and for fixing and addressing the issue. We are just glad that we can continue to use sysmon :)

    Thursday, December 5, 2019 11:15 AM
  • Perfect!

    Great job as always!

    Ciao

    -mario

    Thursday, December 5, 2019 11:15 AM
  • I just installed the new release 10.42 and it looks like a winner :)

    Thanks for your good work!

    Wednesday, December 11, 2019 8:35 PM