none
SysmonDrv.sys causing BSOD RRS feed

  • Question

  • Hello,

    I'm in the process of testing an update from sysmon V10.0.4.1 to V12.0.2.0. Running sysmon.exe -i sysmonconfig.xml is causing a BSOD with Stop Code REFERENCE BY POINTER. I'm using Olaf Hartong's sysmon-modular https://github.com/olafhartong/sysmon-modular with very few changes (a few company specific exclusions). I can provide the full sysmon configuration if required.


    If the host blue screens before sysmon has managed to install it will recover by itself. If not, I have to open Command Prompt in recovery and delete SysmonDrv.sys.

    Windbg preview gives me the following output
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************
    
    PAGE_FAULT_IN_NONPAGED_AREA (50)
    Invalid system memory was referenced.  This cannot be protected by try-except.
    Typically the address is just plain bad or it is pointing at freed memory.
    Arguments:
    Arg1: ffff80050ca8515a, memory referenced.
    Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
    Arg3: fffff80357c52767, If non-zero, the instruction address which referenced the bad memory
        address.
    Arg4: 0000000000000002, (reserved)
    
    Debugging Details:
    ------------------
    
    
    KEY_VALUES_STRING: 1
    
        Key  : Analysis.CPU.mSec
        Value: 2890
    
        Key  : Analysis.DebugAnalysisProvider.CPP
        Value: Create: 8007007e on <hostname>
    
        Key  : Analysis.DebugData
        Value: CreateObject
    
        Key  : Analysis.DebugModel
        Value: CreateObject
    
        Key  : Analysis.Elapsed.mSec
        Value: 5296
    
        Key  : Analysis.Memory.CommitPeak.Mb
        Value: 84
    
        Key  : Analysis.System
        Value: CreateObject
    
        Key  : WER.OS.Branch
        Value: 19h1_release
    
        Key  : WER.OS.Timestamp
        Value: 2019-03-18T12:02:00Z
    
        Key  : WER.OS.Version
        Value: 10.0.18362.1
    
    
    ADDITIONAL_XML: 1
    
    OS_BUILD_LAYERS: 1
    
    VIRTUAL_MACHINE:  VMware
    
    BUGCHECK_CODE:  50
    
    BUGCHECK_P1: ffff80050ca8515a
    
    BUGCHECK_P2: 0
    
    BUGCHECK_P3: fffff80357c52767
    
    BUGCHECK_P4: 2
    
    READ_ADDRESS:  ffff80050ca8515a Paged pool
    
    MM_INTERNAL_CODE:  2
    
    BLACKBOXBSD: 1 (!blackboxbsd)
    
    
    BLACKBOXNTFS: 1 (!blackboxntfs)
    
    
    BLACKBOXPNP: 1 (!blackboxpnp)
    
    
    BLACKBOXWINLOGON: 1
    
    PROCESS_NAME:  Sysmon.exe
    
    TRAP_FRAME:  fffff60137d3ef80 -- (.trap 0xfffff60137d3ef80)
    NOTE: The trap frame does not contain all registers.
    Some register values may be zeroed or incorrect.
    rax=ffff80050ca85152 rbx=0000000000000000 rcx=0000000000000006
    rdx=ffff8003154d4380 rsi=0000000000000000 rdi=0000000000000000
    rip=fffff80357c52767 rsp=fffff60137d3f110 rbp=ffff80030f5ef770
     r8=0000000000000006  r9=fffff60137d3f110 r10=ffffb90094399000
    r11=fffff60137d3f108 r12=0000000000000000 r13=0000000000000000
    r14=0000000000000000 r15=0000000000000000
    iopl=0         nv up ei ng nz na pe nc
    nt!CmpNotifyChangeKey+0xe7:
    fffff803`57c52767 8b4808          mov     ecx,dword ptr [rax+8] ds:ffff8005`0ca8515a=????????
    Resetting default scope
    
    STACK_TEXT:  
    fffff601`37d3ecd8 fffff803`5780819e     : 00000000`00000050 ffff8005`0ca8515a 00000000`00000000 fffff601`37d3ef80 : nt!KeBugCheckEx
    fffff601`37d3ece0 fffff803`5769559f     : ffffffff`146fffff 00000000`00000000 00000000`00000000 ffff8005`0ca8515a : nt!MiSystemFault+0x19dcee
    fffff601`37d3ede0 fffff803`577d0d5e     : 00000000`00000000 fffff803`00000080 00000000`00000001 00000000`00000000 : nt!MmAccessFault+0x34f
    fffff601`37d3ef80 fffff803`57c52767     : 00000000`00000005 ffff8003`07bd6000 00000000`626e4d43 ffff8003`0f5ef770 : nt!KiPageFault+0x35e
    fffff601`37d3f110 fffff803`57c52056     : 00000000`00000000 00000000`00000000 00000000`00000000 ffff8003`0f5ef778 : nt!CmpNotifyChangeKey+0xe7
    fffff601`37d3f1c0 fffff803`57c4e700     : 0000004d`fa1fe8f0 fffff601`37d3f540 00000000`00000001 0000004d`fa1fe8f0 : nt!NtNotifyChangeMultipleKeys+0x296
    fffff601`37d3f3e0 fffff803`577d4555     : ffffc905`142c3080 0000004d`fa1fe8d8 fffff601`37d3f468 00000000`00000000 : nt!NtNotifyChangeKey+0x60
    fffff601`37d3f450 00007fff`31c9e334     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
    0000004d`fa1fe8e8 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007fff`31c9e334
    
    
    SYMBOL_NAME:  nt!CmpNotifyChangeKey+e7
    
    MODULE_NAME: nt
    
    IMAGE_NAME:  ntkrnlmp.exe
    
    STACK_COMMAND:  .thread ; .cxr ; kb
    
    BUCKET_ID_FUNC_OFFSET:  e7
    
    FAILURE_BUCKET_ID:  AV_R_INVALID_nt!CmpNotifyChangeKey
    
    OS_VERSION:  10.0.18362.1
    
    BUILDLAB_STR:  19h1_release
    
    OSPLATFORM_TYPE:  x64
    
    OSNAME:  Windows 10
    
    FAILURE_ID_HASH:  {f54d959b-14cc-1246-e8af-387efa9d8107}
    
    Followup:     MachineOwner
    ---------
    
    

    Thanks


    Tuesday, November 10, 2020 3:22 PM

All replies

  • I managed to narrow down the BSOD's to an ImageLoad exclude rule:

          <ImageLoad onmatch="exclude">
            <Rule groupRelation="and">
              <Image condition="begin with">C:\Windows\System32</Image>
              <Image condition="end with">WMIC.exe</Image>
              <ImageLoaded condition="begin with">C:\ProgramData\Microsoft\Windows Defender\Platform</ImageLoaded>
              <ImageLoaded condition="end with">MpOAV.dll</ImageLoaded>
            </Rule>
            <Rule groupRelation="and">
              <Image condition="begin with">C:\Windows\System32</Image>
              <Image condition="end with">WmiPrvSE.exe</Image>
              <ImageLoaded condition="begin with">C:\ProgramData\Microsoft\Windows Defender\Platform</ImageLoaded>
              <ImageLoaded condition="end with">MpClient.dll</ImageLoaded>
            </Rule>
            <Rule groupRelation="and">
              <Image condition="begin with">C:\Windows\System32</Image>
              <Image condition="end with">WmiPrvSE.exe</Image>
              <ImageLoaded condition="begin with">C:\ProgramData\Microsoft\Windows Defender\Platform</ImageLoaded>
              <ImageLoaded condition="end with">ProtectionManagement.dll</ImageLoaded>
            </Rule>
            <Rule groupRelation="and">
              <Image condition="begin with">C:\ProgramData\Microsoft\Windows Defender\Platform</Image>
              <Image condition="end with">MpCmdRun.exe</Image>
              <ImageLoaded condition="begin with">C:\ProgramData\Microsoft\Windows Defender\Platform</ImageLoaded>
              <ImageLoaded condition="end with">MpClient.dll</ImageLoaded>
            </Rule>
          </ImageLoad>

    Removing this rule has stopped the BSODs. I am now seeing the whole system grind to halt with no useful output from resource monitor which seems to think CPU / memory / disk and network are all fine.

    Thursday, November 12, 2020 4:12 PM
  • In any case, please contact Mark Cook at Syssite@microsfot.com in order to provide the bsod dump so he could eventually fix the problem.

    Thanks

    -mario

    Sunday, November 15, 2020 8:39 AM
  • I tried emailing Syssite@microsoft.com and received a bounceback
    Monday, November 16, 2020 12:25 PM
  • Hi,

    we have seen similar problems where systems grind to a halt with no obvious resources issues or any useful log data anywhere and when the sysmon service is stopped it returns to normal.

    we have also seen that 12.02 stopped logging any EVID 12,13 events all together when it was running and also had an affect on SMB/CIFS service on one system.

    We have reverted back to v11.11 now but have found there that it seems to also have an issue with rule groups and we see application crashes in the Application event logs. We are now looking at also reverting back to and older config schema version or try to refrain to have too many rule groups especially in under the exclusion section.


    • Edited by Gernot_Baar Monday, November 16, 2020 2:03 PM
    Monday, November 16, 2020 2:02 PM
  • I have also reverted back to our old configuration. I can fairly consistently cause both the BSOD and the machine slow down in both a lab vm and on our network. If someone from the sysinternals team would like my configuration for testing I'm happy to provide it. It does seem to be the exclusion rule groups that cause the issues.
    Wednesday, November 25, 2020 9:16 AM