none
BSOD tracked to SysmonDrv

    Question

  • We've been working on repeated BSODs in our environment that Microsoft Support has tracked to SysmonDrv. Per support, Sysmon is attempting to access a memory address that no longer contains data (or data Sysmon is not expecting). This triggers a system access violation. 

    0 fffff80001764bc0 fffff8000191d63f nt!KiFreezeTargetExecution+0x1df 
     1 fffff80001764cb0 fffff800018b3502 nt!KiProcessNMI+0x3f 
     2 fffff80001764d10 fffff800018b331d nt!KxNmiInterrupt+0x82 
     3 fffff80001764e50 fffff88001dc50d8 nt!KiNmiInterrupt+0x19d                                                TrapFrame @ fffff80001764e50
    4 fffff880124670b0 fffff88001dc54f2 SysmonDrv+0x40d8 
     5 fffff88012467210 fffff80001868b15 SysmonDrv+0x44f2 
     6 fffff88012467240 fffff80001850d51 nt!IopUnloadSafeCompletion+0x55 
     7 fffff88012467280 fffff880013cd627 nt!IopfCompleteRequest+0x341 
     8 fffff88012467370 fffff88001dc9cfd Npfs!NpFsdCreateNamedPipe+0x403 
     9 fffff88012467490 fffff80001cff5c2 SysmonDrv+0x8cfd 
     a fffff88012467550 fffff80001c26e54 nt!IopParseDevice+0x14e2

     b fffff880124676b0 fffff80001b01766 nt!ObpLookupObjectName+0x784 
     c fffff880124677b0 fffff80001ccdd38 nt!ObOpenObjectByName+0x306 
     d fffff88012467880 fffff80001b5d86a nt!IopCreateFile+0xa08 
     e fffff88012467930 fffff80001b5d97e nt!IoCreateFile+0x8a 
     f fffff880124679c0 fffff800018b79d3 nt!NtCreateNamedPipeFile+0x106 
    10 fffff88012467a70 000000007793a26a nt!KiSystemServiceCopyEnd+0x13 
     0         03daf098         76a4806d ntdll_77a90000!ZwCreateNamedPipeFile+0x12
    1         03daf0a0         66dbd612 KERNELBASE!CreateNamedPipeW+0x1d0
    2         03daf13c         66dbd3bb IAMAgent!IAMAgentRemoteDisconnected+0xe6342
    3         03daf374         66dbd2d2 IAMAgent!IAMAgentRemoteDisconnected+0xe60eb
    4         03daf39c         66d928bc IAMAgent!IAMAgentRemoteDisconnected+0xe6002
    5         03daf3c0         71f5c3d4 IAMAgent!IAMAgentRemoteDisconnected+0xbb5ec
    6         03daf974         71f5c474 MSVCR100!_callthreadstart+0x1b 
     7         03daf9ac         7635343d MSVCR100!_threadstart+0x58 
     8         03daf9b4         77ac9802 kernel32!BaseThreadInitThunk+0xe 
     9         03daf9c0         77ac97d5 ntdll_77a90000!__RtlUserThreadStart+0x70
    a         03dafa00         00000000 ntdll_77a90000!_RtlUserThreadStart+0x1b 

    More specifically, per the Microsoft support engineer:

    # Child-SP         Return           Call Site                                                                 Info
    0 fffff80001764bc0 fffff8000191d63f nt!KiFreezeTargetExecution+0x1df 
     1 fffff80001764cb0 fffff800018b3502 nt!KiProcessNMI+0x3f 
     2 fffff80001764d10 fffff800018b331d nt!KxNmiInterrupt+0x82 
     3 fffff80001764e50 fffff88001dc50d8 nt!KiNmiInterrupt+0x19d                                                   TrapFrame @ fffff80001764e50
    4 fffff880124670b0 fffff88001dc54f2 SysmonDrv+0x40d8 
     5 fffff88012467210 fffff80001868b15 SysmonDrv+0x44f2 
     6 fffff88012467240 fffff80001850d51 nt!IopUnloadSafeCompletion+0x55 
     7 fffff88012467280 fffff880013cd627 nt!IopfCompleteRequest+0x341 
     8 fffff88012467370 fffff88001dc9cfd Npfs!NpFsdCreateNamedPipe+0x403 
     9 fffff88012467490 fffff80001cff5c2 SysmonDrv+0x8cfd 
     a fffff88012467550 fffff80001c26e54 nt!IopParseDevice+0x14e2 
     b fffff880124676b0 fffff80001b01766 nt!ObpLookupObjectName+0x784 
     c fffff880124677b0 fffff80001ccdd38 nt!ObOpenObjectByName+0x306 
     d fffff88012467880 fffff80001b5d86a nt!IopCreateFile+0xa08 
     e fffff88012467930 fffff80001b5d97e nt!IoCreateFile+0x8a 
     f fffff880124679c0 fffff800018b79d3 nt!NtCreateNamedPipeFile+0x106 
    10 fffff88012467a70 000000007793a26a nt!KiSystemServiceCopyEnd+0x13 
     0         03daf098         76a4806d ntdll_77a90000!ZwCreateNamedPipeFile+0x12
    1         03daf0a0         66dbd612 KERNELBASE!CreateNamedPipeW+0x1d0
    2         03daf13c         66dbd3bb IAMAgent!IAMAgentRemoteDisconnected+0xe6342
    3         03daf374         66dbd2d2 IAMAgent!IAMAgentRemoteDisconnected+0xe60eb
    4         03daf39c         66d928bc IAMAgent!IAMAgentRemoteDisconnected+0xe6002
    5         03daf3c0         71f5c3d4 IAMAgent!IAMAgentRemoteDisconnected+0xbb5ec
    6         03daf974         71f5c474 MSVCR100!_callthreadstart+0x1b 
     7         03daf9ac         7635343d MSVCR100!_threadstart+0x58 
     8         03daf9b4         77ac9802 kernel32!BaseThreadInitThunk+0xe 
     9         03daf9c0         77ac97d5 ntdll_77a90000!__RtlUserThreadStart+0x70
    a         03dafa00         00000000 ntdll_77a90000!_RtlUserThreadStart+0x1b

    0: kd> .cxr 0xfffff880114846d0
    rax=000000000000006f rbx=000000000000006f rcx=fffff8a02e0f60b6
    rdx=006407ccd2339fe6 rsi=fffffa80229cd670 rdi=fffff88011485120
    rip=fffff88001dc21d0 rsp=fffff880114850a8 rbp=fffff880114851b0
    r8=000000000000006f  r9=0000000000000000 r10=fffffa802051f3c8
    r11=fffff8a02e0f6048 r12=fffff8a02e0f6010 r13=0000000000000002
    r14=fffff8a02e0f6048 r15=fffff8a02e0f603c
    iopl=0         nv up ei ng nz na pe nc
    cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
    SysmonDrv+0x11d0:   <
    fffff880`01dc21d0 8a040a          mov     al,byte ptr [rdx+rcx] ds:002b:0064006d`0043009c=?? <- RDX is bad and we are going to crash

    We have further memory dumps from the Microsoft support ticket if they would be helpful.

    Monday, November 26, 2018 8:59 PM

All replies


  • To evaluate the BSOD please post logs for troubleshooting.

    Using administrative command prompt copy and paste this whole command.

    Make sure the default language is English so that the logs can be scanned and read.

    https://www.tenforums.com/tutorials/3813-language-add-remove-change-windows-10-a.html

    The command will automatically collect the computer files and place them on the desktop.

    Then use 7zip to organize the files and one drive or drop box to place share links into the thread for troubleshooting.

    https://support.office.com/en-us/article/Share-OneDrive-files-and-folders-9fcc2f7d-de0c-4cec-93b0-a82024800c07

    This command will automatically collect these files:  msinfo32, mini dumps, drivers, hosts, install, uninstall, services, startup, event viewer files, etc.

    Open administrative command prompt and copy and paste the whole command:

    copy %SystemRoot%\minidump\*.dmp "%USERPROFILE%\Desktop\"&dxdiag /t %Temp%\dxdiag.txt&copy %Temp%\dxdiag.txt "%USERPROFILE%\Desktop\SFdebugFiles\"&type %SystemRoot%\System32\drivers\etc\hosts >> "%USERPROFILE%\Desktop\hosts.txt"&systeminfo > "%USERPROFILE%\Desktop\systeminfo.txt"&driverquery /v > "%USERPROFILE%\Desktop\drivers.txt" &msinfo32 /nfo "%USERPROFILE%\Desktop\msinfo32.nfo"&wevtutil qe System /f:text > "%USERPROFILE%\Desktop\eventlog.txt"&reg export HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall "%USERPROFILE%\Desktop\uninstall.txt"&reg export "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components" "%USERPROFILE%\Desktop\installed.txt"&net start > "%USERPROFILE%\Desktop\services.txt"&REM wmic startup list full /format:htable >"%USERPROFILE%\Desktop\startup.html"&wmic STARTUP GET Caption, Command, User >"%USERPROFILE%\Desktop\startup.txt"

    There are two files for you to find manually:

    a) C:\Windows\MEMORY.DMP

    Use file explorer > this PC > local C: drive > right upper corner search enter the above to find results.

    b) dxdiag:  

    In the left lower corner search type:  dxdiag > When the DirectX Diagnostic Tool opens click on the next page button so that each tab is opened > click on save all information > save to desktop > post one drive or drop box share link into the thread

    For memory dumps if the file size is < 1.5 GB then zip and post individual share links into the thread using one drive, drop box, or google drive

    .

    .

    .

    Please remember to vote and to mark the replies as answers if they help.

    .

    .

    .

    Monday, November 26, 2018 9:59 PM
  • I have a complete memory dump of the system, but we may not have access to the VM snapshot that it was generated from. Would that be enough? Also, is there a secure way to transfer the dump? It could contain sensitive information (e.g. PII/PHI), and I do not want to post it on a public forum.
    Tuesday, November 27, 2018 3:48 PM
  • Pat

    please contact me offline via sysmonsupport@microsoft.com and I will provide you with a location for the dump files.

    MarkC(MSFT)

    Wednesday, November 28, 2018 7:25 PM
  • The share links can be posted and then removed.

    If see if you can post over then next few days for example later this evening or tomorrow sometime.  Once the memory dump is successfully downloaded the share link can be deleted.

    Wednesday, November 28, 2018 11:01 PM