none
netft.sys is the cause for the bugchk blue screen on the server Windows 2008 R2 Datacenter RRS feed

  • Question

  • Hi

    we have the server geting rebooted by a bugchk error for netft.sys Please let me know if we have any fix for this issue. i am not sure wht is causing the issue on the server

    the server is windows 2008 R2 Datacenter and it is on the HyperV cluster

    Thanks in advance

    • Moved by BrianEhMVP Friday, March 8, 2013 8:05 PM
    Friday, March 8, 2013 7:59 PM

Answers

All replies

  • I assume you are referring to a stop 0x9e for netft.sys, which is an intentional bugcheck caused by the cluster service due to a deadlock condition identified. I'd recommend reviewing the following for further tips to troubleshoot this issue:

    http://blogs.technet.com/b/askcore/archive/2009/06/12/why-is-my-2008-failover-clustering-node-blue-screening-with-a-stop-0x0000009e.aspx


    Visit my blog about multi-site clustering

    Friday, March 8, 2013 9:28 PM
    Moderator
  • Thanks John,

    thi was the same error i was refering to. I did go through the MS site and i found some Hotfix for it, Not very sure if that Hot fix is for the same issue.

      http://support.microsoft.com/kb/2135160/en-us error “FIX: "0x0000009E" Stop error when you host Hyper-V virtual machines in a Windows Server 2008 R2-based failover cluster

    do u have any clue on it ?

    Friday, March 8, 2013 10:29 PM
  • I think the most important line in that KB article:

    Not all "0x0000009E" Stop errors are caused by this problem.

    This blue screen just indicates that something caused the cluster to believe that the node was hung. If you are running Hyper-V, this might be a good place to start with this hotfix. Otherwise, you might consider opening a ticket with PSS to troubleshoot this further.


    Visit my blog about multi-site clustering

    Friday, March 8, 2013 11:00 PM
    Moderator
  • Clustering has health detection between the user mode service and the kernel mode NetFT driver.  If user mode goes unresponsive, then clustering bugchecks the box in an effort to force a failover.  A STOP 0x9e is expected cluster behavior.  You should troubleshoot the condition as a user mode hang...

    Thanks!
    Elden

    Saturday, March 9, 2013 3:02 AM
    Owner
  • Techei, I am a developer on the clustering team. if the problem persists then you can share dump file with me (c:\windows\memory.dmp). I will take a look what exactly caused the bugcheck. if you do not want to share it with the world then give me your email, I'll reply with my email and you can share this dump with me. If you prefer work through customer support then they also should be able to look at the dump and tell you what netft is not happy about.

     

    Regards, Vladimir Petter, Microsoft Corporation

    Sunday, March 10, 2013 7:16 PM
  • We're having a very similar problem.. which i'd love for someone to actually investigate the memory dump to give me a real clue as to the problem.  We're in the midst of multiple cases with PSS but a solution couldn't come quick enough.  How can I share this dump with you?

    Wednesday, March 13, 2013 3:02 PM
  • Hi Vladimir, I have the same issue with the bugcheck 0x0000009e

    How can I send you the memory.dmp? Thaanks a lot

    Friday, October 11, 2013 8:20 PM
  • If you can put it to some location I can download it from that would work.

    Thanks,

    Vladimir.


    Regards, Vladimir Petter, Microsoft Corporation

    Saturday, October 12, 2013 3:28 AM
  • I used my skydrive. You can dowload it from there

    https://skydrive.live.com/redir?resid=95BBBD60D3F2B190!117

    thanks!

    Tuesday, October 15, 2013 3:55 PM
  • look at the properties of the files c:\windows\system32\drivers\EmcpXcr.sys, EmcpBase.sys and EmcpGpx.sys to see what company these drivers came from and contact that company for support, and share with them your dump.

    Cluster bugchecked the machine because cluster tried to terminate resource host monitor - the host process where the plug-ins that control applications, disks etc are running in, and this process did not go away in 20 minutes. Termination is stuck because some threads are stuck in the kernel. They all end up waiting for the thread below. That thread has been stuck for 25 minutes

    THREAD fffffa800793f660  Cid 0004.0050  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
        fffffa8008467db8  SynchronizationEvent
        fffffa8008467dd0  NotificationEvent
    IRP List:
        fffffa801452f4b0: (0006,0310) Flags: 00060070  Mdl: 00000000
        fffffa8013d9ba90: (0006,0310) Flags: 00060070  Mdl: 00000000
    Not impersonating
    DeviceMap                 fffff8a000008500
    Owning Process            fffffa80078c49e0       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      262955         Ticks: 96163 (0:00:25:00.152)
    Context Switch Count      10707          IdealProcessor: 12            
    UserTime                  00:00:00.000
    KernelTime                00:00:12.105
    Win32 Start Address nt!ExpWorkerThread (0xfffff800024e0150)
    Stack Init fffff88002b5cdb0 Current fffff88002b5be20
    Base fffff88002b5d000 Limit fffff88002b57000 Call 0
    Priority 13 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    fffff880`02b5be60 fffff800`024cc5f2 nt!KiSwapContext+0x7a
    fffff880`02b5bfa0 fffff800`024d90ea nt!KiCommitThreadWait+0x1d2
    fffff880`02b5c030 fffff880`0146ee3b nt!KeWaitForMultipleObjects+0x272
    fffff880`02b5c2f0 fffff880`01603601 EmcpBase!PowerSleep+0x83
    fffff880`02b5c350 fffff880`016038d9 EmcpXcr!XcryptUpdateAssocCallout+0xc45
    fffff880`02b5c3b0 fffff880`014f8c10 EmcpXcr!XcryptUpdateAssocCallout+0xf1d
    fffff880`02b5c3f0 fffff880`01467adf EmcpGpx!GpxDestroySplitPirp+0x78
    fffff880`02b5c420 fffff880`014f8c30 EmcpBase!PowerDispatchX+0x243
    fffff880`02b5c470 fffff880`01467adf EmcpGpx!GpxDestroySplitPirp+0x98
    fffff880`02b5c4a0 fffff880`01468b41 EmcpBase!PowerDispatchX+0x243
    fffff880`02b5c4f0 fffff880`01475681 EmcpBase!PowerSyncIoTopDispatch+0x89
    fffff880`02b5c520 fffff880`01478686 EmcpBase!PowerWinIsIrpSync+0x965
    fffff880`02b5c570 fffff880`0180d362 EmcpBase!PowerWinIsPseudoBusPDO+0x219a
    fffff880`02b5c5d0 fffff880`0180971f disk!DiskGetPortGeometry+0x92
    fffff880`02b5c630 fffff880`018015ec disk!DiskUpdateGeometry+0x4ef
    fffff880`02b5c670 fffff880`0180164c disk!DiskReadDriveCapacity+0x1c
    fffff880`02b5c6a0 fffff880`0159ce7d disk!DiskDeviceControl+0x2e3
    fffff880`02b5c700 fffff880`012f1244 CLASSPNP!ClassDeviceControlDispatch+0x2d
    fffff880`02b5c730 fffff800`027f745d partmgr!PmFilterDeviceControl+0xd4
    fffff880`02b5c790 fffff800`027f72b4 nt!FstubGetDiskGeometry+0x12d
    fffff880`02b5c810 fffff800`027f76e2 nt!FstubAllocateDiskInformation+0x44
    fffff880`02b5c840 fffff880`012f3512 nt!IoReadPartitionTableEx+0x1a
    fffff880`02b5c870 fffff880`012f1398 partmgr!PmGetDriveLayoutEx+0x5d2
    fffff880`02b5c970 fffff880`0162d48f partmgr!PmFilterDeviceControl+0x228
    fffff880`02b5c9d0 fffff880`0162ae3d ClusDisk!ClusDskpSendIoctl+0x8f
    fffff880`02b5ca70 fffff800`02742eb0 ClusDisk!ClusDskDeviceChangeNotification+0xc9
    fffff880`02b5cae0 fffff800`02741787 nt!PnpNotifyDriverCallback+0x5c
    fffff880`02b5cb70 fffff800`02742ffc nt!PnpNotifyTargetDeviceChange+0x16b
    fffff880`02b5cc20 fffff800`027420ca nt!PnpProcessCustomDeviceEvent+0x2c
    fffff880`02b5cc50 fffff800`024e0261 nt!PnpDeviceEventWorker+0x142
    fffff880`02b5ccb0 fffff800`027732ea nt!ExpWorkerThread+0x111
    fffff880`02b5cd40 fffff800`024c78e6 nt!PspSystemThreadStartup+0x5a
    fffff880`02b5cd80 00000000`00000000 nt!KxStartSystemThread+0x16

    Storage class device fffffa800f2ef060 with extension at fffffa800f2ef1b0

    Classpnp Internal Information at fffffa801371c530

        Transfer Packet Engine:

         Packet          Status  DL Irp          Opcode  Sector/ListId   UL Irp
        --------         ------ --------         ------ --------------- --------

        Pending Idle Requests: 0x0


        Failed Requests:

               Srb    Scsi                                  
        Opcode Status Status Sense Code  Sector/ListId   Time Stamp
        ------ ------ ------ ---------- --------------- ------------
          28     04     02    02 04 03      00000001    12:28:45.684  
          28     04     02    02 04 03      003fffff    12:28:45.684  
          28     04     02    02 04 03      00000001    12:28:45.684  
          28     04     02    02 04 03      003fffff    12:28:45.684  
          28     04     02    02 04 03      00000001    12:28:45.684  
          28     04     02    02 04 03      003fffff    12:28:45.684  
          28     04     02    02 04 03      00000001    12:28:45.699  
          28     04     02    02 04 03      003fffff    12:28:45.699  
          28     04     02    02 04 03      00000001    12:28:45.699  
          28     04     02    02 04 03      003fffff    12:28:45.699  
          28     04     02    02 04 03      00000001    12:28:45.699  
          28     04     02    02 04 03      003fffff    12:28:45.715  
          28     04     02    02 04 03      00000001    12:28:45.824  
          28     04     02    02 04 03      003fffff    12:28:45.824  
          28     04     02    02 04 03      00000001    12:28:45.840  
          28     04     02    02 04 03      003fffff    12:28:45.840  

        -- dt classpnp!_CLASS_PRIVATE_FDO_DATA fffffa801371c530 --

    Classpnp External Information at fffffa800f2ef1b0

        DGC RAID 5 0429 CKM00100500355

        Minidriver information at fffffa800f2ef670
        Attached device object at fffffa800f2ed060
        Physical device object at fffffa800f2ed060

        Media Geometry:

            Bytes in a Sector = 512
            Sectors per Track = 63
            Tracks / Cylinder = 255
            Media Length      = 2147483648 bytes = ~2 GB

        -- dt classpnp!_FUNCTIONAL_DEVICE_EXTENSION fffffa800f2ef1b0 --


    Regards, Vladimir Petter, Microsoft Corporation

    Wednesday, October 16, 2013 5:05 AM
  • Thanks a lot! I ve already talked with EMC and they recomended  to us   do an updated   of drivers and multipath.

    I have another question but  using the debug  tool. I download it from http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx   but always it give me error from symbols

    Microsoft (R) Windows Debugger Version 6.2.9200.20512 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.


    Loading Dump File [C:\Temp\MEMORY.DMP]
    Kernel Summary Dump File: Only kernel address space is available

    Symbol search path is: srv*C:\websymbols*http://msdl.microsoft.com/download/symbols;srv*
    Executable search path is: srv*
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntkrnlmp.exe - 
    Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
    Product: Server, suite: Enterprise TerminalServer SingleUserTS
    Built by: 7601.18247.amd64fre.win7sp1_gdr.130828-1532
    Machine Name:
    Kernel base = 0xfffff800`02461000 PsLoadedModuleList = 0xfffff800`026a46d0
    Debug session time: Fri Oct 11 16:53:45.992 2013 (UTC - 3:00)
    System Uptime: 0 days 1:33:22.285
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntkrnlmp.exe - 
    Loading Kernel Symbols
    ...............................................................

    Loading User Symbols

    Loading unloaded module list
    ...........
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    Use !analyze -v to get detailed debugging information.

    BugCheck 9E, {fffffa8008a5eab0, 4b0, 0, 0}

    ***** Kernel symbols are WRONG. Please fix symbols to do analysis.

    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Either you specified an unqualified symbol, or your debugger   ***
    ***    doesn't have full symbol information.  Unqualified symbol      ***
    ***    resolution is turned off by default. Please either specify a   ***
    ***    fully qualified symbol module!symbolname, or enable resolution ***
    ***    of unqualified symbols by typing ".symopt- 100". Note that   ***
    ***    enabling unqualified symbol resolution with network symbol     ***
    ***    server shares in the symbol path may cause the debugger to     ***
    ***    appear to hang for long periods of time when an incorrect      ***
    ***    symbol name is typed or the network symbol server is down.     ***
    ***                                                                   ***
    ***    For some commands to work properly, your symbol path           ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: nt!_KPRCB                                     ***
    ***                                                                   ***
    *************************************************************************

    *************************************************************************
    Probably caused by : netft.sys ( netft!NetftWatchdogTimerDpc+b9 )

    Followup: MachineOwner
    ---------

    8: kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    USER_MODE_HEALTH_MONITOR (9e)
    One or more critical user mode components failed to satisfy a health check.
    Hardware mechanisms such as watchdog timers can detect that basic kernel
    services are not executing. However, resource starvation issues, including
    memory leaks, lock contention, and scheduling priority misconfiguration,
    may block critical user mode components without blocking DPCs or
    draining the nonpaged pool.
    Kernel components can extend watchdog timer functionality to user mode
    by periodically monitoring critical applications. This bugcheck indicates
    that a user mode health check failed in a manner such that graceful
    shutdown is unlikely to succeed. It restores critical services by
    rebooting and/or allowing application failover to other servers.
    Arguments:
    Arg1: fffffa8008a5eab0, Process that failed to satisfy a health check within the
    configured timeout
    Arg2: 00000000000004b0, Health monitoring timeout (seconds)
    Arg3: 0000000000000000
    Arg4: 0000000000000000

    Debugging Details:
    ------------------

    ***** Kernel symbols are WRONG. Please fix symbols to do analysis.

    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Either you specified an unqualified symbol, or your debugger   ***
    ***    doesn't have full symbol information.  Unqualified symbol      ***
    ***    resolution is turned off by default. Please either specify a   ***
    ***    fully qualified symbol module!symbolname, or enable resolution ***
    ***    of unqualified symbols by typing ".symopt- 100". Note that   ***
    ***    enabling unqualified symbol resolution with network symbol     ***
    ***    server shares in the symbol path may cause the debugger to     ***
    ***    appear to hang for long periods of time when an incorrect      ***
    ***    symbol name is typed or the network symbol server is down.     ***
    ***                                                                   ***
    ***    For some commands to work properly, your symbol path           ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: nt!_KPRCB                                     ***
    ***                                                                   ***
    *************************************************************************


    ADDITIONAL_DEBUG_TEXT:  
    You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

    MODULE_NAME: netft

    FAULTING_MODULE: fffff80002461000 nt

    DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bc48a

    PROCESS_OBJECT: fffffa8008a5eab0

    DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

    BUGCHECK_STR:  0x9E

    CURRENT_IRQL:  0

    LAST_CONTROL_TRANSFER:  from fffff88000e626a5 to fffff800024d6bc0

    STACK_TEXT:  
    fffff880`0253d518 fffff880`00e626a5 : 00000000`0000009e fffffa80`08a5eab0 00000000`000004b0 00000000`00000000 : nt!KeBugCheckEx
    fffff880`0253d520 fffff800`024e185c : fffff880`0253d618 00000000`00000001 00000000`400e0088 00000000`00000001 : netft!NetftWatchdogTimerDpc+0xb9
    fffff880`0253d570 fffff800`024e16f6 : fffff880`00e6f100 00000000`00057ace 00000000`00000000 00000000`00000000 : nt!KeReleaseMutant+0xb2c
    fffff880`0253d5e0 fffff800`024e15de : 0000000d`0b38db3c fffff880`0253dc58 00000000`00057ace fffff880`02518f48 : nt!KeReleaseMutant+0x9c6
    fffff880`0253dc30 fffff800`024e13c7 : 00000003`0ebda1c2 00000003`00057ace 00000003`0ebda10c 00000000`000000ce : nt!KeReleaseMutant+0x8ae
    fffff880`0253dcd0 fffff800`024ce8ca : fffff880`02515180 fffff880`025202c0 00000000`00000000 fffff880`01a95a48 : nt!KeReleaseMutant+0x697
    fffff880`0253dd80 00000000`00000000 : fffff880`0253e000 fffff880`02538000 fffff880`0253dd40 00000000`00000000 : nt!KiCpuId+0x6fa


    STACK_COMMAND:  kb

    FOLLOWUP_IP: 
    netft!NetftWatchdogTimerDpc+b9
    fffff880`00e626a5 cc              int     3

    SYMBOL_STACK_INDEX:  1

    SYMBOL_NAME:  netft!NetftWatchdogTimerDpc+b9

    FOLLOWUP_NAME:  MachineOwner

    IMAGE_NAME:  netft.sys

    BUCKET_ID:  WRONG_SYMBOLS

    Followup: MachineOwner
    ---------

    8: kd> !process fffffa8008a5eab0 3
    NT symbols are incorrect, please fix symbols
    Wednesday, October 16, 2013 4:26 PM
  • Once open the dump first run ".symfix" then ".reload". After that always start from "!analyze -v". If you see
     USER_MODE_HEALTH_MONITOR (9e) then it complains about a process not doing something in time (in the case above not terminating in time). The process address is in Arg1. run "!process <Arg1> 1f", and go from there.

    Regards, Vladimir Petter, Microsoft Corporation

    Wednesday, October 16, 2013 5:10 PM
  • Hi all

    Thanks for this information.
    3. days ago we had a blusceen as well on one of our Exchange 2010 MBX server.
    All drives are connected via iSCSI except C. These drives are not pysical. We are using NetApp and on top VmWare.

    As I could see in the dump file while using WinDBG is:

    Probably caused by : netft.sys ( netft+26a5 )

    What do you suggest to solve this issue?

    Kind regards Matthias

    Tuesday, August 5, 2014 7:35 AM
  • Hello Matthias,

    If you can share the dump with me I'll be happy to take a look.


    Regards, Vladimir Petter, Microsoft Corporation

    Tuesday, August 5, 2014 4:09 PM
  • Hi Vladimir

    Thanks for your reply. I rely appreciate this.
    Here is the link to have a look into the dump: https://drive.google.com/file/d/0BxDBxOjFJ8GLRXpVZE9JN19BXzA/edit?usp=sharing.

    I am waiting for your feedback.

    Kind regards

    Matthias

    Wednesday, August 6, 2014 2:37 PM
  • Looks like your VMWare disks are timing out IO at the moment, and also I see some netapp LUNs were failing. Please check your storage.

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     04     22    00 00 00      01f73420    16:17:20.276  Retried

          28     04     22    00 00 00      018807e0    16:17:20.276  Retried

          2a     04     22    00 00 00      000599c8    16:17:20.276  Retried

          2a     04     22    00 00 00      01aa0bb0    17:23:49.276  Retried

          2a     04     22    00 00 00      02e790f0    17:23:49.276  Retried

          2a     04     22    00 00 00      013bfc50    17:23:49.276  Retried

          2a     04     22    00 00 00      02692250    17:23:49.276  Retried

          2a     04     22    00 00 00      03553f80    17:23:49.276  Retried

          2a     04     22    00 00 00      0311d0a5    17:23:49.276  Retried

          2a     04     22    00 00 00      00608b90    17:23:49.276  Retried

          2a     04     22    00 00 00      02b2a210    17:23:49.276  Retried

          2a     04     22    00 00 00      006327f8    17:23:49.276  Retried

          2a     04     22    00 00 00      0007a050    17:23:49.276  Retried

          2a     04     22    00 00 00      0333eb68    17:23:49.276  Retried

          2a     04     22    00 00 00      0004e538    17:23:49.276  Retried

          28     04     22    00 00 00      009fa980    17:23:49.276  Retried

        VMware Virtual disk 1.0 6000c292caa09c5f634b3b75741142ea

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          28     04     22    00 00 00      006bbac0    16:11:00.276  Retried

          28     04     22    00 00 00      008e1d30    16:12:26.276  Retried

          28     04     22    00 00 00      0099b8c8    16:12:26.276  Retried

          28     04     22    00 00 00      0084d9c8    16:12:26.276  Retried

          28     04     22    00 00 00      006cdc08    16:12:26.276  Retried

          28     04     22    00 00 00      006cdb88    16:12:26.276  Retried

          28     04     22    00 00 00      008e0020    16:14:50.276  Retried

          28     04     22    00 00 00      0099b828    16:14:50.276  Retried

          28     04     22    00 00 00      006bbbb8    16:14:50.276  Retried

          28     04     22    00 00 00      0079ee58    16:14:50.276  Retried

          28     04     22    00 00 00      006ed580    16:14:50.276  Retried

          28     04     22    00 00 00      008fd188    16:14:50.276  Retried

          28     04     22    00 00 00      00701e60    16:17:22.276  Retried

          28     04     22    00 00 00      00757f68    16:17:22.276  Retried

          28     04     22    00 00 00      00a87f28    16:17:22.276  Retried

          28     04     22    00 00 00      00783c60    16:17:22.276  Retried

        VMware Virtual disk 1.0 6000c29be6a5c7d86a9a12fabe644f5d

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          28     04     02    06 3f 0e      0fbac380    13:33:27.572  Retried

          2a     0e     00    00 00 00      137a3c00    16:09:42.041  Retried

          2a     0e     00    00 00 00      137a3e00    16:09:42.041  Retried

          2a     0e     02    06 29 00      137a3c00    16:09:42.119  Retried

          2a     0e     00    00 00 00      00614818    17:24:09.432  Retried

        NETAPP LUN 811a 7SRjT+BTl8Uf

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     00    00 00 00      0e910e60    16:09:42.729  Retried

          2a     0e     02    06 29 00      006407f8    16:09:42.807  Retried

          2a     0e     00    00 00 00      005fd608    16:11:00.291  Retried

          2a     0e     00    00 00 00      00632330    16:11:00.291  Retried

          2a     0e     00    00 00 00      0063d620    16:11:00.291  Retried

          2a     0e     00    00 00 00      0bfc32e0    16:11:00.291  Retried

          2a     0e     02    06 29 00      005fd608    16:11:00.322  Retried

          2a     0e     00    00 00 00      0c35dfe0    16:12:29.994  Retried

          2a     0e     02    06 29 00      0c35dfe0    16:12:29.994  Retried

          2a     0e     00    00 00 00      0060b1b8    16:12:30.010  Retried

          2a     0e     00    00 00 00      0060b1c8    16:14:50.479  Retried

          2a     0e     00    00 00 00      000d0070    16:14:50.479  Retried

          2a     0e     00    00 00 00      0e9e6b60    16:14:50.479  Retried

          2a     0e     02    06 29 00      0060b1c8    16:14:50.510  Retried

          2a     0e     00    00 00 00      0e80ece0    16:17:21.354  Retried

          2a     0e     02    06 29 00      0e80ece0    16:17:21.510  Retried

        NETAPP LUN 811a 7SRjT+BTl8Uh

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     00    00 00 00      000f09d8    16:12:30.088  Retried

          2a     0e     00    00 00 00      002a6be8    16:12:30.088  Retried

          28     0e     02    06 29 00      0e223ce8    16:12:30.088  Retried

          28     0e     02    06 29 00      0e240ce8    16:13:43.276  Retried

        NETAPP LUN 811a 7SRjT+BTl8Uj

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     00    00 00 00      00635770    16:11:02.369  Retried

          2a     0e     02    06 29 00      005fd5f0    16:11:02.369  Retried

          2a     0e     00    00 00 00      0011c320    16:13:42.979  Retried

          2a     0e     02    06 29 00      0011c320    16:13:43.010  Retried

        NETAPP LUN 811a 7SRjT+BTl8Ul

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     02    06 29 00      005fd608    16:13:43.260  Retried

          2a     0e     02    06 29 00      00640a20    16:17:30.916  Retried

        NETAPP LUN 811a 7SRjT+BTl8Un

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     00    00 00 00      0351aef8    16:11:02.463  Retried

          2a     0e     00    00 00 00      0060fb78    16:11:02.463  Retried

          28     0e     02    06 29 00      0b1ad938    16:11:02.463  Retried

          2a     0e     00    00 00 00      06557578    16:12:30.057  Retried

          28     0e     02    06 29 00      0b218f38    16:12:30.213  Retried

          28     0e     02    06 29 00      0b236338    16:13:43.369  Retried

          28     04     02    06 3f 0e      0cec5338    17:08:29.760  Retried

          28     04     02    06 3f 0e      0d187b38    17:21:28.932  Retried

        NETAPP LUN 811a 7SRjT+BTl8Up

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     00    00 00 00      1379fca0    16:09:43.197  Retried

          2a     0e     02    06 29 00      002c8a60    16:09:43.229  Retried

          2a     0e     02    06 29 00      005fd600    16:11:02.307  Retried

          2a     0e     00    00 00 00      1379fca0    16:13:43.026  Retried

          2a     0e     00    00 00 00      00628278    16:13:43.026  Retried

          2a     0e     02    06 29 00      1379fca0    16:13:43.041  Retried

          2a     0e     00    00 00 00      0060d340    16:17:30.713  Retried

          2a     0e     00    00 00 00      002c8a60    16:17:30.713  Retried

          2a     0e     02    06 29 00      0060d340    16:17:30.729  Retried

        NETAPP LUN 811a 7SRjT+BTl8Ut

        Opcode Status Status Sense Code  Sector/ListId   Time Stamp

        ------ ------ ------ ---------- --------------- ------------

          2a     0e     00    00 00 00      02f88e50    16:17:30.572  Retried

          2a     0e     00    00 00 00      012aa038    16:17:30.588  Retried

          2a     0e     00    00 00 00      0063d768    16:17:30.588  Retried

          2a     0e     00    00 00 00      0286d1a0    16:17:30.588  Retried

          2a     0e     00    00 00 00      006199c0    16:17:30.588  Retried

          2a     0e     00    00 00 00      01175778    16:17:30.588  Retried

          2a     0e     00    00 00 00      0004cae0    16:17:30.588  Retried

          2a     0e     02    06 29 00      0323e8c8    16:17:30.791  Retried

          2a     04     02    06 3f 0e      025d736f    16:54:35.307  Retried

        NETAPP LUN 811a 7SRjT+BTl8Uv


    Regards, Vladimir Petter, Microsoft Corporation

    Wednesday, August 6, 2014 6:38 PM
  • Hi Vladimir,

    I've used WinDBG on many occassions, mainly for clr debugging.  however, i'm amazed at the information you keep getting out of the dump files.

    What are the most useful commands to help with this?  It's a science in itself.  Any guidance will be hugely appreciated

    Thanks

    Andreas

    Thursday, August 21, 2014 1:00 PM
  • Hi Vladamir,

    Can you please help me with debugging the minidump file.
    https://drive.google.com/folderview?id=0B0T9i-iE-IVhWDYwY1Z1Nm1yRFk&usp=sharing


    Regards

    Techwiz
    Thursday, August 21, 2014 5:06 PM
  • Hello Techwiz,

    Unfortunately I cannot tell much from the kernel minidump. Please change your OS settings to collect kernel full dump on bugcheck, and share that dump on the next repro.

    From the minidump it looks like a thread of RHS.exe process has been stuck in the kernel for 20 minutes preventing process termination from making forward progresss. After that a watchdog has recycled machine by bugchecking it. Minidump does not contain thread's stack so I cannot tell where is it stuck.


    Regards, Vladimir Petter, Microsoft Corporation

    Friday, August 22, 2014 7:42 AM
  • Vladimir,

    Can you take a look at my DMP file?

    https://drive.google.com/file/d/0BzIR_AB_PyatQmowSkJZX2gyQTg/view?usp=sharing

    I've had two crashes in the past two weeks with Netft.sys listed as the culprit.

    I would appreciate it if you noticed anything else. From a host on a Hyper-V Cluster.

    Thanks,

    Jon

    Monday, December 15, 2014 7:36 PM
  • Please open a case with Microsoft support and they will be able to debug the dump and identify root cause.

    Also see this blog:
    http://blogs.msdn.com/b/clustering/archive/2013/11/13/10467483.aspx

    Thanks!
    Elden

    Monday, December 15, 2014 8:09 PM
    Owner
  • Hello Jon,

    The problem is that you have such high DPC rate coming from network on CPU0 and CPU14 that it is stalling threads, and thread scheduling on CPU0.

    On CPU 0 DPCs were running back-to-back for almost 27 seconds. CPU0 currently handling a DPC, and in the queue there are 4 to go. By the time when these 4 are processed probably new DPCs will be enqued.

    --------------------------------------------------
    CPU#0
    --------------------------------------------------
    Current DPC: NDIS!ndisInterruptDpc (Normal DPC)
    Debugger Saved IRQL: 0
    Cumulative DPC Time Limit: 120.000 seconds
    Current Cumulative DPC Time: 26.906 seconds
    Single DPC Time Limit: 20.000 seconds
    Current Single DPC Time: 0.000 seconds

    Pending DPCs:
    ----------------------------------------
    CPU Type      KDPC       Function
     0: Normal  : 0xffffe00152bafb20 0xfffff800608b6ae0 vmbkmclr!InpProcessingDpcRoutine
     0: Normal  : 0xffffe00149e628e8 0xfffff80060014c60 NDIS!ndisInterruptDpc
     0: Normal  : 0xffffe00149d1f8e8 0xfffff80060014c60 NDIS!ndisInterruptDpc
     0: Normal  : 0xffffe00149ddf8e8 0xfffff80060014c60 NDIS!ndisInterruptDpc

    On CPU14 they were running for almost 1.8 seconds

    --------------------------------------------------

    CPU#14

    --------------------------------------------------

    Current DPC: netft!NetftWatchdogTimerDpc (Normal DPC)

    Debugger Saved IRQL: 2

    Cumulative DPC Time Limit: 120.000 seconds

    Current Cumulative DPC Time: 1.828 seconds

    Single DPC Time Limit: 20.000 seconds

    Current Single DPC Time: 0.000 seconds

    The thread that is currently (RUNING) not making forward progress because CPU0 is busy processing DPCs from the network

    THREAD ffffe00147e1f040  Cid 0004.01a4  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0

    Not impersonating

    DeviceMap                 ffffc000c360dca0

    Owning Process            ffffe001463c3900       Image:         System

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      52759634       Ticks: 52 (0:00:00:00.812)

    Context Switch Count      11411046       IdealProcessor: 0            

    UserTime                  00:00:00.000

    KernelTime                00:09:27.515

    Win32 Start Address vmbusr!AwWorkerThread (0xfffff80060a1d9f0)

    Stack Init ffffd001e478add0 Current ffffd001e478a7b0

    Base ffffd001e478b000 Limit ffffd001e4785000 Call 0

    Priority 15 BasePriority 8 UnusualBoost 7 ForegroundBoost 0 IoPriority 2 PagePriority 5

    # Child-SP          RetAddr           Call Site

    00 fffff803`7df681b0 fffff800`608b9007 vmbusr!BusChSendInterrupt+0x7

    01 (Inline Function) --------`-------- vmbkmclr!KmclSendSignal+0xf

    02 (Inline Function) --------`-------- vmbkmclr!OutpProcessRingResult+0x29

    03 (Inline Function) --------`-------- vmbkmclr!OutpTrySendControlPacket+0x227

    04 (Inline Function) --------`-------- vmbkmclr!OutSendPacket+0x2ac

    05 fffff803`7df681e0 fffff800`6091f89a vmbkmclr!VmbPacketSendWithTransferPageRanges+0x307

    06 fffff803`7df68290 fffff800`6091f2ee vmswitch!VmsVmNicPvtRndisHostMessageSend+0xfa

    07 fffff803`7df68350 fffff800`6091ec7f vmswitch!RndisDevHostDeviceIndicatePackets+0x61e

    08 (Inline Function) --------`-------- vmswitch!RndisDevDeviceIndicatePackets+0x25

    09 fffff803`7df68540 fffff800`6092ed39 vmswitch!VmsVmNicPvtPacketForward+0x17f

    0a fffff803`7df68750 fffff800`6092bb0a vmswitch!VmsRouterDeliverNetBufferLists+0x5c9

    0b fffff803`7df68830 fffff800`60012a53 vmswitch!VmsExtPtReceiveNetBufferLists+0x13a

    0c fffff803`7df68890 fffff800`60012f19 NDIS!ndisMIndicateNetBufferListsToOpen+0x123

    0d (Inline Function) --------`-------- NDIS!ndisMDispatchReceiveNetBufferListsInternal+0x27e

    0e fffff803`7df68950 fffff800`600136b2 NDIS!ndisMTopReceiveNetBufferLists+0x2c9

    0f (Inline Function) --------`-------- NDIS!ndisIterativeDPInvokeHandlerOnTracker+0x2d3

    10 (Inline Function) --------`-------- NDIS!ndisInvokeNextReceiveHandler+0x64d

    11 (Inline Function) --------`-------- NDIS!ndisMIndicateReceiveNetBufferListsInternal+0x6a2

    12 fffff803`7df689e0 fffff800`6091963e NDIS!NdisMIndicateReceiveNetBufferLists+0x732

    13 fffff803`7df68bd0 fffff800`60918aea vmswitch!VmsExtMpIndicatePackets+0x96

    14 fffff803`7df68c10 fffff800`60017f81 vmswitch!VmsExtMpSendNetBufferLists+0x47a

    15 (Inline Function) --------`-------- NDIS!ndisMSendNBLToMiniportInternal+0xca

    16 (Inline Function) --------`-------- NDIS!ndisMSendNBLToMiniport+0xca

    17 (Inline Function) --------`-------- NDIS!ndisCallSendHandler+0x24d

    18 (Inline Function) --------`-------- NDIS!ndisIterativeDPInvokeHandlerOnTracker+0x27c

    19 (Inline Function) --------`-------- NDIS!ndisInvokeNextSendHandler+0x417

    1a (Inline Function) --------`-------- NDIS!ndisSendNBLToFilter+0x497

    1b (Inline Function) --------`-------- NDIS!ndisMTopSendNetBufferLists+0x4a5

    1c fffff803`7df68da0 fffff800`6091e347 NDIS!NdisSendNetBufferLists+0x551

    1d fffff803`7df68f90 fffff800`6091de14 vmswitch!VmsExtPtRouteNetBufferLists+0x377

    1e fffff803`7df69060 fffff800`60012a53 vmswitch!VmsPtNicReceiveNetBufferLists+0x3c4

    1f fffff803`7df691c0 fffff800`60012f19 NDIS!ndisMIndicateNetBufferListsToOpen+0x123

    20 (Inline Function) --------`-------- NDIS!ndisMDispatchReceiveNetBufferListsInternal+0x27e

    21 fffff803`7df69280 fffff800`600136b2 NDIS!ndisMTopReceiveNetBufferLists+0x2c9

    22 (Inline Function) --------`-------- NDIS!ndisIterativeDPInvokeHandlerOnTracker+0x2d3

    23 (Inline Function) --------`-------- NDIS!ndisInvokeNextReceiveHandler+0x64d

    24 (Inline Function) --------`-------- NDIS!ndisMIndicateReceiveNetBufferListsInternal+0x6a2

    25 fffff803`7df69310 fffff800`61a0f814 NDIS!NdisMIndicateReceiveNetBufferLists+0x732

    26 fffff803`7df69500 fffff800`61a0f23e NdisImPlatform!implatTryToIndicateReceiveNBLs+0x1e8

    27 fffff803`7df69570 fffff800`60012a53 NdisImPlatform!implatReceiveNetBufferLists+0x1a2

    28 fffff803`7df695f0 fffff800`60012f19 NDIS!ndisMIndicateNetBufferListsToOpen+0x123

    29 (Inline Function) --------`-------- NDIS!ndisMDispatchReceiveNetBufferListsInternal+0x27e

    2a fffff803`7df696b0 fffff800`60013094 NDIS!ndisMTopReceiveNetBufferLists+0x2c9

    2b (Inline Function) --------`-------- NDIS!ndisInvokeNextReceiveHandler+0x2f

    2c (Inline Function) --------`-------- NDIS!ndisMIndicateReceiveNetBufferListsInternal+0x84

    2d fffff803`7df69740 fffff800`606251c4 NDIS!NdisMIndicateReceiveNetBufferLists+0x114

    2e fffff803`7df69930 fffff800`60625a9d e1i63x64!RECEIVE::RxIndicateNBLs+0xd4

    2f fffff803`7df69970 fffff800`60618150 e1i63x64!RECEIVE::RxProcessInterrupts+0x19d

    30 fffff803`7df699f0 fffff800`6061857e e1i63x64!INTERRUPT::MsgIntDpcTxRxProcessing+0x1c0

    31 fffff803`7df69a60 fffff800`60617b78 e1i63x64!INTERRUPT::MsgIntMessageInterruptDPC+0x13e

    32 fffff803`7df69ac0 fffff800`60014e02 e1i63x64!INTERRUPT::MiniportMessageInterruptDPC+0x28

    33 (Inline Function) --------`-------- NDIS!ndisMiniportDpc+0x110

    34 fffff803`7df69b00 fffff803`7c342cd0 NDIS!ndisInterruptDpc+0x1a3

    35 fffff803`7df69be0 fffff803`7c341f87 nt!KiExecuteAllDpcs+0x1b0

    36 fffff803`7df69d30 fffff803`7c3cbad5 nt!KiRetireDpcList+0xd7

    37 fffff803`7df69fb0 fffff803`7c3cb8d9 nt!KyRetireDpcList+0x5

    38 ffffd001`e478aa10 fffff803`7c3cd9fa nt!KiDispatchInterruptContinue

    39 ffffd001`e478aa40 fffff803`7c343cd3 nt!KiDpcInterrupt+0xca

    3a (Inline Function) --------`-------- nt!KzLowerIrql+0x9

    3b ffffd001`e478abd0 fffff800`608b7d4b nt!KeInsertQueueDpc+0x1e3

    3c (Inline Function) --------`-------- vmbkmclr!InpReleaseLockAndPerformWork+0xe8

    3d ffffd001`e478ac50 fffff800`608b6a90 vmbkmclr!InpTransitionRunningQueue+0x17b

    3e ffffd001`e478ac90 fffff800`60a1daaf vmbkmclr!InpProcessingWorkerRoutine+0xf0

    3f (Inline Function) --------`-------- vmbusr!AwRunWorkItem+0x29

    40 ffffd001`e478ace0 fffff803`7c379c70 vmbusr!AwWorkerThread+0xbf

    41 ffffd001`e478ad40 fffff803`7c3cefc6 nt!PspSystemThreadStartup+0x58

    42 ffffd001`e478ada0 00000000`00000000 nt!KxStartSystemThread+0x16

    There are several threads on the CPU0 ready to run

    Processor 0: Ready Threads at priority 15

        THREAD ffffe8006e2d2080  Cid 0c08.2894  Teb: 00007ff7ac976000 Win32Thread: 0000000000000000 READY on processor 0

        THREAD ffffe0014e53f880  Cid 0c08.0d58  Teb: 00007ff7ac978000 Win32Thread: 0000000000000000 READY on processor 0

        THREAD ffffe8006df75880  Cid 0c08.247c  Teb: 00007ff7ac972000 Win32Thread: fffff901407ac010 READY on processor 0

        THREAD ffffe8006bd24880  Cid 0c08.177c  Teb: 00007ff7ac96a000 Win32Thread: 0000000000000000 READY on processor 0

        THREAD ffffe00148889880  Cid 0004.0dc8  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

        THREAD ffffe8006a321880  Cid 0820.1e64  Teb: 00007ff671f88000 Win32Thread: fffff901406c4b60 READY on processor 0

        THREAD ffffe00150278040  Cid 0004.045c  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

        THREAD ffffe0014d98e300  Cid 0004.2160  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

        THREAD ffffe8006e65b880  Cid 0004.1b00  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

    I see a clussvc thread has been sitting in the READY state on CPU0 for almost 58 seconds. It is possible that this thread eventually was supposed to send the heartbeat.

    THREAD ffffe0014d98e300  Cid 0004.2160  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

    Not impersonating

    DeviceMap                 ffffc000c360dca0

    Owning Process            ffffe001463c3900       Image:         System

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      52755922       Ticks: 3764 (0:00:00:58.812)

    Context Switch Count      56741          IdealProcessor: 0            

    UserTime                  00:00:00.000

    KernelTime                00:00:01.109

    Win32 Start Address nt!ExpWorkerThread (0xfffff8037c314100)

    Stack Init ffffd001eb77efd0 Current ffffd001eb77ebc0

    Base ffffd001eb77f000 Limit ffffd001eb779000 Call 0

    Priority 15 BasePriority 7 UnusualBoost 8 ForegroundBoost 0 IoPriority 2 PagePriority 5

    Child-SP          RetAddr           Call Site

    ffffd001`eb77ec00 fffff803`7c29fe9b nt!KiSwapContext+0x76

    (Inline Function) --------`-------- nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff803`7c29fe9b)

    ffffd001`eb77ed40 fffff803`7c29fb7b nt!KiCheckForThreadDispatch+0x153

    ffffd001`eb77ed80 fffff803`7c29f98d nt!KeSetSystemGroupAffinityThread+0xfb

    ffffd001`eb77edd0 fffff803`7c2f5a4b nt!KeGenericProcessorCallback+0xdd

    ffffd001`eb77ef40 fffff803`7c3cc2f7 nt!KeGenericCallDpc+0x27

    ffffd001`eb77ef80 fffff803`7c3cc2bd nt!KySwitchKernelStackCallout+0x27 (TrapFrame @ ffffd001`eb77ee40)

    ffffd001`eb8f97f0 fffff803`7c2b6a7d nt!KiSwitchKernelStackContinue

    ffffd001`eb8f9810 fffff803`7c3024ab nt!KeExpandKernelStackAndCalloutInternal+0x2fd

    ffffd001`eb8f9900 fffff803`7c32634b nt!MiSwapStackPage+0x2d7

    ffffd001`eb8f99d0 fffff803`7c304cee nt!MiClaimPhysicalRun+0x44f

    ffffd001`eb8f9a50 fffff803`7c3043cd nt!MiFindContiguousPages+0x282

    ffffd001`eb8f9bb0 fffff803`7c723a8c nt!MiRebuildLargePage+0x99

    ffffd001`eb8f9c40 fffff803`7c31438c nt!MiRebuildLargePages+0x88

    ffffd001`eb8f9c90 fffff803`7c379c70 nt!ExpWorkerThread+0x28c

    ffffd001`eb8f9d40 fffff803`7c3cefc6 nt!PspSystemThreadStartup+0x58

    ffffd001`eb8f9da0 00000000`00000000 nt!KxStartSystemThread+0x16

    Because of DPC storm on CPU0 that clussvc did not send to netft heartbeat for 1 minute and netft finally bugchecked the machine.

    I would be curios to learn what scenario are you running that leads to that.

    As a remediation I see several options

    • Increase ClussvcHangTimeout (cluster public property) to a value above “Cumulative DPC Time Limit”. You can set it to 135 (2 minutes 15 seconds). In that case either machine survives or it will be bugchecked by DPC watchdog, in any case it might be better than 9e from netft,sys.
    • Look at you NIC settings. Looks like you are using  4 “Intel(R) Gigabit ET Quad Port Server Adapter” hooked to VMswitch and 3 “Broadcom BCM5709C NetXtreme II GigE” available for the host. Perhaps you can tune up how traffic from these NICs is load balanced across CPUs using RSS (Receive Side Scaling) and VMQ. Please also check that you have latest and greatest drivers for these NICs.       
    • I see you are using Windows 2012 R2. Please make sure you have latest fixes: https://support.microsoft.com/kb/2920151?wa=wsignin1.0.

    I would second Elden’s advise to contact Microsoft support for this case. You should share with them your dump, and you are welcome to share analysis above.


    Regards, Vladimir Petter, Microsoft Corporation

    • Edited by Vladimir Petter Tuesday, December 16, 2014 4:11 AM fixed formating
    Tuesday, December 16, 2014 3:58 AM
  • Hello Vladimir,

    it seems that we are having a very similar problem with a new installation at customer site. Maybe you can help us locating what the Problem is?

    You can find the latest dump file here: https://www.dropbox.com/s/i6m0ic49epbxk6g/MEMORY.zip?dl=0

    There is a 2012 R2 Cluster running on a DataCore SANsymphonyV virtual SAN. Once we start to initialize new virtual disks the cluster crashes down.

    Would be great if you could help us, because the System will not go productive until the problem is fixed...

    Thanks a lot,

    Martin 

    Wednesday, August 26, 2015 8:25 AM
  • Hello Martin,

    Netft bugchecked machine because RHS.exe was not able to complete termination in 20 minutes. One of the threads of that process is stuck in kernel waiting for an IO to complete by the storage.

    I would suggest to share that dump with support of folks who implemented DcsPoll.sys. Or see if they have any updates.

        Loaded symbol image file: DcsPoll.sys
        Image path: \SystemRoot\System32\drivers\DcsPoll.sys
        Image name: DcsPoll.sys
        Timestamp:        Mon Aug 10 11:26:39 2015 (55C8ECDF)
        CheckSum:         00013D53
        ImageSize:        00016000
        File version:     15.0.300.5312
        Product version:  15.0.300.5312
        File flags:       0 (Mask 3F)
        File OS:          40004 NT Win32
        File type:        2.0 Dll
        File date:        00000000.00000000
        Translations:     0409.04b0
        CompanyName:      DataCore Software Corporation
        ProductName:      DcsPoll.sys
        InternalName:     DcsPoll.sys
        OriginalFilename: DcsPoll.sys
        ProductVersion:   15.0.300.5312
        FileVersion:      15.0.300.5312
        FileDescription:  DcsPoll.sys
        LegalCopyright:   Copyright © 1998-2015 DataCore Software Corporation. All Rights Reserved.
        Comments:         All Rights Reserved.

    System Uptime: 4 days 2:58:57.605, and this thread is doing some busy loop.

    THREAD ffffe000e1174880  Cid 0004.0248  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2e
    Not impersonating
    DeviceMap                 ffffc0015500c0b0
    Owning Process            ffffe000e10b05c0       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      22805606       Ticks: 0
    Context Switch Count      1196804        IdealProcessor: 44            
    UserTime                  00:00:00.000
    KernelTime                4 Days 02:06:17.359
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for DcsPoll.sys -
    Win32 Start Address DcsPoll (0xfffff8013fa85504)
    Stack Init ffffd00023569c90 Current ffffd00023569810
    Base ffffd0002356a000 Limit ffffd00023564000 Call 0
    Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`23569b20 fffff801`3fa856a0 DcsPoll+0x5b2d
    ffffd000`23569bc0 fffff803`c931536c DcsPoll+0x66a0
    ffffd000`23569c00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58@ 5906]
    ffffd000`23569c60 00000000`00000000 nt!KxStartSystemThread+0x16

    I see they are also busy spinning on the 3 other CPUs

    THREAD ffffe000e3dd1380  Cid 0004.0644  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 4
    Not impersonating
    DeviceMap                 ffffc0015500c0b0
    Owning Process            ffffe000e10b05c0       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      22805606       Ticks: 0
    Context Switch Count      37419169       IdealProcessor: 4            
    UserTime                  00:00:00.000
    KernelTime                00:23:29.843
    Win32 Start Address DcsPool (0xfffff80140a71d44)
    Stack Init ffffd0002118fc90 Current ffffd0002118f890
    Base ffffd00021190000 Limit ffffd0002118a000 Call 0
    Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`2118f920 fffff801`40a90462 DcsPool+0xe40a
    ffffd000`2118fb20 fffff801`40a903ed DcsPool+0x23462
    ffffd000`2118fb60 fffff801`40a9cd9e DcsPool+0x233ed
    ffffd000`2118fb90 fffff801`40a71dde DcsPool+0x2fd9e
    ffffd000`2118fbc0 fffff803`c931536c DcsPool+0x4dde
    ffffd000`2118fc00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58
    ffffd000`2118fc60 00000000`00000000 nt!KxStartSystemThread+0x16

    THREAD ffffe8008628a040  Cid 0004.26f0  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 10
    Not impersonating
    DeviceMap                 ffffc0015500c0b0
    Owning Process            ffffe000e10b05c0       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      22805606       Ticks: 0
    Context Switch Count      2805765        IdealProcessor: 16            
    UserTime                  00:00:00.000
    KernelTime                00:19:47.625
    Win32 Start Address DcsFcEng (0xfffff8013f609cf4)
    Stack Init ffffd0002b121c90 Current ffffd0002b121810
    Base ffffd0002b122000 Limit ffffd0002b11c000 Call 0
    Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`2b121968 fffff801`3fa6f6a4 DcsSup+0x13eb
    ffffd000`2b121970 fffff801`3f60a3ce DcsSup!DcsSup::memset+0x48
    ffffd000`2b1219a0 fffff801`3f60a208 DcsFcEng+0xa3ce
    ffffd000`2b121a40 fffff801`3f607a6c DcsFcEng+0xa208
    ffffd000`2b121ad0 fffff801`3f607ead DcsFcEng+0x7a6c
    ffffd000`2b121b00 fffff801`3f609c11 DcsFcEng+0x7ead
    ffffd000`2b121b40 fffff801`3f609cfd DcsFcEng+0x9c11
    ffffd000`2b121bd0 fffff803`c931536c DcsFcEng+0x9cfd
    ffffd000`2b121c00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58
    ffffd000`2b121c60 00000000`00000000 nt!KxStartSystemThread+0x16

    THREAD ffffe000e1174040  Cid 0004.0244  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2c
    Not impersonating
    DeviceMap                 ffffc0015500c0b0
    Owning Process            ffffe000e10b05c0       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      22805606       Ticks: 0
    Context Switch Count      767536         IdealProcessor: 16            
    UserTime                  00:00:00.000
    KernelTime                4 Days 02:41:57.718
    Win32 Start Address DcsPoll (0xfffff8013fa85504)
    Stack Init ffffd000269e2c90 Current ffffd000269e26f0
    Base ffffd000269e3000 Limit ffffd000269dd000 Call 0
    Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`269e2970 fffff801`40db3264 DcsIs+0x1d4ba
    ffffd000`269e29a0 fffff801`414d4c33 DcsIs+0x1d264
    ffffd000`269e29d0 fffff801`3fb815aa DcsiMgr+0x20c33
    ffffd000`269e2a10 fffff801`3f621150 DcsShim+0xa5aa
    ffffd000`269e2a80 fffff801`3f61d82a DcsFcEng+0x21150
    ffffd000`269e2ac0 fffff801`3f60ed11 DcsFcEng+0x1d82a
    ffffd000`269e2af0 fffff801`3fa84b71 DcsFcEng+0xed11
    ffffd000`269e2b20 fffff801`3fa856a0 DcsPoll+0x5b71
    ffffd000`269e2bc0 fffff803`c931536c DcsPoll+0x66a0
    ffffd000`269e2c00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58
    ffffd000`269e2c60 00000000`00000000 nt!KxStartSystemThread+0x16

    In this dump I see 3 IOs outstanding to disks

     

      DO ffffe80085c2a4a0   Ext ffffe80085c2a5f0   Adapter ffffe000e48b31a0   Working
       Vendor: DataCore   Product: Mirror Disk        SCSI ID: (0, 3, 1)  
       Claimed Enumerated
       SlowLock Free   RemLock 2   PageCount 0
       QueueTagList: ffffe80085c2a6b0      Outstanding: Head ffffe000e4ff0050  Tail ffffe000e4ff0050  Timeout 50
       DeviceQueue ffffe80085c2a6e0   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0  
       IO Gateway: Busy Count 0   Pause Count 0
       Requests: Outstanding 1   Device 0   ByPass 0


    [Device-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Bypass-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Outstanding Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ffffe80084c69350  [SCSI]     ffffe800850bd220  ffffe000e4ff0020  SCSI/UNMAP        ffffe80084222d00  0000000000000000  50

    [Completed Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

       DO ffffe800851d1480   Ext ffffe800851d15d0   Adapter ffffe000e48b31a0   Working
       Vendor: DataCore   Product: Virtual Disk       SCSI ID: (0, 0, 1)  
       Claimed Enumerated
       SlowLock Free   RemLock 2   PageCount 0
       QueueTagList: ffffe800851d1690      Outstanding: Head ffffe000e53f1c10  Tail ffffe000e53f1c10  Timeout 50
       DeviceQueue ffffe800851d16c0   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0  
       IO Gateway: Busy Count 0   Pause Count 0
       Requests: Outstanding 1   Device 0   ByPass 0


    [Device-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Bypass-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Outstanding Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ffffe80085c142f0  [SCSI]     ffffe80083f7dec0  ffffe000e53f1be0  SCSI/UNMAP        ffffe80085d7b750  0000000000000000  50

    [Completed Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
       DO ffffe80086beb060   Ext ffffe80086beb1b0   Adapter ffffe000e48b31a0   Working
       Vendor: DataCore   Product: Mirror Disk        SCSI ID: (0, 3, 2)  
       Claimed Enumerated
       SlowLock Free   RemLock 2   PageCount 0
       QueueTagList: ffffe80086beb270      Outstanding: Head ffffe000e4ff7050  Tail ffffe000e4ff7050  Timeout 50
       DeviceQueue ffffe80086beb2a0   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0  
       IO Gateway: Busy Count 0   Pause Count 0
       Requests: Outstanding 1   Device 0   ByPass 0

    [Device-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Bypass-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Outstanding Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ffffe000e1392ee0  [SCSI]     ffffe80084579890  ffffe000e4ff7020  SCSI/UNMAP        ffffe000edf20360  0000000000000000  50

    [Completed Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

       DO ffffe80089d6c7f0   Ext ffffe80089d6c940   Adapter ffffe000e48b31a0   Working
       Vendor: DataCore   Product: Virtual Disk       SCSI ID: (0, 1, 0)  
       Claimed Enumerated
       SlowLock Free   RemLock 2   PageCount 0
       QueueTagList: ffffe80089d6ca00      Outstanding: Head ffffe000e53f8c10  Tail ffffe000e53f8c10  Timeout 50
       DeviceQueue ffffe80089d6ca30   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0  
       IO Gateway: Busy Count 0   Pause Count 0
       Requests: Outstanding 1   Device 0   ByPass 0


    [Device-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Bypass-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Outstanding Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ffffe800864be010  [SCSI]     ffffe800856d2df0  ffffe000e53f8be0  SCSI/UNMAP        ffffe80084b7fa10  0000000000000000  50

    [Completed Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

     Since this is scsi/unmap my guess this is some solution that provides SSD/NVME based cache.


    Regards, Vladimir Petter, Microsoft Corporation


    Wednesday, August 26, 2015 7:20 PM
  • Hello Vladimir,

    Could you please help in reviewing memory dump uploaded at below link? 

    https://drive.google.com/file/d/0B1Z6Q5Mfd7nid2lDWGJKLVZDWWs/view?usp=sharing

    I have 2 node windows 2012 R2 active/passive cluster setup on Hyper-V. cluster nodes generate crash dump with bug check 09e very frequently.

    Thanks & Regards,

    Suchit Patil

    Wednesday, September 9, 2015 3:41 PM
  • Hello Suchit,

    Cluster bugchecked machine because Resource Host Monitor has not completed termination in 20 minutes. One of the RHS threads is stuck in the kernel for about 20 minutes. Looks like things are getting stuck in TmXPFlt.sys.

     

    As a remediation you might want to uninstall this product until issue is resolved. I would also suggest to talk to support of the company that provided that solution to see if they have a fix and to make sure they are aware of that issue.

     

    I see lots of threads in the system are stuck with a similar call stack.

     

        Loaded symbol image file: TmXPFlt.sys

        Image path: \??\C:\Program Files (x86)\Trend Micro\OfficeScan Client\TmXPFlt.sys

        Image name: TmXPFlt.sys

        Timestamp:        Sat Aug 30 06:11:38 2014 (5401CD8A)

        CheckSum:         0005DDB6

        ImageSize:        0006C000

        Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

     

            THREAD fffffa806e2d6080  Cid 0d94.0f6c  Teb: 000007f7a0c6e000 Win32Thread: fffff90102e3ab80 WAIT: (Executive) KernelMode Non-Alertable

                fffff88007394440  SynchronizationEvent

            IRP List:

                fffffa8033b24010: (0006,03e8) Flags: 00000884  Mdl: 00000000

            Not impersonating

            DeviceMap                 fffff8a00000c310

            Owning Process            fffffa8032dd9980       Image:         rhs.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      750939         Ticks: 76757 (0:00:19:59.328)

            Context Switch Count      378            IdealProcessor: 5            

            UserTime                  00:00:00.015

            KernelTime                00:00:00.015

            Win32 Start Address 0x000007f7a13cbc24

            Stack Init fffff88007395c90 Current fffff88007394190

            Base fffff88007396000 Limit fffff88007390000 Call 0

            Priority 14 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

            Child-SP          RetAddr           Call Site

            fffff880`073941d0 fffff800`342aff79 nt!KiSwapContext+0x76

            (Inline Function) --------`-------- nt!KiSwapThread+0xfa (Inline Function @ fffff800`342aff79)

            fffff880`07394310 fffff800`342ac21f nt!KiCommitThreadWait+0x229

            fffff880`07394380 fffff880`05050457 nt!KeWaitForSingleObject+0x1cf

            fffff880`07394410 fffff880`050460df TmXPFlt+0xe457

            fffff880`07394470 fffff880`04384df5 TmXPFlt+0x40df

            fffff880`07394590 fffff880`016ae844 TmPreFlt!TmpQueryFullName+0xd61

            fffff880`07394660 fffff880`016afa6c fltmgr!FltpPerformPreCallbacks+0x324

            fffff880`07394770 fffff880`016da349 fltmgr!FltpPassThroughInternal+0x8c

            fffff880`073947a0 fffff800`34655228 fltmgr!FltpCreate+0x339

            (Inline Function) --------`-------- nt!IoCallDriverWithTracing+0xc3 (Inline Function @ fffff800`34655228)

            fffff880`07394850 fffff800`34668470 nt!IopParseDevice+0x173c

            fffff880`07394a30 fffff800`34656978 nt!ObpLookupObjectName+0x644

            fffff880`07394b40 fffff800`3466930e nt!ObOpenObjectByName+0x258

            fffff880`07394c10 fffff800`3463f96c nt!IopCreateFile+0x37c

            fffff880`07394cb0 fffff800`34284d53 nt!NtOpenFile+0x58

            fffff880`07394d40 fffff800`34289f30 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07394db0)

            fffff880`07394f48 fffff800`34626a68 nt!KiServiceLinkage

            fffff880`07394f50 fffff800`34284d53 nt!NtCreateUserProcess+0x400

            fffff880`07395a90 000007fb`572a371b nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07395b00)

            000000a4`76ced028 00000000`00000000 0x000007fb`572a371b

     

     

    I’ve also noticed several threads where TmXPFlt is trying to open a file over SMB. Perhaps all other activity is stuck behind these activities, but it is hard to tell without symbols.

     

    THREAD fffffa806e078080  Cid 0004.0c9c  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable

        fffffa803444e190  SynchronizationEvent

    IRP List:

        fffffa806f440010: (0006,01f0) Flags: 00000884  Mdl: 00000000

    Impersonation token:  fffff8a00dc72270 (Level Impersonation)

    DeviceMap                 fffff8a00e5514b0

    Owning Process            fffffa8030bc9980       Image:         System

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      728673         Ticks: 99023 (0:00:25:47.234)

    Context Switch Count      42000          IdealProcessor: 7            

    UserTime                  00:00:00.000

    KernelTime                00:00:40.156

    Win32 Start Address TmXPFlt (0xfffff8800504dddc)

    Stack Init fffff88009395fd0 Current fffff88009395b80

    Base fffff88009396000 Limit fffff88009390000 Call 0

    Priority 12 BasePriority 8 UnusualBoost 3 ForegroundBoost 0 IoPriority 2 PagePriority 5

    Child-SP          RetAddr           Call Site

    fffff880`09395bc0 fffff800`342aff79 nt!KiSwapContext+0x76

    (Inline Function) --------`-------- nt!KiSwapThread+0xfa (Inline Function @ fffff800`342aff79)

    fffff880`09395d00 fffff800`342ac21f nt!KiCommitThreadWait+0x229

    fffff880`09395d70 fffff880`056483bb nt!KeWaitForSingleObject+0x1cf

    fffff880`09395e00 fffff880`0563ffde mrxsmb10!SmbCeInitiateExchange+0x30f

    fffff880`09395e70 fffff880`043a40db mrxsmb10!MRxSmbCreate+0x8d6

    fffff880`09395f50 fffff800`342804a7 mrxsmb!SmbpShellCreateWithNewStack+0x1b

    fffff880`09395f80 fffff800`3428046d nt!KySwitchKernelStackCallout+0x27 (TrapFrame @ fffff880`09395e40)

    fffff880`0664b880 fffff800`342c786e nt!KiSwitchKernelStackContinue

    fffff880`0664b8a0 fffff800`34243fc5 nt!KeExpandKernelStackAndCalloutInternal+0x20e

    fffff880`0664b9a0 fffff880`043a40aa nt!KeExpandKernelStackAndCallout+0x15

    fffff880`0664b9e0 fffff880`01ba8620 mrxsmb!SmbShellCreate+0x4a

    fffff880`0664ba10 fffff880`01ba547d rdbss!RxCollapseOrCreateSrvOpen+0x210

    fffff880`0664baa0 fffff880`01ba69ab rdbss!RxCreateFromNetRoot+0x63d

    fffff880`0664bbd0 fffff880`01b6e652 rdbss!RxCommonCreate+0x15b

    fffff880`0664bc70 fffff880`01ba059b rdbss!RxFsdCommonDispatch+0x522

    fffff880`0664bdd0 fffff880`043d209c rdbss!RxFsdDispatch+0xcb

    fffff880`0664be30 fffff880`01f37161 mrxsmb!MRxSmbFsdDispatch+0x8c

    fffff880`0664be70 fffff880`01f34215 mup!MupiCallUncProvider+0x1b1

    fffff880`0664bee0 fffff880`01f32475 mup!MupStateMachine+0xb6

    fffff880`0664bf10 fffff880`016b04ee mup!MupCreate+0x165

    fffff880`0664bf80 fffff880`016da35d fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x25e

    fffff880`0664c020 fffff800`34655228 fltmgr!FltpCreate+0x34d

    (Inline Function) --------`-------- nt!IoCallDriverWithTracing+0xc3 (Inline Function @ fffff800`34655228)

    fffff880`0664c0d0 fffff800`34668470 nt!IopParseDevice+0x173c

    fffff880`0664c2b0 fffff800`34656978 nt!ObpLookupObjectName+0x644

    fffff880`0664c3c0 fffff800`3466930e nt!ObOpenObjectByName+0x258

    fffff880`0664c490 fffff800`34669a59 nt!IopCreateFile+0x37c

    fffff880`0664c530 fffff800`34284d53 nt!NtCreateFile+0x79 fffff880`0664c5c0 fffff800`34289f30 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0664c630) fffff880`0664c7c8 fffff880`04fb1651 nt!KiServiceLinkage

    fffff880`0664c7d0 fffff880`04fb243a VSApiNt!VSSwapShortTable+0x721

    fffff880`0664c840 fffff880`05049d16 VSApiNt!VSKDZwCreateFile+0x5a

    fffff880`0664c8b0 fffff880`0504c1b9 TmXPFlt+0x7d16

    fffff880`0664c980 fffff880`0504ce6e TmXPFlt+0xa1b9

    fffff880`0664c9e0 fffff880`0504da7f TmXPFlt+0xae6e

    fffff880`0664cb20 fffff880`0504def1 TmXPFlt+0xba7f

    fffff880`0664cbe0 fffff800`3422f2c5 TmXPFlt+0xbef1

    fffff880`0664cc10 fffff800`3426c656 nt!PspSystemThreadStartup+0x59 [d:\win8_ldr\minkernel\ntos\ps\psexec.c @ 5691]

    fffff880`0664cc60 00000000`00000000 nt!KxStartSystemThread+0x16 [d:\win8_ldr\minkernel\ntos\ke\amd64\threadbg.asm @ 75]

     

        Loaded symbol image file: VSApiNt.sys

        Image path: \??\C:\Program Files (x86)\Trend Micro\OfficeScan Client\VSApiNt.sys

        Image name: VSApiNt.sys

        Timestamp:        Sat Aug 30 06:03:46 2014 (5401CBB2)

        CheckSum:         0024476C

        ImageSize:        00238000

        Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

     

    Here is a list of all opens that are stack over SMB.

     

    RxContext        RDR [  Maj,  Min] Irp              Thread           FCB

     

    fffffa8032dd4bb0   0 [ 0x 0, 0x 0] fffffa806e7df010 fffffa806eea9440 0000000000000000

           16:08.471   CREATE          '\HMEL-BTH-DC03.hmel.int\IPC$'

     

    fffffa8034749950   0 [ 0x 0, 0x 0] fffffa806f624d90 fffffa80342c4b00 fffff88001b998c0

           13:29.967   CREATE          '<<empty>>'

     

    fffffa8033ce44b0   0 [ 0x 0, 0x 0] fffffa806f130d10 fffffa80338bbb00 fffff8a00e5bf010

           25:47.248   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E4\F0E43290E239950FABB7730FEA0B4421.DVS'

     

    fffffa8033ee15e0   0 [ 0x 0, 0x 0] fffffa8033f27400 fffffa806f4a0900 fffff8a00ee432a0

           25:47.248   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-07\6\125\6125F1071BA45DE8BA67A9D1E7004ED1~90~9F3EAD6D~00~1.DVSSP'

     

    fffffa8034459200   0 [ 0x 0, 0x 0] fffffa806ef81be0 fffffa803413cb00 fffff8a00d5d4670

           25:47.247   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E4\F0E4B8A43ECF2FB81443FFE354A7A931.DVS'

     

    fffffa8033ac7950   2 [ 0x e, 0x 0] fffffa8032de64f0 fffffa8033896080 fffff8a00f667610

           76:14.999   IOCTL           '\2'

     

    fffffa8031f94010   2 [ 0x e, 0x 0] fffffa80335c2010 fffffa803216b600 fffff8a00f667610

           44:46.553   IOCTL           '\2'

     

    fffffa80335e05a0   0 [ 0x 0, 0x 0] fffffa803453ec20 fffffa8033ff5080 fffff8a00daac010

           25:47.250   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-16\F\051\F0516C6C8D4D24CED66C01341EBC0F71.DVS'

     

    fffffa8033b59610   0 [ 0x 0, 0x 0] fffffa80342b9600 fffffa806f8af5c0 fffff8a0118532a0

           25:47.248   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D856FB14A3D5416A61EF0D7116911~0F~C97B4131~00~1.DVSSP'

     

    fffffa8033c93240   0 [ 0x 0, 0x 0] fffffa8033b2c580 fffffa8034381b00 fffff8a00da1d600

           25:47.248   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2014\07-17\6\0A5\60A5BB94AFB6B48F2B15E53045337701~35~1FD28490~00~1.DVSSP'

     

    fffffa806ef187f0   0 [ 0x 0, 0x 0] fffffa80340a4630 fffffa806e0c0080 fffff8a010c947e0

           25:46.524   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-17\D\06B\D06B43ABD3EEB44CA7CB2FE2CAB27721~39~6A552458~00~1.DVSSP'

     

    fffffa80336269a0   0 [ 0x 0, 0x 0] fffffa8033528780 fffffa806e0fa080 fffff8a012ccf010

           25:46.499   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\03-05\3\03B\303B66B5ACE86D5696529DF90977A8F1~6D~5FD6F55D~00~1.DVSSP'

     

    fffffa8034749cb0   0 [ 0x 0, 0x 0] fffffa806f030750 fffffa806f53e080 fffff88001b998c0

           13:29.967   CREATE          '<<empty>>'

     

    fffffa806f35fc20   0 [ 0x 0, 0x 0] fffffa80343ced10 fffffa803417fb00 fffff8a00d22e350

           25:47.247   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D8ED60F35534A9D9860B16528A501~C4~46589CF8~00~1.DVSSP'

     

    fffffa803444e010   0 [ 0x 0, 0x 0] fffffa806f440010 fffffa806e078080 fffff8a00dd3e2f0

           25:47.241   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D8CD7E12525CA7EE50328316C7AF1~85~6174E8BD~00~1.DVSSP'

     

    fffffa8034151010   0 [ 0x 0, 0x 0] fffffa8034366690 fffffa806e0fab00 fffff8a00d89d8e0

           25:47.234   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E4\F0E4B2D443D4F8D070887D83728C6411.DVS'

     

    fffffa8033fe7cb0   0 [ 0x 0, 0x 0] fffffa8033bf2d80 fffffa806e031b00 fffff8a00d66b500

           25:47.234   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D83CE4F080D152AB519841E0F9551.DVS'

     

    fffffa806e9acb30   0 [ 0x 0, 0x 0] fffffa8033468940 fffffa806e0bfb00 fffff8a00e6bf010

           25:47.234   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D8585CAAFD30258F5F9204E8B8F21.DVS'

     

    fffffa806f214cb0   0 [ 0x 0, 0x 0] fffffa8033e719a0 fffffa806f56e080 fffff8a012de3010

           25:46.500   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\06-15\5\048\50488155ACE08770781A75BBD1F269C1~29~B78492F0~00~1.DVSSP'

     

    fffffa8034747cb0   0 [ 0x 0, 0x 0] fffffa806f567380 fffffa806ebda480 fffff88001b998c0

           13:29.967   CREATE          '<<empty>>'

     

    fffffa80346e2010   0 [ 0x 0, 0x 0] fffffa8033c48730 fffffa803424ca80 fffff88001b998c0

           13:29.967   CREATE          '<<empty>>'

     

    fffffa80349fd310   0 [ 0x 0, 0x 0] fffffa8033d55b00 fffffa8033dbd100 fffff88001b998c0

            3:29.963   CREATE          '<<empty>>'

     

    fffffa803493e010   0 [ 0x 0, 0x 0] fffffa8033c3c010 fffffa80341a1b00 fffff88001b998c0

            3:29.962   CREATE          '<<empty>>'

     

    fffffa806f6fe010   0 [ 0x 0, 0x 0] fffffa806f32b680 fffffa806e0bdb00 fffff8a010d06730

           25:47.234   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E9\F0E9001E013394D9C07EF02330BFB911.DVS'

     

    fffffa806ec38010   0 [ 0x 0, 0x 0] fffffa806f07e700 fffffa806f366b00 fffff8a012ca2a80

           25:47.225   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E7\F0E756057B9D6D2ABECB8E13E3415CC1.DVS'

     

    fffffa806ee48230   0 [ 0x 0, 0x 0] fffffa806f132010 fffffa806e0be6c0 fffff8a01242a010

           25:47.014   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2014\12-20\D\0C2\D0C292F3FB7A73F70208526C10EAF491.DVS'

     

    fffffa806f78e9a0   0 [ 0x 0, 0x 0] fffffa806e8e12c0 fffffa8034337b00 fffff8a004aea010

           25:46.501   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-17\C\0E1\C0E1AAE496323E417F8B4BEFFAAE1FB1~D5~E770E121~00~1.DVSSP'

     

    fffffa806eb5a450   0 [ 0x 0, 0x 0] fffffa806ef3b6b0 fffffa806ee98b00 fffff88001b998c0

           13:29.967   CREATE          '<<empty>>'

     

    fffffa806f5f87f0   0 [ 0x 0, 0x 0] fffffa806f677010 fffffa806e0bf080 fffff8a00ffa4a60

           25:47.190   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-29\5\0BC\50BCEB2E36C8C4FABB47EC3453CB13B1~F3~00A34DF4~00~1.DVSSP'

     

    fffffa806f1d9770   0 [ 0x 0, 0x 0] fffffa806f46d390 fffffa806f048080 fffff8a01282a010

           25:47.249   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\01-16\1\124\1124F16CCD8694D455954CAABDF10111~97~5E195B14~00~1.DVSSP'

     

    fffffa806f71ecb0   0 [ 0x 0, 0x 0] fffffa806f543370 fffffa806f5cb080 fffff8a012705a80

           25:47.248   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-15\1\00F\100F697A6B99B08748C65EBD6A56BC21.DVS'

     

    fffffa806f703900   0 [ 0x 0, 0x 0] fffffa806f742010 fffffa806e0c06c0 fffff8a012b51380

           25:47.221   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-13\F\011\F0116842118398CEE7462D0AE65D7601.DVS'

     

    fffffa806eeee780   0 [ 0x 0, 0x 0] fffffa80345d6a10 fffffa806f38b300 fffff88001b998c0

            3:29.962   CREATE          '<<empty>>'

     

    fffffa806f529190   0 [ 0x 0, 0x 0] fffffa806f7f8320 fffffa806e0bab00 fffff8a012b145b0

           25:47.249   CREATE          '\Hmelbthdd\bth-evmjavsp01'

     

    fffffa806eb4b910   0 [ 0x 0, 0x 0] fffffa8034263e10 fffffa803438e080 fffff8a010b89a80

           25:47.027   CREATE          '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-16\F\054\F054CC18A0FC36B2B94C5D416455D4B1.DVS'

     

    fffffa803415c010   0 [ 0x 0, 0x 0] fffffa806f800510 fffffa806f267680 fffff88001b998c0

           13:29.967   CREATE          '<<empty>>'

     

    fffffa806ebee830   0 [ 0x 0, 0x 0] fffffa806f6fd010 fffffa8033ee4080 fffff88001b998c0

           13:29.966   CREATE          '<<empty>>'

     

    SMB client has several Irps stuck in the networking stack for a long time

     

    Time Pending  IRP

     

     25:47.251    fffffa806f775010

     16:08.472    fffffa8033fb3b90

     

    6: kd> !irp fffffa806f775010

    Irp is active with 2 stacks 1 is current (= 0xfffffa806f7750e0)

     No Mdl: No System Buffer: Thread 00000000:  Irp stack trace. 

         cmd  flg cl Device   File     Completion-Context

    >[IRP_MJ_INTERNAL_DEVICE_CONTROL(f), N/A(10)]

                0 e1 fffffa80315c6c10 00000000 fffff8800439d8b0-fffffa8033a6fd10 Success Error Cancel pending

                  \Driver\AFD   mrxsmb!SmbWskGetAddressInfoComplete

                         Args: fffffa803209e410 fffff880096e5ae0 fffffa803392d5c0 00000000

     [N/A(0), N/A(0)]

                0  0 00000000 00000000 00000000-00000000   

     

                         Args: fffff88001b97a00 fffff88001b97a00 fffffa806f775010 7184f95f

     

    6: kd> !irp fffffa8033fb3b90

    Irp is active with 2 stacks 1 is current (= 0xfffffa8033fb3c60)

     No Mdl: No System Buffer: Thread 00000000:  Irp stack trace. 

         cmd  flg cl Device   File     Completion-Context

    >[IRP_MJ_INTERNAL_DEVICE_CONTROL(f), N/A(10)]

                0 e1 fffffa80315c6c10 00000000 fffff8800439d8b0-fffffa806f77a240 Success Error Cancel pending

                  \Driver\AFD   mrxsmb!SmbWskGetAddressInfoComplete

                         Args: fffffa803209e410 fffff8800bf71c50 fffffa8033c71ec0 00000000

     [N/A(0), N/A(0)]

                0  0 00000000 00000000 00000000-00000000   

     

                         Args: fffff88001b98b00 fffff88001b98b00 fffffa8033fb3b90 718dce3a

     

    Looks like these Irps are calls from SMB client to DNS client to resolve names. NDIS should send up-call back to the DNS client service in user mode to resolve them. DNS client service is hosted in one of the svchost.exe processes. I see many threads from svchost processes are stuck in TmXPFlt so it is possible that this is what is causing the deadlock.


    Regards, Vladimir Petter, Microsoft Corporation


    Wednesday, September 9, 2015 6:56 PM
  • Thank you very much Vladimir for your immediate review of memory dump, we have taken required action on Trend Micro Antivirus and cluster nodes are under observation.


    Thursday, September 10, 2015 7:12 AM
  • Hello Vladimir,

    Could you please help me to understand what command did you use to get all thread, specifically THREAD fffffa806e078080 , also how did you see the  list of all opens threads that are stack over SMB, and how did you check the stuck IRP's in SMB.

    Thanks & Regards,

    Suchit Patil


    Friday, September 11, 2015 12:25 PM
  • Hello Vladimir.

    I know this is an old post, but I would appreciate if you can analyse another minidump file about 0x0000009E message.

    It´s an Exchange Mailbox Server.

    https://dl.dropboxusercontent.com/u/12198734/091015-233437-01.dmp

    Tks a lot

    Dan Pinheiro

    Rio de Janeiro - Brasil


    Dan Pinheiro RJ - Brasil

    Friday, September 11, 2015 12:26 PM
  • Hello Suchit,

    To dump ongoing SMB requests I am using a debugger extension that unfortunately is not publically released. As a workaround you can always find synchronies SMB requests using "!stacks 2 mrx" or "!stacks 2 rdbss"


    Regards, Vladimir Petter, Microsoft Corporation

    Friday, September 11, 2015 6:51 PM
  • Hello Dan, what you've shared with me is a kernel minidump, which would not have thread stacks, and unfortunately would not be useful. Please configure full memory dump

    http://www.bing.com/search?q=how+to+configure+windows+to+collect+full+kernel+dump&qs=n&form=QBLH&pq=how+to+configure+windows+to+collect+full+kernel+dump&sc=0-36&sp=-1&sk=&cvid=186d14b3517640b8896347f68916c892

    And once you get one feel free to share it with me.


    Regards, Vladimir Petter, Microsoft Corporation

    Friday, September 11, 2015 6:56 PM
  • Vladimir,

    I've also run into this issue and would appreciate you taking a look at a dump file if you are still willing and available?

    Monday, October 19, 2015 4:07 PM
  • Sure, I  can take a look. Please share a kernel dump with me.

    Regards, Vladimir Petter, Microsoft Corporation

    Tuesday, October 20, 2015 2:00 AM
  • Thank you it's much appreciated.   Should be able to download the dump from the link below.


    Tuesday, October 20, 2015 12:25 PM
  • I think you are running into a known issue. My guess this is happening when backup is reading data from snapshot. In this case it is reading 700GB VHD file, which builds a big section. 32GB of RAM on this machine allows that file to fit into the file cache. Eventually memory manager tries to trim active pages from the file cache and move them to stand by list. While doing that it is holding a lock on the thread fffffa803f730080. It has been doing that for about 1 minute on CPU 15

     

    THREAD fffffa803f730080  Cid 0e60.1944  Teb: 000007f7d3a06000 Win32Thread: 0000000000000000 RUNNING on processor 15

    IRP List:

        fffffa803e217010: (0006,01f0) Flags: 00060070  Mdl: 00000000

    Not impersonating

    DeviceMap                 fffff8a00b793f00

    Owning Process            fffffa8020179980       Image:         vmwp.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      9213470        Ticks: 283 (0:00:00:04.414)

    Context Switch Count      99             IdealProcessor: 12            

    UserTime                  00:00:00.000

    KernelTime                00:01:05.286

    Win32 Start Address 0x000007ff24521290

    Stack Init fffff88007075c90 Current fffff88007074a80

    Base fffff88007076000 Limit fffff88007070000 Call 0

    Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

    Child-SP          RetAddr           Call Site

    fffff880`07074e00 fffff803`494593f4 nt!MiTrimSection+0x18e

    fffff880`07074f60 fffff803`49457bda nt!MiTrimSharedPage+0x188

    fffff880`07075000 fffff803`49459059 nt!MiClaimPhysicalRun+0x38a

    fffff880`07075080 fffff803`495f6170 nt!MiFindContiguousPages+0x39d

    fffff880`070751f0 fffff803`49545efd nt!MiAllocatePagesForMdl+0xb1f64

    fffff880`070752f0 fffff880`065d8454 nt!MmAllocateNodePagesForMdlEx+0x75

    fffff880`07075340 fffff880`065d89ca Vid!VsmmpAllocatePagesFromNumaNode+0x58

    fffff880`07075390 fffff880`065bd947 Vid!VsmmAllocatePagesStrictQoS+0x10a

    fffff880`07075440 fffff880`065da6e8 Vid!VsmmReserveBucketReservePages+0xcb

    fffff880`070754b0 fffff880`065b7f50 Vid!VsmmReservePages+0x160

    fffff880`07075560 fffff880`065b761f Vid!VidIoControlPartition+0x7d0

    fffff880`07075630 fffff880`065b7708 Vid!VidIoControlDispatch+0x143

    fffff880`070756e0 fffff880`01078706 Vid!VidIoControlPreProcess+0x5c

    (Inline Function) --------`-------- Wdf01000!PreprocessIrp+0x3f (Inline Function @ fffff880`01078706)

    (Inline Function) --------`-------- Wdf01000!DispatchWorker+0xe7 (Inline Function @ fffff880`01078706)

    fffff880`07075730 fffff880`01078866 Wdf01000!FxDevice::Dispatch+0x106

    fffff880`07075770 fffff803`498822e8 Wdf01000!FxDevice::DispatchWithLock+0xa6

    (Inline Function) --------`-------- nt!IoCallDriverWithTracing+0x1b (Inline Function @ fffff803`498822e8)

    (Inline Function) --------`-------- nt!IopCallDriverReference+0x9b (Inline Function @ fffff803`498822e8)

    fffff880`070757b0 fffff803`49841d23 nt!IopSynchronousServiceTail+0x158

    fffff880`07075880 fffff803`4985b406 nt!IopXxxControlFile+0xde0

    fffff880`07075a20 fffff803`4946b453 nt!NtDeviceIoControlFile+0x56

    fffff880`07075a90 000007ff`244e2bda nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07075b00)

    00000011`b777f828 00000000`00000000 0x000007ff`244e2bda

     

    Meanwhile cluster service is trying to send a heartbeat to the kernel, and the thread that is supposed to do that got blocked on the same lock while trying to lock a page.

    THREAD fffffa802750e080  Cid 09cc.1048  Teb: 000007f73e43c000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable

        fffff880089b6360  SynchronizationEvent

    IRP List:

        fffffa80198536c0: (0006,0118) Flags: 00060000  Mdl: 00000000

    Not impersonating

    DeviceMap                 fffff8a00000c370

    Owning Process            fffffa801eeb1700       Image:         clussvc.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      9209942        Ticks: 3811 (0:00:00:59.451)

    Context Switch Count      10363          IdealProcessor: 5            

    UserTime                  00:00:00.124

    KernelTime                00:00:00.078

    Win32 Start Address 0x000007ff24521290

    Stack Init fffff880089b6c90 Current fffff880089b6060

    Base fffff880089b7000 Limit fffff880089b1000 Call 0

    Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

    Child-SP          RetAddr           Call Site

    fffff880`089b60a0 fffff803`494af54b nt!KiSwapContext+0x76

    (Inline Function) --------`-------- nt!KiSwapThread+0xfa (Inline Function @ fffff803`494af54b)

    fffff880`089b61e0 fffff803`494ae38f nt!KiCommitThreadWait+0x23b

    fffff880`089b62a0 fffff803`494f1746 nt!KeWaitForSingleObject+0x1cf

    fffff880`089b6330 fffff803`494d36d1 nt!ExfAcquirePushLockExclusive+0x16d

    (Inline Function) --------`-------- nt!ExAcquirePushLockExclusive+0x62 (Inline Function @ fffff803`494d36d1)

    fffff880`089b63c0 fffff803`49447a62 nt!LOCK_WORKING_SET+0x79

    fffff880`089b63f0 fffff803`49448218 nt!MiLockCode+0xc2

    fffff880`089b6500 fffff803`4944839a nt!MiLockImageSection+0x58

    fffff880`089b6550 fffff803`49806f58 nt!MiLockPagableImageSection+0xda

    fffff880`089b6590 fffff880`013759c0 nt!MmLockPagableSectionByHandle+0x68

    (Inline Function) --------`-------- NDIS!ndisReferencePackage+0x22 (Inline Function @ fffff880`013759c0)

    fffff880`089b65c0 fffff803`498822e8 NDIS!ndisDeviceControlIrpHandler+0x98

    (Inline Function) --------`-------- nt!IoCallDriverWithTracing+0x1b (Inline Function @ fffff803`498822e8)

    (Inline Function) --------`-------- nt!IopCallDriverReference+0x9b (Inline Function @ fffff803`498822e8)

    fffff880`089b67b0 fffff803`49841d23 nt!IopSynchronousServiceTail+0x158

    fffff880`089b6880 fffff803`4985b406 nt!IopXxxControlFile+0xde0

    fffff880`089b6a20 fffff803`4946b453 nt!NtDeviceIoControlFile+0x56

    fffff880`089b6a90 000007ff`244e2bda nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`089b6b00)

    00000053`a6c9f1f8 00000000`00000000 0x000007ff`244e2bda

     

    Because kernel component has not received heartbeat in 1 minute it bugchecked the machine with USER_MODE_HEALTH_MONITOR (9e).

    As far as I remember this was fixed and a fix was backported to Windows Server 2012 R2. I do not remember if it was ported back to Windows Server 2012 (9200.17528.amd64fre.win8_gdr.150928-0600), which you are running.

    Here is what I suggest to do.

    • Make sure that you have all the latest patches.

    • If you do and issue is still reproducing, then I suggest to call to Microsoft support and present them this analysis and the dump. They should be able to provide you a patch, and if necessary they will initiate backport of the fix to Windows Server 2012.


    Regards, Vladimir Petter, Microsoft Corporation

    Wednesday, October 21, 2015 6:48 AM
  • Thank you so much for taking a look at it at providing suggestions.  I suspected it had something to do with backups due to the time frame I was seeing it happen.  I thought I had applied all relevant patches and hotfixes but I will revisit that and make sure everything is applied and investigate further from there.

    Thanks again for taking your time to look at this.


    Wednesday, October 21, 2015 4:16 PM
  • Hi Vladimir,

    Would you please help me out with taking a look a dump-file ?

    So far I have this:

    Opened log file 'c:\temp\debuglog.txt'

    8: kd> .sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols
    Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
    Expanded Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols

    ************* Symbol Path validation summary **************
    Response                         Time (ms)     Location
    Deferred                                       srv*c:\symbols*http://msdl.microsoft.com/download/symbols
    8: kd> .reload;!analyze -v;r;kv;lmnt;.logclose;q
    Loading Kernel Symbols
    ...............................................................
    ................................................................
    ..................
    Loading User Symbols

    Loading unloaded module list
    ............
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    USER_MODE_HEALTH_MONITOR (9e)
    One or more critical user mode components failed to satisfy a health check.
    Hardware mechanisms such as watchdog timers can detect that basic kernel
    services are not executing. However, resource starvation issues, including
    memory leaks, lock contention, and scheduling priority misconfiguration,
    may block critical user mode components without blocking DPCs or
    draining the nonpaged pool.
    Kernel components can extend watchdog timer functionality to user mode
    by periodically monitoring critical applications. This bugcheck indicates
    that a user mode health check failed in a manner such that graceful
    shutdown is unlikely to succeed. It restores critical services by
    rebooting and/or allowing application failover to other servers.
    Arguments:
    Arg1: fffffa80f7862060, WatchdogSourceRhsCleanup
    Monitors that RHS process goes away when
    terminating on graceful exit
    Arg2: 00000000000004b0, WatchdogSourceRhsResourceDeadlockBugcheckNow
    RHS was asked to immediately bugcheck machine
    on resource deadlock
    Arg3: 0000000000000000, WatchdogSourceRhsExceptionFromResource
    Resource has leaked unhandled exception from an entry point,
    RHS is terminating and this watchdog monitors that
    process will go away
    Arg4: 0000000000000000, WatchdogSourceRhsUnhandledException
    Unhandled exception in RHS.
    RHS is terminating and this watchdog monitors that
    process will go away
    5  - WatchdogSourceRhsResourceDeadlock
    Monitors that RHS process goes away when
    terminating on resource deadlock
    6  - WatchdogSourceRhsResourceTypeDeadlock
    Monitors that RHS process goes away when
    terminating on resource type deadlock
    7  - WatchdogSourceClussvcUnhandledException
    Unhandled exception in clussvc.
    clussvc is terminating and this watchdog monitors that
    process will go away
    8  - WatchdogSourceClussvcBugcheckMessageRecieved
    Another cluster node has send message asking to bugcheck this node.
    9  - WatchdogSourceClussvcWatchdogBugcheck
    User mode watchdog has expired and created netft watchdog
    to bugchecked the node.
    10 - WatchdogSourceClussvcIsAlive
    Cluster service sends heartbeat to netft every 500 millseconds.
    By default netft expects at least 1 heartbeat per second.
    If this watchdog was triggered that means clussvc is o not getting
    CPU to send heartbers.
    101 - WatchdogSourceRhsResourceDeadlockPhysicalDisk
    102 - WatchdogSourceRhsResourceDeadlockStoragePool
    103 - WatchdogSourceRhsResourceDeadlockFileServer
    104 - WatchdogSourceRhsResourceDeadlockSODAFileServer
    105 - WatchdogSourceRhsResourceDeadlockStorageReplica
    106 - WatchdogSourceRhsResourceDeadlockStorageQOS
    107 - WatchdogSourceRhsResourceDeadlockStorageNFSV2
    Values above are subclasses of WatchdogSourceRhsResourceDeadlock
    For the resource types that are known to cause RHS termination hangs
    we isolate them to separate codes so we can quickly triage dumps.
    201 - WatchdogSourceRhsResourceTypeDeadlockPhysicalDisk
    202 - WatchdogSourceRhsResourceTypeDeadlockStoragePool
    203 - WatchdogSourceRhsResourceTypeDeadlockFileServer
    204 - WatchdogSourceRhsResourceTypeDeadlockSODAFileServer
    205 - WatchdogSourceRhsResourceTypeDeadlockStorageReplica
    206 - WatchdogSourceRhsResourceTypeDeadlockStorageQOS
    207 - WatchdogSourceRhsResourceTypeDeadlockStorageNFSV2
    Values above are subclasses of WatchdogSourceRhsResourceTypeDeadlock
    For the resource types that are known to cause RHS termination hangs
    we isolate them to separate codes so we can quickly triage dumps.

    Debugging Details:
    ------------------


    DUMP_CLASS: 1

    DUMP_QUALIFIER: 401

    BUILD_VERSION_STRING:  7601.18798.amd64fre.win7sp1_gdr.150316-1654

    SYSTEM_MANUFACTURER:  HP

    SYSTEM_PRODUCT_NAME:  ProLiant BL460c Gen8

    SYSTEM_SKU:  641016-B21      

    BIOS_VENDOR:  HP

    BIOS_VERSION:  I31

    BIOS_DATE:  12/20/2013

    DUMP_TYPE:  1

    BUGCHECK_P1: fffffa80f7862060

    BUGCHECK_P2: 4b0

    BUGCHECK_P3: 0

    BUGCHECK_P4: 0

    PROCESS_NAME:  rhs.exe

    IMAGE_NAME:  rhs.exe

    DEBUG_FLR_IMAGE_TIMESTAMP:  0

    MODULE_NAME: rhs

    FAULTING_MODULE: 0000000000000000 

    CPU_COUNT: 10

    CPU_MHZ: ce4

    CPU_VENDOR:  GenuineIntel

    CPU_FAMILY: 6

    CPU_MODEL: 2d

    CPU_STEPPING: 7

    CPU_MICROCODE: 6,2d,7,0 (F,M,S,R)  SIG: 710'00000000 (cache) 710'00000000 (init)

    DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

    BUGCHECK_STR:  0x9E

    CURRENT_IRQL:  2

    ANALYSIS_SESSION_HOST:  ATC-WJUMP

    ANALYSIS_SESSION_TIME:  12-03-2015 18:10:10.0807

    ANALYSIS_VERSION: 10.0.10586.567 amd64fre

    STACK_TEXT:  
    fffff880`0253f3d8 fffff880`0512d6a5 : 00000000`0000009e fffffa80`f7862060 00000000`000004b0 00000000`00000000 : nt!KeBugCheckEx
    fffff880`0253f3e0 fffff800`0209c91c : fffff880`0253f4c0 00000000`00000001 00000000`40390088 fffff880`014cbc94 : netft!NetftWatchdogTimerDpc+0xb9
    fffff880`0253f430 fffff800`0209c7b6 : fffff880`0513a100 00000000`010322f9 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x6c
    fffff880`0253f4a0 fffff800`0209c69e : 00000268`d86300d3 fffff880`0253fb18 00000000`010322f9 fffff880`0251b4a8 : nt!KiProcessExpiredTimerList+0xc6
    fffff880`0253faf0 fffff800`0209c487 : 000000c6`cac035c1 000000c6`010322f9 000000c6`cac035cf 00000000`000000f9 : nt!KiTimerExpiration+0x1be
    fffff880`0253fb90 fffff800`0208874a : fffff880`02517180 fffff880`025222c0 00000000`00000001 fffff880`00000000 : nt!KiRetireDpcList+0x277
    fffff880`0253fc40 00000000`00000000 : fffff880`02540000 fffff880`0253a000 fffff880`0253fc00 00000000`00000000 : nt!KiIdleLoop+0x5a


    STACK_COMMAND:  kb

    THREAD_SHA1_HASH_MOD_FUNC:  816f11a87ad4e6cc4ea0c0bae49953386adf648a

    THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  5d45c8c68fd994a4bdace855a86f47ca470766fe

    THREAD_SHA1_HASH_MOD:  62ffe22e249d6ed7a775c30201445f402c1cebbf

    FOLLOWUP_NAME:  MachineOwner

    IMAGE_VERSION:  

    FAILURE_BUCKET_ID:  X64_0x9E_IMAGE_rhs.exe

    BUCKET_ID:  X64_0x9E_IMAGE_rhs.exe

    PRIMARY_PROBLEM_CLASS:  X64_0x9E_IMAGE_rhs.exe

    TARGET_TIME:  2015-11-23T12:43:44.000Z

    OSBUILD:  7601

    OSSERVICEPACK:  1000

    SERVICEPACK_NUMBER: 0

    OS_REVISION: 0

    SUITE_MASK:  274

    PRODUCT_TYPE:  3

    OSPLATFORM_TYPE:  x64

    OSNAME:  Windows 7

    OSEDITION:  Windows 7 Server (Service Pack 1) Enterprise TerminalServer SingleUserTS

    OS_LOCALE:  

    USER_LCID:  0

    OSBUILD_TIMESTAMP:  2015-03-17 00:02:04

    BUILDDATESTAMP_STR:  150316-1654

    BUILDLAB_STR:  win7sp1_gdr

    BUILDOSVER_STR:  6.1.7601.18798.amd64fre.win7sp1_gdr.150316-1654

    ANALYSIS_SESSION_ELAPSED_TIME: 15d0

    ANALYSIS_SOURCE:  KM

    FAILURE_ID_HASH_STRING:  km:x64_0x9e_image_rhs.exe

    FAILURE_ID_HASH:  {c259dc81-ac67-ae55-7e62-cf1c10f90628}

    Followup:     MachineOwner
    ---------

    rax=0000000000000000 rbx=fffffa80f9c091d0 rcx=000000000000009e
    rdx=fffffa80f7862060 rsi=fffff88005139110 rdi=fffff8800253f400
    rip=fffff80002090a40 rsp=fffff8800253f3d8 rbp=0000000000000000
     r8=00000000000004b0  r9=0000000000000000 r10=fffff8000201e000
    r11=fffffa806e4a4c10 r12=fffff8800513a0e0 r13=fffff8800253fb18
    r14=fffff880025222c0 r15=fffff88002517180
    iopl=0         nv up ei pl zr na po nc
    cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
    nt!KeBugCheckEx:
    fffff800`02090a40 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffff880`0253f3e0=000000000000009e
    Child-SP          RetAddr           : Args to Child                                                           : Call Site
    fffff880`0253f3d8 fffff880`0512d6a5 : 00000000`0000009e fffffa80`f7862060 00000000`000004b0 00000000`00000000 : nt!KeBugCheckEx
    fffff880`0253f3e0 fffff800`0209c91c : fffff880`0253f4c0 00000000`00000001 00000000`40390088 fffff880`014cbc94 : netft!NetftWatchdogTimerDpc+0xb9
    fffff880`0253f430 fffff800`0209c7b6 : fffff880`0513a100 00000000`010322f9 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x6c
    fffff880`0253f4a0 fffff800`0209c69e : 00000268`d86300d3 fffff880`0253fb18 00000000`010322f9 fffff880`0251b4a8 : nt!KiProcessExpiredTimerList+0xc6
    fffff880`0253faf0 fffff800`0209c487 : 000000c6`cac035c1 000000c6`010322f9 000000c6`cac035cf 00000000`000000f9 : nt!KiTimerExpiration+0x1be
    fffff880`0253fb90 fffff800`0208874a : fffff880`02517180 fffff880`025222c0 00000000`00000001 fffff880`00000000 : nt!KiRetireDpcList+0x277
    fffff880`0253fc40 00000000`00000000 : fffff880`02540000 fffff880`0253a000 fffff880`0253fc00 00000000`00000000 : nt!KiIdleLoop+0x5a
    start             end                 module name
    fffff800`01faf000 fffff800`01fb9000   kdcom    kdcom.dll    Mon Jul 13 21:31:07 2009 (4A5BDFDB)
    fffff800`0201e000 fffff800`02606000   nt       ntkrnlmp.exe Tue Mar 17 00:02:04 2015 (5507A73C)
    fffff800`02606000 fffff800`0264f000   hal      hal.dll      Sat Nov 20 08:00:25 2010 (4CE7C669)
    fffff880`00c00000 fffff880`00c52e00   mfewfpk  mfewfpk.sys  Fri Jan 30 13:42:08 2015 (54CBD080)
    fffff880`00c53000 fffff880`00c80000   mrxsmb   mrxsmb.sys   Tue Jan 06 20:49:30 2015 (54AC90AA)
    fffff880`00c81000 fffff880`00cd0000   mcupdate_GenuineIntel mcupdate_GenuineIntel.dll Sat Nov 20 08:03:51 2010 (4CE7C737)
    fffff880`00cd0000 fffff880`00ce4000   PSHED    PSHED.dll    Mon Jul 13 21:32:23 2009 (4A5BE027)
    fffff880`00ce4000 fffff880`00d42000   CLFS     CLFS.SYS     Tue Mar 03 22:21:12 2015 (54F67A28)
    fffff880`00d42000 fffff880`00da5000   storport storport.sys Sat Nov 20 05:35:02 2010 (4CE7A456)
    fffff880`00da5000 fffff880`00dd6000   be2iscsi be2iscsi.sys Sun Aug 11 04:57:21 2013 (520751F1)
    fffff880`00e00000 fffff880`00e3a000   involflt involflt.sys Tue Aug 05 22:35:00 2014 (53E19454)
    fffff880`00e3a000 fffff880`00e5e000   hpeaadsm hpeaadsm.sys Mon Jan 10 00:07:38 2011 (4D2A941A)
    fffff880`00e5e000 fffff880`00e89000   HpCISSs2 HpCISSs2.sys Mon Oct 28 11:10:30 2013 (526E7E66)
    fffff880`00e9d000 fffff880`00f5d000   CI       CI.dll       Sat Nov 20 08:12:36 2010 (4CE7C944)
    fffff880`00f78000 fffff880`00fd4000   volmgrx  volmgrx.sys  Sat Nov 20 04:20:43 2010 (4CE792EB)
    fffff880`00fd4000 fffff880`00feb000   HpSAMD   HpSAMD.sys   Tue Apr 20 14:32:18 2010 (4BCDF332)
    fffff880`01000000 fffff880`01060000   NETIO    NETIO.SYS    Tue Nov 26 05:21:01 2013 (5294760D)
    fffff880`01060000 fffff880`0107a000   mountmgr mountmgr.sys Sat Nov 20 04:19:21 2010 (4CE79299)
    fffff880`01082000 fffff880`01175000   NDIS     NDIS.SYS     Sat Nov 20 04:23:30 2010 (4CE79392)
    fffff880`01175000 fffff880`011d3000   msrpc    msrpc.sys    Sat Nov 20 04:21:56 2010 (4CE79334)
    fffff880`011d3000 fffff880`011f9000   msdsm    msdsm.sys    Fri May 31 23:58:14 2013 (51A97156)
    fffff880`01200000 fffff880`0122b000   mpio     mpio.sys     Thu Nov 29 22:55:16 2012 (50B82E24)
    fffff880`0122b000 fffff880`01240000   partmgr  partmgr.sys  Sat Mar 17 01:06:09 2012 (4F641BC1)
    fffff880`01240000 fffff880`01249000   compbatt compbatt.sys Mon Jul 13 19:31:02 2009 (4A5BC3B6)
    fffff880`01249000 fffff880`01255000   BATTC    BATTC.SYS    Mon Jul 13 19:31:01 2009 (4A5BC3B5)
    fffff880`01260000 fffff880`01322000   Wdf01000 Wdf01000.sys Fri Jun 21 23:13:05 2013 (51C51641)
    fffff880`01322000 fffff880`01332000   WDFLDR   WDFLDR.SYS   Wed Jul 25 22:29:04 2012 (5010AB70)
    fffff880`01332000 fffff880`01389000   ACPI     ACPI.sys     Sat Nov 20 04:19:16 2010 (4CE79294)
    fffff880`01389000 fffff880`01392000   WMILIB   WMILIB.SYS   Mon Jul 13 19:19:51 2009 (4A5BC117)
    fffff880`01392000 fffff880`0139c000   msisadrv msisadrv.sys Mon Jul 13 19:19:26 2009 (4A5BC0FE)
    fffff880`0139c000 fffff880`013cf000   pci      pci.sys      Sat Nov 20 04:19:11 2010 (4CE7928F)
    fffff880`013cf000 fffff880`013dc000   vdrvroot vdrvroot.sys Mon Jul 13 20:01:31 2009 (4A5BCADB)
    fffff880`013dc000 fffff880`013f1000   volmgr   volmgr.sys   Sat Nov 20 04:19:28 2010 (4CE792A0)
    fffff880`01418000 fffff880`01588000   ql2300   ql2300.sys   Thu Mar 07 13:19:46 2013 (5138DA42)
    fffff880`01588000 fffff880`01593000   amdxata  amdxata.sys  Fri Mar 19 12:18:18 2010 (4BA3A3CA)
    fffff880`01593000 fffff880`015df000   fltmgr   fltmgr.sys   Sat Nov 20 04:19:24 2010 (4CE7929C)
    fffff880`0161f000 fffff880`016f4700   mfehidk  mfehidk.sys  Fri Jan 30 13:41:34 2015 (54CBD05E)
    fffff880`016f5000 fffff880`01767000   cng      cng.sys      Thu Jan 29 22:17:08 2015 (54CAF7B4)
    fffff880`01767000 fffff880`01793000   ksecpkg  ksecpkg.sys  Fri Apr 03 22:15:42 2015 (551F494E)
    fffff880`01793000 fffff880`017dc000   fwpkclnt fwpkclnt.sys Fri Apr 04 21:23:21 2014 (533F5B09)
    fffff880`017dc000 fffff880`017e9000   TDI      TDI.SYS      Sat Nov 20 04:22:06 2010 (4CE7933E)
    fffff880`017e9000 fffff880`017ff000   intelppm intelppm.sys Mon Jul 13 19:19:25 2009 (4A5BC0FD)
    fffff880`01800000 fffff880`0180a000   Fs_Rec   Fs_Rec.sys   Wed Feb 29 22:41:06 2012 (4F4EEFD2)
    fffff880`0180a000 fffff880`01812000   hpqilo3whea hpqilo3whea.sys Fri Feb 12 18:33:27 2010 (4B75E547)
    fffff880`01812000 fffff880`01822000   vmstorfl vmstorfl.sys Sat Nov 20 04:57:30 2010 (4CE79B8A)
    fffff880`01829000 fffff880`019cb000   Ntfs     Ntfs.sys     Fri Apr 12 07:54:36 2013 (5167F5FC)
    fffff880`019cb000 fffff880`019e6000   ksecdd   ksecdd.sys   Fri Apr 03 21:58:36 2015 (551F454C)
    fffff880`019e6000 fffff880`019f7000   pcw      pcw.sys      Mon Jul 13 19:19:27 2009 (4A5BC0FF)
    fffff880`01a00000 fffff880`01bff000   tcpip    tcpip.sys    Fri Apr 04 21:26:44 2014 (533F5BD4)
    fffff880`01c00000 fffff880`01c22000   tdx      tdx.sys      Sat Nov 20 04:21:54 2010 (4CE79332)
    fffff880`01c26000 fffff880`01c72000   volsnap  volsnap.sys  Sat Nov 20 04:20:08 2010 (4CE792C8)
    fffff880`01c72000 fffff880`01c7a000   spldr    spldr.sys    Mon May 11 12:56:27 2009 (4A0858BB)
    fffff880`01c7a000 fffff880`01c98000   mup      mup.sys      Tue Jan 06 20:48:27 2015 (54AC906B)
    fffff880`01c98000 fffff880`01cae580   mfedisk  mfedisk.sys  Fri Jan 30 13:42:39 2015 (54CBD09F)
    fffff880`01caf000 fffff880`01cb8000   hwpolicy hwpolicy.sys Sat Nov 20 04:18:54 2010 (4CE7927E)
    fffff880`01cb8000 fffff880`01cce000   disk     disk.sys     Mon Jul 13 19:19:57 2009 (4A5BC11D)
    fffff880`01cce000 fffff880`01cfe000   CLASSPNP CLASSPNP.SYS Sat Nov 20 04:19:23 2010 (4CE7929B)
    fffff880`01cfe000 fffff880`01d1c000   bowser   bowser.sys   Tue Feb 22 23:55:04 2011 (4D649328)
    fffff880`01d41000 fffff880`01d61000   DpmFilter DpmFilter.sys Mon Sep 17 06:52:03 2012 (505700D3)
    fffff880`01d61000 fffff880`01d6f000   ClusDisk ClusDisk.sys Sat Nov 20 04:42:24 2010 (4CE79800)
    fffff880`01d6f000 fffff880`01d77000   InCDskFl InCDskFl.sys Tue Aug 05 22:49:45 2014 (53E197C9)
    fffff880`01d77000 fffff880`01d80000   Null     Null.SYS     Mon Jul 13 19:19:37 2009 (4A5BC109)
    fffff880`01d80000 fffff880`01d8e000   vga      vga.sys      Mon Jul 13 19:38:47 2009 (4A5BC587)
    fffff880`01d8e000 fffff880`01db3000   VIDEOPRT VIDEOPRT.SYS Mon Jul 13 19:38:51 2009 (4A5BC58B)
    fffff880`01db3000 fffff880`01dc3000   watchdog watchdog.sys Mon Jul 13 19:37:35 2009 (4A5BC53F)
    fffff880`01dc3000 fffff880`01dcc000   RDPCDD   RDPCDD.sys   Mon Jul 13 20:16:34 2009 (4A5BCE62)
    fffff880`01dcc000 fffff880`01dd5000   rdpencdd rdpencdd.sys Mon Jul 13 20:16:34 2009 (4A5BCE62)
    fffff880`01dd5000 fffff880`01dde000   rdprefmp rdprefmp.sys Mon Jul 13 20:16:35 2009 (4A5BCE63)
    fffff880`01dde000 fffff880`01de9000   Msfs     Msfs.SYS     Mon Jul 13 19:19:47 2009 (4A5BC113)
    fffff880`01de9000 fffff880`01dfa000   Npfs     Npfs.SYS     Mon Jul 13 19:19:48 2009 (4A5BC114)
    fffff880`04c01000 fffff880`04c46000   netbt    netbt.sys    Sat Nov 20 04:23:18 2010 (4CE79386)
    fffff880`04c46000 fffff880`04ccf000   afd      afd.sys      Fri May 30 02:45:48 2014 (5388291C)
    fffff880`04ccf000 fffff880`04cd8000   wfplwf   wfplwf.sys   Mon Jul 13 20:09:26 2009 (4A5BCCB6)
    fffff880`04cd8000 fffff880`04cfe000   pacer    pacer.sys    Sat Nov 20 05:52:18 2010 (4CE7A862)
    fffff880`04cfe000 fffff880`04d0d000   netbios  netbios.sys  Mon Jul 13 20:09:26 2009 (4A5BCCB6)
    fffff880`04d0d000 fffff880`04d2a000   serial   serial.sys   Mon Jul 13 20:00:40 2009 (4A5BCAA8)
    fffff880`04d2a000 fffff880`04d45000   wanarp   wanarp.sys   Sat Nov 20 05:52:36 2010 (4CE7A874)
    fffff880`04d45000 fffff880`04d59000   termdd   termdd.sys   Sat Nov 20 06:03:40 2010 (4CE7AB0C)
    fffff880`04d59000 fffff880`04daa000   rdbss    rdbss.sys    Tue Jan 06 20:49:42 2015 (54AC90B6)
    fffff880`04daa000 fffff880`04db6000   nsiproxy nsiproxy.sys Mon Jul 13 19:21:02 2009 (4A5BC15E)
    fffff880`04db6000 fffff880`04dc1000   mssmbios mssmbios.sys Mon Jul 13 19:31:10 2009 (4A5BC3BE)
    fffff880`04dc1000 fffff880`04dd0000   discache discache.sys Mon Jul 13 19:37:18 2009 (4A5BC52E)
    fffff880`04dd0000 fffff880`04def000   dfsc     dfsc.sys     Tue Jan 06 20:48:10 2015 (54AC905A)
    fffff880`04def000 fffff880`04e00000   blbdrive blbdrive.sys Mon Jul 13 19:35:59 2009 (4A5BC4DF)
    fffff880`04e16000 fffff880`04e92000   be2nd62  be2nd62.sys  Sat Dec 07 01:04:36 2013 (52A2BA74)
    fffff880`04e92000 fffff880`04ea4000   usbehci  usbehci.sys  Tue Nov 26 20:41:11 2013 (52954DB7)
    fffff880`04ea4000 fffff880`04efa000   USBPORT  USBPORT.SYS  Tue Nov 26 20:41:11 2013 (52954DB7)
    fffff880`04efa000 fffff880`04f09000   hpqilo3core hpqilo3core.sys Wed May 22 17:41:45 2013 (519D3B99)
    fffff880`04f09000 fffff880`04f4a000   g200eHm  g200eHm.sys  Wed Apr 11 16:24:18 2012 (4F85E872)
    fffff880`04f4a000 fffff880`04f59000   hpqilo3chif hpqilo3chif.sys Sat Nov 23 21:26:20 2013 (529163CC)
    fffff880`04f59000 fffff880`04f66000   usbuhci  usbuhci.sys  Tue Nov 26 20:41:05 2013 (52954DB1)
    fffff880`04f66000 fffff880`04f7e000   IPMIDrv  IPMIDrv.sys  Sat Nov 20 05:04:53 2010 (4CE79D45)
    fffff880`04f7e000 fffff880`04f8a000   serenum  serenum.sys  Mon Jul 13 20:00:33 2009 (4A5BCAA1)
    fffff880`04f8a000 fffff880`04fa8000   i8042prt i8042prt.sys Mon Jul 13 19:19:57 2009 (4A5BC11D)
    fffff880`04fa8000 fffff880`04fb7000   kbdclass kbdclass.sys Mon Jul 13 19:19:50 2009 (4A5BC116)
    fffff880`04fb7000 fffff880`04fc6000   mouclass mouclass.sys Mon Jul 13 19:19:50 2009 (4A5BC116)
    fffff880`04fc6000 fffff880`04fd5000   tpm      tpm.sys      Mon Jul 13 19:21:48 2009 (4A5BC18C)
    fffff880`04fd5000 fffff880`04fdf000   acpipmi  acpipmi.sys  Sat Nov 20 04:30:42 2010 (4CE79542)
    fffff880`04fdf000 fffff880`04fef000   CompositeBus CompositeBus.sys Sat Nov 20 05:33:17 2010 (4CE7A3ED)
    fffff880`05000000 fffff880`05018000   rspndr   rspndr.sys   Mon Jul 13 20:08:50 2009 (4A5BCC92)
    fffff880`0501e000 fffff880`05060000   cpqteam  cpqteam.sys  Wed Dec 11 05:30:40 2013 (52A83ED0)
    fffff880`05060000 fffff880`05076000   AgileVpn AgileVpn.sys Mon Jul 13 20:10:24 2009 (4A5BCCF0)
    fffff880`05076000 fffff880`0509a000   rasl2tp  rasl2tp.sys  Sat Nov 20 05:52:34 2010 (4CE7A872)
    fffff880`0509a000 fffff880`050a6000   ndistapi ndistapi.sys Mon Jul 13 20:10:00 2009 (4A5BCCD8)
    fffff880`050a6000 fffff880`050d5000   ndiswan  ndiswan.sys  Sat Nov 20 05:52:32 2010 (4CE7A870)
    fffff880`050d5000 fffff880`050f0000   raspppoe raspppoe.sys Mon Jul 13 20:10:17 2009 (4A5BCCE9)
    fffff880`050f0000 fffff880`05111000   raspptp  raspptp.sys  Sat Nov 20 05:52:31 2010 (4CE7A86F)
    fffff880`05111000 fffff880`0512b000   rassstp  rassstp.sys  Mon Jul 13 20:10:25 2009 (4A5BCCF1)
    fffff880`0512b000 fffff880`05147000   netft    netft.sys    Mon Jul 13 19:34:34 2009 (4A5BC48A)
    fffff880`05147000 fffff880`05152000   rdpbus   rdpbus.sys   Mon Jul 13 20:17:46 2009 (4A5BCEAA)
    fffff880`05152000 fffff880`05153480   swenum   swenum.sys   Mon Jul 13 20:00:18 2009 (4A5BCA92)
    fffff880`05154000 fffff880`05197000   ks       ks.sys       Sat Nov 20 05:33:23 2010 (4CE7A3F3)
    fffff880`05197000 fffff880`051a9000   umbus    umbus.sys    Sat Nov 20 05:44:37 2010 (4CE7A695)
    fffff880`051a9000 fffff880`051b5000   Dxapi    Dxapi.sys    Mon Jul 13 19:38:28 2009 (4A5BC574)
    fffff880`051b5000 fffff880`051c3000   monitor  monitor.sys  Mon Jul 13 19:38:52 2009 (4A5BC58C)
    fffff880`051c3000 fffff880`051e6000   luafv    luafv.sys    Mon Jul 13 19:26:13 2009 (4A5BC295)
    fffff880`051e6000 fffff880`051fb000   lltdio   lltdio.sys   Mon Jul 13 20:08:50 2009 (4A5BCC92)
    fffff880`05200000 fffff880`0522b000   dump_HpCISSs2 dump_HpCISSs2.sys Mon Oct 28 11:10:30 2013 (526E7E66)
    fffff880`0522f000 fffff880`05289000   usbhub   usbhub.sys   Tue Nov 26 20:41:36 2013 (52954DD0)
    fffff880`05289000 fffff880`0529e000   NDProxy  NDProxy.SYS  Sat Nov 20 05:52:20 2010 (4CE7A864)
    fffff880`0529e000 fffff880`052eea00   mfeavfk  mfeavfk.sys  Fri Jan 30 13:43:31 2015 (54CBD0D3)
    fffff880`052ef000 fffff880`052fab00   mfeaacsk mfeaacsk.sys Fri Jan 30 13:48:09 2015 (54CBD1E9)
    fffff880`052fb000 fffff880`05307100   mfeplk   mfeplk.sys   Fri Jan 30 13:48:10 2015 (54CBD1EA)
    fffff880`05308000 fffff880`0535fb80   mfeaack  mfeaack.sys  Fri Jan 30 13:48:06 2015 (54CBD1E6)
    fffff880`05360000 fffff880`053dfc80   mfefirek mfefirek.sys Fri Jan 30 13:44:56 2015 (54CBD128)
    fffff880`053e0000 fffff880`053ee000   crashdmp crashdmp.sys Mon Jul 13 20:01:01 2009 (4A5BCABD)
    fffff880`053ee000 fffff880`053f8000   dump_diskdump dump_diskdump.sys Sat Nov 20 05:33:56 2010 (4CE7A414)
    fffff880`06e00000 fffff880`06ea6000   peauth   peauth.sys   Mon Jul 13 21:01:19 2009 (4A5BD8DF)
    fffff880`06ea6000 fffff880`06eb1000   secdrv   secdrv.SYS   Wed Sep 13 09:18:38 2006 (4508052E)
    fffff880`06ebe000 fffff880`06f0c000   mrxsmb10 mrxsmb10.sys Tue Jan 06 20:48:48 2015 (54AC9080)
    fffff880`06f0c000 fffff880`06f30000   mrxsmb20 mrxsmb20.sys Tue Jan 06 20:48:38 2015 (54AC9076)
    fffff880`06f30000 fffff880`06ff9000   HTTP     HTTP.sys     Tue Feb 24 22:17:55 2015 (54ED3EE3)
    fffff880`07c00000 fffff880`07c98000   srv      srv.sys      Thu Apr 28 23:06:06 2011 (4DBA2B1E)
    fffff880`07c98000 fffff880`07ca7000   tssecsrv tssecsrv.sys Sat Nov 20 06:04:09 2010 (4CE7AB29)
    fffff880`07cc3000 fffff880`07cf4000   srvnet   srvnet.sys   Thu Apr 28 23:05:35 2011 (4DBA2AFF)
    fffff880`07cf4000 fffff880`07d06000   tcpipreg tcpipreg.sys Sat Nov 20 05:51:48 2010 (4CE7A844)
    fffff880`07d06000 fffff880`07d3f000   RDPWD    RDPWD.SYS    Fri Apr 27 23:55:20 2012 (4F9B6A28)
    fffff880`07d57000 fffff880`07dc0000   srv2     srv2.sys     Thu Apr 28 23:05:46 2011 (4DBA2B0A)
    fffff880`07dc0000 fffff880`07dee000   rdpdr    rdpdr.sys    Sat Nov 20 06:06:41 2010 (4CE7ABC1)
    fffff880`07dee000 fffff880`07df9000   tdtcp    tdtcp.sys    Thu Feb 16 23:57:32 2012 (4F3DDE3C)
    fffff880`0b472000 fffff880`0b47d000   asyncmac asyncmac.sys Mon Jul 13 20:10:13 2009 (4A5BCCE5)
    fffff960`00040000 fffff960`00363000   win32k   win32k.sys   Mon May 25 13:08:15 2015 (556356FF)
    fffff960`00400000 fffff960`0041e000   dxg      dxg.sys      Mon Jul 13 19:38:28 2009 (4A5BC574)
    fffff960`00710000 fffff960`0071a000   TSDDD    TSDDD.dll    Mon Jul 13 20:16:34 2009 (4A5BCE62)
    fffff960`00860000 fffff960`00896000   G200eHd  G200eHd.dll  Wed Apr 11 16:24:18 2012 (4F85E872)
    fffff960`00af0000 fffff960`00b38000   RDPDD    RDPDD.dll    Sat Nov 20 06:05:00 2010 (4CE7AB5C)

    Unloaded modules:
    fffff880`0b4ee000 fffff880`0b55f000   spsys.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  00071000
    fffff880`0b5b9000 fffff880`0b5e6000   mfeapfk.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0002D000
    fffff880`0b58c000 fffff880`0b5b9000   mfeapfk.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0002D000
    fffff880`0b55f000 fffff880`0b58c000   mfeapfk.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0002D000
    fffff880`0b47d000 fffff880`0b4ee000   spsys.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  00071000
    fffff880`0b401000 fffff880`0b472000   spsys.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  00071000
    fffff880`07d06000 fffff880`07d33000   mfeapfk.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0002D000
    fffff880`07d06000 fffff880`07d57000   mfeavfk01.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  00051000
    fffff880`01cfe000 fffff880`01d0c000   crashdmp.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0000E000
    fffff880`01d0c000 fffff880`01d16000   dump_storport.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0000A000
    fffff880`01d16000 fffff880`01d41000   dump_HpCISSs2.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0002B000
    fffff880`00f5d000 fffff880`00f78000   sacdrv.sys
        Timestamp: unavailable (00000000)
        Checksum:  00000000
        ImageSize:  0001B000
    Closing open log file c:\temp\debuglog.txt
    Friday, December 4, 2015 4:30 AM
  • Please share this dump with me.


    Regards, Vladimir Petter, Microsoft Corporation

    Friday, December 4, 2015 6:58 PM
  • Hi Vladimir

    sorry to hijack this thread. First time posting

    Would you be able to help me decipher my bugcheck crash?

    Running exchange 2010 mailbox server on windows 2008r2. Server crashed yesterday with

    The bugcheck was: 0x0000009e (0xfffffa8014390320, 0x000000000000003c, 0x0000000000000000, 0x0000000000000000).

    the link to the memory dump is here

    https://www.dropbox.com/s/oujf7etgx271a63/MEMORY.DMP?dl=0

    thank you so much

    Thursday, December 31, 2015 6:04 PM
  • Hi, I am getting errors trying to load this dump

    THIS DUMP FILE IS PARTIALLY CORRUPT.
    KdDebuggerDataBlock is not present or unreadable.

    Unfortunately looks like this dump is corrupt.


    Regards, Vladimir Petter, Microsoft Corporation

    Friday, January 1, 2016 1:42 AM
  • Have you made nay changes in Network settings like Teaming or change in Teaming mode from LACP to Auto or any thing?

    because in this case you might be face problem if you have not restarted the server after making this change. 

     


    • Edited by Blog.Patel Friday, January 1, 2016 4:36 AM
    Friday, January 1, 2016 4:36 AM
  • that is weird. server is a virtual machine running under vmware esx.

    i guess i have to wait for another crash then

    thank you for your prompt response

    Monday, January 4, 2016 2:57 PM
  • we run a 5 host hyper-v cluster with a SAN as CSV. we've had a few bugcheck E as well caused by netflt.sys .

    One of the Hype-V hosts seem to suffer from BugCheck 9E, {ffffe800f6529900, 4b0, 5, 0}

    I have uploaded support diagnostics using http://home.diagnostics.support.microsoft.com/SelfHelp

    Not sure how to publish them here, but would be nice if somebody experienced from microsoft take a look at it. I am tired of calling Microsoft support as the support has recently degraded a lot and the support is just a waste of time.

    Tuesday, January 5, 2016 8:42 AM
  • Hi, Hopefully someone from support can take a look at this dump. I am in a completely different part of the org - engineering, and do not have access to the support infrastructure.

    If you have support case opened then support folks should be able to help you with this dump. 


    Regards, Vladimir Petter, Microsoft Corporation

    Tuesday, January 5, 2016 9:33 AM
  • If this issue is causing a business outage, a support case really needs to be opened.  This is not a support forum so hoping that a support engineer wanders over here (on their personal time, because like any support organization, they do have charge for company time) might take a while.  If you are having issues with support, I would request an escalation.  If you are paying for support, you need to demand support.

    One thing you need to start with is looking at the physical environment running the ESX hosts.  Even though they are running ESX, the components must still be certified to run  the version of Windows Server running in the VM, in this case, 2008 R2.  You can check at windowsservercatalog.com.


    . : | : . : | : . tim

    Tuesday, January 5, 2016 2:59 PM

  • Hello Vladimir,
    Could you please help in reviewing memory dump uploaded at below link? 

    https://drive.google.com/file/d/0B7HTVM4VNM3SSEVDZElfZmEtejA/view?usp=sharing

    one of the server rebooted due to prowershell. please find debug details


    CRITICAL_OBJECT_TERMINATION (f4)
    A process or thread crucial to system operation has unexpectedly exited or been
    terminated.
    Several processes and threads are necessary for the operation of the
    system; when they are terminated (for any reason), the system can no
    longer function.
    Arguments:
    Arg1: 0000000000000003, Process
    Arg2: fffffa800d393c10, Terminating object
    Arg3: fffffa800d393e48, Process image file name
    Arg4: fffff80001eed8d0, Explanatory message (ascii)

    Debugging Details:
    ------------------

    Page e156a not present in the dump file. Type ".hh dbgerr004" for details
    Page e1b5e not present in the dump file. Type ".hh dbgerr004" for details

    PROCESS_OBJECT: fffffa800d393c10

    IMAGE_NAME:  csrss.exe

    DEBUG_FLR_IMAGE_TIMESTAMP:  0

    MODULE_NAME: csrss

    FAULTING_MODULE: 0000000000000000 

    PROCESS_NAME:  wsmprovhost.exe

    BUGCHECK_STR:  0xF4_wsmprovhost.exe

    DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

    CURRENT_IRQL:  0

    LAST_CONTROL_TRANSFER:  from fffff80001f69353 to fffff80001c6a450

    STACK_TEXT:  
    fffffa60`0c598b28 fffff800`01f69353 : 00000000`000000f4 00000000`00000003 fffffa80`0d393c10 fffffa80`0d393e48 : nt!KeBugCheckEx
    fffffa60`0c598b30 fffff800`01e82358 : fffffa80`0f23bbb0 fffffa80`0f23bbb0 00000000`1a78e4c0 00000000`00000000 : nt!PspCatchCriticalBreak+0x93
    fffffa60`0c598b70 fffff800`01eb5f50 : fffffa80`0f23bbb0 00000000`00000008 00000000`1a78e4c0 fffff800`00000008 : nt! ?? ::NNGAKEGL::`string'+0x110f6
    fffffa60`0c598bc0 fffff800`01c69ef3 : fffffa80`0d393c10 fffffa80`0f23bbb0 fffffa60`0c598ca0 00000000`1a78e4c0 : nt!NtTerminateProcess+0xd8
    fffffa60`0c598c20 00000000`778e6fda : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
    00000000`1a78e3b8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x778e6fda



    Thanks & Regards,
    Asif Iqbal


    Thursday, January 28, 2016 1:16 PM
  • Hello Asif,

    This thread has been about cluster bugcheking machine with 9e. You have hit a different issue. In your case user mode process csrss.exe is getting terminated. This is a critical OS process, and in response to its termination OS is bugchecking machine with CRITICAL_OBJECT_TERMINATION (f4). It looks like process termination was done by wsmprovhost.exe, which I guess is running some administrative action on behalf of a user with SID S-1-5-21-800109603-1914418702-385554406-17320. You can use PsGetSid (https://technet.microsoft.com/en-us/sysinternals/bb897417.aspx) to lookup user name from the SID.


    Regards, Vladimir Petter, Microsoft Corporation

    Friday, January 29, 2016 6:35 AM
  • Vladimir,

    Would you be able to analyze the following minidump?

    https://rbs.syncedtool.com/shares/file/4e7c8229e80029/

    Please let me know if you need additional information. Thank you very much.

    Brett

    Thursday, February 11, 2016 4:09 AM
  • Hello Brett,

    You have a minidump. Unfortunately there is no much I can tell from. Please configure you servers to collect full kernel dump (http://blogs.msdn.com/b/xiz/archive/2013/11/04/how-to-setup-to-collect-memory-dump.aspx) and share a dump next time you get a repro.

    In your case a resource call took over 5 minutes, and RHS.exe terminated itself. Because process termination did not complete in 20 minutes netft.sys bugchecked machine with 9e. Process termination cannot complete because one of the threads of that process is stuck waiting for the kernel to complete an IO. From my experience most often we find that this IO is stuck somewhere in the storage stack. If you will be able to get full kernel dump for 9e then I should be able to track down why IO is stuck.

    I see this dump is from VMware virtual machine. Please check that at the time of failure VMware did not go through some storage problem.


    Regards, Vladimir Petter, Microsoft Corporation

    Thursday, February 11, 2016 6:30 AM
  • Hi Vladimir,

    I have the same issue - 3 Node Hyper-V cluster - Windows 2012 R2, iSCSI Netapp Storage - 4 CSV disk, which are holding the VM`s.

    Could you please check, if you have some time the dump I have. I have checked some things in it, but I am not so good with the Dump reading and analyzing:

    https://drive.google.com/file/d/0B75QCMrGcihoVnEyY1FGb1g4aEU/view?usp=sharing

     Thank you in advance.

    Monday, July 25, 2016 12:15 PM
  • Something going on with the system disk (HP disk). IOs are stuck on the disk for 30+, and this disk is being reset. Most likely disk reset is stuck. Try to install latest firmaware and drivers from HP. If issue is not resolved then contact HP support.

    Thanks,

    Vladimir. 

    ffffe000b42c3770 [1,2] HP LOGICAL VOLUME Paging Disk      

    Storage class device ffffe000b42c3770 with extension at ffffe000b42c38c0
    Classpnp Internal Information at ffffe000b41f1010
        Transfer Packet Engine:
         Packet          Status  DL Irp          Opcode  Sector/ListId   UL Irp
        --------         ------ --------         ------ --------------- --------
        ffffe000b8eddb60 Queued ffffe000b8edd840   00   ffffe000b8953640 \Windows\System32\winevt\Logs\Microsoft-Windows-MsLbfoProvider%4Operational.evtx
        ffffe000b8eacd40 Queued ffffe000b8eaca20   00   ffffe8007a1b4010 \Windows\Cluster\clussvc.exe
        ffffe8007849d4c0 Queued ffffe8007a32b010   00   ffffe000b8e5bcf0 \ProgramData\Microsoft\Windows\WER\ReportQueue\NonCritical_7.9.9600.18039_40833763cfe4a66227486a838a61544353f89a8_00000000_e3915ddc\Report.wer
        ffffe8007a199e80 Queued ffffe8007a32e4e0   00   ffffe000b89b68d0 \Windows\System32\winevt\Logs\Microsoft-Windows-FailoverClustering%4Diagnostic.evtx
        ffffe8007bc4c670 Queued ffffe8007a2f5350   00   ffffe000ba9f2010 \Windows\System32\wfp\wfpdiag.etl
        ffffe8007a022960 Queued ffffe8007a3b89d0   00   ffffe8007ceb54e0 \Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\edb.log
        ffffe8007a1cc840 Queued ffffe8007a2f5cf0   00   ffffe80079c1d4e0 \Windows\System32\winevt\Logs\Microsoft-Windows-WMI-Activity%4Operational.evtx
        ffffe8007a1b1e80 Queued ffffe8007a1b1b60   00   ffffe8007cf05340 \$LogFile
        ffffe8007a1b19d0 Queued ffffe8007a1b16b0   00   ffffe000ba26a820 \Windows\System32\winevt\Logs\Microsoft-Windows-FailoverClustering%4Diagnostic.evtx
        ffffe80079014e80 Queued ffffe80078cc6010   00   ffffe000bad19cf0 \Windows\System32\wbem\Repository\OBJECTS.DATA
        Pending Idle Requests: 0x0

        Failed Requests:
               Srb    Scsi                                  
        Opcode Status Status Sense Code  Sector/ListId   Time Stamp
        ------ ------ ------ ---------- --------------- ------------
          1a     04     02    05 24 00                  07:37:37.124  
    Classpnp External Information at ffffe000b42c38c0
        HP LOGICAL VOLUME 6.64 5001438018152350
        Minidriver information at ffffe000b42c3d80
        Attached device object at ffffe000b44477c0
        Physical device object at ffffe000b44477c0
        Media Geometry:
            Bytes in a Sector = 512
            Sectors per Track = 32
            Tracks / Cylinder = 255
            Media Length      = 299966445568 bytes = ~279 GB

        -- dt classpnp!_FUNCTIONAL_DEVICE_EXTENSION ffffe000b42c38c0 --

    8: kd> !irp ffffe000b8953640
    Irp is active with 8 stacks 4 is current (= 0xffffe000b89537e8)
     Mdl=ffffd000524757e0: No System Buffer: Thread ffffe000ba251880:  Irp stack trace. 
         cmd  flg cl Device   File     Completion-Context
     [N/A(0), N/A(0)]
                0  0 00000000 00000000 00000000-00000000   
       Args: 00000000 00000000 00000000 00000000
     [N/A(0), N/A(0)]
                0  0 00000000 00000000 00000000-00000000   
       Args: 00000000 00000000 00000000 00000000
     [N/A(0), N/A(0)]
                0  0 00000000 00000000 00000000-00000000   
       Args: 00000000 00000000 00000000 00000000
    >[IRP_MJ_WRITE(4), N/A(34)]
               10 e0 ffffe000b42c3770 00000000 fffff801659e7360-00000000 Success Error Cancel
            \Driver\disk partmgr!PmIoCompletion
       Args: 0000a000 00000000 4bf55000 00000000
     [IRP_MJ_WRITE(4), N/A(3)]
               10 e0 ffffe000b42c5a40 00000000 fffff801656011b0-ffffe000b41f6a30 Success Error Cancel
            \Driver\partmgr volmgr!VmpReadWriteCompletionRoutine
       Args: 51af0eed0a 00000000 4bf55000 00000000
     [IRP_MJ_WRITE(4), N/A(0)]
                0 e0 ffffe000b41f68e0 00000000 fffff8016639c410-ffffe000b4600180 Success Error Cancel
            \Driver\volmgr volsnap!VspRefCountCompletionRoutine
       Args: 0000a000 00000000 51af0eed08 00000000
     [IRP_MJ_WRITE(4), N/A(0)]
                0 e1 ffffe000b4600030 00000000 fffff80165c34a14-ffffd000524751c0 Success Error Cancel pending
            \Driver\volsnap Ntfs
       Args: 0000a000 00000000 36055000 00000000
     [IRP_MJ_WRITE(4), N/A(0)]
                0  0 ffffe000b3864030 ffffe000b49282c0 00000000-00000000   
            \FileSystem\Ntfs
       Args: 0000a000 00000000 000c1000 00000000

    THREAD ffffe000ba251880  Cid 0004.2984  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
        ffffd000524751c8  NotificationEvent
    IRP List:
        ffffe000b8953640: (0006,0310) Flags: 00060043  Mdl: ffffd000524757e0
    Not impersonating
    DeviceMap                 ffffc00157a0c320
    Owning Process            ffffe000b3816480       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      9579627        Ticks: 133171 (0:00:34:40.796)
    Context Switch Count      64043          IdealProcessor: 5            
    UserTime                  00:00:00.000
    KernelTime                00:00:01.359
    Win32 Start Address nt!ExpWorkerThread (0xfffff801fe4ce390)
    Stack Init ffffd00052475dd0 Current ffffd00052474a70
    Base ffffd00052476000 Limit ffffd00052470000 Call 0
    Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`52474ab0 fffff801`fe4d24ee nt!KiSwapContext+0x76
    ffffd000`52474bf0 fffff801`fe4d1f69 nt!KiSwapThread+0x14e
    ffffd000`52474c90 fffff801`fe487563 nt!KiCommitThreadWait+0x129
    ffffd000`52474d10 fffff801`65c2e422 nt!KeWaitForSingleObject+0x373
    (Inline Function) --------`-------- Ntfs!NtfsWaitOnIo+0x1e (Inline Function @ fffff801`65c2e422)
    ffffd000`52474da0 fffff801`65c30734 Ntfs!NtfsNonCachedIo+0x412
    ffffd000`52475010 fffff801`65c33b63 Ntfs!NtfsCommonWrite+0xd54
    ffffd000`52475250 fffff801`653a2b1e Ntfs!NtfsFsdWrite+0x203
    ffffd000`52475520 fffff801`653a10c2 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x2ce
    ffffd000`524755c0 fffff801`fe4a2d80 fltmgr!FltpDispatch+0xb2
    ffffd000`52475620 fffff801`fe4a2baa nt!IoSynchronousPageWrite+0x138
    ffffd000`52475660 fffff801`fe51b2cd nt!MiIssueSynchronousFlush+0x66
    ffffd000`524756d0 fffff801`fe4a25e5 nt!MiFlushSectionInternal+0x775
    ffffd000`524758e0 fffff801`fe45f0ab nt!MmFlushSection+0xa9
    ffffd000`524759a0 fffff801`fe45f7cb nt!CcFlushCachePriv+0x493
    (Inline Function) --------`-------- nt!CcFlushCache+0x1d (Inline Function @ fffff801`fe45f7cb)
    ffffd000`52475ab0 fffff801`fe4a63f5 nt!CcWriteBehindInternal+0x17b
    ffffd000`52475b40 fffff801`fe4a67ad nt!CcWriteBehind+0x95
    ffffd000`52475be0 fffff801`fe4cea2f nt!CcWorkerThread+0x22d
    ffffd000`52475c90 fffff801`fe514c10 nt!ExpWorkerThread+0x69f
    ffffd000`52475d40 fffff801`fe56e8c6 nt!PspSystemThreadStartup+0x58
    ffffd000`52475da0 00000000`00000000 nt!KxStartSystemThread+0x16

       DO ffffe000b44477c0   Ext ffffe000b4447910   Adapter ffffe000b42241a0   Working
       Vendor: HP         Product: LOGICAL VOLUME     SCSI ID: (0, 4, 0)  
       Claimed Enumerated
       SlowLock Free   RemLock 13   PageCount 1
       QueueTagList: ffffe000b44479d0      Outstanding: Head ffffe000b4438040  Tail ffffe000b4448040  Timeout -2 (Timed Out)
       DeviceQueue ffffe000b4447a00   Depth: 24   Status: Not Frozen   PauseCount: 1   BusyCount: 0  
       IO Gateway: Busy Count 0   Pause Count 0
       Requests: Outstanding 6   Device 4   ByPass 0

    [Device-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ffffe8007a2f5350  [SCSI]     ffffe80078803820    n/a         SCSI/WRITE (10)   ffffe000b92e3f40    n/a             60s
    ffffe000b8eaca20  [SCSI]     ffffe000b9278e80    n/a         SCSI/READ (10)    ffffe000b6d06b30    n/a             60s
    ffffe8007a32b010  [SCSI]     ffffe8007935e230    n/a         SCSI/READ (10)    ffffe000b8cb7010    n/a             60s
    ffffe8007a32e4e0  [SCSI]     ffffe800787f5710    n/a         SCSI/WRITE (10)   ffffe8007a152f48    n/a             60s

    [Bypass-Queued Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------

    [Outstanding Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ffffe8007a3b89d0  [SCSI]     ffffe80077e65a10  ffffe000b4438010  SCSI/WRITE (10)   ffffe000b8f1ddf0  ffffe000b44380e0  60s
    ffffe80078cc6010  [SCSI]     ffffe80077f01990  ffffe000b4422010  SCSI/WRITE (10)   ffffd000548f02b0  ffffe000b44220e0  60s
    ffffe8007a2f5cf0  [SCSI]     ffffe800788c17b0  ffffe000b443c010  SCSI/WRITE (10)   ffffd000212357e0  ffffe000b443c0e0  60s
    ffffe8007a1b1b60  [SCSI]     ffffe80077312090  ffffe000b442c010  SCSI/WRITE (10)   ffffe000b4253720  ffffe000b442c0e0  60s
    ffffe8007a1b16b0  [SCSI]     ffffe80078823d10  ffffe000b441a010  SCSI/READ (10)    ffffe8007874e420  ffffe000b441a0e0  60s
    ffffe000b8edd840  [SCSI]     ffffe000b4714b40  ffffe000b4432010  SCSI/WRITE (10)   ffffd000524757e0  ffffe000b44320e0  60s
    ffffe800773256f0  [SCSI]     ffffe000b8c7c770  ffffe000b4448010  RESET LUN         0000000000000000  0000000000000000  0s

    [Completed Requests]
    IRP               SRB Type   SRB               XRB               Command           MDL               SGList            Timeout
    -----------------------------------------------------------------------------------------------------------------------------------
    ERROR: 7 counted requests > 6 outstanding requests


    Regards, Vladimir Petter, Microsoft Corporation

    Tuesday, July 26, 2016 2:12 AM
  • Hi Vladimir,

    Thank you so much for your time. You are machine.

    I`ll check with HP, because we are already with the latest SPP from HP.

    Tuesday, July 26, 2016 6:21 AM
  • Hi Vladimir,

    I've got the same problem on my Windows 2012R2 HyperV cluster.

    I checked the memory dump but could not identify the real problem. Here are the dump information. Could you help me to see what's going wrong ?

    10: kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    USER_MODE_HEALTH_MONITOR (9e)
    One or more critical user mode components failed to satisfy a health check.
    Hardware mechanisms such as watchdog timers can detect that basic kernel
    services are not executing. However, resource starvation issues, including
    memory leaks, lock contention, and scheduling priority misconfiguration,
    may block critical user mode components without blocking DPCs or
    draining the nonpaged pool.
    Kernel components can extend watchdog timer functionality to user mode
    by periodically monitoring critical applications. This bugcheck indicates
    that a user mode health check failed in a manner such that graceful
    shutdown is unlikely to succeed. It restores critical services by
    rebooting and/or allowing application failover to other servers.
    Arguments:
    Arg1: ffffe0002424b900, Process that failed to satisfy a health check within the
        configured timeout
    Arg2: 0000000000000690, Health monitoring timeout (seconds)
    Arg3: 0000000000000005
    Arg4: 0000000000000000

    Debugging Details:
    ------------------


    PROCESS_OBJECT: ffffe0002424b900

    DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

    BUGCHECK_STR:  0x9E

    PROCESS_NAME:  System

    CURRENT_IRQL:  2

    ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre

    LAST_CONTROL_TRANSFER:  from fffff800202c2468 to fffff802f9f519a0

    STACK_TEXT:  
    ffffd000`cd2bf968 fffff800`202c2468 : 00000000`0000009e ffffe000`2424b900 00000000`00000690 00000000`00000005 : nt!KeBugCheckEx
    ffffd000`cd2bf970 fffff800`202c20f2 : 00000000`00000000 00000000`00000001 ffffd000`cd2aa180 00000000`00000000 : netft!NetftProcessWatchdogEvent+0xe4
    ffffd000`cd2bf9b0 fffff802`f9ecb3a8 : ffffd000`cd2bfb20 00000000`00000000 ffffe000`240d49f0 fffff800`2013acd5 : netft!NetftWatchdogTimerDpc+0x36
    ffffd000`cd2bf9e0 fffff802`f9f554ea : ffffd000`cd2aa180 ffffd000`cd2aa180 ffffd000`cd2b6bc0 ffffe000`2c08e040 : nt!KiRetireDpcList+0x4f8
    ffffd000`cd2bfc60 00000000`00000000 : ffffd000`cd2c0000 ffffd000`cd2ba000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x5a


    STACK_COMMAND:  kb

    FOLLOWUP_IP:
    netft!NetftProcessWatchdogEvent+e4
    fffff800`202c2468 cc              int     3

    SYMBOL_STACK_INDEX:  1

    SYMBOL_NAME:  netft!NetftProcessWatchdogEvent+e4

    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: netft

    IMAGE_NAME:  netft.sys

    DEBUG_FLR_IMAGE_TIMESTAMP:  5434e860

    BUCKET_ID_FUNC_OFFSET:  e4

    FAILURE_BUCKET_ID:  0x9E_netft!NetftProcessWatchdogEvent

    BUCKET_ID:  0x9E_netft!NetftProcessWatchdogEvent

    ANALYSIS_SOURCE:  KM

    FAILURE_ID_HASH_STRING:  km:0x9e_netft!netftprocesswatchdogevent

    FAILURE_ID_HASH:  {fc992d70-4714-ccd6-c6b5-601c2a57cb6c}

    Followup: MachineOwner
    ---------

    10: kd> !process ffffe0002424b900
    PROCESS ffffe0002424b900
        SessionId: 0  Cid: 153c    Peb: 7ff6f6277000  ParentCid: 09cc
        DirBase: 13da7a000  ObjectTable: ffffc0016bb10040  HandleCount: <Data Not Accessible>
        Image: rhs.exe
        VadRoot ffffe00022e05380 Vads 87 Clone 0 Private 592. Modified 12. Locked 0.
        DeviceMap ffffc0015d00c3a0
        Token                             ffffc00165ce9060
        ElapsedTime                       02:00:11.093
        UserTime                          00:00:00.031
        KernelTime                        00:00:00.031
        QuotaPoolUsage[PagedPool]         76408
        QuotaPoolUsage[NonPagedPool]      11248
        Working Set Sizes (now,min,max)  (1952, 50, 345) (7808KB, 200KB, 1380KB)
        PeakWorkingSetSize                1891
        VirtualSize                       2097191 Mb
        PeakVirtualSize                   2097193 Mb
        PageFaultCount                    2285
        MemoryPriority                    BACKGROUND
        BasePriority                      13
        CommitCharge                      639

            THREAD ffffe0002424e080  Cid 153c.1654  Teb: 00007ff6f627e000 Win32Thread: fffff901406ddb60 WAIT: (UserRequest) UserMode Non-Alertable
                ffffe8006ee78960  NotificationEvent
                ffffe0002349f080  ProcessObject
                ffffe000031f11b0  NotificationEvent
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      329056         Ticks: 144294 (0:00:37:34.593)
            Context Switch Count      60             IdealProcessor: 23             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.046
            Win32 Start Address 0x00007ff6f6ce1c80
            Stack Init ffffd000d503dc90 Current ffffd000d503cfc0
            Base ffffd000d503e000 Limit ffffd000d5038000 Call 0
            Priority 14 BasePriority 13 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Kernel stack not resident.
            Child-SP          RetAddr           Call Site
            ffffd000`d503d000 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d503d140 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d503d1e0 fffff802`f9e31f43 nt!KiCommitThreadWait+0x129
            ffffd000`d503d260 fffff802`fa1c9929 nt!KeWaitForMultipleObjects+0x403
            ffffd000`d503d320 fffff802`fa1c967a nt!ObWaitForMultipleObjects+0x289
            ffffd000`d503d830 fffff802`f9f5d1b3 nt!NtWaitForMultipleObjects+0xd6
            ffffd000`d503da90 00007ffb`b3cb11fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d503db00)
            0000006d`24a5edc8 00000000`00000000 0x00007ffb`b3cb11fa

            THREAD ffffe8006ee7f800  Cid 153c.1650  Teb: 00007ff6f627c000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
                ffffe8006ee7f750  NotificationEvent
                ffffe8006ee7f6d0  SynchronizationEvent
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      338992         Ticks: 134358 (0:00:34:59.343)
            Context Switch Count      181            IdealProcessor: 25             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.000
            Win32 Start Address 0x00007ff6f6cdf550
            Stack Init ffffd000d5acec90 Current ffffd000d5acdfc0
            Base ffffd000d5acf000 Limit ffffd000d5ac9000 Call 0
            Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Kernel stack not resident.
            Child-SP          RetAddr           Call Site
            ffffd000`d5ace000 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d5ace140 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d5ace1e0 fffff802`f9e31f43 nt!KiCommitThreadWait+0x129
            ffffd000`d5ace260 fffff802`fa1c9929 nt!KeWaitForMultipleObjects+0x403
            ffffd000`d5ace320 fffff802`fa1c967a nt!ObWaitForMultipleObjects+0x289
            ffffd000`d5ace830 fffff802`f9f5d1b3 nt!NtWaitForMultipleObjects+0xd6
            ffffd000`d5acea90 00007ffb`b3cb11fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5aceb00)
            0000006d`252bf678 00000000`00000000 0x00007ffb`b3cb11fa

            THREAD ffffe8006ee84080  Cid 153c.16d8  Teb: 00007ff6f627a000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
                ffffe8006dd2f060  SynchronizationEvent
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      329056         Ticks: 144294 (0:00:37:34.593)
            Context Switch Count      16             IdealProcessor: 27             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.000
            Win32 Start Address 0x00007ffbb3c533a0
            Stack Init ffffd000d5506c90 Current ffffd000d5506760
            Base ffffd000d5507000 Limit ffffd000d5501000 Call 0
            Priority 15 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Kernel stack not resident.
            Child-SP          RetAddr           Call Site
            ffffd000`d55067a0 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d55068e0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d5506980 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
            ffffd000`d5506a00 fffff802`fa1eeec2 nt!KeWaitForSingleObject+0x2c0
            ffffd000`d5506a90 fffff802`f9f5d1b3 nt!NtWaitForSingleObject+0xb2
            ffffd000`d5506b00 00007ffb`b3cb0c8a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5506b00)
            0000006d`2533f6a8 00000000`00000000 0x00007ffb`b3cb0c8a

            THREAD ffffe8006ee86080  Cid 153c.16c8  Teb: 00007ff6f6275000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
                ffffe8006ee7f590  SynchronizationEvent
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      329056         Ticks: 144294 (0:00:37:34.593)
            Context Switch Count      15             IdealProcessor: 31             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.000
            Win32 Start Address 0x00007ffbaa0c1dc0
            Stack Init ffffd000d54fcc90 Current ffffd000d54fc760
            Base ffffd000d54fd000 Limit ffffd000d54f7000 Call 0
            Priority 15 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Kernel stack not resident.
            Child-SP          RetAddr           Call Site
            ffffd000`d54fc7a0 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d54fc8e0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d54fc980 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
            ffffd000`d54fca00 fffff802`fa1eeec2 nt!KeWaitForSingleObject+0x2c0
            ffffd000`d54fca90 fffff802`f9f5d1b3 nt!NtWaitForSingleObject+0xb2
            ffffd000`d54fcb00 00007ffb`b3cb0c8a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d54fcb00)
            0000006d`2549fe58 00000000`00000000 0x00007ffb`b3cb0c8a

            THREAD ffffe8006ee87080  Cid 153c.1708  Teb: 00007ff6f6273000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
                ffffe0002bd3b360  SynchronizationEvent
            IRP List:
                ffffe80072954cf0: (0006,0310) Flags: 00000884  Mdl: 00000000
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      365873         Ticks: 107477 (0:00:27:59.328)
            Context Switch Count      132            IdealProcessor: 33             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.000
            Win32 Start Address 0x00007ff6f6cbe880
            Stack Init ffffd000d563ac90 Current ffffd000d5638e60
            Base ffffd000d563b000 Limit ffffd000d5635000 Call 0
            Priority 14 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    *** ERROR: Module load completed but symbols could not be loaded for EX64.SYS
    *** ERROR: Module load completed but symbols could not be loaded for SRTSP64.SYS

            Child-SP          RetAddr           Call Site
            ffffd000`d5638ea0 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d5638fe0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d5639080 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
            ffffd000`d5639100 fffff802`fa290399 nt!KeWaitForSingleObject+0x2c0
            ffffd000`d5639190 fffff802`fa1e55b7 nt!IopAcquireFileObjectLock+0x85
            ffffd000`d56391e0 fffff802`f9f5d1b3 nt!NtReadFile+0x8d7
            ffffd000`d56392d0 fffff802`f9f55600 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5639340)
            ffffd000`d56394d8 fffff800`1f4bc73c nt!KiServiceLinkage
            ffffd000`d56394e0 fffff800`1f4bc89d EX64+0x273c
            ffffd000`d5639550 fffff800`1f4bdf87 EX64+0x289d
            ffffd000`d56395a0 fffff800`1f4c0087 EX64+0x3f87
            ffffd000`d56395f0 fffff800`1f4c0468 EX64+0x6087
            ffffd000`d56396b0 fffff800`1f4c0350 EX64+0x6468
            ffffd000`d5639840 fffff800`1f4c0b35 EX64+0x6350
            ffffd000`d5639910 fffff800`1f4c0e0d EX64+0x6b35
            ffffd000`d5639a50 fffff800`1f4c2a4c EX64+0x6e0d
            ffffd000`d5639af0 fffff800`1f4c888e EX64+0x8a4c
            ffffd000`d5639cb0 fffff800`1f4c971b EX64+0xe88e
            ffffd000`d5639e00 fffff800`1f4c991e EX64+0xf71b
            ffffd000`d5639ee0 fffff800`1ed5240f EX64+0xf91e
            ffffd000`d5639f40 fffff800`1ed525c7 SRTSP64+0x7940f
            ffffd000`d5639fa0 fffff800`1eda031d SRTSP64+0x795c7
            ffffd000`d563a000 fffff800`1ed9e198 SRTSP64+0xc731d
            ffffd000`d563a070 fffff800`1ed9debc SRTSP64+0xc5198
            ffffd000`d563a0f0 fffff800`1ed08718 SRTSP64+0xc4ebc
            ffffd000`d563a150 fffff800`1ed08a40 SRTSP64+0x2f718
            ffffd000`d563a180 fffff800`1ed406b1 SRTSP64+0x2fa40
            ffffd000`d563a1b0 fffff800`1e6fc9d7 SRTSP64+0x676b1
            ffffd000`d563a200 fffff800`1e6fd14d fltmgr!FltpPerformPostCallbacks+0x2d7
            ffffd000`d563a2d0 fffff800`1e6fbbc1 fltmgr!FltpPassThroughCompletionWorker+0x7d
            ffffd000`d563a340 fffff800`1e724349 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x371
            ffffd000`d563a3e0 fffff802`fa240b91 fltmgr!FltpCreate+0x339
            ffffd000`d563a490 fffff802`fa1ecc39 nt!IopParseDevice+0x6c1
            ffffd000`d563a6b0 fffff802`fa1eaa63 nt!ObpLookupObjectName+0x6b9
            ffffd000`d563a830 fffff802`fa2637ab nt!ObOpenObjectByName+0x1e3
            ffffd000`d563a960 fffff802`fa263438 nt!IopCreateFile+0x36b
            ffffd000`d563aa00 fffff802`f9f5d1b3 nt!NtCreateFile+0x78
            ffffd000`d563aa90 00007ffb`b3cb119a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d563ab00)
            0000006d`2561de18 00000000`00000000 0x00007ffb`b3cb119a

            THREAD ffffe8006ee8b080  Cid 153c.1774  Teb: 00007ff6f614a000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
                ffffe8006d89d710  NotificationEvent
                ffffe8006ec92d60  NotificationEvent
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      436170         Ticks: 37180 (0:00:09:40.937)
            Context Switch Count      421            IdealProcessor: 39             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.000
            Win32 Start Address 0x00007ffbb30f0b70
            Stack Init ffffd000d5adfc90 Current ffffd000d5adefc0
            Base ffffd000d5ae0000 Limit ffffd000d5ada000 Call 0
            Priority 14 BasePriority 13 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Kernel stack not resident.
            Child-SP          RetAddr           Call Site
            ffffd000`d5adf000 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d5adf140 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d5adf1e0 fffff802`f9e31f43 nt!KiCommitThreadWait+0x129
            ffffd000`d5adf260 fffff802`fa1c9929 nt!KeWaitForMultipleObjects+0x403
            ffffd000`d5adf320 fffff802`fa1c967a nt!ObWaitForMultipleObjects+0x289
            ffffd000`d5adf830 fffff802`f9f5d1b3 nt!NtWaitForMultipleObjects+0xd6
            ffffd000`d5adfa90 00007ffb`b3cb11fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5adfb00)
            0000006d`25a7f8f8 00000000`00000000 0x00007ffb`b3cb11fa

            THREAD ffffe000263ee880  Cid 153c.151c  Teb: 00007ff6f614e000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
                ffffe000263eeeb0  Semaphore Limit 0x1
            Waiting for reply to ALPC Message ffffc0015e5b9d00 : queued at port ffffe00023a68e40 : owned by process ffffe00023469900
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      338992         Ticks: 134358 (0:00:34:59.343)
            Context Switch Count      545            IdealProcessor: 31             
            UserTime                  00:00:00.031
            KernelTime                00:00:00.031
            Win32 Start Address 0x00007ffbb3c533a0
            Stack Init ffffd000d7fe6c90 Current ffffd000d7fe6400
            Base ffffd000d7fe7000 Limit ffffd000d7fe1000 Call 0
            Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Kernel stack not resident.
            Child-SP          RetAddr           Call Site
            ffffd000`d7fe6440 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`d7fe6580 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`d7fe6620 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
            ffffd000`d7fe66a0 fffff802`f9e3d9c3 nt!KeWaitForSingleObject+0x2c0
            ffffd000`d7fe6730 fffff802`fa1cb3c9 nt!AlpcpSignalAndWait+0x443
            ffffd000`d7fe67e0 fffff802`fa1ca8c7 nt!AlpcpReceiveSynchronousReply+0x55
            ffffd000`d7fe6840 fffff802`fa1ec520 nt!AlpcpProcessSynchronousRequest+0x377
            ffffd000`d7fe69c0 fffff802`f9f5d1b3 nt!NtAlpcSendWaitReceivePort+0x270
            ffffd000`d7fe6a90 00007ffb`b3cb14ca nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d7fe6b00)
            0000006d`2569e138 00000000`00000000 0x00007ffb`b3cb14ca

            THREAD ffffe8006ec4e880  Cid 153c.1784  Teb: 00007ff6f6278000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Alertable
                ffffe8006d89ec80  QueueObject
            Not impersonating
            DeviceMap                 ffffc0015d00c3a0
            Owning Process            ffffe0002424b900       Image:         rhs.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      472644         Ticks: 706 (0:00:00:11.031)
            Context Switch Count      329            IdealProcessor: 33             
            UserTime                  00:00:00.000
            KernelTime                00:00:00.000
            Win32 Start Address 0x00007ffbb3c533a0
            Stack Init ffffd000222f9c90 Current ffffd000222f9420
            Base ffffd000222fa000 Limit ffffd000222f4000 Call 0
            Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
            Child-SP          RetAddr           Call Site
            ffffd000`222f9460 fffff802`f9e32d7e nt!KiSwapContext+0x76
            ffffd000`222f95a0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
            ffffd000`222f9640 fffff802`f9e314cd nt!KiCommitThreadWait+0x129
            ffffd000`222f96c0 fffff802`f9e3104a nt!KeRemoveQueueEx+0x26d
            ffffd000`222f9750 fffff802`f9e306fb nt!IoRemoveIoCompletion+0x8a
            ffffd000`222f9870 fffff802`f9f5d1b3 nt!NtWaitForWorkViaWorkerFactory+0x30b
            ffffd000`222f9a90 00007ffb`b3cb273a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`222f9b00)
            0000006d`2541f3f8 00000000`00000000 0x00007ffb`b3cb273a

    Friday, November 11, 2016 5:19 PM
  • Please open a case with Microsoft support, and they will be able to analyze the memory dump for you and root cause what went wrong.

    Thanks!
    Elden

    Saturday, November 12, 2016 12:03 AM
    Owner
  • Hello Vladimir,  Sorry for replying to an old thread.  But we just had a cluster server stop error with a 9E with the watch dog event.  I've run the memory dump through the debugger and I can't find anything obvious about why it reached the timeout / bug-check threshold.  I was wondering if someone could look at it with more knowledge in the debugger than I have?

    Here is the 7-zip'd file.

    https://1drv.ms/u/s!Aj_NX6R0CE7qeXLZMJXSCGH-Lqg

    Thanks,

    Dean


    Friday, May 19, 2017 8:37 PM
  • Hello Vladimir,

    Would you be helping me in analyzing a memory dump with bugcheck 19, {0x22, ffffe00017fb3000, 0, 0}?

    server is a VM running windows 2012 on Esxi 6.0, it is rebooting after every 2-3 days with above mentioned bugcheck.

    Thanks & Regards,

    Suchit

    Wednesday, September 27, 2017 2:53 PM
  • HI Vladimir,

    Can you please analyze the memory dump file generated.

    location: https://ibm.box.com/s/dnvntrh4dg2kd8gexqhbo9yiizg6pqnn

    Wednesday, February 14, 2018 8:42 AM
    • Edited by LaMerk Sunday, March 25, 2018 1:21 AM
    Sunday, March 25, 2018 1:20 AM
  • Hi, I am having similar issue. Can I share the dump file?
    Thursday, October 4, 2018 8:12 PM
  • HI,Vladimir:

           What command did you use for this dump file?I also suspect that it is a vmware disk problem.Thank you!

    Monday, June 10, 2019 3:11 PM
  • Hello Hardy.

    If you are looking at bugcheck 9e then first parameter of bugcheck is address of the process this watchdog was for. Start by running "!process <address> 1f". Stepping back a bit, always start with

    .symfix

    .reload

    and then start trying other commands. This will fix symbols for you. Use lml to see what symbols are loaded.

    Not going back. Once you dump that process look at the threads, and see if they are blocked on something in kernel. For instance if they are waiting on ERESOURCE then try to get address of ERESOURCE and use !locks <address> to see who owns it. Keep following wait chain.

    If you get stuck then you can try your like by checking what is going on in storage using

    !storagekd.unit

    !storagekd.classext

    I am inside Microsoft corpnet so I have access to private symbols, while you might be limited to public symbols only. As such I am not sure if storagekd will work for you. I am also not sure if it ships with public version of windbg. If you are blocked and can find a way to give me access to the dump I can take a look.

    Regards,

    Vladimir.


    Regards, Vladimir Petter, Microsoft Corporation

    Monday, June 10, 2019 3:23 PM
  • HI,Vladimir:

           Thanks for your help, I have tried many times using windbg to prompt DBGHELP: Timeout to store: c:\symbols*http://msdl.microsoft.com/download/symbols. Please help you analyze the dmp file, thank you!

    https://1drv.ms/u/s!ApZTZXkGWOEB2zmFbxqB072hED7A?e=dut0Bx

    Tuesday, June 11, 2019 4:42 PM
  • What you have is a kernel minidump. Minidump would not be helpful for this class of issues because it includes only stack of the thread that triggered bugcheck, and we need all kernel threads and non-paged pool so we can follow threads wait chain. You need to change settings to collect full kernel dump on the next repro https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/enabling-a-kernel-mode-dump-file

    Regards,

    Vladimir Petter.


    Regards, Vladimir Petter, Microsoft Corporation

    Tuesday, June 11, 2019 6:43 PM
  • HI,Vladimir:

           I have a system failure, I hope you can help analyze the problem! I hope you can open the link: https://pan.baidu.com/s/1zz6Xkj_qzlwiK2QrbkO2VQ   Extraction code: if3l   Thank you!

    Wednesday, July 24, 2019 5:25 PM
  • HI,Vladimir:

          Not sure if you can open the Baidu network disk, I will change it to onedrive: https://1drv.ms/u/s!ApZTZXkGWOEB3DKLkm7MwHd-i_Wu   .Thank you!

    Thursday, July 25, 2019 2:48 AM
  • HI,Vladimir

    We have system failure at the same time at two Hyperv Cluster Windows 2012R2. Some hosts in the cluster restarted with bugcheck  code 0x0000009E.  In memory dump I can onli get this.  All host are fw and software up tu date.

    I put all dumps on share (mini and some full memory dump)

    https://izum-my.sharepoint.com/:f:/g/personal/gorazd_taciga_izum_si/Eg3EH-lS-o9LntvTfxvfLqQBSvOCnEk6hs1FhXmv2lE6Ew?e=a1ii15

    I hope you can help analyze the problem. 

    4: kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************
    
    USER_MODE_HEALTH_MONITOR (9e)
    One or more critical user mode components failed to satisfy a health check.
    Hardware mechanisms such as watchdog timers can detect that basic kernel
    services are not executing. However, resource starvation issues, including
    memory leaks, lock contention, and scheduling priority misconfiguration,
    may block critical user mode components without blocking DPCs or
    draining the nonpaged pool.
    Kernel components can extend watchdog timer functionality to user mode
    by periodically monitoring critical applications. This bugcheck indicates
    that a user mode health check failed in a manner such that graceful
    shutdown is unlikely to succeed. It restores critical services by
    rebooting and/or allowing application failover to other servers.
    Arguments:
    Arg1: ffffe000234e7080, Process that failed to satisfy a health check within the
    	configured timeout
    Arg2: 000000000000003c, Health monitoring timeout (seconds)
    Arg3: 000000000000000a, WatchdogSourceClussvcIsAlive
    	Cluster service sends heartbeat to netft every 500 millseconds.
    	By default netft expects at least 1 heartbeat per second.
    	If this watchdog was triggered that means clussvc is o not getting
    	CPU to send heartbers.
    Arg4: 0000000000000000

    Regards

    Gorazd 


    Gorazd

    Tuesday, October 1, 2019 8:08 AM
  • You have DPC storm on CPU0 from network activity. DPC storm is starving cluster service timer that is triggering sending heart to netft.sys. Since netft.sys has not recieved HB in 60 seconds it bugchecked machine.

    To solve this issue you need to setup RSS/vRSS/VMQ to make sure network traffic is equally spread across all CPUs instead of all happening on CPU0.

    In Windows Server 2019 we made number of improvements to reduce number of false positives watchdog timeouts.

    - We increased timeout to 135 to be larger than commulative DPC timeout 120 seocnds so you would get a bugcheck from commulative DPC watchdog before the cluster watchdog.

    - We changed default watchdog action from bugchck to take a LiveDump, terminate process, and extent watchdog to 20 minutes. Only if process termination does not complete in 20 minutes we would bugcheck machine.

    --------------------------------------------------

    CPU#0
    --------------------------------------------------
    Current DPC: NDIS!ndisInterruptDpc (Normal DPC)
    Debugger Saved IRQL: 0
    Cumulative DPC Time Limit: 120.000 seconds
    Current Cumulative DPC Time: 60.156 seconds
    Single DPC Time Limit: 20.000 seconds
    Current Single DPC Time: 0.000 seconds
    
    DPC History:
    ----------------------------------------
         Deferred Routine                                   Start Tick      End Tick        Duration (in seconds)
    ------------------------------------------------------------------------------------------------
    [00] NDIS!ndisInterruptDpc                              0x3EDB          0x3EDB          -
    [01] NDIS!ndisInterruptDpc                              0x3EDB          0x3EDB          -
    [02] NDIS!ndisInterruptDpc                              0x3EDB          (DPC Active)    
    [03] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [04] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [05] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [06] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [07] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [08] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [09] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [10] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [11] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [12] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [13] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [14] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
    [15] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDB          0.0156
    

    You can also see number of threads are sitting on CPU0 in a ready state for almost 60 seconds

    Processor 0: Ready Threads at priority 30
        THREAD ffffe800edf29080  Cid 0c58.1cfc  Teb: 00007ff692a72000 Win32Thread: 0000000000000000 READY on processor 0
        IRP List:
            ffffe0009b376ee0: (0006,0118) Flags: 00060030  Mdl: 00000000
        Not impersonating
        DeviceMap                 ffffc0000260c370
        Owning Process            ffffe0008fc27080       Image:         vmms.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      12261          Ticks: 3830 (0:00:00:59.843)
        Context Switch Count      931            IdealProcessor: 0             
        UserTime                  00:00:00.015
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ffeeaaf8a10
        Stack Init ffffd0002846dc90 Current ffffd0002846cc80
        Base ffffd0002846e000 Limit ffffd00028468000 Call 0000000000000000
        Priority 30 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`2846ccc0 fffff801`d123519b nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
        ffffd000`2846ce00 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
        ffffd000`2846ce40 fffff801`d1234c8d nt!KeSetSystemGroupAffinityThread+0xfb
        ffffd000`2846ce90 fffff801`d1234a3d nt!KeGenericProcessorCallback+0xdd
        ffffd000`2846d000 fffff801`166326cc nt!KeFlushQueuedDpcsEx+0x11d
        ffffd000`2846d1a0 fffff801`16632627 vmswitch!VmsOmpPortUninitializeFlowTable+0x1c
        ffffd000`2846d1d0 fffff801`1662b52b vmswitch!VmsOmpPortDeleted+0xcb
        ffffd000`2846d220 fffff801`1661ef22 vmswitch!VmsOmObjectRefCountDecrement+0x2f
        ffffd000`2846d250 fffff801`1661d32f vmswitch!VmsOmPortDelete+0x3fa
        (Inline Function) --------`-------- vmswitch!VmsCdpPortDeleteByName+0x24 (Inline Function @ fffff801`1661d32f)
        ffffd000`2846d330 fffff801`1662bcca vmswitch!VmsCdpPortDelete+0x11f
        ffffd000`2846d5c0 fffff801`1533018c vmswitch!VmsCdpDeviceControl+0x416
        ffffd000`2846d5f0 fffff801`15331176 NDIS!ndisDummyIrpHandler+0x88
        ffffd000`2846d620 fffff801`d168e0af NDIS!ndisDeviceControlIrpHandler+0x666
        (Inline Function) --------`-------- nt!IoCallDriverWithTracing+0x7e (Inline Function @ fffff801`d168e0af)
        (Inline Function) --------`-------- nt!IopCallDriverReference+0x187 (Inline Function @ fffff801`d168e0af)
        ffffd000`2846d810 fffff801`d168f018 nt!IopSynchronousServiceTail+0x32b
        ffffd000`2846d8e0 fffff801`d165e8a6 nt!IopXxxControlFile+0xdb8
        ffffd000`2846da20 fffff801`d13592e3 nt!NtDeviceIoControlFile+0x56
        ffffd000`2846da90 00007ffe`eab507ca nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`2846db00)
        00000082`eeefe958 00000000`00000000 0x00007ffe`eab507ca
    
        THREAD ffffe800eded4040  Cid 0004.17e8  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
        IRP List:
            ffffe800f2b85ca0: (0006,0358) Flags: 00000404  Mdl: 00000000
        Not impersonating
        DeviceMap                 ffffc0000260c370
        Owning Process            ffffe00084020480       Image:         System
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      13221          Ticks: 2870 (0:00:00:44.843)
        Context Switch Count      426            IdealProcessor: 0             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address nt!ExpWorkerThread (0xfffff801d1269430)
        Stack Init ffffd00026d3fc90 Current ffffd00026d3f3c0
        Base ffffd00026d40000 Limit ffffd00026d3a000 Call 0000000000000000
        Priority 30 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`26d3f400 fffff801`d123519b nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
        ffffd000`26d3f540 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
        ffffd000`26d3f580 fffff801`d1234c8d nt!KeSetSystemGroupAffinityThread+0xfb
        ffffd000`26d3f5d0 fffff801`d1234a3d nt!KeGenericProcessorCallback+0xdd
        ffffd000`26d3f740 fffff801`d1597e64 nt!KeFlushQueuedDpcsEx+0x11d
        ffffd000`26d3f8e0 fffff801`1620529f nt!MmPageEntireDriver+0xa0
        (Inline Function) --------`-------- Msfs!MsDereferenceVcb+0x26 (Inline Function @ fffff801`1620529f)
        (Inline Function) --------`-------- Msfs!MsCommonClose+0x5b (Inline Function @ fffff801`1620529f)
        ffffd000`26d3f910 fffff801`15474101 Msfs!MsFsdClose+0x6f
        ffffd000`26d3f950 fffff801`d15cde0c fltmgr!FltpDispatch+0xf1
        ffffd000`26d3f9b0 fffff801`d15c1e5c nt!IopDeleteFile+0x128
        ffffd000`26d3fa30 fffff801`d125c9bf nt!ObpRemoveObjectRoutine+0x64
        (Inline Function) --------`-------- nt!ObfDereferenceObjectWithTag+0x80 (Inline Function @ fffff801`d125c9bf)
        ffffd000`26d3fa90 fffff801`1549eb9a nt!ObfDereferenceObject+0x8f
        ffffd000`26d3fad0 fffff801`d1269acf fltmgr!FltpManualDeviceAttachWorker+0xc2
        ffffd000`26d3fb50 fffff801`d12ccdc6 nt!ExpWorkerThread+0x69f
        ffffd000`26d3fc00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
        ffffd000`26d3fc60 00000000`00000000 nt!KxStartSystemThread+0x16
    
    Processor 0: Ready Threads at priority 15
        THREAD ffffe000a3bf9080  Cid 0004.2ffc  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
        Not impersonating
        DeviceMap                 ffffc0000260c370
        Owning Process            ffffe00084020480       Image:         System
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      12940          Ticks: 3151 (0:00:00:49.234)
        Context Switch Count      7              IdealProcessor: 0             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address nt!EtwpLogger (0xfffff801d16afc78)
        Stack Init ffffd0002bad9c90 Current ffffd0002bad9780
        Base ffffd0002bada000 Limit ffffd0002bad4000 Call 0000000000000000
        Priority 15 BasePriority 15 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`2bad97c0 fffff801`d123519b nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
        ffffd000`2bad9900 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
        ffffd000`2bad9940 fffff801`d1234c8d nt!KeSetSystemGroupAffinityThread+0xfb
        ffffd000`2bad9990 fffff801`d1234147 nt!KeGenericProcessorCallback+0xdd
        ffffd000`2bad9b00 fffff801`d169d08e nt!KeGenericCallDpc+0x27
        (Inline Function) --------`-------- nt!EtwpSynchronizeWithElevatedIrqlLogging+0xe (Inline Function @ fffff801`d169d08e)
        ffffd000`2bad9b40 fffff801`d16afef5 nt!EtwpFreeLoggerContext+0x8e
        ffffd000`2bad9b80 fffff801`d12ccdc6 nt!EtwpLogger+0x27d
        ffffd000`2bad9c00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
        ffffd000`2bad9c60 00000000`00000000 nt!KxStartSystemThread+0x16
    
    Processor 0: Ready Threads at priority 14
        THREAD ffffe800ee157880  Cid 1878.2c2c  Teb: 00007ff71d04c000 Win32Thread: fffff90140740610 READY on processor 0
        Not impersonating
        DeviceMap                 ffffc000078c9eb0
        Owning Process            ffffe0008f9b8900       Image:         vmwp.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      12271          Ticks: 3820 (0:00:00:59.687)
        Context Switch Count      85             IdealProcessor: 0             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ffeeaaf8a10
        Stack Init ffffd0002a173c90 Current ffffd0002a1737c0
        Base ffffd0002a174000 Limit ffffd0002a16e000 Call 0000000000000000
        Priority 14 BasePriority 8 PriorityDecrement 16 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`2a173800 fffff801`d12724e7 nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KzCheckForThreadDispatch+0x101 (Inline Function @ fffff801`d12724e7)
        (Inline Function) --------`-------- nt!KiCheckForThreadDispatch+0x101 (Inline Function @ fffff801`d12724e7)
        ffffd000`2a173940 fffff801`d1233439 nt!KiProcessDeferredReadyList+0x13b
        ffffd000`2a173980 fffff801`d160dcbb nt!KeSetLegacyAffinityThread+0xd9
        ffffd000`2a1739e0 fffff801`d13592e3 nt!NtSetInformationThread+0x9ab
        ffffd000`2a173b00 00007ffe`eab5082a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`2a173b00)
        00000088`03cff018 00000000`00000000 0x00007ffe`eab5082a
    
    Processor 0: Ready Threads at priority 12
        THREAD ffffe00085b60880  Cid 0004.01c4  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
        Not impersonating
        DeviceMap                 ffffc0000260c370
        Owning Process            ffffe00084020480       Image:         System
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      12261          Ticks: 3830 (0:00:00:59.843)
        Context Switch Count      7542           IdealProcessor: 0             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.062
        Win32 Start Address nt!ExpWorkerThread (0xfffff801d1269430)
        Stack Init ffffd00077b31c90 Current ffffd00077b31700
        Base ffffd00077b32000 Limit ffffd00077b2c000 Call 0000000000000000
        Priority 12 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`77b31740 fffff801`d123519b nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
        ffffd000`77b31880 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
        ffffd000`77b318c0 fffff801`158023c2 nt!KeSetSystemGroupAffinityThread+0xfb
        (Inline Function) --------`-------- vmbusr!WdmlibKeSetSystemGroupAffinityThread+0x20 (Inline Function @ fffff801`158023c2)
        ffffd000`77b31910 fffff801`15817d24 vmbusr!PncPullMessage+0x4e
        ffffd000`77b31980 fffff801`d126a920 vmbusr!XPartReceiveMessageWorkItem+0x7c
        ffffd000`77b31ae0 fffff801`d1269acf nt!IopProcessWorkItem+0x80
        ffffd000`77b31b50 fffff801`d12ccdc6 nt!ExpWorkerThread+0x69f
        ffffd000`77b31c00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
        ffffd000`77b31c60 00000000`00000000 nt!KxStartSystemThread+0x16
    
    Processor 0: Ready Threads at priority 8
        THREAD ffffe00085e49880  Cid 0004.01f8  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
        Not impersonating
        DeviceMap                 ffffc0000260c370
        Owning Process            ffffe00084020480       Image:         System
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      12261          Ticks: 3830 (0:00:00:59.843)
        Context Switch Count      5              IdealProcessor: 0             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address NDIS!ndisReceiveWorkerThread (0xfffff801152aedd0)
        Stack Init ffffd000775c6c90 Current ffffd000775c67d0
        Base ffffd000775c7000 Limit ffffd000775c1000 Call 0000000000000000
        Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`775c6810 fffff801`d13c7417 nt!KiSwapContext+0x76
        ffffd000`775c6950 fffff801`d134c2af nt!KiQuantumEnd+0x52b
        ffffd000`775c69a0 fffff801`d134eb69 nt!KiDispatchInterruptContinue+0x16
        ffffd000`775c69d0 fffff801`d125daa7 nt!KiDpcInterrupt+0x2a9 (TrapFrame @ ffffd000`775c69d0)
        ffffd000`775c6b68 fffff801`152aefc1 nt!KzLowerIrql+0x7
        ffffd000`775c6b70 fffff801`d12ccdc6 NDIS!ndisReceiveWorkerThread+0x1f1
        ffffd000`775c6c00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
        ffffd000`775c6c60 00000000`00000000 nt!KxStartSystemThread+0x16


    Regards, Vladimir Petter, Microsoft Corporation

    Tuesday, October 1, 2019 6:06 PM
  • Thanks for your help

    Regards

    Gorazd


    Gorazd

    Thursday, October 3, 2019 6:04 AM