none
PE does not start or crashes on heavy loaded Win10Pro machine RRS feed

  • Question

  • Hi all,

    using PE since years to administrate several machines I can't think beeing without such a tool :-)

    Problem: On heavy utilized machines (Win10 64, ~50 users connected via Remote Desktop, approxx. 2000 processes, 1500000 handles) PE does not start. 
    procexp64.exe shows up for ~30 sec in the process list after it disappears. Startet from an elevated CMD, Errorlevel = 0.

    Right after Reboot of the machine and if not that much ressources are used, PE works as intended.

    Is there some limit where usage of PE is not feasible any more?

    Regards,
    Franz



    Edit Nov. 14, 2019:

    Now I observed, that if machine is 'medium' loaded an PE was able to start, it crashes after some minutes. This is what event logging tells me:

    Name der fehlerhaften Anwendung: procexp64.exe, Version: 16.26.0.0, Zeitstempel: 0x5d1680dc
    Name des fehlerhaften Moduls: procexp64.exe, Version: 16.26.0.0, Zeitstempel: 0x5d1680dc
    Errorcode: 0xc0000005
    Erroroffset: 0x0000000000050476
    process ID: 0x17584
    Startzeit der fehlerhaften Anwendung: 0x01d59ac7519da274
    Pfad der fehlerhaften Anwendung: C:\Programme_Admin\ProcessExplorer\procexp64.exe
    Pfad des fehlerhaften Moduls: C:\Programme_Admin\ProcessExplorer\procexp64.exe
    Berichtskennung: e78760c2-1cbb-460a-98e5-dcf396e691b4


    • Edited by Franz_D Thursday, November 14, 2019 9:05 AM
    Wednesday, September 11, 2019 6:42 AM

All replies

  • If Errorlevel = 0 then probably PE choose to exit because it was unable to allocate memory or load the device driver or both.. in that case you may find something logged in the Application or System Event Log.

    Clearly PE is a huge application using a lot of RAM especially if there are so many process running.. so you have to look at Task Manager or RamMap at the Free column.

    If Free is 0 or almost 0, try empty the different sets of memory.. in most cases, you should be able to gain a lot of free memory and at that point PE will start correctly.

    In my case a quick empty and then refresh reached this results:

    Is just for cases like your that I'm asking from several years to implement in RamMap the command abvailable in the Empty menu as command line items, so to quickly release and free the memory when the system is under stress.., but it is still in the To Do List.. :-(

    HTH
    -mario

    Wednesday, September 11, 2019 7:13 AM
  • Hi Mario

    actually we recently added the command line switched to RAMMAP but Mark wanted to defer the publication of this and changes to other tools for the latest tranche of updates to reduce the number of changes we publish in one go. If you ping me offline at syssite@microsoft.com I can make a copy available to you for review..

    MarkC(MSFT)

    Monday, September 16, 2019 9:03 AM
  • RAMMap did not indicate a problem.

    Thursday, November 14, 2019 9:16 AM
  • There is a crash dump (4 of them with v16.26, 1 with v16.30) available. I'm trying to gather information how to provide them to the experts (mailed to syssite at microsoft.com one minute ago to ask for upload location)
    • Edited by Franz_D Thursday, November 14, 2019 9:37 AM Additional information
    Thursday, November 14, 2019 9:25 AM
  • This is what Visual Studio 2013 Express tells me when opening the .dmp (v16.30, PE startet and crashed after some minutes).
    What I don't understand is:
    'Error Information: Debugging a 64-bit dump of a 32-bit process requires full heap information'
    ProcessExplorer.exe is shown as a 64bit process, why does Visual Studio think it's from a 32bit app?

    Dump Summary
    ------------
    Dump File: procexp64.exe.86640.dmp : C:\Temp\procexp64.exe.86640.dmp
    Last Write Time: 14.11.2019 10:36:13
    Process Name: procexp64.exe : C:\Programme_Admin\ProcessExplorer\procexp64.exe
    Process Architecture: x86 (64-bit dump)
    Exception Code: 0xC0000005
    Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access.
    Heap Information: Not Present

    System Information
    ------------------
    OS Version: 10.0.17763
    CLR Version(s): 


    Edit: Please note, my next possibility to reply is on Mon., 18th Nov

    • Edited by Franz_D Thursday, November 14, 2019 9:58 AM noted earliest next reply date
    Thursday, November 14, 2019 9:53 AM
  • Try instaling windbg from the "debugging tools for windows" in the Windows 10 SDK..

    Visual studio debugger (especially one so old) may misinterpret something in the header of the dump.

    HTH
    -mario

    Thursday, November 14, 2019 10:03 AM
  • Hi Mario,

    here you are (hope, I did it right, was the 1st time using Windbg)


    Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.


    Loading Dump File [C:\Temp\procexp64.exe.86640.dmp]
    User Mini Dump File: Only registers, stack and portions of memory are available

    Symbol search path is: srv*
    Executable search path is:
    Windows 10 Version 17763 MP (6 procs) Free x64
    Product: WinNt, suite: SingleUserTS
    17763.1.amd64fre.rs5_release.180914-1434
    Machine Name:
    Debug session time: Thu Nov 14 10:36:13.000 2019 (UTC + 1:00)
    System Uptime: not available
    Process Uptime: 0 days 0:04:54.000
    ................................................................
    .................................................
    Loading unloaded module list
    .................................
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (15270.18040): Access violation - code c0000005 (first/second chance not available)
    For analysis of this file, run !analyze -v
    ntdll!NtWaitForMultipleObjects+0x14:
    00007ffc`be8002b4 c3              ret
    0:008> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Exception Analysis                                   *
    *                                                                             *
    *******************************************************************************


    KEY_VALUES_STRING: 1

        Key  : AV.Dereference
        Value: NullClassPtr

        Key  : AV.Fault
        Value: Write

        Key  : Timeline.Process.Start.DeltaSec
        Value: 294


    PROCESSES_ANALYSIS: 1

    SERVICE_ANALYSIS: 1

    STACKHASH_ANALYSIS: 1

    TIMELINE_ANALYSIS: 1

    Timeline: !analyze.Start
        Name: <blank>
        Time: 2019-11-14T11:38:26.887Z
        Diff: 7333887 mSec

    Timeline: Dump.Current
        Name: <blank>
        Time: 2019-11-14T09:36:13.0Z
        Diff: 0 mSec

    Timeline: Process.Start
        Name: <blank>
        Time: 2019-11-14T09:31:19.0Z
        Diff: 294000 mSec


    DUMP_CLASS: 2

    DUMP_QUALIFIER: 400

    CONTEXT:  (.ecxr)
    rax=0000000000000000 rbx=000000000000000d rcx=00000000000003f4
    rdx=0000000000ff8080 rsi=0000007a17afc6d8 rdi=000000000000000f
    rip=00007ff641620446 rsp=0000007a17afc600 rbp=00000000008080ff
     r8=0000000000000000  r9=000000000000000f r10=0000000000000080
    r11=0000000000000000 r12=000000000000000f r13=0000000000000002
    r14=0000000000000000 r15=0000000000000000
    iopl=0         nv up ei pl nz na pe nc
    cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
    procexp64+0x50446:
    00007ff6`41620446 891401          mov     dword ptr [rcx+rax],edx ds:00000000`000003f4=????????
    Resetting default scope

    FAULTING_IP:
    procexp64+50446
    00007ff6`41620446 891401          mov     dword ptr [rcx+rax],edx

    EXCEPTION_RECORD:  (.exr -1)
    ExceptionAddress: 00007ff641620446 (procexp64+0x0000000000050446)
       ExceptionCode: c0000005 (Access violation)
      ExceptionFlags: 00000000
    NumberParameters: 2
       Parameter[0]: 0000000000000001
       Parameter[1]: 00000000000003f4
    Attempt to write to address 00000000000003f4

    DEFAULT_BUCKET_ID:  NULL_CLASS_PTR_WRITE

    PROCESS_NAME:  procexp64.exe

    FOLLOWUP_IP:
    procexp64+50446
    00007ff6`41620446 891401          mov     dword ptr [rcx+rax],edx

    WRITE_ADDRESS:  00000000000003f4

    ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%p verwies auf Arbeitsspeicher bei 0x%p. Der Vorgang %s konnte im Arbeitsspeicher nicht durchgef hrt werden.

    EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%p verwies auf Arbeitsspeicher bei 0x%p. Der Vorgang %s konnte im Arbeitsspeicher nicht durchgef hrt werden.

    EXCEPTION_CODE_STR:  c0000005

    EXCEPTION_PARAMETER1:  0000000000000001

    EXCEPTION_PARAMETER2:  00000000000003f4

    WATSON_BKT_PROCSTAMP:  5d70cac3

    WATSON_BKT_PROCVER:  16.30.0.0

    PROCESS_VER_PRODUCT:  Process Explorer

    WATSON_BKT_MODULE:  procexp64.exe

    WATSON_BKT_MODSTAMP:  5d70cac3

    WATSON_BKT_MODOFFSET:  50446

    WATSON_BKT_MODVER:  16.30.0.0

    MODULE_VER_PRODUCT:  Process Explorer

    BUILD_VERSION_STRING:  17763.1.amd64fre.rs5_release.180914-1434

    MODLIST_WITH_TSCHKSUM_HASH:  19f64f26c71b2a917a0a8259b4a234a8ea7e3fd7

    MODLIST_SHA1_HASH:  ee08f1a9574d914a005e0d616e22dd89a22ff5e0

    NTGLOBALFLAG:  0

    PROCESS_BAM_CURRENT_THROTTLED: 0

    PROCESS_BAM_PREVIOUS_THROTTLED: 0

    APPLICATION_VERIFIER_FLAGS:  0

    DUMP_FLAGS:  94

    DUMP_TYPE:  1

    ANALYSIS_SESSION_HOST:  xxxxxxxxxxxx

    ANALYSIS_SESSION_TIME:  11-14-2019 12:38:26.0887

    ANALYSIS_VERSION: 10.0.18362.1 amd64fre

    THREAD_ATTRIBUTES:
    OS_LOCALE:  DEU

    BUGCHECK_STR:  APPLICATION_FAULT_NULL_CLASS_PTR_WRITE_NULL_CLASS_PTR_DEREFERENCE_INVALID_POINTER_WRITE

    PRIMARY_PROBLEM_CLASS:  APPLICATION_FAULT

    PROBLEM_CLASSES:

        ID:     [0n313]
        Type:   [@ACCESS_VIOLATION]
        Class:  Addendum
        Scope:  BUCKET_ID
        Name:   Omit
        Data:   Omit
        PID:    [Unspecified]
        TID:    [0x18040]
        Frame:  [0] : procexp64

        ID:     [0n286]
        Type:   [INVALID_POINTER_WRITE]
        Class:  Primary
        Scope:  BUCKET_ID
        Name:   Add
        Data:   Omit
        PID:    [Unspecified]
        TID:    [0x18040]
        Frame:  [0] : procexp64

        ID:     [0n309]
        Type:   [NULL_CLASS_PTR_DEREFERENCE]
        Class:  Primary
        Scope:  BUCKET_ID
        Name:   Add
        Data:   Omit
        PID:    [Unspecified]
        TID:    [0x18040]
        Frame:  [0] : procexp64

        ID:     [0n311]
        Type:   [NULL_CLASS_PTR_WRITE]
        Class:  Primary
        Scope:  DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
                BUCKET_ID
        Name:   Add
        Data:   Omit
        PID:    [0x15270]
        TID:    [0x18040]
        Frame:  [0] : procexp64

    LAST_CONTROL_TRANSFER:  from 00007ff641621931 to 00007ff641620446

    STACK_TEXT: 
    0000007a`17afc600 00007ff6`41621931 : 00000000`00000010 00000000`00000080 0000007a`00000000 00000000`00f0f0f0 : procexp64+0x50446
    0000007a`17afc680 00007ff6`416747d9 : 00000000`00000000 00007ff6`416ca408 00007ff6`416ca430 00000000`00000010 : procexp64+0x51931
    0000007a`17afc810 00007ff6`4167242b : 0000007a`17afcc30 00000000`00000000 0000021f`a6313930 0000021f`a40bfa80 : procexp64+0xa47d9
    0000007a`17afcb30 00007ff6`4164bdd8 : 00000000`00000000 00000000`00000102 00000000`00000000 00000000`29184900 : procexp64+0xa242b
    0000007a`17aff8e0 00007ff6`4168c8b9 : 00000000`00000000 00000000`00000000 00007ff6`4164bbd0 00000000`00000000 : procexp64+0x7bdd8
    0000007a`17aff990 00007ffc`bdcf7974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : procexp64+0xbc8b9
    0000007a`17aff9c0 00007ffc`be7ca271 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
    0000007a`17aff9f0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


    THREAD_SHA1_HASH_MOD_FUNC:  f89ca06b18e1bb370713a18a4ff7d68bb413f969

    THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  8bcb2e8ebd2c79c7aa78cdf9fa5377168e100a45

    THREAD_SHA1_HASH_MOD:  4d096c9068c871d855c714edd8b6fb28af767f69

    FAULT_INSTR_CODE:  44011489

    SYMBOL_STACK_INDEX:  0

    SYMBOL_NAME:  procexp64+50446

    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: procexp64

    IMAGE_NAME:  procexp64.exe

    DEBUG_FLR_IMAGE_TIMESTAMP:  5d70cac3

    STACK_COMMAND:  ~8s ; .ecxr ; kb

    FAILURE_BUCKET_ID:  NULL_CLASS_PTR_WRITE_c0000005_procexp64.exe!Unknown

    BUCKET_ID:  APPLICATION_FAULT_NULL_CLASS_PTR_WRITE_NULL_CLASS_PTR_DEREFERENCE_INVALID_POINTER_WRITE_procexp64+50446

    FAILURE_EXCEPTION_CODE:  c0000005

    FAILURE_IMAGE_NAME:  procexp64.exe

    BUCKET_ID_IMAGE_STR:  procexp64.exe

    FAILURE_MODULE_NAME:  procexp64

    BUCKET_ID_MODULE_STR:  procexp64

    FAILURE_FUNCTION_NAME:  Unknown

    BUCKET_ID_FUNCTION_STR:  Unknown

    BUCKET_ID_OFFSET:  50446

    BUCKET_ID_MODPRIVATE: 1

    BUCKET_ID_MODTIMEDATESTAMP:  5d70cac3

    BUCKET_ID_MODCHECKSUM:  170058

    BUCKET_ID_MODVER_STR:  16.30.0.0

    BUCKET_ID_PREFIX_STR:  APPLICATION_FAULT_NULL_CLASS_PTR_WRITE_NULL_CLASS_PTR_DEREFERENCE_INVALID_POINTER_WRITE_

    FAILURE_PROBLEM_CLASS:  APPLICATION_FAULT

    FAILURE_SYMBOL_NAME:  procexp64.exe!Unknown

    WATSON_STAGEONE_URL:  watson.microsoft.com/StageOne/procexp64.exe/16.30.0.0/5d70cac3/procexp64.exe/16.30.0.0/5d70cac3/c0000005/00050446.htm?Retriage=1

    TARGET_TIME:  2019-11-14T09:36:13.000Z

    OSBUILD:  17763

    OSSERVICEPACK:  475

    SERVICEPACK_NUMBER: 0

    OS_REVISION: 0

    SUITE_MASK:  256

    PRODUCT_TYPE:  1

    OSPLATFORM_TYPE:  x64

    OSNAME:  Windows 10

    OSEDITION:  Windows 10 WinNt SingleUserTS

    USER_LCID:  0

    OSBUILD_TIMESTAMP:  1989-09-09 23:58:46

    BUILDDATESTAMP_STR:  180914-1434

    BUILDLAB_STR:  rs5_release

    BUILDOSVER_STR:  10.0.17763.1.amd64fre.rs5_release.180914-1434

    ANALYSIS_SESSION_ELAPSED_TIME:  56d

    ANALYSIS_SOURCE:  UM

    FAILURE_ID_HASH_STRING:  um:null_class_ptr_write_c0000005_procexp64.exe!unknown

    FAILURE_ID_HASH:  {93f47178-dc66-a80f-d802-be9d18317091}

    Followup:     MachineOwner
    ---------

    0:008> .ecxr
    rax=0000000000000000 rbx=000000000000000d rcx=00000000000003f4
    rdx=0000000000ff8080 rsi=0000007a17afc6d8 rdi=000000000000000f
    rip=00007ff641620446 rsp=0000007a17afc600 rbp=00000000008080ff
     r8=0000000000000000  r9=000000000000000f r10=0000000000000080
    r11=0000000000000000 r12=000000000000000f r13=0000000000000002
    r14=0000000000000000 r15=0000000000000000
    iopl=0         nv up ei pl nz na pe nc
    cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
    procexp64+0x50446:
    00007ff6`41620446 891401          mov     dword ptr [rcx+rax],edx ds:00000000`000003f4=???????

    Thursday, November 14, 2019 11:45 AM
  • Hi Franz

    thanks for sending me the dump files. I will take a look and get back to you.

    MarkC(MSFT)

    Thursday, November 14, 2019 1:14 PM