Sign in
 
HomeWindows Server 2012Windows Server 2008 R2Windows Server 2003LibraryForums
Windows Server TechCenter >
> 
> 
Windows Server 2008 R2 - All Applications Crash 0xc0000005 Within 24 Hours

תשובה Windows Server 2008 R2 - All Applications Crash 0xc0000005 Within 24 Hours

  • Tuesday, July 10, 2012 7:15 PM
     
     
    Sign In to Vote
    0

    We're currently running Windows Server 2008 R2 SP2. After about 24 hours of operation, all applications will crash with the following error:

    "Application popup: [Name of Application].exe - Application Error : The exception unknown software exception (0xc0000005) occurred in the application at location 0x764f50b9.

    Click on OK to terminate the program"

    The application name and the location is the only thing that's different in the errors.

    A reboot is the only way to clear the issue up. Within 24 hours, the error manifests itself again, and we won't be able to continue working until we reboot. During this time, SQL Server is still functional. No other applications will run.

    This server currently runs Windows SQL Server 2008 SP2, Trend Micro OfficeScan 10.5, eVault Backup Agent, and Automate 7. I've tried uninstalling OfficeScan and eVault, but the error persisted. I've also tried setting DEP to "Turn on DEP for essential Windows programs and services only".

    Dell Poweredge 2850

    1x Intel Xeon 7030 2.8GHz CPU

    16GB PC2-3200 DDR RAM

    Any help would be greatly appreciated! Thank you!

     

All Replies

  • Tuesday, July 10, 2012 7:23 PM
    Moderator
     
     
    Sign In to Vote
    0

    It appears that, some of the system files have gone corrupt.

    You can try couple of things here

    1. from elevated cmd (run as administrator ) run command sfc /scannow 

    2. If sfc/scannow doesn't help,  try in-place upgrade of OS

    How to Perform an In-Place Upgrade on Windows Vista, Windows 7, Windows Server 2008 & Windows Server 2008 R2

    http://support.microsoft.com/kb/2255099

    P.S. As the server also runs SQL, take take full DB backup before doing in-place upgrade

    I do not represent the organisation I work for, all the opinions expressed here are my own.

    This posting is provided "AS IS" with no warranties or guarantees and confers no rights.

     
  • Tuesday, July 10, 2012 8:35 PM
     
     
    Sign In to Vote
    0

    I just rebooted and ran sfc /scannow. According to the command, the system file integrity was fine, but the errors immediately appeared.

    Would you say that confirms your suspicion?

     
  • Tuesday, July 10, 2012 9:16 PM
    Moderator
     
     
    Sign In to Vote
    0

    I just rebooted and ran sfc /scannow. According to the command, the system file integrity was fine, but the errors immediately appeared.

    Would you say that confirms your suspicion?

    In my perception, Yes.

    However, you might want to wait for a while and see if anyone else have other suggestions to offer.

    Thanks

    I do not represent the organisation I work for, all the opinions expressed here are my own.

    This posting is provided "AS IS" with no warranties or guarantees and confers no rights.

     
  • Tuesday, July 10, 2012 11:30 PM
     
     
    Sign In to Vote
    0

    It sounds like Santosh is right on in the assumption that there's some corruption in either a widely used module or some type of corruption being caused by a driver etc.. Have there been any recent patches, driver updates or applications installed on the system in question?

    Doug Kentner

     
  • Wednesday, July 11, 2012 5:58 PM
     
     
    Sign In to Vote
    0
    No driver updates, but we usually run all Windows patches within the week they're released.
     
  • Wednesday, July 11, 2012 8:42 PM
     
     
    Sign In to Vote
    0
    Are there dumps being generated for any of the processes that could indicate if there's a common function or module that's being called when the crash is happening?

    Doug Kentner

     
  • Thursday, July 12, 2012 12:07 PM
     
     
    Sign In to Vote
    0
    Where would I look for a crash dump?
     
  • Thursday, July 12, 2012 2:26 PM
     
     
    Sign In to Vote
    0

    You can check the following locations for dumps.

    C:\ProgramData\Microsoft\Windows\WER\ (.mdmp and .hdmp if they exist will be in a reportarchive or reportqueue folder)

    C:\Users\XXX\AppData\Local\Microsoft\Windows\WER (same caveat on location of files)

    If there aren't any dumps there's a process you could follow to enable dumps for app crashes located here:

    http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

    There are instructions on using WinDbg to analyze the dumps here:

    http://msdn.microsoft.com/en-us/library/windows/hardware/ff538058(v=vs.85).aspx

    If you could post the output of a couple !analyze -v commands we could look at the call stacks to see if there's some commonality

    Doug Kentner

     
  • Wednesday, July 18, 2012 12:07 PM
     
     
    Sign In to Vote
    0
    I executed the in-place upgrade as Santosh suggested, but after 48 hours, the errors occurred again and the server required a reboot.
     
  • Thursday, July 19, 2012 11:58 AM
     
     
    Sign In to Vote
    0

    I followed dkentner's instructions on enabling application dumps. This morning applications were failing again, but no dumps were created.

    Any other suggestions?

     
  • Friday, July 20, 2012 4:29 PM
     
     
    Sign In to Vote
    1

    I'm curious if there may be a 3rd party dll being loaded into the process space of all these apps, can you attempt the following and paste the results?:

    1. Install the Windows Debugging Tools (If you haven't already)

    2. Launch notepad

    3. Launch Windbg (x64 if this is an x64 system, x86 if it's x86 system)

    4. Choose File/Attach to process

    5. Select notepad.exe from the list that displays

    6. in the command window in Windbg type lm and press enter

    7. please paste the results here to the forum, This will tell us what modules are being loaded. If there's a 3rd party module loaded, we may be able to tell which one may be causing it based on comparison with another system

    Link with instructions on obtaining the debugging tools here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff538058(v=vs.85).aspx

    Doug Kentner

     
  • Friday, July 20, 2012 6:53 PM
     
      Has Code
    Sign In to Vote
    0

    Here's the results:

    Microsoft (R) Windows Debugger Version 6.2.8400.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

*** wait with pending attach
Symbol search path is: *** Invalid ***
****************************************************************************
* Symbol loading may be unreliable without a symbol search path.           *
* Use .symfix to have the debugger choose a symbol path.                   *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************
Executable search path is:
ModLoad: 00000000`ffd80000 00000000`ffdaf000   C:\Windows\System32\notepad.exe
ModLoad: 00000000`77600000 00000000`77786000   C:\Windows\system32\ntdll.dll
ModLoad: 00000000`774d0000 00000000`775fd000   C:\Windows\system32\kernel32.dll
ModLoad: 000007fe`ff0e0000 000007fe`ff1e8000   C:\Windows\system32\ADVAPI32.dll
ModLoad: 000007fe`ff310000 000007fe`ff453000   C:\Windows\system32\RPCRT4.dll
ModLoad: 000007fe`fefd0000 000007fe`ff034000   C:\Windows\system32\GDI32.dll
ModLoad: 00000000`77400000 00000000`774cd000   C:\Windows\system32\USER32.dll
ModLoad: 000007fe`ff040000 000007fe`ff0dc000   C:\Windows\system32\msvcrt.dll
ModLoad: 000007fe`fead0000 000007fe`feb5c000   C:\Windows\system32\COMDLG32.dll
ModLoad: 000007fe`ff570000 000007fe`ff5e3000   C:\Windows\system32\SHLWAPI.dll
ModLoad: 000007fe`fc430000 000007fe`fc629000   C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_1509f8bef40ee4da\COMCTL32.dll
ModLoad: 000007fe`fdd60000 000007fe`fe9b3000   C:\Windows\system32\SHELL32.dll
ModLoad: 000007fe`faae0000 000007fe`fab38000   C:\Windows\System32\WINSPOOL.DRV
ModLoad: 000007fe`fedf0000 000007fe`fefc8000   C:\Windows\system32\ole32.dll
ModLoad: 000007fe`ff490000 000007fe`ff563000   C:\Windows\system32\OLEAUT32.dll
ModLoad: 000007fe`ff460000 000007fe`ff48d000   C:\Windows\system32\IMM32.DLL
ModLoad: 000007fe`fe9c0000 000007fe`feac2000   C:\Windows\system32\MSCTF.dll
ModLoad: 000007fe`ff6a0000 000007fe`ff6ad000   C:\Windows\system32\LPK.DLL
ModLoad: 000007fe`ff820000 000007fe`ff8ba000   C:\Windows\system32\USP10.dll
ModLoad: 000007fe`fc3d0000 000007fe`fc421000   C:\Windows\System32\UxTheme.dll
ModLoad: 000007fe`f8c10000 000007fe`f8c3d000   C:\Windows\syswow64\cptlwa64.dll
ModLoad: 000007fe`fcc20000 000007fe`fcc2c000   C:\Windows\System32\WTSAPI32.dll
ModLoad: 000007fe`fdbb0000 000007fe`fdbcc000   C:\Windows\System32\Secur32.dll
(1770.13c): Break instruction exception - code 80000003 (first chance)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\system32\ntdll.dll -
ntdll!DbgBreakPoint:
00000000`77646060 cc              int     3
0:002> lm
start             end                 module name
00000000`77400000 00000000`774cd000   USER32     (deferred)             
00000000`774d0000 00000000`775fd000   kernel32   (deferred)             
00000000`77600000 00000000`77786000   ntdll      (export symbols)       C:\Windows\system32\ntdll.dll
00000000`ffd80000 00000000`ffdaf000   notepad    (deferred)             
000007fe`f8c10000 000007fe`f8c3d000   cptlwa64   (deferred)             
000007fe`faae0000 000007fe`fab38000   WINSPOOL   (deferred)             
000007fe`fc3d0000 000007fe`fc421000   UxTheme    (deferred)             
000007fe`fc430000 000007fe`fc629000   COMCTL32   (deferred)             
000007fe`fcc20000 000007fe`fcc2c000   WTSAPI32   (deferred)             
000007fe`fdbb0000 000007fe`fdbcc000   Secur32    (deferred)             
000007fe`fdd60000 000007fe`fe9b3000   SHELL32    (deferred)             
000007fe`fe9c0000 000007fe`feac2000   MSCTF      (deferred)             
000007fe`fead0000 000007fe`feb5c000   COMDLG32   (deferred)             
000007fe`fedf0000 000007fe`fefc8000   ole32      (deferred)             
000007fe`fefd0000 000007fe`ff034000   GDI32      (deferred)             
000007fe`ff040000 000007fe`ff0dc000   msvcrt     (deferred)             
000007fe`ff0e0000 000007fe`ff1e8000   ADVAPI32   (deferred)             
000007fe`ff310000 000007fe`ff453000   RPCRT4     (deferred)             
000007fe`ff460000 000007fe`ff48d000   IMM32      (deferred)             
000007fe`ff490000 000007fe`ff563000   OLEAUT32   (deferred)             
000007fe`ff570000 000007fe`ff5e3000   SHLWAPI    (deferred)             
000007fe`ff6a0000 000007fe`ff6ad000   LPK        (deferred)             
000007fe`ff820000 000007fe`ff8ba000   USP10      (deferred)

     
  • Saturday, July 21, 2012 3:28 PM
     
     Answered
    Sign In to Vote
    1

    So comparing the list of modules loaded against a test machine that I've got, the only difference is the cptlwa64.dll it looks like that module may be related to a piece of enterprise audit software called softrack.

    Is that piece of software installed here?

    If it is we could be looking at a regularly scheduled scan kicking off that causes an exception in the processes.

    to double check you could look at some of the DLLs loaded by other processes that are crashing using process explorer to see if this module is loaded in all of them.

    Doug Kentner

    • Marked As Answer by DrSteppo Thursday, July 26, 2012 3:44 PM
    •  
     
  • Monday, July 23, 2012 10:51 AM
     
     
    Sign In to Vote
    0

    You read my mind! I actually successfully got a dump file and that was the exact service that did it. I've disabled the tracker for now and am rebooting the server as we speak.

    I'll check in if this makes the system stable.

     
  • Monday, July 23, 2012 3:10 PM
     
     
    Sign In to Vote
    0

    Perfect. Just to make sure that disabling the tracker doesn't leave the module loaded you could either use the debugger or process explorer to make sure that the module doesn't show up in notepad anymore as well. Sounds like we may have the culprit though.

    Doug Kentner



    • Edited by dkentner Monday, July 23, 2012 4:39 PM
    • Edited by dkentner Monday, July 23, 2012 4:40 PM
    •  
     
  • Thursday, July 26, 2012 3:45 PM
     
     
    Sign In to Vote
    0

    Perfect. Just to make sure that disabling the tracker doesn't leave the module loaded you could either use the debugger or process explorer to make sure that the module doesn't show up in notepad anymore as well. Sounds like we may have the culprit though.

    Doug Kentner



    Thanks, Doug! That was it. We'll be re-installing that application once we've had it out with the vendor.

    Thanks again!

     
  • Monday, August 06, 2012 8:08 AM
    Moderator
     
     
    Sign In to Vote
    0
    Thanks for all people's time and efforts.