none
DPMRA crashing after update to DPM 2012 R2 UR2

    Question

  • Hi,

    we have a problem on our DPM2012 R2 server. 

    The DPM servers OS is Windows Server 2012.

    After applying Update Rollup 2, the DPMRA service on the DPM server started crashing when trying to create a new recovery point or trying to synchronize:

    Faulting application name: DPMRA.exe, version: 4.2.1226.0, time stamp: 0x5346902f
    Faulting module name: DPMRA.exe, version: 4.2.1226.0, time stamp: 0x5346902f
    Exception code: 0xc0000005
    Fault offset: 0x000000000023442c
    Faulting process id: 0x630
    Faulting application start time: 0x01cf606a8d5b3262
    Faulting application path: C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\DPMRA.exe
    Report Id: ed0d579b-cc5d-11e3-9436-00155d0f6501
    Faulting package full name: 
    Faulting package-relative application ID: 

    When DPMRA crashes, the DPM jobs fail:

    The replica of CDC-RND-SQL01\Tfs_RnD_Configuration on CDC-RND-SQL01.dev.local is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
    DPM failed to communicate with the protection agent on WP-DPM02.alpha.local because the agent is not responding. (ID: 43)
    Creation of recovery points for WPCLUSTERSQL\WSS_Content_WP-SP-INTRA_TEMP on SQL Server (MSSQLSERVER).WPSQLCluster.alpha.local have failed. The last recovery point creation failed for the following reason: (ID: 3114)
    DPM failed to communicate with the protection agent on WP-DPM02.alpha.local because the agent is not responding. (ID: 43)
    Creation of recovery points for WPCLUSTERSQL\CDC-INTRA-SPS01_WSS_Content_80_Changes01 on SQL Server (MSSQLSERVER).WPSQLCluster.alpha.local have failed. The last recovery point creation failed for the following reason: (ID: 3114)
    The protection agent on WP-DPM02.alpha.local was temporarily unable to respond because it was in an unexpected state. (ID: 60)

    Already tried disabling UAC, firewall is disabled, checked network connectivity, reinstalled agents on the protected servers.

    Does anyone have encountered this? How did you solve it?

    Here are a few DPMRA logs:

    DPMRA_log_01.log

    DPMRA_log_02.log


    • Edited by Tadas K Friday, April 25, 2014 10:48 AM
    Friday, April 25, 2014 10:19 AM

Answers

  • Hi,

    For those experiencing the below DPMRA crash after installing DPM 2012 Sp1 UR6 or DPM 2012 R2 UR2

    DPMRA.exe, version: 4.1.3441.0 = DPM 2012 Sp1 UR6
    DPMRA.exe, version: 4.2.1226.0 =  DPM 2012 R2 UR2

    Faulting application name: DPMRA.exe, version: 4.2.1226.0, time stamp: 0x5346902f
    Faulting module name: DPMRA.exe, version: 4.2.1226.0, time stamp: 0x5346902f
    Exception code: 0xc0000005
    Fault offset: 0x000000000023442c
    Faulting process id: 0x630
    Faulting application start time: 0x01cf606a8d5b3262
    Faulting application path: C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\DPMRA.exe
    Report Id: ed0d579b-cc5d-11e3-9436-00155d0f6501
    Faulting package full name: 
    Faulting package-relative application ID: 

    You can run the below Windows Powershell script on the DPM Server to clean up the *DPMRA.CRASHDETECT files.  This should eliminate the dpmra crashes until a new fix is released to address the original problem.

    1) Stop MSDPM and DPMRA Services.
    2) Copy / Paste the below script into notepad and save as deletecrashfiles.ps1
    3) Open Windows PowerShell using administrative privilege.
    4) Run the deletecrashfiles.ps1
    5) Restart the DPM services.  

    #get DPM install path
    $dpmsetup = "HKLM:\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Setup"
    $dpminstallregkey = get-itemproperty $dpmsetup
    $installpath = $dpminstallregkey.installpath
    $path = $installpath + "\Volumes\Replica"
    
    cd $path
    function GetFolders($path)
    {
        $folders = Get-ChildItem "."
        foreach ($folder in $folders)
        {
            if (Test-path $folder.Name -pathtype Container)
            {
               #write-host $folder.name
                GetCrashitems(".\"+$folder.Name)
            }
            else
            {
                #write-host $folder.Name
            }
        }
    }
    
    function GetCrashItems($path)
    {
        $crashfile = "dpmra.crashdetect"
        $folders = Get-ChildItem $path
        foreach ($folder in $folders)
        {
            #write-host $folder 
            if (Test-path $folder.fullname -pathtype Container)
            {
                write-host 'getting items from ' +($folder.fullname)
                $items = Get-ChildItem $folder.fullname
                foreach($item in $items)
                {
                    if ($item.name -match $crashfile)
                    {
                    write-host 'Deleting *dpmra.crashdetect file: '$item.fullname
                    $item.delete()
    
                    }
                }
            }
        }
    }
    
    GetFolders $path


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Thursday, May 15, 2014 6:39 PM
    Moderator
  • Hi,

    we were able to resolve this issue.

    I noticed that in DPMs "Volumes" folder (in our environment its "C:\Program Files\Microsoft System Center 2012\DPM\DPM\Volumes") multiple "{*}dpmra.crashdetect" (e.g., "{BF4D5F9E-8B01-44C8-B190-BC2F73B71599}dpmra.crashdetect") files were getting created.
    Sometimes, when a successful job would complete, an old {*}dpmra.crashdetect file would get deleted.

    There were some "{*}dpmra.crashdetect" files that were older than the Update Rollup 2 was installed. And DPM jobs related to these files would always crash and bringing down all other running jobs.

    We stopped DPMs services, deleted all of these "{*}dpmra.crashdetect" files, and started the services up again. It started working without any problems - all recovery point or synchronization jobs complete successfully.

    • Marked as answer by Tadas K Wednesday, May 7, 2014 12:26 PM
    Wednesday, May 7, 2014 12:26 PM

All replies

  • Hello Tadas,

    Is your DPMAM service starting? If not, then this could be a cascading effect.

    An issue with the DPMAM has been identified and in fact the UR2 has been pulled out for now - http://support.microsoft.com/kb/2958100/en-gb

    As for customers who are already affected by this issue (DPMAM not starting / crashing), we have a private fix. Please open a case with Microsoft & we will be able to help you go through the steps.

    Regret the inconvenience.

    Regards


    Regards, Rajeev Narshana [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights

    Monday, April 28, 2014 2:33 AM
  • When will Microsoft re-release UR2? Or will Microsoft just skip it and release UR3?  If so when?  I am waiting for this update rollup so that I can move forward with upgrading to R2.

    Thank you


    renei

    Tuesday, May 6, 2014 4:28 PM
  • Hi,

    we were able to resolve this issue.

    I noticed that in DPMs "Volumes" folder (in our environment its "C:\Program Files\Microsoft System Center 2012\DPM\DPM\Volumes") multiple "{*}dpmra.crashdetect" (e.g., "{BF4D5F9E-8B01-44C8-B190-BC2F73B71599}dpmra.crashdetect") files were getting created.
    Sometimes, when a successful job would complete, an old {*}dpmra.crashdetect file would get deleted.

    There were some "{*}dpmra.crashdetect" files that were older than the Update Rollup 2 was installed. And DPM jobs related to these files would always crash and bringing down all other running jobs.

    We stopped DPMs services, deleted all of these "{*}dpmra.crashdetect" files, and started the services up again. It started working without any problems - all recovery point or synchronization jobs complete successfully.

    • Marked as answer by Tadas K Wednesday, May 7, 2014 12:26 PM
    Wednesday, May 7, 2014 12:26 PM
  • Deleting the crashdetect files resolved the same issue we were having with DPM 2012 Update Rollup 6.

    Thanks for the solution!

    Thursday, May 15, 2014 2:26 PM
  • Hi,

    For those experiencing the below DPMRA crash after installing DPM 2012 Sp1 UR6 or DPM 2012 R2 UR2

    DPMRA.exe, version: 4.1.3441.0 = DPM 2012 Sp1 UR6
    DPMRA.exe, version: 4.2.1226.0 =  DPM 2012 R2 UR2

    Faulting application name: DPMRA.exe, version: 4.2.1226.0, time stamp: 0x5346902f
    Faulting module name: DPMRA.exe, version: 4.2.1226.0, time stamp: 0x5346902f
    Exception code: 0xc0000005
    Fault offset: 0x000000000023442c
    Faulting process id: 0x630
    Faulting application start time: 0x01cf606a8d5b3262
    Faulting application path: C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\DPMRA.exe
    Report Id: ed0d579b-cc5d-11e3-9436-00155d0f6501
    Faulting package full name: 
    Faulting package-relative application ID: 

    You can run the below Windows Powershell script on the DPM Server to clean up the *DPMRA.CRASHDETECT files.  This should eliminate the dpmra crashes until a new fix is released to address the original problem.

    1) Stop MSDPM and DPMRA Services.
    2) Copy / Paste the below script into notepad and save as deletecrashfiles.ps1
    3) Open Windows PowerShell using administrative privilege.
    4) Run the deletecrashfiles.ps1
    5) Restart the DPM services.  

    #get DPM install path
    $dpmsetup = "HKLM:\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Setup"
    $dpminstallregkey = get-itemproperty $dpmsetup
    $installpath = $dpminstallregkey.installpath
    $path = $installpath + "\Volumes\Replica"
    
    cd $path
    function GetFolders($path)
    {
        $folders = Get-ChildItem "."
        foreach ($folder in $folders)
        {
            if (Test-path $folder.Name -pathtype Container)
            {
               #write-host $folder.name
                GetCrashitems(".\"+$folder.Name)
            }
            else
            {
                #write-host $folder.Name
            }
        }
    }
    
    function GetCrashItems($path)
    {
        $crashfile = "dpmra.crashdetect"
        $folders = Get-ChildItem $path
        foreach ($folder in $folders)
        {
            #write-host $folder 
            if (Test-path $folder.fullname -pathtype Container)
            {
                write-host 'getting items from ' +($folder.fullname)
                $items = Get-ChildItem $folder.fullname
                foreach($item in $items)
                {
                    if ($item.name -match $crashfile)
                    {
                    write-host 'Deleting *dpmra.crashdetect file: '$item.fullname
                    $item.delete()
    
                    }
                }
            }
        }
    }
    
    GetFolders $path


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Thursday, May 15, 2014 6:39 PM
    Moderator
  • Hi

    2 weeks i tried to fix problem.

    Win2008R2  - DPM 2012R2 UR2

    Errors

    after long protection backup to LTO-4. i have got issues. all job was finished  with error:

    The protection agent on server.domain.com was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

    The DPMRA service terminated unexpectedly.  It has done this 104 time(s).

    This steps resolve problem:

    1) Stop MSDPM and DPMRA Services.
    2) Copy / Paste the below script into notepad and save as deletecrashfiles.ps1
    3) Open Windows PowerShell using administrative privilege.
    4) Run the deletecrashfiles.ps1
    5) Restart the DPM services. 


    Than you!!

    Monday, May 19, 2014 1:02 PM
  • Hi,

    The new re-released UR2 for DPM 2012 R2 is now available. Please find the links:

    Old version:

    Update Rollup 2 for System Center 2012 R2 Data Protection Manager - http://support.microsoft.com/kb/2958100

    Redirecting to the new KB:

    Description of Update Rollup 2 for System Center 2012 R2 Data Protection Manager - http://support.microsoft.com/kb/2963543

    Available on the MS Catalog:

    Catalog Microsoft Update - http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2963543


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, May 21, 2014 1:50 PM
    Moderator
  • I had this same issue even after running the script and called a ticket in on it...Tech ran the same script but did so from the DPM Powershell instead of the Windows PS and it worked.  Just an FYI.
    Thursday, July 10, 2014 8:58 PM
  • Thanks a lot !

    Rod Verdie

    Thursday, July 10, 2014 10:00 PM
  • We had the same issue.  This helped.  Thanks for the posting!
    Tuesday, July 22, 2014 7:34 PM
  • Still valid for UR8. Thank you, Mike!

    http://exchange12rocks.org/ | http://about.me/exchange12rocks

    Thursday, November 26, 2015 11:39 AM
  • PS script must start on dpm server or protected computer?

    Ignas Butrimas

    Tuesday, July 26, 2016 6:21 PM
  • Hi, the script/fix needs to be done on the DPM server.
    Tuesday, July 26, 2016 6:48 PM
  • Still valid for UR12?

    Thank you.

    Thursday, May 4, 2017 1:26 PM