none
The protection agent on was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

    Question

  • I am having problems backing up data with DPM 2012. I keep getting this meesage. The protection agent on dpmServer was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

    I can't understand what my error logs are trying to tell me please help.

    NORMAL Begin new execution batch at task:(True)-a2c49c70-696b-423d-8aaf-7898c1feb403
    042C 28E0 10/07 22:47:42.979 16 ActiveJob.cs(851) NORMAL Dispatching task for execution:True-a2c49c70-696b-423d-8aaf-7898c1feb403
    042C 2AAC 10/07 22:47:42.979 16 ActiveJob.cs(581) NORMAL Begin ExecuteTask:JobID=eab9800f-4eb7-46b2-83bc-c2ff2c6adb0d, ItemID=a2c49c70-696b-423d-8aaf-7898c1feb403
    042C 2AAC 10/07 22:47:42.990 27 DatasourceServerTypeLock.cs(395) NORMAL Server e9a3e229-5f73-4223-845f-e2038860e74b is not already registed to Acquire a lock, Registering the server now for the task a2c49c70-696b-423d-8aaf-7898c1feb403, IsDatasoureServerTypeLockNeeded = False
    042C 2AAC 10/07 22:47:42.991 01 DPMTaskController.cs(70) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL (job=eab9800f-4eb7-46b2-83bc-c2ff2c6adb0d, taskId=a2c49c70-696b-423d-8aaf-7898c1feb403) DPMTaskControllerHelper : ExecuteTask - CheckTaskStatus returned ReadyToExecute
    042C 2AAC 10/07 22:47:42.993 01 TaskExecutor.cs(692) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Setting up Fsm: verb[3d859d8c-d0bb-4142-8696-c0d215203e0d]
    042C 2AAC 10/07 22:47:43.009 27 CommonLoop.cs(343) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL DatamovePollingCmdTimeoutInMin = 30
    042C 2AAC 10/07 22:47:43.009 27 CommonLoop.cs(343) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL DatamovePollingCmdTimeoutInMin = 30
    042C 2AAC 10/07 22:47:43.021 01 TaskExecutor.cs(487) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL ExecuteTask(job=eab9800f-4eb7-46b2-83bc-c2ff2c6adb0d) - Starting task instance
    042C 092C 10/07 22:47:43.023 02 EventManager.cs(98) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Publishing event from TaskInstance.cs(406): TaskStart, [TaskID=a2c49c70-696b-423d-8aaf-7898c1feb403]
    042C 092C 10/07 22:47:43.115 01 TaskInstance.cs(442) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Task: Initial transition enqueued, startState='Backup.StartState'
    042C 1A54 10/07 22:47:43.116 27 BackupMachine.cs(1919) A2C49C70-696B-423D-8AAF-7898C1FEB403 ACTIVITY BackupMachine : Starting machine execution
    042C 2FC8 10/07 22:47:43.135 27 FindPrincipalServerBlock.cs(245) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Found Principal server for DatasourceId = a87a4dea-2e19-4456-ae3b-e5ce01c10351 Server= fs.org
    042C 2AAC 10/07 22:47:43.146 27 DatasourceServerTypeLock.cs(395) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Server e9a3e229-5f73-4223-845f-e2038860e74b is not already registed to Acquire a lock, Registering the server now for the task a2c49c70-696b-423d-8aaf-7898c1feb403, IsDatasoureServerTypeLockNeeded = False
    042C 2AAC 10/07 22:47:43.147 27 OperationTypeLock.cs(582) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL In LockServer() of ReplicationOperationTypeLock, returning lockStatus = AcquiredLock
    042C 2AAC 10/07 22:47:43.156 27 OperationTypeLock.cs(628) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL UnlockServer() of ReplicationOperationTypeLock, returning
    042C 2AAC 10/07 22:47:43.159 27 OperationTypeLock.cs(224) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Setting detailed error for operation lock failure to = PrmReplicaAgentDirtyShutdownDetected
    042C 2AAC 10/07 22:47:43.160 27 OperationTypeLock.cs(487) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL In Lock() of ReplicationOperationTypeLock, returning lockStatus = DoNotWaitToAcquireLock
    042C 2AAC 10/07 22:47:43.162 27 CheckConcurrencyBlock.cs(151) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Check concurrency block - Failure (abortedOnConflictingServerOperation=True)
    042C 1F90 10/07 22:47:43.165 27 FsmBlock.cs(130) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING Backup.CheckConcurrency.PT : <-- Exited FSM block with FAILURE (errorCode = ReplicaInvalidWithReason)
    042C 1F90 10/07 22:47:43.419 27 BackupMachine.cs(2156) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING BackupMachine : FAILURE - BACKUP, errorCode=ReplicaInvalidWithReason
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING Task Diagnostic Information - <?xml version="1.0" encoding="utf-16"?>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING <TaskExecutionContext>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmWriterId>00000000-0000-0000-0000-000000000000</PrmWriterId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmDatasourceId>a87a4dea-2e19-4456-ae3b-e5ce01c10351</PrmDatasourceId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmActiveNodeName>skafs.srvs.org</PrmActiveNodeName>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmLogicalReplicaId>342784c2-f783-487d-be78-154a5a17193f</PrmLogicalReplicaId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmDatasetId>0cbb5339-0af5-4e70-9882-ed118a191890</PrmDatasetId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmPhysicalReplicaId>f440f18e-9436-4fed-9e7c-96cc1a9ebec1</PrmPhysicalReplicaId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmReplicaValidity>Invalid</PrmReplicaValidity>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmReplicaStatus>Idle</PrmReplicaStatus>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmOwnerLockId>00000000-0000-0000-0000-000000000000</PrmOwnerLockId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <TEVerb>DeltaReplicate</TEVerb>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <TEErrorState>Backup.CheckConcurrencyPending</TEErrorState>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <TEErrorDetails>&lt;?xml version="1.0" encoding="utf-16"?&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING &lt;q1:ErrorInfo ErrorCode="1073774947" DetailedCode="0" DetailedSource="0" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd"&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="datasourceid" Value="a87a4dea-2e19-4456-ae3b-e5ce01c10351" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="dpmerrorcode" Value="30187" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="protectedgroup" Value="fs" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="datasourcename" Value="P:\" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="servername" Value="fs.org" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="replicaid" Value="342784c2-f783-487d-be78-154a5a17193f" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING &lt;/q1:ErrorInfo&gt;</TEErrorDetails>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING </TaskExecutionContext>
    042C 1F90 10/07 22:47:43.423 02 EventManager.cs(98) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Publishing event from TaskInstance.cs(823): TaskStop, [TaskID=a2c49c70-696b-423d-8aaf-7898c1feb403]
    042C 1F90 10/07 22:47:43.446 01 TaskExecutor.cs(843) A2C49C70-696B-423D-8AAF-7898C1FEB403 FATAL Task stopped (state=Failed, error=ReplicaInvalidWithReason; 0; None), search "Task Diagnostic Information" for details.
    042C 1F90 10/07 22:47:43.457 16 ActiveJob.cs(745) WARNING Fail: Task 'a2c49c70-696b-423d-8aaf-7898c1feb403' failed with error during execution.
    042C 1F90 10/07 22:47:43.457 16 Task.cs(235) NORMAL Changing task state from 'GenerateWorkplan' -> 'Failed' (a2c49c70-696b-423d-8aaf-7898c1feb403)

    Tuesday, October 08, 2013 1:31 AM

All replies

  • Same problem on our two file-server: "The protection agent on was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)"

    Problem started about 04 October 2013.

    DPM Server and agents: 4.1.3417

    There is event in fileservers application log:

    Log Name:      Application
    Source:        Application Error
    Date:          09.10.2013 17:15:17
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      host1.ourlocaldomain.local

    Description:
    Faulting application name: DPMRA.exe, version: 4.1.3417.0, time stamp: 0x52011087
    Faulting module name: FileWriterHelperPlugin.DLL, version: 4.1.3313.0, time stamp: 0x50ad994d
    Exception code: 0xc0000005
    Fault offset: 0x000000000002ba9f
    Faulting process id: 0xe7c
    Faulting application start time: 0x01cec4d870046a62
    Faulting application path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL
    Report Id: b1b1f6ed-30cb-11e3-acc1-00155d079701
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-09T10:15:17.000000000Z" />
        <EventRecordID>109798</EventRecordID>
        <Channel>Application</Channel>
        <Computer>host1.ourlocaldomain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>DPMRA.exe</Data>
        <Data>4.1.3417.0</Data>
        <Data>52011087</Data>
        <Data>FileWriterHelperPlugin.DLL</Data>
        <Data>4.1.3313.0</Data>
        <Data>50ad994d</Data>
        <Data>c0000005</Data>
        <Data>000000000002ba9f</Data>
        <Data>e7c</Data>
        <Data>01cec4d870046a62</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL</Data>
        <Data>b1b1f6ed-30cb-11e3-acc1-00155d079701</Data>
      </EventData>
    </Event>

    Log Name:      Application
    Source:        Application Error
    Date:          09.10.2013 17:15:22
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      host2.ourlocaldomain.local

    Description:
    Faulting application name: DPMRA.exe, version: 4.1.3417.0, time stamp: 0x52011087
    Faulting module name: FileWriterHelperPlugin.DLL, version: 4.1.3313.0, time stamp: 0x50ad994d
    Exception code: 0xc0000005
    Fault offset: 0x000000000002ba9f
    Faulting process id: 0x71c
    Faulting application start time: 0x01cec4d870669e98
    Faulting application path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL
    Report Id: b4680c67-30cb-11e3-ba10-001dd8b71c0e
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-09T10:15:22.000000000Z" />
        <EventRecordID>54725</EventRecordID>
        <Channel>Application</Channel>
        <Computer>host2.ourlocaldomain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>DPMRA.exe</Data>
        <Data>4.1.3417.0</Data>
        <Data>52011087</Data>
        <Data>FileWriterHelperPlugin.DLL</Data>
        <Data>4.1.3313.0</Data>
        <Data>50ad994d</Data>
        <Data>c0000005</Data>
        <Data>000000000002ba9f</Data>
        <Data>71c</Data>
        <Data>01cec4d870669e98</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL</Data>
        <Data>b4680c67-30cb-11e3-ba10-001dd8b71c0e</Data>
      </EventData>
    </Event>


    Wednesday, October 09, 2013 10:30 AM
  • HI DID YOU GUYS GET ANY ANSWERS ON THIS?? IM ALSO FACING THE SAME PROBLEMS
    Friday, September 05, 2014 11:22 AM
  • Hi,

    If you are on DPM 2012 R2 and experiencing a backup failure that reports unexpected
    state. (ID 60 Details: Internal error code: 0x809909B0) in addition you identify that application event 1000 Description Faulting application name: DPMRA.exe then please ensure you review the below

    Backup application process crashes in Windows Server 2012 R2 or Windows Server 2012

    http://support.microsoft.com/kb/2934897/en-us

    Snip from above hotfix symptoms:

    Assume that you have a file that contains lots of extents (such as Microsoft Outlook .PST files) on a server that runs Windows Server 2012 R2 or Windows Server 2012. When you try to back up the server by using a backup application that uses the native backup APIs, the application process crashes, and the backup operation fails. For example, the Data Protection Manager Replication Agent process (DPMRA.exe) crashes during the backup.

    **************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."


    Friday, September 05, 2014 11:49 AM
  • Thanks Dwayne, i'm currently deployed Server 2012 R2 and deployed SCDPM 2012 R2 and a secondary site with the same specs also. i normally experience this one the secondary site yet there is a dedicated 16Meg line too.

    on the other hand i have 9 protection groups that replicate at different times but they are ever running consistency checks everyday when it comes time to replicate.

    most of the recovery point creations fail on the secondary site and idon't know why

    Friday, September 05, 2014 12:02 PM
  • again in my instance the servers that are protected are Windows 2008 R2 servers
    Friday, September 05, 2014 12:05 PM
  • Hi,

    Thanks for the additional information can you confirm the following?

    -The version of DPM 2012R2 you are on for both DPM servers example: <4.2.1205.0>

    -The agent version on one of the problem servers? Version information example: <4.2.1205.0>

    First two items can be identified  by

    1) Selecting 'About DPM' within the DPM Console.

    2)Protection agent version information can be found by selecting 'Management' then select a problem server in question and note the agent version information.

    Also on the problem DPM server when you face the error in question do you have any event 1000 logged in the application events on DPM server or protected server?

    **************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."


    Friday, September 05, 2014 1:27 PM