none
The protection agent on was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

    Question

  • I am having problems backing up data with DPM 2012. I keep getting this meesage. The protection agent on dpmServer was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

    I can't understand what my error logs are trying to tell me please help.

    NORMAL Begin new execution batch at task:(True)-a2c49c70-696b-423d-8aaf-7898c1feb403
    042C 28E0 10/07 22:47:42.979 16 ActiveJob.cs(851) NORMAL Dispatching task for execution:True-a2c49c70-696b-423d-8aaf-7898c1feb403
    042C 2AAC 10/07 22:47:42.979 16 ActiveJob.cs(581) NORMAL Begin ExecuteTask:JobID=eab9800f-4eb7-46b2-83bc-c2ff2c6adb0d, ItemID=a2c49c70-696b-423d-8aaf-7898c1feb403
    042C 2AAC 10/07 22:47:42.990 27 DatasourceServerTypeLock.cs(395) NORMAL Server e9a3e229-5f73-4223-845f-e2038860e74b is not already registed to Acquire a lock, Registering the server now for the task a2c49c70-696b-423d-8aaf-7898c1feb403, IsDatasoureServerTypeLockNeeded = False
    042C 2AAC 10/07 22:47:42.991 01 DPMTaskController.cs(70) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL (job=eab9800f-4eb7-46b2-83bc-c2ff2c6adb0d, taskId=a2c49c70-696b-423d-8aaf-7898c1feb403) DPMTaskControllerHelper : ExecuteTask - CheckTaskStatus returned ReadyToExecute
    042C 2AAC 10/07 22:47:42.993 01 TaskExecutor.cs(692) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Setting up Fsm: verb[3d859d8c-d0bb-4142-8696-c0d215203e0d]
    042C 2AAC 10/07 22:47:43.009 27 CommonLoop.cs(343) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL DatamovePollingCmdTimeoutInMin = 30
    042C 2AAC 10/07 22:47:43.009 27 CommonLoop.cs(343) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL DatamovePollingCmdTimeoutInMin = 30
    042C 2AAC 10/07 22:47:43.021 01 TaskExecutor.cs(487) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL ExecuteTask(job=eab9800f-4eb7-46b2-83bc-c2ff2c6adb0d) - Starting task instance
    042C 092C 10/07 22:47:43.023 02 EventManager.cs(98) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Publishing event from TaskInstance.cs(406): TaskStart, [TaskID=a2c49c70-696b-423d-8aaf-7898c1feb403]
    042C 092C 10/07 22:47:43.115 01 TaskInstance.cs(442) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Task: Initial transition enqueued, startState='Backup.StartState'
    042C 1A54 10/07 22:47:43.116 27 BackupMachine.cs(1919) A2C49C70-696B-423D-8AAF-7898C1FEB403 ACTIVITY BackupMachine : Starting machine execution
    042C 2FC8 10/07 22:47:43.135 27 FindPrincipalServerBlock.cs(245) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Found Principal server for DatasourceId = a87a4dea-2e19-4456-ae3b-e5ce01c10351 Server= fs.org
    042C 2AAC 10/07 22:47:43.146 27 DatasourceServerTypeLock.cs(395) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Server e9a3e229-5f73-4223-845f-e2038860e74b is not already registed to Acquire a lock, Registering the server now for the task a2c49c70-696b-423d-8aaf-7898c1feb403, IsDatasoureServerTypeLockNeeded = False
    042C 2AAC 10/07 22:47:43.147 27 OperationTypeLock.cs(582) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL In LockServer() of ReplicationOperationTypeLock, returning lockStatus = AcquiredLock
    042C 2AAC 10/07 22:47:43.156 27 OperationTypeLock.cs(628) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL UnlockServer() of ReplicationOperationTypeLock, returning
    042C 2AAC 10/07 22:47:43.159 27 OperationTypeLock.cs(224) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Setting detailed error for operation lock failure to = PrmReplicaAgentDirtyShutdownDetected
    042C 2AAC 10/07 22:47:43.160 27 OperationTypeLock.cs(487) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL In Lock() of ReplicationOperationTypeLock, returning lockStatus = DoNotWaitToAcquireLock
    042C 2AAC 10/07 22:47:43.162 27 CheckConcurrencyBlock.cs(151) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Check concurrency block - Failure (abortedOnConflictingServerOperation=True)
    042C 1F90 10/07 22:47:43.165 27 FsmBlock.cs(130) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING Backup.CheckConcurrency.PT : <-- Exited FSM block with FAILURE (errorCode = ReplicaInvalidWithReason)
    042C 1F90 10/07 22:47:43.419 27 BackupMachine.cs(2156) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING BackupMachine : FAILURE - BACKUP, errorCode=ReplicaInvalidWithReason
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING Task Diagnostic Information - <?xml version="1.0" encoding="utf-16"?>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING <TaskExecutionContext>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmWriterId>00000000-0000-0000-0000-000000000000</PrmWriterId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmDatasourceId>a87a4dea-2e19-4456-ae3b-e5ce01c10351</PrmDatasourceId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmActiveNodeName>skafs.srvs.org</PrmActiveNodeName>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmLogicalReplicaId>342784c2-f783-487d-be78-154a5a17193f</PrmLogicalReplicaId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmDatasetId>0cbb5339-0af5-4e70-9882-ed118a191890</PrmDatasetId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmPhysicalReplicaId>f440f18e-9436-4fed-9e7c-96cc1a9ebec1</PrmPhysicalReplicaId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmReplicaValidity>Invalid</PrmReplicaValidity>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmReplicaStatus>Idle</PrmReplicaStatus>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <PrmOwnerLockId>00000000-0000-0000-0000-000000000000</PrmOwnerLockId>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <TEVerb>DeltaReplicate</TEVerb>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <TEErrorState>Backup.CheckConcurrencyPending</TEErrorState>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  <TEErrorDetails>&lt;?xml version="1.0" encoding="utf-16"?&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING &lt;q1:ErrorInfo ErrorCode="1073774947" DetailedCode="0" DetailedSource="0" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd"&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="datasourceid" Value="a87a4dea-2e19-4456-ae3b-e5ce01c10351" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="dpmerrorcode" Value="30187" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="protectedgroup" Value="fs" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="datasourcename" Value="P:\" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="servername" Value="fs.org" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING  &lt;q1:Parameter Name="replicaid" Value="342784c2-f783-487d-be78-154a5a17193f" /&gt;
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING &lt;/q1:ErrorInfo&gt;</TEErrorDetails>
    042C 1F90 10/07 22:47:43.422 01 TaskInstance.cs(798) A2C49C70-696B-423D-8AAF-7898C1FEB403 WARNING </TaskExecutionContext>
    042C 1F90 10/07 22:47:43.423 02 EventManager.cs(98) A2C49C70-696B-423D-8AAF-7898C1FEB403 NORMAL Publishing event from TaskInstance.cs(823): TaskStop, [TaskID=a2c49c70-696b-423d-8aaf-7898c1feb403]
    042C 1F90 10/07 22:47:43.446 01 TaskExecutor.cs(843) A2C49C70-696B-423D-8AAF-7898C1FEB403 FATAL Task stopped (state=Failed, error=ReplicaInvalidWithReason; 0; None), search "Task Diagnostic Information" for details.
    042C 1F90 10/07 22:47:43.457 16 ActiveJob.cs(745) WARNING Fail: Task 'a2c49c70-696b-423d-8aaf-7898c1feb403' failed with error during execution.
    042C 1F90 10/07 22:47:43.457 16 Task.cs(235) NORMAL Changing task state from 'GenerateWorkplan' -> 'Failed' (a2c49c70-696b-423d-8aaf-7898c1feb403)

    Tuesday, October 08, 2013 1:31 AM

All replies

  • Same problem on our two file-server: "The protection agent on was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)"

    Problem started about 04 October 2013.

    DPM Server and agents: 4.1.3417

    There is event in fileservers application log:

    Log Name:      Application
    Source:        Application Error
    Date:          09.10.2013 17:15:17
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      host1.ourlocaldomain.local

    Description:
    Faulting application name: DPMRA.exe, version: 4.1.3417.0, time stamp: 0x52011087
    Faulting module name: FileWriterHelperPlugin.DLL, version: 4.1.3313.0, time stamp: 0x50ad994d
    Exception code: 0xc0000005
    Fault offset: 0x000000000002ba9f
    Faulting process id: 0xe7c
    Faulting application start time: 0x01cec4d870046a62
    Faulting application path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL
    Report Id: b1b1f6ed-30cb-11e3-acc1-00155d079701
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-09T10:15:17.000000000Z" />
        <EventRecordID>109798</EventRecordID>
        <Channel>Application</Channel>
        <Computer>host1.ourlocaldomain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>DPMRA.exe</Data>
        <Data>4.1.3417.0</Data>
        <Data>52011087</Data>
        <Data>FileWriterHelperPlugin.DLL</Data>
        <Data>4.1.3313.0</Data>
        <Data>50ad994d</Data>
        <Data>c0000005</Data>
        <Data>000000000002ba9f</Data>
        <Data>e7c</Data>
        <Data>01cec4d870046a62</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL</Data>
        <Data>b1b1f6ed-30cb-11e3-acc1-00155d079701</Data>
      </EventData>
    </Event>

    Log Name:      Application
    Source:        Application Error
    Date:          09.10.2013 17:15:22
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      host2.ourlocaldomain.local

    Description:
    Faulting application name: DPMRA.exe, version: 4.1.3417.0, time stamp: 0x52011087
    Faulting module name: FileWriterHelperPlugin.DLL, version: 4.1.3313.0, time stamp: 0x50ad994d
    Exception code: 0xc0000005
    Fault offset: 0x000000000002ba9f
    Faulting process id: 0x71c
    Faulting application start time: 0x01cec4d870669e98
    Faulting application path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL
    Report Id: b4680c67-30cb-11e3-ba10-001dd8b71c0e
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-09T10:15:22.000000000Z" />
        <EventRecordID>54725</EventRecordID>
        <Channel>Application</Channel>
        <Computer>host2.ourlocaldomain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>DPMRA.exe</Data>
        <Data>4.1.3417.0</Data>
        <Data>52011087</Data>
        <Data>FileWriterHelperPlugin.DLL</Data>
        <Data>4.1.3313.0</Data>
        <Data>50ad994d</Data>
        <Data>c0000005</Data>
        <Data>000000000002ba9f</Data>
        <Data>71c</Data>
        <Data>01cec4d870669e98</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe</Data>
        <Data>C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL</Data>
        <Data>b4680c67-30cb-11e3-ba10-001dd8b71c0e</Data>
      </EventData>
    </Event>


    Wednesday, October 09, 2013 10:30 AM
  • HI DID YOU GUYS GET ANY ANSWERS ON THIS?? IM ALSO FACING THE SAME PROBLEMS
    Friday, September 05, 2014 11:22 AM
  • Hi,

    If you are on DPM 2012 R2 and experiencing a backup failure that reports unexpected
    state. (ID 60 Details: Internal error code: 0x809909B0) in addition you identify that application event 1000 Description Faulting application name: DPMRA.exe then please ensure you review the below

    Backup application process crashes in Windows Server 2012 R2 or Windows Server 2012

    http://support.microsoft.com/kb/2934897/en-us

    Snip from above hotfix symptoms:

    Assume that you have a file that contains lots of extents (such as Microsoft Outlook .PST files) on a server that runs Windows Server 2012 R2 or Windows Server 2012. When you try to back up the server by using a backup application that uses the native backup APIs, the application process crashes, and the backup operation fails. For example, the Data Protection Manager Replication Agent process (DPMRA.exe) crashes during the backup.

    **************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."


    Friday, September 05, 2014 11:49 AM
  • Thanks Dwayne, i'm currently deployed Server 2012 R2 and deployed SCDPM 2012 R2 and a secondary site with the same specs also. i normally experience this one the secondary site yet there is a dedicated 16Meg line too.

    on the other hand i have 9 protection groups that replicate at different times but they are ever running consistency checks everyday when it comes time to replicate.

    most of the recovery point creations fail on the secondary site and idon't know why

    Friday, September 05, 2014 12:02 PM
  • again in my instance the servers that are protected are Windows 2008 R2 servers
    Friday, September 05, 2014 12:05 PM
  • Hi,

    Thanks for the additional information can you confirm the following?

    -The version of DPM 2012R2 you are on for both DPM servers example: <4.2.1205.0>

    -The agent version on one of the problem servers? Version information example: <4.2.1205.0>

    First two items can be identified  by

    1) Selecting 'About DPM' within the DPM Console.

    2)Protection agent version information can be found by selecting 'Management' then select a problem server in question and note the agent version information.

    Also on the problem DPM server when you face the error in question do you have any event 1000 logged in the application events on DPM server or protected server?

    **************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."


    Friday, September 05, 2014 1:27 PM
  • I too am seeing this on a 2008 R2 Box

    The DPM Server is on Windows 2012 DPM is 2012R2 4.2.1273.0

    The Server being backed up is 2008 R2 with DPM Agent 4.2.1273.0


    Event log shows ID 1000

    Faulting application name: DPMRA.exe, version: 4.2.1273.0, time stamp: 0x543f9460
    Faulting module name: FileWriterHelperPlugin.DLL, version: 4.2.1273.0, time stamp: 0x543f945f
    Exception code: 0xc0000005
    Fault offset: 0x000000000002ef1b
    Faulting process id: 0x880
    Faulting application start time: 0x01d081cc79ac6b55
    Faulting application path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMRA.exe
    Faulting module path: C:\Program Files\Microsoft Data Protection Manager\DPM\bin\FileWriterHelperPlugin.DLL
    Report Id: bae0a86d-edc1-11e4-85dd-000423deddc5

    Tuesday, April 28, 2015 4:39 PM
  • This Morning i updated the DPM server to 4.2.1312.0 and the Agent to 4.2.1312.0 as well.  but the errors still persist.
    Wednesday, April 29, 2015 10:01 AM
  • Hello, I am also experiencing this on a Windows Server 2008 R2 machine - notably, the only 2008 R2 machine that I have deployed the new DPM Agent on.

    DPM Version: 4.2.1373.0

    Agent version: 4.2.1373.0

    There are indeed a lot of .PST files on this machine that would use a lot of extents, as identified on the hotfix page 2934897 for  Server 2012 R2. What is the hotfix required for Server 2008 R2?

    Monday, November 23, 2015 12:36 AM
  • The error status 0x809909B0 is a generic DPM Agent error code. You can't find out what the issue is from this code alone.

    Instead you need to look earlier in the logs, located in C:\Program Files\Microsoft Data Protection Manager\DPM\Temp. Your case will be different but here's what fixed it for me. Here are my notes on this issue:

    DPM uses Microsoft SQL server to store records. The interface contains a lot of information, but it does not display the Task ID, which is critical for identifying tasks in the DPM logs and troubleshooting a failed task specifically.

    I found this SQL query to be particularly useful for listing recent synchronization tasks, based on the view "vw_DPM_Jobs" in the DPMDB database.

    SELECT task.taskID, max(task.ExecutionState) AS Status, 
    max(task.ErrorCode & 0x2ffffff) AS ErrorCode, 
    SUBSTRING(datasource.DataSourceName, 0, 32767) As DatasourceName, 
    ser.ServerName AS ServerName, 
    max(task.StoppedDateTime ) As CreationTime,
    SUM(RepTrail.receivertransferredbytes) AS DataTransferred,
    Datasource.AppId AS AppId,
    0 AS JobType, – Index for sync jobs 
    jobDefinition.RetryAttemptNumber-- AS RetryAttemptNumber
    FROM tbl_rm_replicatrail RepTrail 
    INNER JOIN tbl_te_tasktrail task 
    ON task.taskId = RepTrail.TaskId 
    INNER JOIN tbl_IM_datasource AS datasource 
    ON datasource.DatasourceId = RepTrail.DatasourceId 
    INNER JOIN tbl_AM_Server AS ser 
    ON ser.ServerId = datasource.ServerId 
    INNER JOIN tbl_JM_TaskDefinition taskDefinition WITH (NOLOCK)
    ON task.TaskDefinitionId = taskDefinition.TaskDefinitionId
    INNER JOIN tbl_JM_JobDefinition jobDefinition WITH (NOLOCK)
    ON taskDefinition.JobDefinitionId = jobDefinition.JobDefinitionId
    WHERE datasource.AppId IN ('00000000-0000-0000-0000-000000000000', '7B2E4C3D-876F-4535-B5C9-B344D4F05120') 
    AND ( task.VerbId = '3d859d8c-d0bb-4142-8696-c0d215203e0d' 
    OR task.VerbId = '3F2E6E90-34F8-4de2-958E-8EE8CA94780C') 
    AND RepTrail.IsGCed = 0
    AND task.IsGCed = 0
    AND datasource.IsGCed = 0 
    GROUP BY task.taskID, task.JobId,
    ser.ServerName,
    SUBSTRING(datasource.DataSourceName, 0, 32767),
    Datasource.AppId,
    jobDefinition.RetryAttemptNumber 
    ORDER BY CreationTime DESC

    The DPM logs are stored in C:\Program Files\Microsoft Data Protection Manager\DPM\Temp. The failures were generating a generic error code, 0x809909B0. This was a red herring as there are many pages on the internet referring to this error code but none of them helped in this case and always had generic troubleshooting suggestions like "uninstall and reinstall", nothing that actually diagnosed the issue. Usually the lines before this error code appears in the logs has information related to the issue.

    I spent a lot of time troubleshooting the most recent synchronization failure, which had the error line:
    refcountedContainers.hpp(44) <Task ID> WARNING Assertion Failure :: Index < m_Size
    However I wasn't getting anywhere with this issue. There was no reference to this error anywhere on the internet.

    So I went back and looked at the first instance of failure. This one had the error code 3180, and looking at the error logs, it had the unique error lines:
    agentutils.cpp(178) <Task ID> WARNING Failed: Hr: = [0x80070020] CreateFile failed for \\?\C:\Program Files\Microsoft Data Protection Manager\DPM\ActiveOwner\D-++++-+Volume
    {f8348d27-576d-11e3-bcd4-005056b16826}
    +
    fsmtransition.cpp(111) [0000000000A33210] <Task ID> WARNING Failed: Hr: = [0x80070020] HasEventErrorCode: completion: 0xa10c, signature: 0xaabbcc00

    That error code 0x80070020 has the error: "The process cannot access the file because it is being used by another process"

    So the issue seems to be related to that ActiveOwner Volume file. According to one website, these files "are used to manage ownership of databases, important for ensuring multiple DPM servers aren't attempting to backup/restore resources contemporaneously." We have only a single DPM server so these files are not a problem to delete for us. "This fix literally takes 3 minutes, yet it took us an entire week of investigation to come to this conclusion."

    I moved that file out of that directory and initiated a synchronization task. That task was successful and this volume is now again protected.

    As far as what other process was specifically using that file and prevented the DPM Agent from accessing it, putting the protection for that volume into a failed state from which it couldn't recover, that I don't have an answer for. At least we have a process to follow now to identify if issue has reoccured and reinitiate protection by regenerating that ActiveOwner file.

    This may not solve your specific issue but if you investigate the initial occurence of the issue and step through this troubleshooting process to identify the specific error message, you may be able to 

    Wednesday, January 06, 2016 4:53 AM