none
Replica is inconsistent RRS feed

  • Question

  • Hi Everyone!

    I am kinda stuck on an issue and have not been able to find an answer to that solves the problem. I was wondering if anyone can point me in the right direction.

    I currently have a Hyper-V 2016 server that I am backing up using DPM 2016, Now the Hyper-V host its self has roughly about 50 VM's and about 35 of them are backing up without issue however I have about 15 guests that are all in a "Replica is inconsistent" State with the following error in even log :

    The replica of <Guest> on <Server> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
    An unexpected error occurred while the job was running. (ID: 104)

    Performing a consistency check does nothing and the alert pops back into the console. Any ideas or help ?

    Stuart.

    Wednesday, June 19, 2019 6:37 PM

All replies

  • Hello Stuart,

    Could you share some more details about your environment?

    What build is your DPM 2016 running on? (DPM Console -> Help -> About).

    What operating system version and SQL Server version is your DPM 2016 running on?

    Are you backing up on host-level or guest-level backups?

    What are the Hyper-V configuration versions of the virtual machines that are failing the consistency checks?

    Please also check the DPM logs for any more details/clues about the errors, you'll find the logs here:

    • %ProgramFiles%\Microsoft System Center 2016\DPM\DPM\Temp\MSDPMCurr.errlog

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, June 19, 2019 6:44 PM
  • Also the error via console is as follows : 

    The replica of Microsoft Hyper-V <guest> on <server> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)

    An unexpected error occurred while the job was running. (ID 104 Details: Call context cannot be accessed after call completed (0x80010117))

    Wednesday, June 19, 2019 6:44 PM
  • Could you share some more details about your environment?

    What build is your DPM 2016 running on? (DPM Console -> Help -> About).

    5.0.342.0

    What operating system version and SQL Server version is your DPM 2016 running on?

    2016 stand. and SQL 2016 base.

    Are you backing up on host-level or guest-level backups?

    Host Level

    What are the Hyper-V configuration versions of the virtual machines that are failing the consistency checks?

    Gen 2 

    Please also check the DPM logs for any more details/clues about the errors, you'll find the logs here:

    • %ProgramFiles%\Microsoft System Center 2016\DPM\DPM\Temp\MSDPMCurr.errlog

    I get a few errors show up when I run a sync.. 

    1F78 98C0 06/19 20:08:04.896 18 fsutils.cpp(4047) WARNING Failed: Hr: = [0x80070003] : Encountered Failure: : lVal : HRESULT_FROM_WIN32(dwError)
    1F78 C328 06/19 20:08:11.257 27 FsmBlock.cs(178) 6B0A1C39-797F-4F75-901F-039E4476A050 WARNING Backup.ReplicaPreBackupBlock : RAPreBackup, StatusReason = Error (StatusCode = -2147417833, ErrorCode = GenericAgentFailure, workitem = 2301cf52-7d5d-4ff0-8e0d-293c82594689)
    1F78 C328 06/19 20:08:11.257 27 FsmBlock.cs(178) 6B0A1C39-797F-4F75-901F-039E4476A050 WARNING <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147417833" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="b2c95f15-3ac0-45d0-9d6a-568caec3f447" GuidWorkItem="2301cf52-7d5d-4ff0-8e0d-293c82594689" TETaskInstanceID="6b0a1c39-797f-4f75-901f-039e4476a050"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147417833" DetailedSource="2"/><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>0</BackupTime><RAPreBackupSubTaskStatus>Completed</RAPreBackupSubTaskStatus><DSStatus><ComponentName>8BCF92A3-DA31-413A-B25E-1781530C15F2</ComponentName><LogicalPath></LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    1F78 C328 06/19 20:08:11.257 27 CommonErrorHandler.cs(173) 6B0A1C39-797F-4F75-901F-039E4476A050 WARNING AgentStatus[RAForRead] - (CommandID=RAPreBackup, StatusReason=Error) failed with HRESULT 0x80010117, error -2147417833.
    1F78 C328 06/19 20:08:11.257 27 CommonErrorHandler.cs(275) 6B0A1C39-797F-4F75-901F-039E4476A050 NORMAL Unmapped agent error code = GenericAgentFailure
    1F78 70B4 06/19 20:08:11.289 27 FsmBlock.cs(130) 6B0A1C39-797F-4F75-901F-039E4476A050 WARNING Backup.ReplicaPreBackupBlock : <-- Exited FSM block with FAILURE (errorCode = RmGenericError) ErrorInfo = RmGenericError 

    Wednesday, June 19, 2019 8:10 PM
  • I didn't mean the generation of the virtual machines, sorry if I was unclear, what I meant was the Hyper-V configuration versions that are listed over here:
    https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/deploy/upgrade-virtual-machine-version-in-hyper-v-on-windows-or-windows-server#supported-vm-configuration-versions-for-long-term-servicing-hosts

    Please check the differences of the virtual machines that are working and those that are not.

    Could you also please do the following:

    1. Move all the current MSDPMCurr.errlog files from the %ProgramFiles%\Microsoft System Center 2016\DPM\DPM\Temp\ folder to a safe location.

    2. Once moved, there should be no MSDPMCurr.errlog in the folder, a new one will eventually be created.

    3. Try performing a consistency check until it fails.

    4. Upload the newly created MSDPMCurr.errlog to OneDrive and share the link here.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, June 19, 2019 8:16 PM
  • Hi Leon, 

    I can not see a difference in the VM configs however I have noticed in the logs ( as you can see in the below log ) that some of the guests are considered offline. 

    The MSDPMCurr.errlog is provided here Log   

    Wednesday, June 19, 2019 9:56 PM
  • Thanks for the log, it wasen't much of help I'm afraid.

    Are there any differences between the working VMs and the non-working VMs?

    What workloads are they running within the VM?

    Has this recently started or is this a new environment?

    Please run the following command on the Hyper-V cluster nodes and paste the result here

    vssadmin list writers

    The DPM logs on the Hyper-V cluster nodes might tell us something, you'll find them over here:

    • %ProgramFiles%\Microsoft Data Protection Manager\DPM\Temp\DPMRACurr.errlog



    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, June 19, 2019 10:14 PM
  • Not really, the box I am backing up only has replica's in an offline state. (Not even a cluster) So there is no real workloads either to speak of. 

    This is an existing environment.

    LOG   DPMRACurr.errlog

    VSS list writers : 

    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error

    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error

    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error

    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {0acf4119-3ca8-4a56-b831-47102b638e32}
       State: [1] Stable
       Last error: No error

    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {3f4b231f-46b7-4aec-a444-1b310d3f63ff}
       State: [1] Stable
       Last error: No error

    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {cb5fcb5b-10a2-4142-af8b-d02c01cf72ef}
       State: [1] Stable
       Last error: No error

    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {3f1dabaa-7215-412b-a65a-8236a38161be}
       State: [1] Stable
       Last error: No error

    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {19d51958-f50d-450a-b7b3-f66fece74e21}
       State: [1] Stable
       Last error: No error

    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {f3e70b3e-f119-4580-9fd2-d33cfcb1fa33}
       State: [1] Stable
       Last error: No error

    Writer name: 'MSSearch Service Writer'
       Writer Id: {cd3f2362-8bef-46c7-9181-d62844cdc0b2}
       Writer Instance Id: {91eafce0-cf6b-44cf-8bed-c4ff9e4e2196}
       State: [1] Stable
       Last error: No error

    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {303ed6f6-35c4-468c-be98-49e7d0ba5e3e}
       State: [1] Stable


    Thursday, June 20, 2019 11:14 PM
  • The VSS appears to be in a good state, I couldn't download the DPMRACurr.errlog log, it says it's no longer available.

    Have you tried switching cluster nodes for the problematic VMs?

    Have you tired removing them from the protection group, refresh the datasources, and then re-add the VMs to the protection group?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Sunday, June 23, 2019 10:03 PM
  • Please try again via here 

    The servers are on a standalone box  not part of a cluster and yes I did try removing them and readding them same issue. 


    Monday, June 24, 2019 9:20 PM
  • It seems that you are backing up Hyper-V Replicas?

    When using Hyper-V Replica, some of the replicated VMs will be offline, so this explains why you saw some guests as offline.

    For backing up Hyper-V replica virtual machines, please refer to the conditions in the documentation below:
    Back up replica virtual machines

    You can backup Hyper-V VMs even if they are replica VMs, when you configure DPM to protect both primary and recovery hosts, VM will appear on any of the Servers as it will have the same GUID.

    To protect both primary and recovery hosts using different DPM or if you want to protect both primary and recovery using same DPM, make sure that you check both servers so as to discover VM.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, June 25, 2019 9:34 AM
  • Hi,

    •  search for stale Checkpoints (i.e. AHVDX files)
    • delete old VM checkpoints
    • for testing: switch between production and standard checkpoints and back - delete old checkpoints on both options.

    Please make sure you know what you are deleting / merging and about the consequences

    regards

    /bkpfast


    My postings are provided &quot;AS IS&quot; with no warranties and confer no rights

    Wednesday, June 26, 2019 1:37 PM