none
Replica is inconsistent RRS feed

  • Question

  • I have System Center 2012 R2 installed on Windows server 2008. I am trying to protect a VM that is running SQL Server 2008. The VM is being managed by a Failover Cluster in Windows 2012 R2 standard and has 4 VHDx's attached which are located on a SAN. Now, when ever I try to run an consistency check on that protection group member, I receive the following error:

    ________________________________________________

    The replica of Microsoft Hyper-V \Online\VM on "Target.com" is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.

    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)

    The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried,
    the error is likely to reoccur.
     (0x800423F4))

    ________________________________________________

    Now when I check the event log from the server in question I receive 5 separate event logs pointing to Hyper-v-VMMS.

    The First:

    'VM' cannot create the storage required for the checkpoint using disk C:\TARGET\TARGET\VM.vhdx: General access denied error (0x80070005). (Virtual machine ID 97333BCA-130E-4F20-934A-CB26A19DDBF6) - Event ID 16370

    The Second:

    Checkpoint operation for 'VM' failed. (Virtual machine ID 97333BCA-130E-4F20-934A-CB26A19DDBF6) - Event ID 18012

    The Third:

    Could not create backup checkpoint for virtual machine 'VM': General access denied error (0x80070005). (Virtual machine ID 97333BCA-130E-4F20-934A-CB26A19DDBF6) - Event ID 10150

    The Fourth:

    VSS writers inside virtual machine 'VM' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: Catastrophic failure (0x8000FFFF). (Virtual machine ID 97333BCA-130E-4F20-934A-CB26A19DDBF6) - Event ID 10172

    The Fifth:

    The operation failed. - Event ID 16010

    ________________________________________________

    As suggested online, I ran vssadmin list writers from an elevated command prompt and every writer appears to be stable with no errors. The weird thing is, I am not receiving any errors from the other 3 VHDX's. In the recovery tab, there are no history of any recovery point for that VM. I am able to protect the VM databases but not the VM itself. I also gave full rights in the folder where all the VHDx's are located. Any ideas?

    Thank you

    Monday, November 3, 2014 2:52 PM

Answers

  • Hello Dwayne,

    Just to let you know that I have bypassed the issue by moving the checkpoint location onto another folder on the SAN and now it backs up ok. I still do not understand though why I get an access denied. I compared both locations and they all have identical permissions. I will check into this in the future. Thank you for your help.

    • Marked as answer by DavidLocweld Tuesday, November 4, 2014 4:58 PM
    Tuesday, November 4, 2014 4:58 PM

All replies

  • Hi,

    Suggestion (if not already done so) ensure there is sufficient disk space on the guest Vm (Problem SQL Server)?

    Also does running a backup via Windows Server Backup directly on the guest Vm selecting to back up the system state and C: (OS drive) complete with success?

    Is my understanding correct that each of the other three Vms ( has its own VHDX are backing up fine) and are located in the same folder as the problem Vms VHDx?

    **********************************************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."

    Monday, November 3, 2014 4:34 PM
  • I checked the disk space on all drives and they all have over 20% disk space free. C: drive being the lowest at 10.5GB free of 44.4. As for the 4 VHDX, they are all attached to the same VM. They are not separate VM's. And I don't think any of the VHD's are being backed up because the recovery has no recovery points. It fails almost right away after forcing a consistency check. As for the SQL server itself, it is running fine for the company. They are having no issues. I tried moving the VM from 1 server to another using failover cluster and still the same issue. I have not tried the windows server backup since that is not the way the company wants to backup. They want the VM backed up through DPM. Unless of course you think it may be a good deduction to test it that way?
    Monday, November 3, 2014 7:30 PM
  • Hi,

    Appears the disk space area is well. The windows server backup was included for testing. Thanks for the clarification on the four disks.Two items come to mind:

    1. For the SQL servers four VHDXs is the folder structure (on the backend) the same compared against Vms that are backing up with success?


    Example are Vms that work VHDXs stored within the CSV root or a subfolder

    1. Also on the backend have we compared the permissions where VHDXs reside for Vms backing up with success vs where VHDXs reside for Vms failing to backup with success?

    Overall trying to understand if we have any difference between the VHDXs for Vms that are backing up with success vs the VHDXs for the SQL server which fails backup.

    **********************************************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."

    Monday, November 3, 2014 8:10 PM
  • First, I want to mention that all 4 VHDX are located in a folder based partition. (targets to a physical SAN) Plus 1 or 2 others which belong to a different VM. All of them which are being inherited from the parent. Some backups do work in that area. Everything appear the same in the security tab if I compare with others. To be sure, I gave full rights to everyone on all folders. So I am not really sure it's a permission issue. Any other ideas?

    Monday, November 3, 2014 9:06 PM
  • Hi,

    Thanks for the detailed update. So if we have other VHDXs for other residing in that same folder working then that tanks that theory. All the permissions you have referenced sound to be at the OS level are permissions on the backend SAN also aligned between working and non-working Vms?

    **********************************************

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Dwayne Jackson II. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights."

    Monday, November 3, 2014 9:32 PM
  • It isn't configured as a hardware SAN snapshot if that is what you are asking. All of our VHD's are on the SAN and have no issues except for this one VHD.
    Tuesday, November 4, 2014 2:16 PM
  • Hello Dwayne,

    Just to let you know that I have bypassed the issue by moving the checkpoint location onto another folder on the SAN and now it backs up ok. I still do not understand though why I get an access denied. I compared both locations and they all have identical permissions. I will check into this in the future. Thank you for your help.

    • Marked as answer by DavidLocweld Tuesday, November 4, 2014 4:58 PM
    Tuesday, November 4, 2014 4:58 PM