locked
DPM 2012 SP1 Hyperv Cluster CSV - Generating errors RRS feed

  • Question

  • We are using DPM2012 SP1 to back up 2 nodes hyperV 2008 R2 cluster (two CSVs) on the host level. 

    From time to time the backup generates errors, however most of the time all backups are successful in the end regardless the error message. Is this normal?? 

    Back up 8 VMs.

    4 each node.

    hyperV-1 is currently the owner of both CSVs.

    1 protection group in DPM

     

    All VMs backup starts at the same time (2:00am), 6 of them complete without issues, two (random VMs) generates error after 40-45 minutes, but successful in the end (it seems the backup retries automatically) 

    Error2: (more frequent) Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Back up is in progress. Please wait for backup completion before trying this operation again (0x8007173D)) 

    Error2: (less frequent)

    The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The shadow copy provider had an unexpected error while trying to process the specified operation. (0x8004230F)) 

    We did not configure serialization(not sure if DPM2012 Sp1 & Hyperv2008 R2 need this). The MD3200 Storage comes with hardware VSS, but not sure if it works(I can see it creates snapshot volume in the storage though).

    The error does not happen every day, but at least once or twice a week, the backup completes successfully in the end. Is this normal behavior?



    • Edited by ohohoh Tuesday, May 12, 2015 4:59 AM
    Tuesday, May 12, 2015 4:00 AM

Answers

  • Hi,

    Ok - so it seems hardware provider is being used - else the VM would stay in redirected mode for the duration of the backup.

    You can add these registry entries on the Hyper-V cluster nodes and see if that helps, but as you said the errors seem to be benign since the backups are successful.

    1) Copy the following in notepad, then save the file as csvretry.reg

    Windows Registry Editor Version 5.00

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent\CSV]
    "CsvMaxRetryAttempt"=dword:00000064
    "CsvAttemptWaitTime"=dword:0000ea60

    2) Copy the csvretry.reg file to each node in the cluster.

    3) Logon to each node in the cluster as an administrator, then right-click the csvretry.reg file and select "open with" - then "Registry Editor" option to import the registry settings.

     


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, May 13, 2015 2:03 PM

All replies

  • Hi,

    Yes, DPM 2012 SP1 protecting a Windows 2008 R2 hyper-V cluster using system VSS provider does require serialization unless a hardware VSS provider is being used.  Can you check to see if a hardware provider is installed ?

    From administrative command prompt run: VSSADMIN LIST PROVIDERS and see if more than one provider is listed.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, May 12, 2015 8:33 PM
  • Hi Moderator. yes, there is a hardware VSS listed. but i am not sure if it works or not.\

    - I could see storage creating a snapshot volume by itself.

    - during backup, the CSV is put to redirection mode when each VM is being backed up, but for a very short time.

    does this mean it works?

    last night, the backup generates this error again Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Back up is in progress. Please wait for backup completion before trying this operation again (0x8007173D))  

    but the backup is completed successfully for all VMs.

    Tuesday, May 12, 2015 11:03 PM
  • Hi,

    Ok - so it seems hardware provider is being used - else the VM would stay in redirected mode for the duration of the backup.

    You can add these registry entries on the Hyper-V cluster nodes and see if that helps, but as you said the errors seem to be benign since the backups are successful.

    1) Copy the following in notepad, then save the file as csvretry.reg

    Windows Registry Editor Version 5.00

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent\CSV]
    "CsvMaxRetryAttempt"=dword:00000064
    "CsvAttemptWaitTime"=dword:0000ea60

    2) Copy the csvretry.reg file to each node in the cluster.

    3) Logon to each node in the cluster as an administrator, then right-click the csvretry.reg file and select "open with" - then "Registry Editor" option to import the registry settings.

     


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, May 13, 2015 2:03 PM
  • Thanks, i have not applied the registry change yet, because the backup stopped generating errors for the past few days. 

    But last night.. I found out that a VM restart would trigger a consistency check on that VM, is this normal behavior?

    I had a VM restarted yesterday (software required), last night DMP initiated a consistency check on that VM.

    A couple of weeks ago, we shut down both cluster hosts, after turn them back on and all VMs started... That night, all VM started consistency check and failed, causing CSV offline, and as a consequence CSV hosted VMs offline.

    I wonder why a VM restart would cause consistency check? since this is a resource demanding process, this could cause CSV offline when multiple VMs running consistency check at the same time.

     

    Wednesday, May 20, 2015 11:00 PM
  • Hi,

    <snip>
    But last night.. I found out that a VM restart would trigger a consistency check on that VM, is this normal behavior?
    >snip<

    That is not normal, and most likely not related but a coincidence.  A CC will only be initiated if the replica goes inconsistent, which can happen if a scheduled recovery point fails while in progress or block level tracking is interrupted (IE a node crashes).  Check the job history for that single VM and see what led up to the consistency check job running. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, May 20, 2015 11:34 PM
  • Changes for Microsoft Hyper-V \Backup Using Child Partition Snapshot\server...cannot be applied to C:\ClusterStorage\Volume1\server\Virtual Machines\xxxxxxxxxxxxxxx.bin. (ID 112 Details: The system cannot find the file specified (0x80070002))

    This the only error that i have received last night, also Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Back up is in progress. Please wait for backup completion before trying this operation again (0x8007173D))  on another VM.

    both backup and consistency checked completed successfully eventually after auto-retry.

    The only thing that I can think of is when the VM was triggered for backup, CSV was somehow in an offline status, so the "the system cannot find the file specified (0x80070002)" error occurred. Probably because another VM backup was in progress, because i noticed when scheduled backup is triggered, CSV changes its status from online -> redirection mode -> offline -> online, the whole process happens very quick, in less than 10 seconds, but occurs multiple times. I believe this process happens on every VM in the protection group. say if there are 10 VMs in a protection group, this process will occur 10 times during the scheduled backup.(maybe I'm wrong on this, and i wonder why this is the case?).   

    So I think when that particular VM was triggered for backup, somehow CSV coincidentally was in offline status, again not sure..

    I have used other backup software before eg. shadowprotect.  when backup starts, software take a point in time snapshot of the volume, then start the copying process to repository. Simple and smooth, never had to consider parallel or serialization stuff.. 

     

    Thursday, May 21, 2015 2:49 AM