none
KB975354 (Hyper-V update rollup, integrated in 2008R2 SP1) and DPM 2010 Cluster Shared Volume (CSV) Backup parallel RRS feed

  • Question

  • Hi there,

    Circumstances

    • 4-Node-HyperV-Cluster, all 2008 R2 SP1 with the most recent Hotfixes post SP1 installed (updated recently)
    • We have the problem that we backup lots of VMs on Cluster Shared Volumes with DPM 2010 and lots of them are failing every night because of parallel backup (with more and more VMs this is getting worse)
    • Due to external factors there is no way to get working hardware providers in the near future
    • The workarounds for serializing CSV-Backups with merging xml-files for DPM is not a good option because everytime you add new VMs to the cluster nodes in DPM you have to do the whole serializing/merging thing again as i understand it

    Due to these circumstances i found the Hotfix KB975354 (http://support.microsoft.com/kb/975354/en-us) and i was hoping that by installing it, it would be the best solution for now. Then i discovered that it is a part of 2008 R2 SP1 and it's already installed an all the hosts.

    In the hotfix description is written, that you have to update your VM guest services, too, to get this to work. So we did this for most of them.

    But even, if the guest services are up to date, the backups fail in DPM with this alert:

    "Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Back up is in progress. Please wait for backup completion before trying this operation again (0x8007173D))"

    Here is an excerpt from the KB975354 site:

    "After you install this rollup, Hyper-V will only use the VSS system provider inside the virtual machine when performing a host-level backup."

    From my understanding, that means, that everytime the host triggers a backup, instead of the vss hyperv writer the local system writer is used. So in theory it shouldn't matter anymore, if the CSV is not available for the host, as the backup is performed inside the vm.

    But the backups keep failing so my questions are:

    1. Is my assumption right with the hotfix, that dpm should now backup from inside the vm?
    2. What could still be causing the backup to fail?
    3. Shouldn't every VM with SP1-Guest Services and on SP1-Hosts no longer get these errors?

    Thanks very much in advance!


    Gracjan Zawadzki, IT-Consultant
    • Edited by phat_gz Monday, September 26, 2011 12:10 PM
    Monday, September 26, 2011 12:09 PM

Answers

  • If the above suggested answer does not help in resolving the thread please re-open it.
    --------------------------------------------------------------------------------
    Regards, Michael V [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, January 25, 2012 12:01 AM
    Moderator

All replies

  • Hi,

    Yes, updating the XML in a multi-cluster protection senario is no fun, and we're working on automating that whole process.  In the meantime, there are two things you can do to mitigate the backup failures.

    As per this blog: http://blogs.technet.com/b/dpm/archive/2011/09/12/using-system-center-data-protection-manager-2010-to-perform-a-host-level-backup-fails-with-error-0x8007173d.aspx

     

    A) Add these keys to each node in the cluster.

    Regardless if using hardware or software snapshot provider , the following registry settings allow you to make adjustments to how DPM performs retries to claim the CSV in order to get reliable backups.

    CsvMaxRetryAttempt - Adjust the maximum number of times (Default is 1) the DPM agent will attempt to claim the CSV volume. The value 0xC8 = 200 times.
    CsvAttemptWaitTime - Adjusts the amount of time in milliseconds to wait between retry attempts. The value 0x2bf20 = 3 minutes.

    1) Copy the following in notepad, then save the file as csvretry.reg

    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent\CSV]
    "CsvMaxRetryAttempt"=dword:000000C8
    "CsvAttemptWaitTime"=dword:0002bf20

    2) Copy the csvretry.reg file to each node in the cluster.

    3) Logon to each node in the cluster as an administrator, then right-click the csvretry.reg file and select "open with" - then "Registry Editor" option to import the registry settings.

     

    B) Should the Host level backups fail intermittently for other reasons, by default DPM will auto-retry the backup one hour after the original failure. Should that backup also fail, no additional retries are attempted.

    To change the default auto-retry interval and count, you can adjust the following registry settings. Note: These will effect ALL DPM jobs, not just Hyper-V Guest backup jobs.

    AutoRerunDelay - The delay in time in minutes before which DPM will attempt to automatically rerun failed jobs. If multiple reruns are configured, this is the gap between the reruns as well. The value of 0x3c = 60 minutes.
    AutoRerunNumberOfAttempts - The number of times a failed job will be retried before giving up if it consistently fails. Default is one. Increasing this value may increase the load on your system. The reruns are done at the gap of AutoRerunDelay.

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Configuration]
    "AutoRerunDelay"=dword:0000003c
    "AutoRerunNumberOfAttempts"=dword:00000005

     

    Between the two changes, that should help you get your backups done, but I would still update the XML periodically.

     


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, September 30, 2011 4:25 AM
    Moderator
  • Hey Mike,

    thank you very much for your detailed response.

    Right now we are waiting for the last VMs to be rebootet and it seems right now that even without serializing and just with upgrading the guest services the failed backups declined. But we will try to serialize the next days and then see what the best move might be.

    I just want to add that the description of this hotfix should be revised as it sounds like a solution to not having a hardware provider but seemingly is not.

     


    Gracjan Zawadzki, IT-Consultant
    Tuesday, October 4, 2011 1:08 PM
  • Hi,

     

    The hotfix "A Hyper-V update rollup is available for Windows Server 2008 R2" is just a general rollup fix for hyper-v and has not related to DPM.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Tuesday, October 4, 2011 5:26 PM
    Moderator
  • If the above suggested answer does not help in resolving the thread please re-open it.
    --------------------------------------------------------------------------------
    Regards, Michael V [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, January 25, 2012 12:01 AM
    Moderator