none
Hyper-V Cluster Hosts BMR Backups take over 12 hours RRS feed

  • Question

  • Hello, 

    We are using 2012 R2 DC to create a Hyper-V  2 node cluster.  The backup is DPM 2012 R2 running on 2012 R2 DC.  The DPM system was upgraded to 2012 R2 from DPM 2010 about a year ago and has worked for the most part.

    Two problems we are having.  I am using the VSS writer from the SAN manufacture.  On our old cluster I used the Microsoft VSS writer and had no problems unless I loaded the C: up with too many files on the cluster node.  I saw another post that says to disable the HW VSS writer but that was unsuccessful the last time I tried it.  I am going to try it again though.

    1.  I currently get a random VM that hangs during the backup.  The VM still runs but I cannot migrate it and backups fail from that point on.  In Hyper-V backup the Status field shows "Backing up...".  I am waiting for this to happen on a non critical VM but alas that is not happening. <sigh>  To correct the problem I have to pull power from the cluster node the system was on and then about 90% of the time I have to replace to different .sys files in c:\windows\system23\drivers and then run the startup repair utility from the windows install disk.

    2.  I did not notice this until this week but my BMR backups of the Hyper-V hosts are taking over 12 hours to complete.  The BMR on a different stand alone server is taking less than 10 min.  What is causing this and if these are taking this long is it interfering with the backup of the VMs on the cluster and causing the problem in paragraph 1 above?  I have tried to stagger my backups so they don't interfere with each other but if this one is taking 12-14 hours that will never happen.

    Tuesday, October 14, 2014 9:15 PM

All replies

  • Hi,

    Are you on DPM 2012 R2 UR3 ?    If so - there is no need to a vss hardware provider, you can safely un-install it and DPM backups will continue to work fine.


    Problem-1)  I currently get a random VM that hangs during the backup.  The VM still runs but I cannot migrate it and backups fail from that point on.  In Hyper-V backup the Status field shows "Backing up...". 

    Response-1) Is there still an active backup job for that VM on the DPM server - and does it show that it's still transferring changes ?   If not, then wait for other VM backups to complete on that host, then manually stop the DPMRA service - that will reset the "backup in progress" flag and allow the VM to be managed.  We have an open bug that we're working on to fix that issue. 

    Problem-2) My BMR backups of the Hyper-V hosts are taking over 12 hours to complete. 

    Response-2)  DPM is not responsible for the amount of time a BMR backup takes to complete.  To troubleshoot this outside of DPM perform the following.


    First make sure no active BMR job is running by looking at running jobs on the DPM server.
    Second - using task manager make sure there are no WBENGINE.EXE process running on the DC you want to test - if so - kill it with task manager.


     To test BMR backup outside of DPM, try this command:

    1) Set up a network share on a remote machine \\server\bmrshare

    2) From an administrative command prompt on the PS, type:

                  wbadmin.exe start backup -allcritical -backuptarget:\\server\bmrshare


    This should show you the list of volumes included in the BMR backup and ask "Do you want to start the backup operation?. - Type Y to continue..

    See how long it takes to complete or see if it hangs - kill it if you need to after X hours - but look on the share and see how much it copied before killing it.

    You can try disabling chimney on both DPM and DC to see if that helps.

        c:\>Netsh int tcp set global chimney=disabled


     


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, October 14, 2014 11:02 PM
    Moderator
  • 1. We have also had this issue in the past, our resolution was to restart the "Hyper-V Virtual Machine Management" service. This forced VMMS to do a reevaluation of the VM's status and which should clear the backing up status.

    2. What does your network utilization look like? 

    Wednesday, October 15, 2014 2:27 PM
  • Were you ever able to resolve this issue in the end?

    I am having the exact same problem and initially I attributed it to a rogue Windows Update that was installed as we had similar behaviour from our 6 Hyper-V Nodes after the August updates which, when uninstalled, let the BMR backups complete in an acceptable amount of time.

    I noticed that one of the August updates was reinstalled as part of our October update window which appeared to be when this issue reared its head again, however even after uninstalling that update, the problem persisted.

    I currently have one of the nodes out of action as it was often failing to backup the BMR at all in DPM 2012 R2 UR4 and am in the process of troubleshooting it with MS.

    Any advice would be very much appreciated.

    Tuesday, December 30, 2014 12:23 PM
  • Happy New Year (belatedly anyway)

    Just to update this, we have had our monthly downtime and as part of that I installed the December Windows Updates. Included in the list of updates was the December Update Rollup (KB3013769) which, after a bit of testing, appears to have solved the BMR backup taking a hideous amount of time.

    There were a few cluster-related tweaks as part of this update and it appears that one of these has sorted it.

    Just something for you to try in case you're still experiencing the same problem.

    Andy

    • Proposed as answer by Barratt84 Friday, January 23, 2015 3:16 PM
    Friday, January 23, 2015 3:16 PM