none
Backing up Hyper-V server slows down VMs tremendously RRS feed

  • Question

  • I'm using DPM 2010 to back up a couple of servers running Hyper-V VMs.  These machines are not heavily loaded.  However, they have only one disk, on which all of the VHDs are stored.

    When backup runs, the VMs (especially SQL servers) slow to a crawl because DPM is hogging all the disk I/O bandwidth.  Resmon shows that VSS data is being read at the rate of > 50 MB/s (even though I throttled DPM to 25 MB/s), and disk queue length is 5-10.

    Is there any way to make DPM (or is it VSS?) play nicely?

    It's so bad that my mirrored SQL server pair (high-safety mode) queries often time out during backups.  Other times, their performance is pretty good.

    • Moved by Praveen D [MSFT] Monday, July 19, 2010 6:56 AM Moving to DPM Hyper-V Protection Forum (From:Data Protection Manager)
    Thursday, July 1, 2010 1:59 AM

Answers

  •  

    Does this occur because you are backing up more than one at a time ?   If you do an ad-hock backup of just one VM, does the resmon show the same results ?   If not, put VM's in seperate PG's and stagger the backup times. 

    If network bandwidth throttling is working correctly, then the backup should take longer and disk I/O will be reduced. Double-check to be sure QOS is enabled on both ends, then just for a test, throttle the network bandwidth way down, to lets say 10mbs and see if that works.  If so, increase it until it's acceptable.  You need change network throttling when no jobs are running, then refresh the agent to be sure the new settings are honored.

     


    Regards, Mike J [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, July 1, 2010 2:14 AM
    Moderator

All replies

  •  

    Does this occur because you are backing up more than one at a time ?   If you do an ad-hock backup of just one VM, does the resmon show the same results ?   If not, put VM's in seperate PG's and stagger the backup times. 

    If network bandwidth throttling is working correctly, then the backup should take longer and disk I/O will be reduced. Double-check to be sure QOS is enabled on both ends, then just for a test, throttle the network bandwidth way down, to lets say 10mbs and see if that works.  If so, increase it until it's acceptable.  You need change network throttling when no jobs are running, then refresh the agent to be sure the new settings are honored.

     


    Regards, Mike J [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, July 1, 2010 2:14 AM
    Moderator
  • We had this same issue with DPM 2007. Even though the OS supports it, for some reason the DPM team still lets the backup process run at normal IO priority on reads instead of setting it to background which should prevent it (or at least help) it from interfering with production IO.

    What we found is that set the network bandwidth throttle on the hosts in DPM did help with backups - at the expense of restores. We had to set it to 30Mbps (Mbps not MB/s) to ensure it didn't strain the disk layer during backups. I believe the setting only takes affect after all backups are finished on the server, so don't expect to see immediate results when changing the setting, but dropping this to 30Mbps (~4MB/s) per HV host seems to keep performance acceptable during backup times (which ends up being all day, every day for us because we have so many VM's and now DPM has been throttled so low).


    Jeff Graves, ORCS Web, Inc.
    Thursday, July 1, 2010 1:39 PM
  • Yes, even one at a time causes problems. 

    I have implemented network throttling (200 Mbps), but it seems that during the beginning of the backup, DPM (maybe it's actually VSS?) is reading 50 MB/s from the disk, even though there is little network traffic.  My guess is that it's doing something to prepare for the backup to start, and that process is what is clobbering other applications.

    Tuesday, July 13, 2010 2:21 AM
  • Thanks for the suggestion; i'll try throttling even lower. 
    Tuesday, July 13, 2010 2:22 AM
  • If the above suggested answer does not help in resolving the thread please re-open it
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, September 3, 2010 8:15 PM
    Moderator