none
Problem with backing up supported linux virtual machines on WS 2012 RRS feed

  • Question

  • Hi,

    we have Windows Server 2012 standard, and SC DPM 2012 SP1 with latest updates.

    When on Hyper-V host, we have several Linux / CentOS 6.3 or Ubuntu Server 12, with latest integration tools installed, virtualization is running fine.
    We do not have failover clustor or shared storage.

    However, when DPM agent, starts to backing up hyper-v host, all virtual machines (including windows servers on same host) stops responding, and virtual machines gets no access to disk IO

    On Ubuntu console message is:

    Read-error on swap-device (252:0:319496)

    sd 2:0:0:0 > rejecting I/O to offline device

    after backup of all virtual machines is finished everything again works.

    Some things happen if you do not use DPM at all and use native Windows Server Backup ..

    Also, while machine state is "Backing up.." , you can not modify and/or work with any virtual machine on this host (can not change network or any settings, shutdown virtual machines etc.) , however virtual machines are up and running inside ram and network (ping) is working, but they do not have disk access... . Also, on Hyper-V host, it seems that access to disk (where virtual machines is placed) is not available, and in resource monitor disk usage is at maximum.

    On Hyper-V 2.0 / Windows Server 2008 R2 we did not have such a problems...

    Problem with this persist on all our windows servers 2012 / hyper-v hosts where there exists virtual machines...

    Please help, how to properly backup, including supported linux virtual machines, without downtime.

    Kind regards


    -- Hrvoje Kusulja



    Wednesday, May 1, 2013 7:34 PM

All replies

  • Hi Hrvoje,

    DPM uses VSS to take the backups. If you backup a Non-Vss aware OS using DPM - We put it under saved state/turn it off and then back it up.

    See this

    http://social.technet.microsoft.com/Forums/en-US/dataprotectionmanager/thread/bc6a96d0-5a1f-4093-b84d-6d960afecf72/

    Regards,

    Siddharth Jha

    Friday, May 3, 2013 5:04 AM
  • I understand that there is no DPM Agent inside Linux, and that DPM Agent on Hyper-V host, must save state, backup and then restore.

    However, during backup, I can not work with any other virtual machine in mean of change any settings or power on/off.., also, all of them (including virtual windows servers) does not have access to its virtual disk... :/

    maybe problem is that VSS takes all available disk IO (in my case about 150MB/s constantly), and there is no IO capacity for other hosts.., is there a way for limiting disk IO ?

    Thank you


    -- Hrvoje Kusulja

    Friday, May 3, 2013 6:26 AM
  • Hi,

    A couple of possible workarounds.

    1) Use more CSV disks and put fewer guests per CSV.
    2) Install and configure a VSS hardware provider for the SAN to offload snapshots from Windows.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, May 3, 2013 10:31 PM
    Moderator
  • Hi,

    I do not have CSV, and I do not have Hyper-V cluster. All servers are stand alone with their local internal storage (RAID mirror with battery inside IBM server, there are 2x SAS 15k rpm drives) and total of 10 VMs on one physical server with about 128 GB RAM and two socket CPU. The load on virtual macines is not hight and there is no such a number of disk write IO operations...

    Anyway is there a way to determine exactly where the problem is ? and / or to configure maximum usage of disk IO per virtual machine ?

    thank you


    -- Hrvoje Kusulja

    Friday, May 3, 2013 10:48 PM
  • Hi,

    You would need to use performance monitor to see what the disk performance looks like.

    You can try adding another disk - then configure VSS to use that other disk for shadow copies.

    example:  Hyper-V guests reside on volume G:  - New disk added and new volume H: is created.

    Open administrative command prompt and type:

    C:\>vssadmin add shadowstorage /for=g: /on=h: /maxsize=unbounded 

    This will take some write workload off the G: volume and maintain the snapshot on H:


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Saturday, May 4, 2013 12:09 AM
    Moderator
  • Ok, thank you, I understand this new hardware (hdd) will take some write loads..., however, is there some software solution for limiting disk IO ?

    -- Hrvoje Kusulja

    Saturday, May 4, 2013 4:14 PM
  • I think we should try to duplicate this using diskshadow.exe to create a persistent snapshot and manually copy a VM.

    1) On the hyper-V host open an administrative command prompt.
    2) Run diskshadow.exe - and after it starts enter the following commands as written except for the drive letters used:

       Diskshadow> Set Verbose on
       Diskshadow> Set context Persistent
       Diskshadow> Add Volume X: ALIAS xvol    (Where X: is the drive letter of the volume containing the Hyper-V guests you want to make a snapshot for)
       Diskshadow> Begin backup
       Diskshadow> create                   
       Diskshadow>end backup
       DiskShadow> Expose %xvol% S:              (Where S: is a free drive letter)
      
    3) Now Go into explorer and copy out VM's from the S: Drive which is the live shadow copy and see if that duplicates your problem.

    When you are done, delete the shadow copy in diskshadow using the following command.
       
       
       DiskShadow>Delete shadows exposed S:  


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, May 6, 2013 3:15 PM
    Moderator
  • In our environment we’re backing up "non-native machines" (a couple of Ubuntu VMs, actually) that reside on Windows 2012 host using free tool called VeeamZIP.

    It allows us to perform either so-called “offline” backup (with a little downtime required for quiescing VMs prior to taking a volume snapshot) or crash-consistent one (without any downtime; since these VMs don’t run any specific application, in overwhelming majority of cases crash-consistency is more than enough for us).

    Kind regards, Leonardo.




    Monday, May 13, 2013 11:35 AM
  • hey did you disable backup under integration services in the vm configuration? this is not supported with non windows vm's.

    also check out centos 6.4. It's a major improvement over 6.3 in terms of reliability with hyper-v.

    stefan


    • Edited by Stefan SP Saturday, May 18, 2013 7:56 PM
    Saturday, May 18, 2013 7:56 PM
  • Hi,

    I have CentOS 6.3 (which is supported), I will plan upgrade to CentOS 6.4. I have disabled backup under integration services (and I understand this is not supported).


    -- Hrvoje Kusulja

    Tuesday, May 21, 2013 4:46 PM
  • I have disabled backup under integration services in Hyper-V console. And issue is gone.


    Have a nice day !!! DPM 2012 R2: Remove Recovery Points


    • Edited by EugeneLeitanMVP Thursday, January 29, 2015 12:53 PM add picture
    Thursday, January 29, 2015 12:51 PM