none
Backup CentOS Linux VMs using hyperv-daemons RRS feed

  • Question

  • Hi all,

    I am running a couple of Linux VMs on a Hyper-V 2012R2 Host used Distributions are CentOS 7 and CentOS 6.6 and I am unable to backup these VMs with DPM 2012 R2. In the past, I was able to backup my CentOS VMs through offline backup. IIRC this stopped after I've applied the upgrade to CentOS 6.4.

    Now I found out that DPM 2012 R2 should be able to backup my Linux VMs regarding to this article: 

    http://technet.microsoft.com/en-US/library/dn531026.aspx

    From this article I got the hint to use the hyperv-daemons on my VMs. After installing these packages in DPM the VMs changed from offline to online datasources. So I was confident but it still didn't work. DPM reports the following errors:

    Description:
    The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried, the error is likely to reoccur. (0x800423F4))

    On the Linux VM there is no entry in /var/log/messages about the backup attempt. SELinux is disabled and of course the services hypervfcopyd, hyprvkvpd and hypervvssd are running. Related to error 0x800423F4 i found hints, that there is not enough space to create shadow copy, but on the root filesystem is enough space available (only 20% used).

    What am I doing wrong? Any hints to get this running?

    Thanks in advance,
    Carsten

    Monday, January 12, 2015 9:44 AM

All replies

  • Hi all,

    missed to share informations about my DPM Installation with you. I'm running DPM 2012 R2 with build number 4.2.1273.0. Hyper-V Servers are updated with latest DPM agents.

    On my CentOS VMs I installed the hyperv-daemons that are provided from default repositories:

    # yum info hypervkvpd
    Geladene Plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.de.leaseweb.net
     * extras: mirror2.hs-esslingen.de
     * updates: mirror.skylink-datacenter.de
    Installierte Pakete
    Name       : hypervkvpd
    Architektur : x86_64
    Version    : 0
    Ausgabe    : 0.24.20130826git.el7
    Größe : 31 k
    Quelle      : installed
    Aus Quelle  : base
    Zusammenfassung: HyperV key value pair (KVP) daemon
    URL        : http://www.kernel.org
    Lizenz      : GPLv2
    Beschreibung: Hypervkvpd is an implementation of HyperV key value pair (KVP)
                : functionality for Linux. The daemon first registers with the
                : kernel driver. After this is done it collects information
                : requested by Windows Host about the Linux Guest. It also supports
                : IP injection functionality on the Guest.

    Any help appreciated!

    Carsten


    • Edited by Carsten Lenz Wednesday, January 14, 2015 9:27 AM
    Wednesday, January 14, 2015 8:59 AM
  • Can you please try following and let us know what are the results:

    Method 1:

    Check theVolume shadow copy service in services window. To check follow these steps:

    1.      Click on Start

    2.      In the start Search box type Services

    3.      Right Click on – Run as Administrator

    Check if the below mentioned services has the mentioned status and start up type.

    Volume Shadow Copy Service                   Started              Automatic

    Also check the dependency services that are required for Volume Shadows Copy Service to run.

    To check that double click on the service and click on the dependency tab.

    Method 2:

    Disable the firewall/ anti-virus temporarily installed on your computer and then check if the issue still persists. If the issue resolves, then you might need to uninstall the software and reinstall it back or change the settings and check.


    Regards, Trinadh [MSFT] This posting is provided AS IS with no warranties, and confers no rights. If you found the reply helpful, please MARK IT AS ANSWER. Looking for source of information for DPM? http://blogs.technet.com/b/dpm/ http://technet.microsoft.com/en-in/library/hh758173.aspx

    Monday, February 2, 2015 6:09 PM
    Moderator
  • Hi Trinadh,

    thank's for your reply! On all my Hyper-V Hosts the VSS the StartMode for service VSS is set to manual. Due to your recommendation I started the VSS service on the related Hyper-V Host (without setting startmode to automatic). Trying to backup my Linux CentOS VMs still fails with the same error message.

    On my Hyper-V Hosts (Core installation) there are no antivirus tools installed.

    Any help appreciated,
    Regards Carsten


    Tuesday, February 3, 2015 10:25 AM
  • Hi Carsten,

    I'm having the same issue. I've a number of CentOS VMs, 6.6 & 7.0 on Sever 2012R2 Hyper-V. These were happily backing up until late December when they stopped working. Worse still attempting a backup is crashing the client VMs and has corrupted databases on them.

    I'm not sure what changed that caused these issues. It seems odd that the error is talking about VSS errors. As I understand it Linux backups on server 2012r2 don't use VSS - DPM makes a checkpoint (vhd snapshot) and backs this up. I believe with Windows guests VSS is sometimes used in the guest to ensure a consistent state as part of the checkpoint. So I wonder if the root of the problem is that Hyper-V doesn't realise the guests are Linux and is trying to invoke VSS on the guests.

    I've tried installing the KVP daemon as suggested but this hasn't helped.

    Have you made any progress on this? I'll update this thread if I find a solution - good luck!

    Tim

    Thursday, February 5, 2015 5:01 PM
  • Hi Carsten,

    I've resolved my problem which had the exact symptoms of yours. MS have written a VSS daemon for Linux which is part of the integration components and this seems to be the bit causing the problem.

    We had 2 versions of this on different machines, either microsoft-hyper-v-vss.x86_64 or hypervvssd.x86_64

    You can search to see what you have installed with

    yum list installed | grep hyperv

    or

    yum list installed | grep microsoft

    Then to remove it

    yum remove microsoft-hyper-v-vss.x86_64

    Reboot and try the backup again.

    Anyone able to escalate this to the MS DPM team to look at? Everything worked happily for me for a long time until recently when multiple VMs started failing. Don't know what changed but could be DPM 2012 UR 4, December patch Tuesday or updates on the Linux side.

    Thanks,

    Tim

    Wednesday, February 11, 2015 5:34 PM
  • Hi Tim,

    your workaround didn't work for me:

    # yum list installed | grep hyperv
    hyperv-daemons.x86_64                 0-0.24.20130826git.el7           @base
    hyperv-daemons-license.noarch         0-0.24.20130826git.el7           @base
    hypervkvpd.x86_64                     0-0.24.20130826git.el7           @base
    hypervvssd.x86_64                     0-0.24.20130826git.el7           @base

    # yum list installed | grep microsoft
    #

    There is no microsoft-hyper-v-vss.x86_64 package installed, nor available in default CentOS 7|6 repositories.

    It would be nice to get a solution for this issue, due to the fact, that this constelation is supported according to http://technet.microsoft.com/en-US/library/dn531026.aspx

    Still in hope for a solution. Someday.

    Cheers Carsten

    Thursday, February 12, 2015 7:59 AM
  • Hi Carsten,

    You are looking for either microsoft-hyper-v-vss.x86_64 or hypervvssd.x86_64

    You have the latter, so you need

    yum remove hypervvssd.x86_64

    Disclaimer - obviously the VSS daemon was here for a reason - I think it makes the file system read only to get it into a consistent state, this seemed to be what was crashing the VMs, they never went writeable again. So without this I guess there is a risk of consistency errors in the backups. But this is preferably to no backups and the backup system crashing and corrupting the VMs until the issue is fixed.

    Good luck,

    Tim

    Thursday, February 12, 2015 9:45 AM
  • Hi Tim,

    removing hypervvssd.x86_64 doesn change the behavior on my site, except of changing the VM state in DPM console from online to offline.

    Even an vanilla minimal CentOS 7 Installation cannot be backuped up with DPM.

    I've tried everything. I quit. Until then I'm running something like suspend-vm, export-vm, resume-vm.

    Cheers Carsten


    Thursday, February 12, 2015 10:46 AM