none
Backup of a VM causes 2008 Hyper-V host to crash RRS feed

  • Question

  • We have DPM2010 backing up several VMs on a Windows 2008 host.

    Things work OK for several weeks but every now and again I come in after an overnight backup to find that the host has crashed. Not completely but basically all the VMs don't work (but can be pinged) and the host itself becomes unresponsive to such an extent that it cannot be rebooted. I have no option but to hard power off the server.

    This has occurred 3 times in the last few months. Needless to say that it has quite a negative impact on users.

    It always happens while starting to backup one particular VM (a Windows 2003 VM), and it stalls on it (DPM shows that it is still stuck on it in the morning).

    Previoulsy, I had an issue with this VM as it always backed up Saved State but I managed to resolve this (C drive had a shadow location set to D which I removed). It now backups up as Child Partition (online), but it didn't resolve this host crashing issue, as I discovered this morning.

    On investigation, the event log on the hyper V host shows

    Volsnap, Error 16.

    The shadow copies of volume \\?...e09-11e1-b626-001e4f389e34} were aborted because volume \\?...e09-11e1-b626-001e4f389e34}, which contains shadow copy storage for this shadow copy, was force dismounted.

    There doesn't appear to any special event logged on DPM server around same time.

    Also, http://technet.microsoft.com/en-us/library/ff399057.aspx suggests one possible cause/resolution to the HyperV host crashing, but I don't think it is the same cause/resolution as I don't get the error: "You may experience a Stop 0x7E when you backup a server that has the Hyper-V role installed. " And I cannot find the 64bit version of the hotfix referred to.

    I just wondered if anyone could shed any light on what might be causing this?

    Many Thanks,

    Bruce.

    Thursday, May 3, 2012 7:04 PM

All replies

  • Hi,

    Please review the following to correct problems with the volume.

    http://technet.microsoft.com/en-us/library/cc734315(WS.10).aspx

    You can run mountvol.exe in the guest and on the host to see what volume that maps to then run chkdsk /f againt that volume.  Make sure you have enough free space and defrag the volume.

    If that does not help can you upgrade the Windows 2003 to Windows 2008 ? 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Saturday, May 5, 2012 12:56 AM
    Moderator
  • Hmm, running mountvol.exe on the host shows the volume as E:\ which is where all the Hyper-V VMs (i.e. the VHDs) are stored, but not a location in the DPM Storage Pool, as I expected. Scheduled a chkdsk following a reboot (from with GUI rather than commandline).

    Wednesday, May 9, 2012 2:00 PM
  • Would upgrading the VM to 2008 actually help? Is the problem actually related to the Guest OS version?

    In any case, I can't upgrade the Guest OS due to lack of disk space on C (only 2Gb free), but also I am not aware if application that it is is hosting is compatible with Windows 2008 Server.

    What option I am considering is migrating the VM to one of our 3 Windows 2008 R2 Hyper-V servers as (as I suspect this issue only affects Windows 2008 Hyper-V). The other Hyper-V servers host other critical VMs, and I wouldn't want the same problem to be transferred to another Hyper-V host, so I am not 100% on this option.

    Regarding my post above, the mountvol.exe command would seem to suggest that the Shadow copy storage location for this is E:\ on the Host server (which is also where the VM is stored). I am guessing that this could be a Red Herring; as far as the Host is concerned that is the correct Shadow copy storage location, but in reality, when it comes to backups, DPM is telling VSS to write directly to it's own Storage Pool?

    So, should I be running CHKDSK on the parition where DPM stores the replica/snapshots of this VM?

    Someone in the General Thread came up with some suggestions, and I am beginning to wonder if the simplest option might be to remove the VM from the protection group and delete all the disk-to-disk backups (partitions), and start the protection again. I backup to tape each day so will still have backups of this server. Can I deduce that there is not a hardware/sector level issue with any of the Storage Pool disks as this would show up in the System event log? Therefore, letting DPM re-create the partition would resolve this if there is a (logical) file system corruption, even if it decides to create the new partition on the same section of the same disk.

    I appreciate that this action may have consequences.

    My main concern, would DPM remove all tapes catalogues from the DPMDB for this VM if I remove it from the PG?

    If yes, would this would only be an issue if I need to recover this VM from a tape which has not been overwritten since adding and removing the the VM from the PG? Presumably I would only be able to perform a flat restore of the files (e.g. VHD, XML)?

    Any advice is appreciated.

    Regards,

    Bruce.




    • Edited by Redbruce Friday, May 11, 2012 12:04 PM
    Friday, May 11, 2012 8:49 AM
  • Hi Bruce,

    Lots of thought in that last post, but I'll answer what I can.

    Since the failure only occurs for that single Windows 2003 VM, and all other VM's backup fine that are also located on the E: drive, I'm not sure why you are getting that error for the E: volume during a backup.   Are you certain you don't have no volsnap errors in the system event log in either the guest or the host ?

    When DPM does a backup of a hyper-V guest, the Hyper-V writer initiates a snaphop inside the guest for all of it's volumes.  Once that is done, a snapshot is taken of the host volume (E:) and each of the guest VHD's are mounted locally on the host and their snaphots reverted.   You can see this in Windows disk management while doing a backup, disks will be added and removed. Finally DPM transfers changed blocks on the guests VHD to the DPM server and applies the changes to the replica volume on the DPM server.

    The error you are seeing says that the snapshop for the E: volume was force dismounted and therefore we could not copy out the changed blocks.   

    If you remove the problematic guest from protection and delete replica, but keep the tapes, then you can restore from the tapes.  However, if the problem is either in the guest, or the E: volume on the host, then reprotecting it will probably not help.  Moving the Guest to another Hyper-V server should not be a problem, if the issue follows it, you know it's a guest problem.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, May 11, 2012 2:40 PM
    Moderator
  • The error that I reported was a volsnap on the Hyper-V host:

    A summary of the events starting with the VOLSNAP (all these happen within one minute).

    VOLSNAP: The shadow copies of volume \\?...e09-11e1-b626-001e4f389e34} were aborted because volume \\?...e09-11e1-b626-001e4f389e34}, which contains shadow copy storage for this shadow copy, was force dismounted.

    VSS: Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details RevertToSnashot [hr = 0x80042308].

    Hyper-V-VMMS: Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine '%1'. (Virtual machine ID %2)

    PlugPlayManager: The device 'Storage miniport driver' (VMBUS\{59b9dd9b-a870-44dd-bbd7-34dcfdfef560}\1&3189fc23&0&{59b9dd9b-a870-44dd-bbd7-34dcfdfef560}) disappeared from the system without first being prepared for removal.

    There are other events prior to the VOLSNAP, but they are "Information" events.

    Thanks,

    Bruce.

    Monday, May 14, 2012 6:25 PM
  • We have the Schools/College Inspection body OFTSED (England/Wales) in this week (very important).

    As a temporary measure I have removed the VM from the PG and am backing it up at the Guest level.

    If there is a Disaster that requires a recovery of the VM I can recover the existing VM snapshot (last successfully last week) and then recover any revelent files from the Guest level back into that (the VM is a library DB).

    I can then attempt to address this issue properly next week (after they have gone).

    Thanks,

    Bruce.

    Monday, May 14, 2012 6:30 PM
  • Hi,

    The errors are showing we're having trouble reverting the snapshots  for the guest, so this is a guest problem like I suspected.  Make sure latest integration components are installed in that guest. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, May 14, 2012 6:45 PM
    Moderator
  • Hi Mike,

    I'll take a look tomorrow to see if the Integrated Services are up-to-date on this VM.

    Background:

    One reason why I didn't think that there was an issue on the Guest was that there was no warnings or errors related to the backups in the event logs on the Guest.

    Also, for backing up VMs, I was informed that only the servers that had the agent installed (e.g. the Hyper-V hosts) required the prerequsites, so this 2003 VM had SP2, but none of the DPM prerequisites.

    On Friday I installed the DPM prerequisite KB940349 onto the Guest so that I could backup at the Guest level.

    Was this correct? I.e. Do VMs backed-up at the host level require the DPM prerequisities installing?

    Since I setup DPM this VM was being backed up Saved State (when this problem still ocurred). I worked out that this was because the VSS Storage location for C: was set to D: (which doesn't meet the requirements for Child Partition backups) so I fixed this and the VM server then backed up at the Child Partition level.

    Thanks,

    Bruce.

    Monday, May 14, 2012 7:52 PM
  • Hi,

    You do not need DPM prerequisite fixes installed in Guests when doing host level backups.  Only servers that have a DPM agent running on them need the prerequisites installed.  


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, May 14, 2012 11:25 PM
    Moderator