none
30112 VSS Error + Event ID 10111 Failed to mount the virtual hard disk RRS feed

  • Question

  • Since applying security updates to our Hyper-V server (2008R2 Ent. Core), DPM 2010 has been unable to create recovery points of any of the Hyper-V guests.

    DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer experienced a transient error.  If the backup process is retried, the error may not reoccur. (0x800423F3))

    I also see in the Hyper-V server System event logs:

    Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
    Source:        Microsoft-Windows-Hyper-V-VMMS
    Date:          1/18/2011 2:02:02 AM
    Event ID:      10111
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      server.lan.com
    Description:
    Failed to mount the virtual hard disk '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy24\hyper-v...

    When I do a vssadmin list writers on the Hyper-V server is shows that the Hyper-V VSS Writer is in a stable state, but it did have a retryable error:

    Writer name: 'Microsoft Hyper-V VSS Writer'

       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

       Writer Instance Id: {de88412e-6663-4486-ac77-3781a8556a5f}

       State: [1] Stable

       Last error: Retryable error

     I tried rebooting our Hyper-V server which cleared the Hyper-V VSS writer 'Retryable error'.  But once DPM ran it's scheduled recovery points, the 'Retryable error' returned, same errors as before.  I upgraded SQL 2008 to SP2 which updated the Microsoft SQL Server VSS Writer, but that didn’t help either when I attempt to retry a recovery point on one of our Hyper-V guests.  I then applied hotfix KB 2250444 as I was also seeing the MSDPM error (see below) addressed in this hotfix… but the problem still persists. 

    I am able to use Windows Backup within the Hyper-V guest and create snapshots from Hyper-V, which leads me to believe that this is a DPM issue.

    Log Name:      Application
    Source:        MSDPM
    Date:          1/19/2011 9:26:26 AM
    Event ID:      945
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      server.lan.com

    Description:The description for Event ID 945 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    Any ideas would be greatly appreciated.... Thanks!

     

    • Edited by rosenstc Wednesday, January 19, 2011 7:13 PM edit
    Tuesday, January 18, 2011 6:10 PM

Answers

  • Microsoft helped me resolve this issue:

    • Applied hotfix KB2155024 (this did not resolve the issue)
    • They discovered that my vmbus.sys file on the host Hyper-V server was not at the same level as the Hyper-V guests, so they had me apply hotfix KB981836 which has the latest version of the vmbus.sys file
    • They also had me apply hotfix KB977096 to update my vmms.exe file to the latest levels.
    • They also found that I was missing a registry entry for 'NoAutoMount'.  So, I had to manually create a REG_DWORD registry entry under: HKLM\System\CCS\Services\Mountmgr and set it to 0.

    That seemed to do the trick!

    • Marked as answer by rosenstc Tuesday, February 1, 2011 11:42 PM
    Tuesday, February 1, 2011 11:42 PM

All replies

  • Enable automount (mountvol.exe /E) on the Hyper-V host and retry the DPM backups. It should work.
    Thanks Shyama Hembram[MSFT] This posting is provided AS IS, with no warranties, and confers no rights.
    Thursday, January 20, 2011 8:25 PM
    Moderator
  • I tried the mountvol.exe /E on our Hyper-V host server and then attempted a recovery point for one our Hyper-V guests, but I am still getting the same errors.  Do you suppose I will need to reboot the Hyper-V host server?

    Friday, January 21, 2011 12:20 AM
  • Additional Info:

    I tried installing the WindowsServerBackup feature on the Hyper-V host server and attempted to backup the .vhd file of one of the Hyper-V guests.  It failed with the following error:

    Log Name:      Application
    Source:        Microsoft-Windows-Backup
    Date:          1/21/2011 11:35:03 AM
    Event ID:      521
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      server.lan.com
    Description:
    The backup operation that started at '‎2011‎-‎01‎-‎21T19:25:03.605000000Z' has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code '2155348001'. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

    Friday, January 21, 2011 7:44 PM
  • There are two hotfixes needed for DPM that address Hyper-V.  I found that hotfix KB975921 was not installed on the Hyper-V host server.  Hotfix KB975354 was installed.  Although, now when you go to Microsoft for KB975354 (http://support.microsoft.com/kb/975354)  they only offer one file for fix287961 which extracts the patch: Windows6.1-KB975354-v2-x64.msu.

     

    Patch Windows6.1-KB975354-x64.msu (which should be installed first) is no longer available to download.    I have hotfix KB975354, so I’m not worried about reinstalling this hotfix.   

     

    I have to reboot tonight and reapply integration services, so I will see if this makes any difference.

    Saturday, January 22, 2011 12:15 AM
  • Hotfix KB975921 did not help, same errors as before.
    • Marked as answer by rosenstc Tuesday, February 1, 2011 11:36 PM
    • Unmarked as answer by rosenstc Tuesday, February 1, 2011 11:36 PM
    Monday, January 24, 2011 12:24 AM
  • Microsoft helped me resolve this issue:

    • Applied hotfix KB2155024 (this did not resolve the issue)
    • They discovered that my vmbus.sys file on the host Hyper-V server was not at the same level as the Hyper-V guests, so they had me apply hotfix KB981836 which has the latest version of the vmbus.sys file
    • They also had me apply hotfix KB977096 to update my vmms.exe file to the latest levels.
    • They also found that I was missing a registry entry for 'NoAutoMount'.  So, I had to manually create a REG_DWORD registry entry under: HKLM\System\CCS\Services\Mountmgr and set it to 0.

    That seemed to do the trick!

    • Marked as answer by rosenstc Tuesday, February 1, 2011 11:42 PM
    Tuesday, February 1, 2011 11:42 PM
  • We started getting Event ID 10111 "Failed to mount the virtual hard disk" during DPM backup jobs on multiple Hyper-V guests after installing the latest Windows Updates on the host and guest images.  The situation improved somewhat after installing the latest DPM / Hyper-V / VSS related updates.  The action that seem to resolve the issue is defragmenting the hard drives in the Hyper-V guests for which the DPM jobs were failing, and rebooting those guests.  Somehow the Hyper-V VSS writer seems to be vulnerable for disk fragmentation within the guest operating system.

    Here's a list of other actions you can take in order to (attempt) to resolve event 10111 (thanks to Microsoft support technician):

    • Install the last VMMS.EXE service that host the Hyper-V VSS Writer with the following Hotfix:
      Hyper-V Export function consumes all available memory in Windows Server 2008 or in Windows Server 2008 R2 - http://support.microsoft.com/kb/2547551/en-us
    • Reinstall the Integration Services on ALL Virtual Machines and reboot these virtual machine: From the Virtual Machine Connection windows, go the toolbar menu » Action » Insert Integration Service Setup Disk » Click on the Install…

    • Check on Hyper-V Host and on VM that the Automount volume is enable: You can check that running DISKPART » AUTOMOUNT (if it’s not enable, run AUTOMOUNT ENABLE).
    • Even if the host only uses the VSS System Provider (SWPRV), you can  “force” the use of it by DPM by adding the following key on every Hyper-V host:
      HKLM\Software\Microsoft\Microsoft Data Protection Manager\Agent\UseSystemSoftwareProvider

    • Volsnap.sys - Windows6.1-KB2632149-x64.msu:
      "fvevol!FveFilterDeviceControl+1d0" Stop error when you create a VSS snapshot backup in Windows Server 2008 R2 SP1 - http://support.microsoft.com/kb/2632149
    • Disk Space on Virtual Machine
      Each time we launch a RP of a VM that is running, we create simultaneously a Snapshot of the volume hosting the VM’s (i.e you I drive) and also we create snapshot inside the VM for all drive. This is the normal mechanism when we make online backups.
      If there’s lack of space inside the volume of your VM, it could be the reason why the backup fail. So please check the free space you have on all VM, particularly on those where the RP were failing.
    • Storage Area for Snapshots
      Another issue that could happen is when a recovery point is created, a snapshot could not feed on the available disk space allocated to it during the backup. If during the backup, the VM that is currently under a backup job have an high I/O activity, the shadow copies that was created and used for hosting change during the backup, will grow rapidly. By default, each volume have a Maximum size of Storage Area (ie, location that will host shadow copies) of 10% of the total size of the volume.
      You can check this storage area configuration looking at the Shadow Copy Tag of the property of the volume, or launching the VSSUIRUN.EXE.
      On those VM that have huge VHD and free space, you can grow the value of the limit (incrementing to 10% each time).
    • Forefront TMG 2010 known issue
      There’s a known issue of RP failure due to lack of available disk space caused by the Web Caching configuration of TMG 2010. The following article, in spite of applied to SCDM 2007, is also valid for SCDPM 2010, as it’s a TMG issue that cause this side effect to the protection mechanism.
      981948  "DPM encountered a retryable VSS error" error message when you use System Center Data Protection Manager 2010 or System Center Data Protection Manager 2007 to protect a Hyper-V virtual machine that has Forefront TMG 2010 installed
      http://support.microsoft.com/default.aspx?scid=kb;EN-US;981948
    • Balancing the workload
      Create multiple protection groups for the Hyper-V host servers and serialize the DPM backup (sync) jobs: for example one set for virtual machines, and another set for System Protection of the Hyper-V Host and application data (Exchange, SQL Server, etc). Also spread the time for recovery point creation along the time you have for backup maintenance, balancing the workload may improve performance and resolve possible conflicts.
    • Antivirus Exclusion
      Make sure that you configure exclusions for the DPM and VSS related executables on all machines (Hyper-V hosts and guests) connected to the DPM server.  For example you should/could exclude the following processes in Forefront:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Forefront\Client Security\1.0\AM\Exclusions\Processes\
      C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMClient.exe
      C:\Program Files\Microsoft Data Protection Manager\DPM\bin\DPMClientService.exe
      C:\Program Files\Microsoft DPM\DPM\bin\DPMLA.exe
      C:\Program Files\Microsoft DPM\DPM\bin\DPMRA.exe
      C:\Windows\system32\vmms.exe
      C:\Windows\system32\Vmwp.exe
      C:\Windows\system32\vssvc.exe

      Virtual machines are missing in the Hyper-V Manager Console or when you create or start a virtual machine, you receive one of the following error codes: "0x800704C8", "0x80070037" or "0x800703E3" - http://support.microsoft.com/kb/961804/en-us

    Can't wait to upgrade to DPM 2012, which will contain some significant improvements on handling Hyper-V guests.

    • Edited by Applied Maths NV Tuesday, February 28, 2012 2:38 PM Discussed issues with Microsoft support technician
    Saturday, January 7, 2012 8:54 PM