none
Hyper-V host and all guests become unresponsive, event ID 129 fills system log

    Question

  • Server 2008 Standard, running on Dell 1950 with MD1000 attached storage.

    When running a backup exec 12.5 job using the hyper-v agent, during the snapshot process, the system event log starts to fill up with event ID 129.  System and all guests become unresponsive, VSS service becomes completely unresponsive and the only way to get everything up and running again is a power cycle of the host hardware.  System will not shutdown gracefully.

    So far I have updated backupexec and all agents and worked with their support, but this appears to be some kind of VSS issue.  I have also updated RAID controller firmware and drivers for the storage.

    Details from the event log entry below:

    The description for Event ID 129 from source storvsc cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    \Device\RaidPort38

    Saturday, August 08, 2009 9:30 AM

Answers

  • Hi,

     

    Q: Can you clarify?

     

    >>Volume Shadow Copy Service must be enabled on all volumes used by the virtual machine with a specific configuration. Each volume must itself as the storage location for its shadow copies and that mapping must be available to the Hyper-V VSS writer. In other words, the shadow copy storage of C: should be on C:, the shadow copy storage of D: should be on D:, so on an so forth.

     

    For more information, you can refer to:

     

    Backing Up Hyper-V Virtual Machines

    http://blogs.technet.com/virtualization/archive/2008/08/29/backing-up-hyper-v-virtual-machines.aspx

     

    In addition, Symantec Backup Exec 12.5 can backup an individual VM, please backup the problematic VM to test whether it works.

     

     

    Best Regards,

    Vincent Hu

     

    Tuesday, August 11, 2009 7:20 AM
  • I think I have everything working, there was another VSS error to resolve on the guest.  I've been able to run two sucessfull backups now with no system hang or other VSS issues...  I hope it will stay stable now.

    thanks,
    Thursday, August 13, 2009 7:38 PM

All replies

  • I've narrowed it down to one VM causing this problem, when it's not selected in the backup job everything works fine.  When it is, about 5 minutes in to the snapshot process I start getting the storvsc, ID 129 events on the host, all VMs become unresponsive and I have to power cycle the server.

    Not sure where to go from here...
    Saturday, August 08, 2009 4:26 PM
  • Hi,

     

    First of all, please ensure that you have installed the RTM version of Hyper-V. In addition, please apply and install the following Hyper-V backup related hot fixes:

     

    959962  An update is available for Windows Server 2008-based computers to address issues with backing up and restoring Hyper-V virtual machines

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;959962

     

    960038  You receive a "0x0000007E" Stop error on Windows Server 2008-based computers that host Hyper-V virtual machines when you use the Hyper-V writer to back up virtual machines

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;960038

     

    956697  Windows Server 2008 Hyper-V VSS writer is not used during a backup job because of corrupted or invalid virtual machine configuration files

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;956697

     

    After installing the hot fixes, please ensure to update the Integration Component in all guests to test again. In addition, online backup is not possible if one of the following is true. Please verify whether the problematic VM meets these requirements:

     

    1.Integration components is installed and is running the latest version

    2. No Dynamic disks inside the guest.

    3. All volumes are NTFS

    4. All NTFS volumes must be >1GB and have >300MB free space.

    5. Shadow copies within the VM are on the same volume or are Disabled

     

    According to the symptom, this problem only happens for a particular VM. Please help me to collect the following information:

     

    1. What is OS version of the problematic VM? Windows Server 2000 VM does not support online backup.

    2. Are there any differences between the problematic VM and the working VM?

    3. Please use the .vhd file of the problematic VM to create a new guest machine. Does it work for the newly created VM?

    4. Please collect the exact Event 129 error in detail.

     

     

    Best Regards,

    Vincent Hu

    Monday, August 10, 2009 11:56 AM
  • All of those updates are installed, SP2 is installed on the host.

    Online backup criteria...VM meets all except for #5.  Can you clarify?  I have 3 volumes with shadow copies turned on, are you saying you have to store the shadow copies all on the same volume or that you can only have shadow copies turned on, on one volume?

    Thank you,
    Monday, August 10, 2009 3:11 PM
  • Hi,

     

    Q: Can you clarify?

     

    >>Volume Shadow Copy Service must be enabled on all volumes used by the virtual machine with a specific configuration. Each volume must itself as the storage location for its shadow copies and that mapping must be available to the Hyper-V VSS writer. In other words, the shadow copy storage of C: should be on C:, the shadow copy storage of D: should be on D:, so on an so forth.

     

    For more information, you can refer to:

     

    Backing Up Hyper-V Virtual Machines

    http://blogs.technet.com/virtualization/archive/2008/08/29/backing-up-hyper-v-virtual-machines.aspx

     

    In addition, Symantec Backup Exec 12.5 can backup an individual VM, please backup the problematic VM to test whether it works.

     

     

    Best Regards,

    Vincent Hu

     

    Tuesday, August 11, 2009 7:20 AM
  • Making changes to the shadow copies config to the volumes inside the VM seems to have gotten rid of the 129 events and system hang, but since the problem is intermittent I can't be sure until it runs for awhile.


    However, I am now having other VSS issues.  Backup jobs fail and running "vssadmin list writers" on the host returns these results for the hyper-v writer...


    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.

    Waiting for responses.
    These may be delayed if a shadow copy is being prepared.

    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {f4ae53ba-f458-457a-85db-b4825c9ba696}
       State: [8] Failed
       Last error: Retryable error

    • Proposed as answer by Brian Borg Friday, August 14, 2009 1:09 AM
    Thursday, August 13, 2009 1:04 PM
  • I think I have everything working, there was another VSS error to resolve on the guest.  I've been able to run two sucessfull backups now with no system hang or other VSS issues...  I hope it will stay stable now.

    thanks,
    Thursday, August 13, 2009 7:38 PM
  • Hi,

     

    Glad to hear that you have resolved the issue.

     

    Best Regards,

    Vincent Hu

     

    Monday, August 17, 2009 6:22 AM
  • Have you had any other issues with this?  I'm having the same problem you had, however the issues are very intermittent in comparison - it happens once every 2-6 weeks - so it's rather difficult to figure out which VM (if indeed it's a VM) that's causing the trouble.

    Was the setting you changed located in the Shadow Copies tab of the Drive's properties window?
    Tuesday, August 25, 2009 5:21 AM
  • Hi - I have the exact same issue (see event below) - came out of the blue sky since last weeks backup.

    Can you remember how you solved your other VSS error?

    The RaidPort number mentioned in the eventlog changes from backup to backup (but remains the same until I reboot and start the backup job again).

     

    The description for Event ID 129 from source storvsc cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

     

    If the event originated on another computer, the display information had to be saved with the event.

     

    The following information was included with the event:

     

    \Device\RaidPort2

    Wednesday, October 19, 2011 2:05 PM
  • Narrowed it down to a single VM just like Sbradbury. When that guest is selected in the backup job all VMs become unresponsive and the Hyper-V hosts starts logging the above mentioned event ID 129 every 30 seconds. The backup job freezes with status "snapshot processing".

    Will investigate any possible VSS issues on that specific VM and update you with the results


    • Edited by SteenKirkby Thursday, October 20, 2011 5:46 AM
    Thursday, October 20, 2011 5:42 AM
  • Tried the following - still no luck (and System Writer was not listed before on the VM):

    - Overwrote permissions as follows:

    Takeown /f %windir%\winsxs\filemaps\* /a
    icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
    icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
    icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)

    - Stopped and started Cryptographic service

    net stop cryptsvc
    net start cryptsvc

    - Verified “System Writer” showed up in list of writers using "vssadmin list writers"

    Thursday, October 20, 2011 6:22 AM
  • SteenKirkby,

    I know this post is over 2 years old, but do you remember what you did to fix this issue? We are running into the same problem you did with one particular 2008 VM that runs Exchange being backed up with Symantec BE 2012.  It is freezing our entire Hyper-V server and only a power cycle will cure it. "System Writer" shows up when running "vssadmin list writers" and there are no errors and the status is stable for every writer. Shadow Copies on every volume on the Hyper-V server are disabled as well.  

    Wednesday, June 26, 2013 8:00 PM
  • The backup job is also stuck on Snapshot Processing and Event ID 129 shows up when this is occurring.
    Wednesday, June 26, 2013 8:02 PM