none
Microsoft Hyper-V VSS Writer Issues

    Question

  • I'm running Server 2008 with Hyper-V.
     
    I have a simple script that uses diskshadow to utilize VSS to copy my virtual machines to a NAS device:

    echo Delete Shadows Volume C:\                             > Script.dsh  
    echo Set Context Persistent                               >> Script.dsh  
    echo Begin Backup                                         >> Script.dsh  
    echo Writer Verify {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} >> Script.dsh  
    echo Add Volume C:\ ALIAS MyShadow                        >> Script.dsh  
    echo Create                                               >> Script.dsh  
    echo End Backup                                           >> Script.dsh  
    echo Expose %%MyShadow%% X:                                 >> Script.dsh  
    echo Exec CopyVM.CMD                                      >> Script.dsh  
    echo Unexpose X:                                          >> Script.dsh  
    echo Exit                                                 >> Script.dsh  
    echo MD "\\192.168.1.14\backups\MYSERVER\MONDAY\Volume{148e48a7-543a-11dd-b6ed-806e6f6e6963}\" > CopyVM.CMD  
    echo MD \\192.168.1.14\backups\MYSERVER\MONDAY\Volume{148e48a7-543a-11dd-b6ed-806e6f6e6963}\Virtual Machines\MYVIRTUALSERVER >> copyVM.CMD  
    echo COPY "X:\Virtual Machines\MYVIRTUALSERVER\MYVIRTUALSERVER.vhd"  "\\192.168.1.14\backups\MYSERVER\MONDAY\Volume{148e48a7-543a-11dd-b6ed-806e6f6e6963}\Virtual Machines\MYVIRTUALSERVER\MYVIRTUALSERVER.vhd"  >> CopyVM.Cmd  
    DiskShadow /s Script.dsh  
    Exit 

    When i tested this on the server, it worked fine. It created the shadow, exposed it, copied it to my NAS and finished.

    I added the job to scheduled tasks and set it to run in the evening. For some reason, it failed with the following errors in my event log:

    App: E 'Mon Aug 04 20:12:33 2008': VSS - " An unhandled exception was encountered while processing a VSS writer event callback.  The VSS writer infrastructure is in an unstable state.  The writer hosting  
    process must be restarted in order to resume VSS functionality.        Writer name:            Microsoft  
    Hyper-V VSS Writer    Writer id:              {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}    Writer instance:  
           {ffee4b9e-2efa-4e65-b1ba-f90a2766b0d9}    Process command line:   C:_Windows_system32_vmms.exe  
       Process ID:             1788    Writer operation:       1013    Writer state:           4    Exception  
    code:         0xe06d7363    Exception location:     02    "  
     
    Sys: E 'Mon Aug 04 20:12:28 2008': PlugPlayManager - " The device 'Msft Virtual Disk SCSI Disk Device'  
    (SCSI_Disk_Ven_Msft_Prod_Virtual_Disk_2_9a9c854_0_000000) disappeared from the system without first being prepared for removal.  "  
    Sys: E 'Mon Aug 04 20:12:28 2008': PlugPlayManager - " The device 'Storage miniport driver' (VMBUS_{9e7cea05-51c7-4d9f-815f-9c19b87c62ce}_1_3189fc23_0_{9e7cea05-51c7-4d9f-815f-9c19b87c62ce})  
    disappeared from the system without first being prepared for removal.  "  
    Sys: E 'Mon Aug 04 20:12:24 2008': volsnap - " The shadow copies of volume __?_Volume{e4435086-5ff7-11dd-a59d-00188b426d49}  
    were aborted because the shadow copy storage volume was not present.  "   
     

    I did some googling, but nothing specific comes up for any of these errors on Server 2008 or Hyper-v.

    I did go into diskshadow and noticed that my Writer for Hyper-V did indeed dissapear (as indicated by the error message). I restarted Hyper-V Virtual Machine management and was able to manually fire off the backup with no issue.

    Anyone have any thoughts on what happened during my backups? I'm trying to stabilize the process until another backup solution is available that can do this process for me.

    Thanks!

    Dan
    Tuesday, August 5, 2008 3:07 PM

Answers

  •  

    Hello,

     

    Do you mean that, after the failure, you see the error "The VSS writer infrastructure is in an unstable state" in the Event Viewer and then the Hyper-V writer doesn't show up in the output of "vssadmin list writers"?

     

    Well, this seems a known issue and is being investigated. We have seen this error if some broken symbolic links presented due to unplanned failure.

     

    The hotfix is in work and some tests are still needed. For now, I am sorry to say that there is no workaround for it and I am sorry if this brings any inconvenience.

     

    Thank you for your understanding.

     

    Best regards,

    Chang Yin
    • Marked as answer by Chang Yin Wednesday, August 13, 2008 1:50 AM
    Thursday, August 7, 2008 11:32 AM

All replies

  •  

    Hello,

     

    Do you mean that, after the failure, you see the error "The VSS writer infrastructure is in an unstable state" in the Event Viewer and then the Hyper-V writer doesn't show up in the output of "vssadmin list writers"?

     

    Well, this seems a known issue and is being investigated. We have seen this error if some broken symbolic links presented due to unplanned failure.

     

    The hotfix is in work and some tests are still needed. For now, I am sorry to say that there is no workaround for it and I am sorry if this brings any inconvenience.

     

    Thank you for your understanding.

     

    Best regards,

    Chang Yin
    • Marked as answer by Chang Yin Wednesday, August 13, 2008 1:50 AM
    Thursday, August 7, 2008 11:32 AM
  • Ok as long as I know it is nothing I am doing wrong. Is one of the symptoms that the Hyper-V writer more or less dissapears permanently from the list of writers? At one point I was able to restart the hyper-v virtual machine management and have it re-appear, but now I can't get it back when I list writers even after a reboot.
    Thursday, August 7, 2008 2:18 PM
  • Chang,
    I recently loaded Server 2008 with Hyper-V and am encountering similar issues.  Two questions....

    1)  Is the hotfix you mentioned available yet?
    2)  Could these errors during Windows Server backups lead to system instablity and hangs?

    I'm asking question #2 since everytime I run a backup with Windows Server Backup the system either hangs (no mouse, no keyboard, no nothing.... requires a hard reboot) during the backup or within a couple of hours of completion.

    Thanks,
    Todd
    Tuesday, August 26, 2008 10:45 PM
  • Hi,
    i have the same here since months. When creating a backup, the Hyper-V Host hangs complete. The only way is to reboot the Server.
    In the eventlog there are mostly no entries, but last time the server hangs, i found eventlog-entries which says that the SCSI device is now longer available ?!
    Does anybody know if the Update is now available?

    Best regards,
    Andreas Luebbers
    Saturday, September 13, 2008 8:23 AM
  • I'm  having following in my Server 2008 event log;

    The device 'Storage miniport driver' (VMBUS\{2033c383-0eb6-4314-831c-5c085a738037}\1&3189fc23&0&{2033c383-0eb6-4314-831c-5c085a738037}) disappeared from the system without first being prepared for removal.

    - which i believe leads to or is caused by this;

    Backup started at '9/30/2008 9:30:29 AM' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348001'. Please rerun backup once issue is resolved.

    - so backup fails ... If I shut down virtual machines the backup runs just fine; so VSS has a problem accessing hypervisor while virtual machines is loaded. This just started about 1-2 weeks ago; I never had this problem before so perhaps some update in either Server 2008 or inside any if the virtual machines could cause this; anyway some change caused this and a solution would be nice.

    in the mean time i suggest shutting down virtual machines then backup; this seems to work ok.

    //Uli


    Helping people with computers
    Tuesday, September 30, 2008 6:52 PM
  • Did you guys get this resolved yet?

    I am also now having this same problem.

    have any of you opened a ticket with PSS yet?

    I may if you havent

    omar
    Monday, October 6, 2008 7:05 PM
  •  Andreas

    Has anybody ever come up with a solution for this?  I have been having the same problem with the host hanging during the VSS process when backing up a Hyper-V host.  We now have 5 hosts and I'm in major need of a solution.  I really don't want to have to possibly jump ship to VMWare.
    Friday, October 10, 2008 12:36 PM
  • I opened a case with PSS to an issue similar to this but I am not using DiskShadow directly- i am using Windows Server Backup.

    Either way I do believe these issues are related- I should have some answers in the next few days once they (PSS) confirm their suspicions

    hang tight- lets not give up on hyper-v just yet- they almost have it right(almost)

    Chang- are the registry keys to make wsb and hyper-v interact fully supported by MS now since the release of the vss- hyper-v update released in Early Sept?

    Examples:
    http://www.windowsvistaplace.com/enabling-vss-for-hyper-v-backups-with-windows-server-backup/virtualization

    http://blogs.technet.com/askcore/archive/2008/08/20/how-to-enable-windows-server-backup-support-for-the-hyper-v-vss-writer.aspx
    Friday, October 10, 2008 6:58 PM
  • Omar Droubi said:

    I opened a case with PSS to an issue similar to this but I am not using DiskShadow directly- i am using Windows Server Backup.

    Either way I do believe these issues are related- I should have some answers in the next few days once they (PSS) confirm their suspicions

    hang tight- lets not give up on hyper-v just yet- they almost have it right(almost)

    Chang- are the registry keys to make wsb and hyper-v interact fully supported by MS now since the release of the vss- hyper-v update released in Early Sept?

    Examples:
    http://www.windowsvistaplace.com/enabling-vss-for-hyper-v-backups-with-windows-server-backup/virtualization

    http://blogs.technet.com/askcore/archive/2008/08/20/how-to-enable-windows-server-backup-support-for-the-hyper-v-vss-writer.aspx



    I hope they do have a hotfix soon, Omar.  Your are exactly right.  They've got it very very close.  The VSS service is great.  We have been using it to backup our virtual server hosts since it showed up in Virtual Server 2005 SP1.  We've had pretty good success with just backing up the virtual host when using Virtual Server 2005.  And I really hoped it would be even better with 2008 and Hyper-V.  But considering it hangs the host and requires a power-off to restore the host, it looks like right now it's worse than Virtual Server 2005.  The vendor of the backup software that we use has also duplicated the problem.  And with other folks here experiencing major failures with VSS and Hyper-V, I know it isn't just me.  I think the problem is with the Hyper-V writer, as I have not experienced this same failure when running Virtual Server 2005 SP1 on Windows Server 2008.  We've got 2 servers running Vitual Server 2005 on Server 2008, because the hardware didn't support the required virtualization that Hyper-V requires.  Speaking of Windows Server Backup, I'm glad you mentioned that you were using it, Omar.  I had not tried using Windows Server Backup with the Hyper-V writer.  Sounds like it's pretty much the Hyper-V writer.  Please do let us know your findings, Omar.  I have really been struggling with this.  You're going to have much better success with your case since you're using their backup solution.  You don't get much help when you're using a third-party backup solution.  They always want to point the finger at the third-party.
    • Edited by CiscoNut Saturday, October 11, 2008 6:47 PM
    Saturday, October 11, 2008 6:30 PM
  • - well Hypervisor with Windows Backup using VSS has worked for 6 months ok for me; it backed up ok onto a USB drive every night, then for some reason it stopped working. Backup will still work ok if I save all the virtual clients; and that also saves the host from crashing.

    When I read thread http://social.technet.microsoft.com/forums/en-US/winserverfiles/thread/d5daada2-b78b-4935-bd9e-80726c26c4a8/ I came to remember that I did though update both BIOS and drivers for both RAID controllers = so for me it seems that driver changes has caused this.

    So the problem must be in 'Microsoft Hyper-V VSS Writer' ....    ... maybe I will revert to prior RAID drivers.




    Helping people with computers
    Monday, October 13, 2008 11:28 AM
  • Maui Tech Guru said:

    - well Hypervisor with Windows Backup using VSS has worked for 6 months ok for me; it backed up ok onto a USB drive every night, then for some reason it stopped working. Backup will still work ok if I save all the virtual clients; and that also saves the host from crashing.

    When I read thread http://social.technet.microsoft.com/forums/en-US/winserverfiles/thread/d5daada2-b78b-4935-bd9e-80726c26c4a8/ I came to remember that I did though update both BIOS and drivers for both RAID controllers = so for me it seems that driver changes has caused this.

    So the problem must be in 'Microsoft Hyper-V VSS Writer' ....    ... maybe I will revert to prior RAID drivers.




    Helping people with computers



    In my case, the Hyper-V VSS writer has consistantly failed since we started working with Hyper-V.  It occurred before and after BIOS updates to the hardware.  Sometimes the backup completes, but most times it doesn't.  That is, if the virtual clients are running, rather than being saved.  Having to save the clients is not always a good option...especially for production servers.  I need the clients to remain up during the snap.

    For those of you having problems, here's how you can watch what's going on when the failure occurs.  On the host server, bring up Disk Administrator in Server Manager and make sure you can watch the list of mounted volumes.  Also, bring up the Hyper-V Manager and select the Hyper-V server host that you will be backing up.  Then, launch your backup.  Start by watching the Hyper-V Manager.  For the running guests you should soon see a message come up in the 'Operations' column showing that the VSS snap is occurring.  Once the snaps of all running guests have completed, flip over to Disk Administrator and watch the list of mounted volumes.  You should soon see the snap volumes being mounted and they will appear on the mounted volume list.  If the backup is going to complete successfully, all snaps will be mounted and then successfully dismounted.  But, when it fails, 1 or more of the snaps will hang in the mounted state.  At that point, at least for me, the host server begins to fall apart and the only way to recover is a power-off on the host.  It's not a pretty sight......
    • Edited by CiscoNut Monday, October 13, 2008 5:33 PM
    Monday, October 13, 2008 12:57 PM
  • Hi,

    I'm getting the same error when DPM 2007 tries to backup the system state of an Windows 2008 Server.

    Backup started at '14/10/2008 19:04:17' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348001'. Please rerun backup once issue is resolved.

    Any ideas?

    Thank you,

    Joao Paulo Sevegnani.


    Joao Paulo Sevegnani
    Tuesday, October 14, 2008 8:25 PM
  • Please make sure that you have updated the DPM server and the DPM agents on the client with the DPM roll up patch- or are you using DPM 2007 Sp1 beta??

    here is the roll up:

    x86

    http://www.microsoft.com/downloads/details.aspx?FamilyID=e9e1fe35-b175-40a8-8378-2f306ccc9e28&DisplayLang=en 

    or

    x64
    http://www.microsoft.com/downloads/details.aspx?FamilyID=ad5cd1a2-9b87-4a2c-90a2-9dbaf1024310&DisplayLang=en

    DPM should not have an issue backing up VM Guests if the agent is installed- DPM does not necessarily interact with the Hyper-v VSS writer-until DPM 2007 SP1- when you are backing up the parent partition.

    omar
    Tuesday, October 14, 2008 8:36 PM
  • Hello,
     
    Yes, I've installed the feature pack for DPM 2007 and updated the agents, but the problem persists.

    Where can I get the DPM 2007 SP1 BETA? Can install on a production server? Can I uninstall when the SP1 going to RTM?

    Thanks,

    Joao Paulo Sevegnani.
    Joao Paulo Sevegnani
    Tuesday, October 14, 2008 8:44 PM
  • Sorry I do not know how to get the DPM Sp1 beta- if it even exists in that state yet. I checked the connect directory and I do not see it there-but someone may have it.

    Wednesday, October 15, 2008 4:09 PM
  • Trying to collect some data....
    I am guessing that for those of you seeing the hang you see the following event logs in the parent partition. Am I correct?
     

    Maui Tech Guru said:

    I'm  having following in my Server 2008 event log;

    The device 'Storage miniport driver' (VMBUS\{2033c383-0eb6-4314-831c-5c085a738037}\1&3189fc23&0&{2033c383-0eb6-4314-831c-5c085a738037}) disappeared from the system without first being prepared for removal.

    - which i believe leads to or is caused by this;

    Backup started at '9/30/2008 9:30:29 AM' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348001'. Please rerun backup once issue is resolved.

    - so backup fails ... If I shut down virtual machines the backup runs just fine; so VSS has a problem accessing hypervisor while virtual machines is loaded. This just started about 1-2 weeks ago; I never had this problem before so perhaps some update in either Server 2008 or inside any if the virtual machines could cause this; anyway some change caused this and a solution would be nice.

    in the mean time i suggest shutting down virtual machines then backup; this seems to work ok.

    //Uli


    Helping people with computers



    Thanks,
    Soumya
    This posting is provided "AS IS" with no warranties, and confers no rights
    Friday, October 17, 2008 2:44 AM
  • I'm beginning to move some servers to virtual machines within a Hyper-V environment and have just begun to experience this and some related issues.

    Since the server was setup (about a week ago) Windows Server Backup has run a scheduled backup of the host and running guests.  As expected the Hyper-V console showed Win 2003 and 2008 guests performing a volume shadow copy and Linux hosts entering save mode.

    Yesterday ISA 2006 was installed as a guest and schedules and manual backups all began to fail.  Event Logs show errors such as:

    The device 'Storage miniport driver' (VMBUS\{d79db2f0-6a77-4164-b6e5-fe831c09f66e}\1&3189fc23&0&{d79db2f0-6a77-4164-b6e5-fe831c09f66e}) disappeared from the system without first being prepared for removal.

    The device 'Msft Virtual Disk SCSI Disk Device' (SCSI\Disk&Ven_Msft&Prod_Virtual_Disk\2&21c56301&0&000000) disappeared from the system without first being prepared for removal.

    The application log also shows:

    The COM+ Event System failed to fire the RequestWriterInfo method on subscription {5F150B1B-BA02-4EFF-AD53-85924FE10636}-{00000000-0000-0000-0000-000000000000}-{00000000-0000-0000-0000-000000000000}. The subscriber returned HRESULT 80042318.

    My suspicion is the MSDE 2005 installed with ISA is somehow interfering with the shadow copy.

    This morning I applied windows updates and restarted (got "The Hyper-V Virtual Machine Management service did not shut down properly after receiving a preshutdown control." in the event log) and attempted to perform a manual backup multiple times with the same errors.

    I have made the registry addition detailed here, removed and recreated the same schedule. Now the backup appears to be working although there don't seem to be any volume snaps or saving of Linux hosts.  Is this normal?

    On a final, and somewhat related note, these backups destinations are a couple of 1TB usb drives.  I can write to these at over 50MB/sec using normal file copy but Windows Server Backup doesn't ever get above 5 MB/sec.  Because of this a full backup is taking more than 24 HOURS.  Again, is this normal?

    Thanks

    John

    Friday, October 17, 2008 11:39 AM
  • Follow up:

    restarting various services (Block level backup, vss) and rebooting the server has backup working for now, but still at <5 MB/sec.

    The Hyper-V VSS writer had disappeared and it looked like backups were (occasionally) working without any volume shadow copies.

    If it's of any help below is a selection of error messages from various event logs.  Until 2.00am this morning none of these had ever appeared.

    KB956697 not applicable "This update does not apply to your system", there are no orphans.

    For reference, server is Dell PE T606; dual quad cores; 16GB RAM, PERC 6/i in RAID5

    Thanks

    John




    Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine '%1'. (Virtual machine ID %2)

    Failed to create the backup of virtual machine '[removed]'. (Virtual machine ID CCF1C842-566A-446A-96CF-22FDE3744205)

    Cannot find the specified snapshot. (Snapshot ID B09DFE8A-2417-4630-89B9-10424ED2C1FA)

    The COM+ Event System failed to fire the RequestWriterInfo method on subscription {1DFF5AB8-C507-4E92-8036-3D3610528949}-{00000000-0000-0000-0000-000000000000}-{00000000-0000-0000-0000-000000000000}.  The subscriber returned HRESULT 80042318.

    Backup started at '17/10/2008 12:14:50' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348129'. Please rerun backup once issue is resolved.

    Volume Shadow Copy Service error: Writer Microsoft Hyper-V VSS Writer did not respond to a GatherWriterStatus call.

    Backup started at '{5c8b8e5c-158f-40fa-a248-8f58af073bed}' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348001'.

    An unhandled exception was encountered while processing a VSS writer event callback.  The VSS writer infrastructure is in an unstable state. The writer hosting process must be restarted in order to resume VSS functionality.
     
     Writer name:            Microsoft Hyper-V VSS Writer
     Writer id:              {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
     Writer instance:        {f812970f-205b-44d4-8f60-06f4e0ca4a8c}
     Process command line:   C:\Windows\system32\vmms.exe
     Process ID:             1340
     Writer operation:       1013
     Writer state:           4
     Exception code:         0xe06d7363
     Exception location:     02

    After an earlier restart vssadmin showed:

    vssadmin list writers hyper-v vss writer; state: [8] failed, last error: not responding

    and event log showed:

    The Hyper-V Virtual Machine Management service did not shut down properly after receiving a preshutdown control.





    Friday, October 17, 2008 2:41 PM
  • Soumya Das Bhaumik [MSFT] said:

    Trying to collect some data....
    I am guessing that for those of you seeing the hang you see the following event logs in the parent partition. Am I correct?
     


    Maui Tech Guru said:

    I'm  having following in my Server 2008 event log;

    The device 'Storage miniport driver' (VMBUS\{2033c383-0eb6-4314-831c-5c085a738037}\1&3189fc23&0&{2033c383-0eb6-4314-831c-5c085a738037}) disappeared from the system without first being prepared for removal.

    - which i believe leads to or is caused by this;

    Backup started at '9/30/2008 9:30:29 AM' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348001'. Please rerun backup once issue is resolved.

    - so backup fails ... If I shut down virtual machines the backup runs just fine; so VSS has a problem accessing hypervisor while virtual machines is loaded. This just started about 1-2 weeks ago; I never had this problem before so perhaps some update in either Server 2008 or inside any if the virtual machines could cause this; anyway some change caused this and a solution would be nice.

    in the mean time i suggest shutting down virtual machines then backup; this seems to work ok.

    //Uli


    Helping people with computers



    Thanks,
    Soumya
    This posting is provided "AS IS" with no warranties, and confers no rights



    I actually get these 2 events around the time of the hang....

    The device 'Msft Virtual Disk SCSI Disk Device' (SCSI\Disk&Ven_Msft&Prod_Virtual_Disk\2&1cd962fa&0&000000) disappeared from the system without first being prepared for removal.

    The device 'Storage miniport driver' (VMBUS\{06c36f9e-24d0-435e-9040-6ef3a51b18a5}\1&3189fc23&0&{06c36f9e-24d0-435e-9040-6ef3a51b18a5}) disappeared from the system without first being prepared for removal.

    Friday, October 17, 2008 8:36 PM
  • - Soumya; I'm having all these error occur if the guests are running or saved; if I shut down all guest then there is no problem with backup. At some point Hyper-V-VMMS didn't show registered as a VSS Writer; its back now though but still backup won't work if a single guest is on; this is what happens;

    getting event ID 10103 'Failed to revert to VSS snapshot on the virtual hard disk' after 8 minutes of backup,

    then event 14 'The shadow copies of volume \\?...6a0-11dd-b9d9-001a4d5ef8d5} were aborted because of an IO failure on volume \\?...6a0-11dd-b9d9-001a4d5ef8d5} about 30 seconds later.

    then lots of events 57 'The system failed to flush data to the transaction log. Corruption may occur.' within a second,

    then event 1000 'Faulting application vmms.exe, version 6.0.6001.18125, time stamp 0x48ab767f, faulting module vmms.exe, version 6.0.6001.18125, time stamp 0x48ab767f, exception code 0xc0000005, fault offset 0x000000000016f267, process id 0x8e8, application start time 0x01c93179917009cb.'

    then some more event 57,

    then event 1 'The Storage VSP failed to flush a drive mounted in loopback mode.'

    more 57,

    then event 12 'The device 'Storage miniport driver' (VMBUS\{2f52270d-5543-4699-8e21-e96c7b5ef398}\1&3189fc23&1&{2f52270d-5543-4699-8e21-e96c7b5ef398}) disappeared from the system without first being prepared for removal.'

    more 57 + 12,

    then finally event 12305 'Volume Shadow Copy Service error: Volume/disk not connected or not found. Error context: CreateFileW(\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy46,0xc0000000,0x00000003,...).'

    ----
    I have tried to rip Hyper-V role out, then backup in, create new virtual machine from existing drives and still got problem. So VSS starts and then forgets to remove its temp volumes the system overloads or cant find room to create next temp volume ...

    hope a solutions is to be found soon. The backup application worked just fine for about 8 months so some change crashed it.... can i re-install VSS services?


    Helping people with computers
    Tuesday, October 21, 2008 11:08 AM
  • Hello All,

    After much communication with PSS they have acknowledged that the hyper-v vss writer issues is due to a known Windows Bug

    This bug was first discovered in Windows 7 and has now been reported in Windows Server 2008.

    Now the good news.

    The fix for this bug will be incorporated in Windows Server 2008 Service Pack 2 (ETA currently unknown)

    If they get a fix out sooner I will try to share as best I can- if they happen to released it as a separate hotfix and they notify me about it- since I am now on the list of affected clients/partners who have open cases.

    For the mean time I have implemented the work around of simply disabling the VSS backup integration on each of my VMs. This has proved to work well enough for me since I backup at the physical server level using Windows backup.

    My guest machines go into saved state and start back up with out issue- their downtime is only about 2 to 4 minutes total per physical host and me and my client find this acceptable as the backup runs off hours.

    (I am running Win2k8 VMs only-plus a single W2k3 VM)- within each VM I have assigned its own dedicated disk for backup using Windows Server backup (NTBACKUP on W2k3) in case I need to restore at the VM level. and if the parent partition goes corrupt I can recover the entire server using the Windows backup I run to external USB drives.

    Poor mans backup- but in this economy- for this particular client- 5 physical servers-plus 10 hyper-V guests- this solution works very well. (backup to external USB drives that are swapped a few times per week.

    Does anyone know of issues or unsupported scenarios where putting a machine in a saved state is risky or not recommended?

    Thanks,

    Omar



    • Proposed as answer by Omar Droubi Tuesday, October 21, 2008 3:46 PM
    Tuesday, October 21, 2008 3:46 PM
  • Hi Omar,

    Your work around that you described sounds perfect for my scenario. would you mind sharing the script you use to put all of your Hyper-V VM's into a saved and resumed state?

    Thanks in advance,
    -James
    Tuesday, October 21, 2008 7:58 PM
  • Omar Droubi said:

    Hello All,

    After much communication with PSS they have acknowledged that the hyper-v vss writer issues is due to a known Windows Bug

    This bug was first discovered in Windows 7 and has now been reported in Windows Server 2008.

    Now the good news.

    The fix for this bug will be incorporated in Windows Server 2008 Service Pack 2 (ETA currently unknown)

    If they get a fix out sooner I will try to share as best I can- if they happen to released it as a separate hotfix and they notify me about it- since I am now on the list of affected clients/partners who have open cases.

    For the mean time I have implemented the work around of simply disabling the VSS backup integration on each of my VMs. This has proved to work well enough for me since I backup at the physical server level using Windows backup.

    My guest machines go into saved state and start back up with out issue- their downtime is only about 2 to 4 minutes total per physical host and me and my client find this acceptable as the backup runs off hours.

    (I am running Win2k8 VMs only-plus a single W2k3 VM)- within each VM I have assigned its own dedicated disk for backup using Windows Server backup (NTBACKUP on W2k3) in case I need to restore at the VM level. and if the parent partition goes corrupt I can recover the entire server using the Windows backup I run to external USB drives.

    Poor mans backup- but in this economy- for this particular client- 5 physical servers-plus 10 hyper-V guests- this solution works very well. (backup to external USB drives that are swapped a few times per week.

    Does anyone know of issues or unsupported scenarios where putting a machine in a saved state is risky or not recommended?

    Thanks,

    Omar





    Omar, I can't thank you enough for sharing this.  We have really struggled with this and it's good to know that a fix will be coming at some point.  We too have also chosen to backup from the physical host server, rather than backup each individual virtual guest.  I have also applied the band-aid of turning off the VSS integration for each virtual guest and have not exprienced any further host hang-ups.   Having a production server go into a saved state during the backup window might be a problem for some servers, but I'm hoping it will suffice until the fix arrives.

    Thanks again,

    Cisco
    Friday, October 24, 2008 6:27 PM
  • Hi all, i'm experiencing problems with Hyper-V on the Backup/Restore part.

    the major trouble i have is that, in order :

    1 - Installed Windows Server Enterprise 2008 (SP1 included which is not a SP1 but let it go ...)
    2 - Installed Updates from Windows Update
    3 - Installed Drivers
    4 - Added Hyper-V role
    5 - Added WSB funcitonality
    6 - Added the register key to enable Hyper-V VSS writer in my host VSS service

    Now that i've done everything right, i try to backup my system with VMs runing.
    Theses VMs contains Integration Services with every options enabled in Hyper-V Manager.
    The backup with WSB appears to be fine, no errors appeared in the logs of the VMs nor the logs of the host except this one :
    -------------------------------------------------------------------------------
    Nom du journal :System
    Source :       PlugPlayManager
    Date :         23/10/2008 16:16:27
    ID de l'événement :12
    Catégorie de la tâche :Aucun
    Niveau :       Erreur
    Mots clés :    Classique
    Utilisateur :  N/A
    Ordinateur :   SRV2008ENT
    Description :
    Le périphérique 'Msft Virtual Disk SCSI Disk Device' (SCSI\Disk&Ven_Msft&Prod_Virtual_Disk\2&2ed7f6e0&0&000000) a disparu du système sans que sa suppression ait tout d'abord été préparée.
    XML de l’événement :
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="PlugPlayManager" />
        <EventID Qualifiers="49152">12</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2008-10-23T14:16:27.000Z" />
        <EventRecordID>20361</EventRecordID>
        <Channel>System</Channel>
        <Computer>SRV2008ENT</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Msft Virtual Disk SCSI Disk Device</Data>
        <Data>SCSI\Disk&amp;Ven_Msft&amp;Prod_Virtual_Disk\2&amp;2ed7f6e0&amp;0&amp;000000</Data>
        <Binary>00000000</Binary>
      </EventData>
    </Event>
    ------------------------------------------------------------------------------
    And this one
    ------------------------------------------------------------------------------
    Nom du journal :System
    Source :       PlugPlayManager
    Date :         23/10/2008 16:16:27
    ID de l'événement :12
    Catégorie de la tâche :Aucun
    Niveau :       Erreur
    Mots clés :    Classique
    Utilisateur :  N/A
    Ordinateur :   SRV2008ENT
    Description :
    Le périphérique 'Pilote de miniport de stockage' (VMBUS\{586aaece-0119-490a-9ea0-a3387d80b25e}\1&3189fc23&0&{586aaece-0119-490a-9ea0-a3387d80b25e}) a disparu du système sans que sa suppression ait tout d'abord été préparée.
    XML de l’événement :
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="PlugPlayManager" />
        <EventID Qualifiers="49152">12</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2008-10-23T14:16:27.000Z" />
        <EventRecordID>20360</EventRecordID>
        <Channel>System</Channel>
        <Computer>SRV2008ENT</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Pilote de miniport de stockage</Data>
        <Data>VMBUS\{586aaece-0119-490a-9ea0-a3387d80b25e}\1&amp;3189fc23&amp;0&amp;{586aaece-0119-490a-9ea0-a3387d80b25e}</Data>
        <Binary>00000000</Binary>
      </EventData>
    </Event>
    ------------------------------------------------------------

    Sorry the logs are in french langage as i'm french ...
    I contacted M$ for knowing if theses errors where part of a problem of backup that was not doing properly its tasks.
    But they sent me this KB article :

    http://support.microsoft.com/kb/958669/en-us

    Now, that the backup went fine until the end, i tried to launch a restore but ended with the fact that i couldn't select the "Application" Restore mode which was gray.
    Does somebody got the same problem of me, but i think i'm experiencing the same as yours and the only solution we've got is to wait for a fix that will eventually get released on 12/11/2008 and is called SP2 ...

    hope that helps to get the problem solutioned as fast as possible !
    Monday, October 27, 2008 10:40 AM
  • CiscoNut said:

    Omar Droubi said:

    Hello All,

    After much communication with PSS they have acknowledged that the hyper-v vss writer issues is due to a known Windows Bug

    This bug was first discovered in Windows 7 and has now been reported in Windows Server 2008.

    Now the good news.

    The fix for this bug will be incorporated in Windows Server 2008 Service Pack 2 (ETA currently unknown)

    If they get a fix out sooner I will try to share as best I can- if they happen to released it as a separate hotfix and they notify me about it- since I am now on the list of affected clients/partners who have open cases.

    For the mean time I have implemented the work around of simply disabling the VSS backup integration on each of my VMs. This has proved to work well enough for me since I backup at the physical server level using Windows backup.

    My guest machines go into saved state and start back up with out issue- their downtime is only about 2 to 4 minutes total per physical host and me and my client find this acceptable as the backup runs off hours.

    (I am running Win2k8 VMs only-plus a single W2k3 VM)- within each VM I have assigned its own dedicated disk for backup using Windows Server backup (NTBACKUP on W2k3) in case I need to restore at the VM level. and if the parent partition goes corrupt I can recover the entire server using the Windows backup I run to external USB drives.

    Poor mans backup- but in this economy- for this particular client- 5 physical servers-plus 10 hyper-V guests- this solution works very well. (backup to external USB drives that are swapped a few times per week.

    Does anyone know of issues or unsupported scenarios where putting a machine in a saved state is risky or not recommended?

    Thanks,

    Omar





    Omar, I can't thank you enough for sharing this.  We have really struggled with this and it's good to know that a fix will be coming at some point.  We too have also chosen to backup from the physical host server, rather than backup each individual virtual guest.  I have also applied the band-aid of turning off the VSS integration for each virtual guest and have not exprienced any further host hang-ups.   Having a production server go into a saved state during the backup window might be a problem for some servers, but I'm hoping it will suffice until the fix arrives.

    Thanks again,

    Cisco



    OK, Microsoft, please tell us you are working fast on this one.  This is a big one for my company and is keeping us from jumping feet first into a major move toward Hyper-V.  It's not too big of a deal for our test environments to be saved during a host backup, but it's not going to be possible for the production environments.  Please help us out.....
    Tuesday, November 11, 2008 8:47 PM
  • I have exact same problem and what i've found so far is this:


    Machine that vss writer works without problems has these updates:

    HotFixID
    KB941693
    KB947562
    KB947864
    KB948590
    KB949189
    KB950050
    KB950582
    KB951308
    KB951978
    KB956697

    and machines that don't have these installed:

    HotFixID
    KB938464
    KB941693
    KB947562
    KB947864
    KB948590
    KB949189
    KB950050
    KB950582
    KB950974
    KB951072
    KB951308
    KB951978
    KB952287
    KB953733
    KB953838
    KB954211
    KB954366
    KB955302
    KB956697
    KB956841
    KB957095
    KB958644


    I will start doing some testing with removing excess updates and hopefully i'll get to the bottom of this (i suspect the time when KB956697 is installed in second machine i forgot to install windowsserverbackup and installed KB956... before backup which might be the cause) waiting for sp2 or hotfix is not an option for me since i run hyper-v in production enviroment (yeah stupid me :) ). Luckly other nodes work fine so i tend to do a failover to good nodes before i start backup.
    Wednesday, November 12, 2008 7:14 AM
  • Sharac1 said:

    I have exact same problem and what i've found so far is this:


    Machine that vss writer works without problems has these updates:

    HotFixID
    KB941693
    KB947562
    KB947864
    KB948590
    KB949189
    KB950050
    KB950582
    KB951308
    KB951978
    KB956697

    and machines that don't have these installed:

    HotFixID
    KB938464
    KB941693
    KB947562
    KB947864
    KB948590
    KB949189
    KB950050
    KB950582
    KB950974
    KB951072
    KB951308
    KB951978
    KB952287
    KB953733
    KB953838
    KB954211
    KB954366
    KB955302
    KB956697
    KB956841
    KB957095
    KB958644


    I will start doing some testing with removing excess updates and hopefully i'll get to the bottom of this (i suspect the time when KB956697 is installed in second machine i forgot to install windowsserverbackup and installed KB956... before backup which might be the cause) waiting for sp2 or hotfix is not an option for me since i run hyper-v in production enviroment (yeah stupid me :) ). Luckly other nodes work fine so i tend to do a failover to good nodes before i start backup.



    Thanks for that info, Sharac1.  We only have about 6 Hyper-V hosts at this time.  I know that each one was upgraded as current as possible when they were setup.  Keep us updated on your findings........
    Wednesday, November 12, 2008 4:23 PM
  • I had one really bad experience; I saved my guests and ran backup with VSS still enabled; oops that led to guests that couldn't start. Restarted host but still couldn't start any saved guests, which now includes SBS R2 PDC so that was frustrating as users started complaining about emails & smartphones not syncing, thus internet not working; and I started scrambling for solutions... long story short; I ended up removing all guests, removing Hyper-V role, clean registry, reboot, install Hyper-V role, creating all new guests ... and then I could start my network again. Not a good start of day but good excercise ;-)

    Hyper-V is truly the magic of the millinium for Microsoft (as Vista wasn't) as one piece of hardware can require many Windows licenses; but if whole enviroment crash, hang or die then we are in limbo and cannot commit to the magic.

    We have disabled VSS snapshot support in all VM guest settings, thus disabled the corresponding service inside all guests; now we can let them run and backup nightly as planned; though guests pauses during part of the process its better than not having backup or crashing/hanging host/guests ..

    - Glad to hear SP2 is being brewed up.
    Helping people with computers
    Thursday, November 13, 2008 5:23 PM
  • Mr Maui Tech Guru,

    Are you really so fortunate to live in Maui?-lucky.


    Food for thought-- FYI

    I have seen this before and while it is basic my issue was that my parent partition Antivirus was not excluding all of the necessary folders from AV protection.

    My parent is the C drive and in there I exclude- c:\program data\Windows\Hyper-V

    All Guests are on drive E, F,G & H in a root folder called VirtualMachines- and this root folder is excluded-including all subfolders. Once I did this I was able to restart a saved machine that would not start.

    Furthermore- since the more we know about this the better- I have seen the xml file get corrupted and opening this file in IE quickly revealed additional unterminated entries in the XML file. A quick clean up of the file and the saved state is lost- but the machine then boots up fine and no recreation of VM guest configs or reinstallation of parent OS or hyper-V role.

    This is referenced in the following link as well and corruption in this file has caused the following issues for me:

    1. Go to save a machine- it fails and the machine is gone from the console

    2. Go to start a successfully saved machine- but it will not restart

    Here is the article:
    http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/d872c7a9-8755-4919-ba60-4bd42ebe0cc8
    Thursday, November 13, 2008 8:30 PM
  • Sharac1 said:

    I have exact same problem and what i've found so far is this:


    Machine that vss writer works without problems has these updates:

    HotFixID
    KB941693
    KB947562
    KB947864
    KB948590
    KB949189
    KB950050
    KB950582
    KB951308
    KB951978
    KB956697

    and machines that don't have these installed:

    HotFixID
    KB938464
    KB941693
    KB947562
    KB947864
    KB948590
    KB949189
    KB950050
    KB950582
    KB950974
    KB951072
    KB951308
    KB951978
    KB952287
    KB953733
    KB953838
    KB954211
    KB954366
    KB955302
    KB956697
    KB956841
    KB957095
    KB958644


    I will start doing some testing with removing excess updates and hopefully i'll get to the bottom of this (i suspect the time when KB956697 is installed in second machine i forgot to install windowsserverbackup and installed KB956... before backup which might be the cause) waiting for sp2 or hotfix is not an option for me since i run hyper-v in production enviroment (yeah stupid me :) ). Luckly other nodes work fine so i tend to do a failover to good nodes before i start backup.




    any ETA on this hotfix, as i would prefer to stay aligned to the MS SC suite of tools.any ETA on this hotfix, as i would prefer to stay aligned to the MS SC suite of tools.


    any temporary solutions? will disabling the VSS service on the host provide some stability?

    Friday, November 21, 2008 4:23 AM
  •  Hi All:

    skimming through this post, there seem to be two issues that are standing out:

    1. a failure in post snapshot after which the VSS writer goes into a bad state.
    The likely cause of this is that the VM has multiple volumes configured in the guest and the volume shadowstorage area for each of the volumes are being stored in a way that the Hyper-V VSS writer cannot work with. While the fix for this is being rolled into SP2, the workaround for this is to use vssadmin to explicitly configure the shadowstorage of each volume to be in the same volume. It is a onetime thing that you need to do. Here are the commands that you need to run:
    vssadmin Add ShadowStorage /For=C: /On=C:
    vssadmin Add ShadowStorage /For=D: /On=D:
    ....
    for each of the volumes in the VM

    2.
    a Stop 0x7E when performing a backup using WSB
    We are currently working on a fix for this.

    This posting is provided "AS IS" with no warranties, and confers no rights
    Friday, November 21, 2008 5:48 AM
  • Soumya Das Bhaumik [MSFT] said:

     Hi All:

    skimming through this post, there seem to be two issues that are standing out:

    1. a failure in post snapshot after which the VSS writer goes into a bad state.
    The likely cause of this is that the VM has multiple volumes configured in the guest and the volume shadowstorage area for each of the volumes are being stored in a way that the Hyper-V VSS writer cannot work with. While the fix for this is being rolled into SP2, the workaround for this is to use vssadmin to explicitly configure the shadowstorage of each volume to be in the same volume. It is a onetime thing that you need to do. Here are the commands that you need to run:
    vssadmin Add ShadowStorage /For=C: /On=C:
    vssadmin Add ShadowStorage /For=D: /On=D:
    ....
    for each of the volumes in the VM

    2.
    a Stop 0x7E when performing a backup using WSB
    We are currently working on a fix for this.


    This posting is provided "AS IS" with no warranties, and confers no rights



    Thanks for your reply, Soumya.  I think issue 1 is what has been affecting me....where during the backup of the Hyper-V VMs via the Hyper-V host, the host completely freezes.  In looking at your workaround, isn't that the default for shadowstorage?  In my case, the Hyper-V VMs do have 2 or 3 basic volumes...each one being a seperate VHD file.  When I look at the 'Shadow Copy' configuration tab on the drive properties within 'My Computer', the storage for each volume is defined on that volume and is set to the default size limit of a percentage of that drives total storage.  Isn't that the same as the 'vssadmin add shadowstorage' commands that you mentioned or am I missing something?

    Thanks again,

    Mike
    Friday, November 21, 2008 1:36 PM
  • I've successfully solved my problem with adding shadowstorage. So in fact windows updates have nothing to do with the problem. Problem (atleast in my case) was as follows:

    On machines that did not work i had no shadowstorage assigned for drive letter. Added some storage to cluster at later time assigned it letter g: and on some servers shadowstorage was assigned and sized and on one node shadowstorage was missing (how is that possible i have no clue) and consequently backups failed. I've manually assigned and sized shadowstorage vssadmin add shadow.... and now it seems everything works.

    Monday, November 24, 2008 9:50 AM
  • Sharac1 said:

    I've successfully solved my problem with adding shadowstorage. So in fact windows updates have nothing to do with the problem. Problem (atleast in my case) was as follows:

    On machines that did not work i had no shadowstorage assigned for drive letter. Added some storage to cluster at later time assigned it letter g: and on some servers shadowstorage was assigned and sized and on one node shadowstorage was missing (how is that possible i have no clue) and consequently backups failed. I've manually assigned and sized shadowstorage vssadmin add shadow.... and now it seems everything works.



    Hi Sharac1:

    Glad to hear that adding shadowstorage might be the fix.  What size did you set your shadowstorage to?  Did you use what looks like a default of 10% of the total size of the disk?  Also, did you only add storage to the VMs or did you also add it to the host?  I'm still trying to understand the difference between adding shadowstorage and the default.  If you look at my prior post, I had asked Soumya a question about adding shadowstorage.

    Thanks for your post,

    Mike
    Monday, November 24, 2008 2:22 PM
  •  Hi CiscoNut:

    If you do not explicitly set the shadowstorage associations, VSS makes the decision for you. By default, it does not necessarily set the shadowstorage to be in the same volume. It makes its decision based on which volume has the maximum available storage among other factors. As far as you observations are concerned, it does seem like the associations are indeed 1 to 1. Could you run the following command:

    vssadmin list shadowstorage

    and see if the mappings are the same or if nothing is displayed? if nothing is displayed, I would guess that VSS is still making the decisions for you at the time of backup. On the other hand, if the diff area associations are indeed 1 to 1, could you also check the provider that you are running on the host? are you using the system provider or a hardware provider that ships with the storage vendor's storage hardware that you are using?
    This posting is provided "AS IS" with no warranties, and confers no rights
    Tuesday, November 25, 2008 7:35 AM
  • I had no listings for drives add added after the initial install of node. So i manually created those and by default they had "unbounded" size which i reset to default 10% (6gig in my case).


    Tuesday, November 25, 2008 8:40 AM
  • Soumya Das Bhaumik [MSFT] said:

     Hi CiscoNut:

    If you do not explicitly set the shadowstorage associations, VSS makes the decision for you. By default, it does not necessarily set the shadowstorage to be in the same volume. It makes its decision based on which volume has the maximum available storage among other factors. As far as you observations are concerned, it does seem like the associations are indeed 1 to 1. Could you run the following command:

    vssadmin list shadowstorage

    and see if the mappings are the same or if nothing is displayed? if nothing is displayed, I would guess that VSS is still making the decisions for you at the time of backup. On the other hand, if the diff area associations are indeed 1 to 1, could you also check the provider that you are running on the host? are you using the system provider or a hardware provider that ships with the storage vendor's storage hardware that you are using?


    This posting is provided "AS IS" with no warranties, and confers no rights



    Thanks for your reply, Soumya...

    I think maybe I'm finally starting to understand the VSS environment a little better.  I ran the 'vssadmin list shadowstorage' command on the VMs on 1 of my Hyper-V hosts.  Interestingly, some showed the mappings and some didn't.  On the VMs that didn't, I went to 'My Computer', right clicked one of the volumes, clicked the 'Shadow Copies' tab, and then checked the settings for each volume.  Each volume had the storage area set to be on the same volume, but then when I clicked the 'Details' button, no storage was defined.  When going through the same steps to check the storage on a VM that does show storage from the 'vssadmin list shadowstorage' command, the storage is shown after clicking the 'Details' button.  I have never manually defined shadowstorage for any of my VMs, but it appears that in some cases it has been created for me.  The VMs that have no storage defined are probably the ones that are killing the host.  I'm going to make sure each VM has shadowstorage defined and then enable snapshot backup in integration services on each VM and see if my Hyper-V host will complete a backup without freezing.

    That brings up a few questions....

    How would the shadowstorage have been defined for some VMs and not others?  Is that something that is inconsistent and is that what SP2 or a hotfix would fix?

    And what would Microsoft's recommended size or percentage of total size be for defining shadowstorage for the host and the VMs to allow online snap backups from a Hyper-V host.


    Thanks again, Soumya,

    Mike
    Tuesday, November 25, 2008 1:15 PM
  • Sharac1 said:

    I had no listings for drives add added after the initial install of node. So i manually created those and by default they had "unbounded" size which i reset to default 10% (6gig in my case).




    Thanks again, Sharac1.  Sounds like a solution has come together.....
    Tuesday, November 25, 2008 1:16 PM
  • Soumya Das Bhaumik [MSFT] said:

     Hi All:

    skimming through this post, there seem to be two issues that are standing out:

    1. <deleted>

    2.
    a Stop 0x7E when performing a backup using WSB
    We are currently working on a fix for this.


    This posting is provided "AS IS" with no warranties, and confers no rights



    Is there a workaround for #2?
    My server went down because of it during the backup. Not good when there is 12 VMs running on it.
    Tuesday, November 25, 2008 8:31 PM
  • I found the cause for my problem. One of my VMs uses a SCSI disk. I disabled the Backup integration service for that VM, and the Host server does not blue screena anymore during the backup.
    Thursday, November 27, 2008 4:15 PM
  • Soumya Das Bhaumik [MSFT] said:

     Hi CiscoNut:

    If you do not explicitly set the shadowstorage associations, VSS makes the decision for you. By default, it does not necessarily set the shadowstorage to be in the same volume. It makes its decision based on which volume has the maximum available storage among other factors. As far as you observations are concerned, it does seem like the associations are indeed 1 to 1. Could you run the following command:

    vssadmin list shadowstorage

    and see if the mappings are the same or if nothing is displayed? if nothing is displayed, I would guess that VSS is still making the decisions for you at the time of backup. On the other hand, if the diff area associations are indeed 1 to 1, could you also check the provider that you are running on the host? are you using the system provider or a hardware provider that ships with the storage vendor's storage hardware that you are using?


    This posting is provided "AS IS" with no warranties, and confers no rights



    Thanks again for your postings, Soumya.  It looks like you've nailed this one.  We currently have 6 Hyper-V hosts.  The VMs on all 6 of the hosts have had their 'Backup (volume snapshot)'  integration service disabled so that the VMs would go into a saved state and the host backup would not fail.  For 2 of the Hyper-V hosts I went back and checked the shadowstorage for the host, as well as each VM on the host.  I found several of the VMs did not have an explicit shadow volume association for each volume.  So, I made sure each volume on each VM had a mapping to shadowstorage that was for that volume, as well as being on that volume.    I set the limit for each to be the default 10% of volume size.  I also did the same for each volume on the Hyper-V hosts.  I then verified my settings by running

    vssadmin list shadowstorage

    on each host and VM.  I then enabled the 'Backup (volume snapshot)' integration service on each VM.  The snapshots have been working great and the backup has not failed on either of the 2 Hyper-V hosts that I fixed.  I'm now going to do the same thing for the other 4 Hyper-V hosts and VMs.

    I will keep everybody posted if anything changes, but it looks like the workaround is definitely working.

    Thanks again for all your help, Soumya, as well as everybody else that has participated in this thread.
    Thursday, December 4, 2008 3:23 PM
  • I've never fixed my problem but found an interim solution; turn off Hyper-V management service from within the Hyper-V console ... then backup VSS etc. all works just fine ....  you can't manage the virtual machines though, but hey I don't really do that daily anyway .... I do want to have good daily backup ......

    Guests all have all integration services enabled + host has management service disabled = Server 2008 backup just fine ....

    Helping people with computers in Maui Hawaii.
    Sunday, November 15, 2009 9:41 PM
  • Interesting work around.

     I still have one server that has this issue. strange thing is that all but one virtual guest can work with the backup guest services but one.

    One thing I learned through my last 1.5 years with hyper-v- If there are any Host server updates applied that are hyper-v or VSS related- always update the integration components on each guest machine and reboot the hosts and guests.

    This has fixed many of my issues- but the one stubborn guest just wont comply- so that host server has all guests set with no backup/vss integration in the guests and I get along with that. My plans were to just rebuild the host and or necessary guests at one point or another since they are all childred of the beta hyper-v to Rc1 to Rc2 to RTM hyper-v upgrades and I just think it may be a fight I cant win to continue to try and get this working.

    One very helpful page for me is the following-but I dont install all of the updates on all of my hyper-v hosts- just the ones that I think apply:
    http://technet.microsoft.com/en-us/library/dd430893(WS.10).aspx

    Omar
    Monday, November 16, 2009 3:40 PM
  • Update;

    I've just completed the upgrade to Server 2008 R2. everything wen't well, but guess what; backup still fails on Hyper-V.

    I'm still able to turn off Hyper-V management service and then backup ok. I'm somewhat happy about that, but still I have to keep Hyper-V management services stopped.

    I can agree that it's a good practice to update host and all guests components to 'cleanse' out any descrepencies but it does constitute a lot of work though and time is precious and I'd rather be making money ...


    Helping people with computers in Maui Hawaii.
    Tuesday, November 24, 2009 6:40 PM
  • Time to awaken this topic!

    Windows Server Standard 64BIT - Backup Drives using eSATA

    STILL I am having problems with VSS timing out. On many of our servers. MS why are you taking so long to resolve this issue????

    Im FED up of seeing this error

    ERROR - The shared restore point operation failed with error (0x81000101)
    The creation of a shadow copy has timed out. Try again this operation.
    

    I have changed the timeout as suggested on many sites from 10mins to 20mins and still the same issue... Only really work round is disabling Hyper V management Service.

    This in my eyes is NOT an acceptable workround! We pay for a product and expect it to work to its fullest!! NOT disable some services so we can kinda make it work.

    MS PLEASE GET YOUR FINGER OUT AND RESOLVE THIS ISSUE THATS BEEN GOING ON FOR YEARS 

    Monday, January 24, 2011 3:41 PM
  • I've been havnig this issue for over a year and I decided to upgrade to R2 last week.  Unfortuantely, this did not resolve my backup issue.  I get event id 521 with error 2155348001.  Sometimes the backup works sometimes it doesn't.
    Monday, January 24, 2011 6:38 PM