locked
Server Backup errors on Hyper-V R2 RRS feed

  • General discussion

  • Hello,

    I posted this issue in Hyper-V because the backup problems seems to be related to it, but did not get any feedback:

    We are using a Hyper-V R2 server. There are several Servers (2008 [R2]) on the host running SQL (express), Exchange, DC, etc...
    Hyper-V VSS service is registered (as described http://support.microsoft.com/kb/958662/en-us)

    When we are using the built-in Windows server backup (WSB), shortly after starting, two different errors are logged several times (with different SIDs and volume IDs as shown)

    Error1 (Source volsnap, Event ID 27)
    "The shadow copies of volume \\?\Volume{ba882ffb-e66e-11de-8f15-001517cf3032} were aborted during detection because a critical control file could not be opened."

    Error2 (Source FilterManager, Event ID 3)
    "Filter Manager failed to attach to volume '\Device\HarddiskVolume126'. This volume will be unavailable for filtering until a reboot. The final status was 0xc03a001c."

    These errors do not appear when I shut down the guests before starting the backup.

    The backup runs and finishes without claiming any error on the host (except the logged ones). Also there are no errors on the guest machines during the time frame of the backup. However, on one VM a couple of errors occured: ("VDS Basic Provider", Event ID 1, Unexpected failure. Error code: 490@01010004) but the did not reappear after a chkdsk operation.

    The volume IDs shown in the errors are NOT from local drives (I checked this) so they must be from the VHDs, but I do not know how to get the information which ID is assigned to which VHD.

    I am not sure if the backups are running normal and safe due to this error, so I need help on this to solve the issues.

    Thanks for any hint,
    Rainer

     

    Friday, April 2, 2010 7:41 AM

All replies

  • Hi,

    Online backups: The backup failure when the VMs are online, seems to be during the VSS shadow copy creation, due a failure in volsnap. After mounting the VHD using diskmgmt.msc, the mountvol.exe utility can be used to check the volume guid for the surfaced volumes.

    Offline backups: If the VMs are offline for the backup, the backups should succeed. But the from the backups only the .VHD files can be recovered, the VMs would not be restorable as an application (as described http://support.microsoft.com/kb/958662/en-us)

    Will try to contact the parter team to query for the investigation.

    Thanks,

    Abhinav


    This posting is provided "AS IS" with no warranties, and confers no rights
    Monday, April 5, 2010 5:13 AM
  • Hi,

    Online backups: The backup failure when the VMs are online, seems to be during the VSS shadow copy creation, due a failure in volsnap. After mounting the VHD using diskmgmt.msc, the mountvol.exe utility can be used to check the volume guid for the surfaced volumes.

    Offline backups: If the VMs are offline for the backup, the backups should succeed. But the from the backups only the .VHD files can be recovered, the VMs would not be restorable as an application (as described http://support.microsoft.com/kb/958662/en-us)

    Will try to contact the parter team to query for the investigation.

    Thanks,

    Abhinav


    This posting is provided "AS IS" with no warranties, and confers no rights
    Monday, April 5, 2010 5:13 AM
  • Hi,

    Online backups: The backup failure when the VMs are online, seems to be during the VSS shadow copy creation, due a failure in volsnap. After mounting the VHD using diskmgmt.msc, the mountvol.exe utility can be used to check the volume guid for the surfaced volumes.

    Offline backups: If the VMs are offline for the backup, the backups should succeed. But the from the backups only the .VHD files can be recovered, the VMs would not be restorable as an application.

    Will try to contact the parter team to query for the investigation.

    Thanks,

    Abhinav


    This posting is provided "AS IS" with no warranties, and confers no rights
    Monday, April 5, 2010 5:13 AM
  • Hello Abhinav,

    thanks for your reply and the hint to get the relevant drives. I will try that outm but need to do this at late ours, because the machines need to be shut down to attache the vhd in the host.

    However, now a third drive is affected ... and I have check all guest machines for VSS errors or warnings, but there are none.

    A further investigation from the partner team would be highly appreciated.

    Thanks,
    Rainer

    Tuesday, April 6, 2010 5:46 PM
  • Hi Rainer,

    Please mail me on abhim-nospam@microsoft.com (remove -nospam to get the actual id) with the thread id in the subject line. I need to pass on instructions for collecting the logs we need to investigate the issue. One query here, is the VM hosted on a clustered environment ?

    Thanks,

    Abhinav


    This posting is provided "AS IS" with no warranties, and confers no rights
    Wednesday, April 14, 2010 1:34 PM
  • Hi Rainer,

    Please mail me on abhim-nospam@microsoft.com (remove -nospam to get the actual id) with the thread id in the subject line. I need to pass on instructions for collecting the logs we need to investigate the issue. One query here, is the VM hosted on a clustered environment ?

    Thanks,

    Abhinav


    This posting is provided "AS IS" with no warranties, and confers no rights
    Wednesday, April 14, 2010 1:35 PM
  • Hello Abhinav, hello Rainer,

    I'm experiencing very similar problems with WSB on a Hyper-V Server 2008 R2 with Windows Server 2008 x64 VMs (SBS 2008 premium). The SBS 2008 server is being paused at the beginning of the backup, the W2k8 SQL server stays online but gives a couple of errors.

    So I would be very interested to know the status of this issue.
    Did you find the root cause?
    How severe is the problem? Will the revovery of the backups work?
    Is there a workarround to prevent the problems?

    Thanks in advance
      Klaus

    Monday, May 3, 2010 11:16 AM
  • Hello Klaus,

    the problem is still in investigation. The partner team currently examins the issue. So, there is no solution so far on this ...

    Your first problem (pausing) sounds a little bit like a integrations services issue. Have you double checked that the integration services are installed/updated correctly and the backup feature is selected in the VM settings.

    Bye,

    Rainer

    Monday, May 3, 2010 4:04 PM
  • Hi Rainer,

    thanks for the info!

    Yes, I double and triple checked the versions and they are all identical: the VM Bus driver version ist "6.1.7600.16385" on all VMs (2x W2k8 x64, 1x Win7 x86). The SBS server has been working normally without pausing at the backup start in the beginning but changed its behavior at a certain time. Now it is being paused without logging any errors while the others run continously but throwing a couple of errors when being backed up.

    Maybe it has something to do with these volsnap and filtermanager error messages...

    Thanks
      Klaus

    Monday, May 3, 2010 4:39 PM
  • Hi Rainer,Hello Abhinav

    I'm experiencing the same problems with WSB on a Hyper-V Server 2008 R2 with Windows Server 2008 x64 VMs .

    Error1 (Source volsnap, Event ID 27)

    The shadow copies of volume \\?\Volume{c19d2792-30e8-11df-ae2b-d8d3855c5d49} were aborted during detection because a critical control file could not be opened.

    Error2 (Source FilterManager, Event ID 3)

    Filter Manager failed to attach to volume '\Device\HarddiskVolume36'. This volume will be unavailable for filtering until a reboot. The final status was 0xc03a001c.

    have you manage to get a solution for this problem?

    BR

    Alberts Vilcans

     

     

    Monday, August 9, 2010 1:02 PM
  • Hi Alberts,

    unfortunatly not. The rror seems hard to be investigated by the Hyper-V team (last information 4 weeks ago). I work around the issue pausing relevant machines for 10 minutes when starting backup. Then the errors do not appear.

    Have you any solutions tried / founs other information. The error seems to occour only on very few machines.

    Bye

    Rainer

    Monday, August 9, 2010 1:12 PM
  • The error seems to occour only on very few machines.

    I've been having some variation of this problem since Sept 2009 , and I still don't have a fix.  The only work-around seems to be shutting down the VMs before using WSB.

    If anyone from MS is reading this, please don't write this off to an isolated, unreproducible, machine-specific anomaly.  There is a problem with WSB and Hyper-V, and a fix would be greatly appreciated.

    Monday, August 9, 2010 2:35 PM
  • Hi Rainer, Hi Doug
    What kind of hardvare do you use?

    I have HP ml350g6, one intel xeon cpu e5506 2,13 Ghz, 6 Gb memory, raid controler Smart array P410i,C
    drive 2x146 GB SAS hdd (mirror) for hyper -v and one VM (windows 7 64Bit),and another E drive 2x146 GB SAS hdd (mirror) for just VM (server 2008 64bit, DC and fileserver)

    I try to backup only E drive, where is one WM (server 2008 64bit dc,fileserver) this errors dont show up,  but when I try just C: drive where is Hyper v and one WM (windows 7 64bit) the errors show up

    have you tried to restore a backup with this kind of error?

    BR
    Alberts

    Tuesday, August 10, 2010 11:15 AM
  • Strebla84,

    The machine that's giving us troubles is a Dell T710, Xeon E5520 2.27 Ghz, 12 Gb ECC memory, LSI RAID controller (integrated), 2x1 Tb SAS HDD (RAID 1 mirror) formatted as single C: drive.

    Host Win2008 R2 -AD DC Global, DNS, Hyper-V running 4 VMs:

    (1) Win2003 R2 - Database/Source Control Server

    (2) Win2003 R2 - Product Build/Test Servers

    (1) Win2008 R2 - AD DC, AD CS, DHCP, DNS, AV Server, FMSO

    Have not tried a restore because, when it fails, there is no backup to restore from.

    Best regards, Doug

    Tuesday, August 10, 2010 3:07 PM
  • Hi  Doug,

    Tanks for information.
    For backup are you using usb external hard drive?

    br
    alberts

    Wednesday, August 11, 2010 8:33 AM
  • Hi,

    seem that some live is coming up here now.

    We are using Supermicro Server:

    Supermicro Mainboard X8DT3-F

    2x Xeon Nehalem 4-Core E5520

    12x 4096 MB ECC Registered DDR3 RAM 4 Rank ATP (1x 4096 MB)

    Controller 3Ware 9690SA with 7x 300 GB SAS with 7x 300 GB SAS Fujitsu 3,5" 15k

    We are using an iSCSI and external SATA drives for backup. No difference.

    From my point of view the issue is during creating the Volume Snapshot. The error are only occuring when the back starts and creating the VS, not during backup process.

    The error does not appear when pausing the VMs during snapshot

    Rainer

    Wednesday, August 11, 2010 10:17 AM
  • Yes - using external USB for backup target
    Wednesday, August 11, 2010 2:20 PM
  • Not sure if this plays into it, but try this:

    1. Start a WSB backup with your VMs running
    2. In Server Manager, go to Roles - Hyper-V and watch the list of VMs
    3. I'm seeing the Snapshots being created...and each VM returns a "Successful" message
    4. After all the VMs create their snapshots, there is a long period of inactivity...and then the backup fails
    Is everybody else getting the same thing?
    Wednesday, August 11, 2010 2:23 PM
  • Hi, Doug

    first three steps works the same, but the 4th step is different in my case.
    after Snapshots being created WSB is transferring data to usb external HDD, the backup doesn’t fail.

    But in event log I got 3x warning messages ( DISK ID 51 An error was detected on device \Device\Harddisk3\DR39 during a paging operation.)
    and 1x error message (volsnap ID 27 The shadow copies of volume \\?\Volume{c19d2792-30e8-11df-ae2b-d8d3855c5d49} were aborted during detection because a critical control file could not be opened.) I notice that  volume \\?\Volume{c19d2792-30e8-11df-ae2b-d8d3855c5d49} is external hdd. but I know that I exclude external hdd from backup.

    br 
    alberts

    Thursday, August 12, 2010 7:28 AM
  • Hi,

    I am seeing similar errors to both Doug Cutter and Alberts. I start a backup and it seems to be working and then just hangs. When I look at the event log, I see a number of volsnap events culminating with a "volsnap ID 27: The shadow copies of volume \\?\Volume{xxxxxx} were aborted during detection because a critical control file could not be opened". On some ocassions, I also get 3 off "disk id 51: An error was detected on device \Device\Harddisk4\DRXX during a paging operation."

    Having read your posts, I have a number of questions that I hope you guys can help with:

    1) I think that the event ID 51's are probably a symptom of the volsnap failure and do not indicate a disk failure. Would you agree?

    2) Each group of event ID 51 messages reports the same Harddisk4, but a different DRxx value (e.g. DR13, DR18, DR23). What is this?

    3) I thought that the backup was using VSS to snapshot a disc, but I have seen elsewhere

    (http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Q_24009118.html)

    that the resolution was reated to the removal of an orphaned snapshot, but they were talking about a virtual machine snapshot. Can anyone clarify for me the relation between a VSS snapshot and a VM snapshot.

    4) If the problem is related to having an "orphaned" VM snapshot, can anyone tell me how I find out if I have one/some?

    Thanks

    Alan Mason

     

    Sunday, October 3, 2010 6:21 PM
  • We have the same issue happening during the windows server backup startup (Win Srv 2K8 R2 Enterprise w/Hyper-V):

    Filter Manager failed to attach to volume '\Device\HarddiskVolume14'. This volume will be unavailable for filtering until a reboot. The final status was 0xc03a001c.

    The error does not seem to affect the backup or the system, the backups complete with the success status, the server is fully functional.

    The error appeared only after we included the entire partition in the backup. When the backup was doing only the system state this error was not present.

    The difference from other posters is that on the server there is no Hyper-V (no Virtual Machines).

    Thursday, October 14, 2010 3:21 PM
  • I am seeing similar results as described by strebla84 but I'm using Shadow Protect for backup of a Server 2008 R2 with Hyper-V role and 2 VMs (SBS 2008 and W2K3). I suspect this is some type of VSS issues related to Hyper-V and may not be something specific to WSB since I'm having similar problems using a 3rd party backup which also leverages VSS...

     

    This thread has been open a while and we all need a fix for this.

     

    -/\/\ark

    Friday, October 22, 2010 12:45 AM
  • Same problems here: Any solutions after nine months?

    Franz

    Wednesday, December 22, 2010 2:20 PM
  • Same problem here. 

    Server 2008 R2

      2 VMs

      WSB

      FilterManager and VolSnap errors

      Backups succeed

      System Hangs

    Please update with a fix.

    Mike

    Wednesday, January 5, 2011 2:23 PM
  • Another month gone and still no resolution.  I have this issue on different Hyper-V hosts, in different environments, using different backup software (Windows Server Backup and ShadowProtect).  The only common threads appear to be Win2008 R2, Hyper-V, and VSS.

    Regards,

    Darren

    Sunday, February 6, 2011 6:38 PM
  • Another month gone and still no resolution.  I have this issue on different Hyper-V hosts, in different environments, using different backup software (Windows Server Backup and ShadowProtect).  The only common threads appear to be Win2008 R2, Hyper-V, and VSS.

    Regards,

    Darren


    Same issue here guys. I use Backup Assist and it has been intermitttently coming up with these errors. Whenever this error comes up the backups fail annd on close inspection of the Files Successfully backed up, I find that out of 4 vhds that should be backed up, 1 has been skipped. This quite strangely is not depicted in the logs for "Skipped Files".

    This is not just happening with one customer but at 4 places where we have used the HYPER-V bare metal installation.

    Any help is appreciated.

    Tuesday, March 1, 2011 1:37 PM
  • [Solved]!

    We finally solved these issues with two Hyper-V R2 Hosts:

    1. Added Registry Key HKey_Local_Maschine\ SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization, New Reg_DWORD, Name "VssRevertTimeout", Value 360 (decimal). I'm not sure if this one is required. Have added this key first, didn't change anything, added the second key that is mentioned below and haven't removed this key.
    2. Added Registry Key "HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP"
      DWord : CreateTimeout
      Value decimal: 12000000    (2*10*60*1000 = 20 mins).
      It tooks more that two months until Microsoft found this Registry Key during our PSS call!!!

    From that point, we were able to successfully backup one Hyper-V host. The other doesn't work yet:

    Had to install Hotfix http://support.microsoft.com/kb/976461 in every Windows 2003 VM. Before, we got Shadow Copy Errors inside the VM's. After implementing this Hotfix in one VM, the error apperead in another 2003 VM.

    After that, we got error 2155347997, and Windows Backup was unable to save the system state. Had to install Hotfix http://support.microsoft.com/kb/2182466/en-us on the Hyper-V host. The Hotfix doesn't require a reboot, but we had to reboot until this error disappeared.

    After that, we were able to sucessfully run Backups with Windows Backup on our two 2008 R2 Hyper-V hosts.

     

     

     

    Monday, March 7, 2011 10:46 AM
  • Franz,

    Thank you for the detailed and well documented steps to resolving this problem.  We performed them on one host and one 2003 VM and the WBS Backup worked without shutting down the VM. 

    We have two more 2003 VMs on this host so we will update them today and report back with the results.

    When we ran KB 2182466 on the host, we got a message saying it didn't apply.  Perhaps MS has rolled this update out so our machine already has the update.

    Again, a hundred thank yous for sharing the solution to this frustrating and long-standing problem.

    Best regards, Doug

    Wednesday, March 16, 2011 12:38 PM
  • We have two more 2003 VMs on this host so we will update them today and report back with the results

     

    Hi, Doug

    result?

     

    Tuesday, April 12, 2011 8:49 AM
  • Some additional information:

    Windows Backup still failed sometimes. Microsoft couldn't find the reason in any tracelog. But we also had increasing performance issues on all our Hyper-V VM's. I have then found that the performance counter \Logical Disk(*)\Avg. sec/Write of our Hyper-V Host was critical, about 50ms. According the Microsoft document "Measuring Performance on Hyper-V", http://technet.microsoft.com/en-us/library/cc768535(BTS.10).aspx (very good!), this value should not be greater than 26ms.

    We had this high value despite there were nearly no write operations of all VM's. Further investigations shows that the cache of our RAID controller was configured for 100% Read and 0% Write. The reason is that HP sells US$ 12'000 servers with an integrated RAID controller P410, but WITHOUT a battery!!! (ML370 G6). So we had to set an option "enable write cache even if battery is degraded" in the array configuration utility ACU, and got then a new option in ACU where we could set the write cache to 75% and read to 25% (makes sense when having RAID 5). From this moment, \Logical Disk(*)\Avg. sec/Write felt to about 2ms, and the performance of all VM improved DRAMATICALLY! and from this moment on, we never had any failure in the Windows Backup again, now since two weeks.

    Conclusion: It seems that Windows Backup or the shadow copy processes don't work when the disk performance is not so good.

    Franz

     
    Thursday, April 14, 2011 7:04 AM
  • Sorry I didn't respond sooner...never got the email notification on this thread.

    It has worked almost flawlessly.  In the last month, only ONE backup has failed.  Once again, huge thanks to Franz for sharing the solution in such a detailed and professional manner!

    Wednesday, April 20, 2011 5:14 PM
  • thank you franz, having the same issue here on a single win2k8r2 blade for some reason.  the other blades all have the same OS, the same hardware, etc.  these registry keys (or one of them?) appears to have solved the issue.  my backup which hasn't run in 2 weeks is now at 14%

    1. Added Registry Key HKey_Local_Maschine\ SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization, New Reg_DWORD, Name "VssRevertTimeout", Value 360 (decimal). I'm not sure if this one is required. Have added this key first, didn't change anything, added the second key that is mentioned below and haven't removed this key.
    2. Added Registry Key "HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP"
      DWord : CreateTimeout
      Value decimal: 12000000    (2*10*60*1000 = 20 mins).
      It tooks more that two months until Microsoft found this Registry Key during our PSS call!!!
    Tuesday, October 25, 2011 4:48 PM
  • On a server 2008 r2 box i do not see the  "HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP" SPP folder is not there did you create it and then create the Dword?

    The first one wasn't a problem. 

    Thanks

    Tuesday, December 6, 2011 3:34 PM
  • On a server 2008 r2 box i do not see the  "HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP" SPP folder is not there did you create it and then create the Dword?

    The first one wasn't a problem. 

    Thanks


    Same question here?
    rgds Sven
    Wednesday, December 28, 2011 2:32 PM
  • Have checked another customer with a 2008 R2 Server (now with SP1) and enabled Hyper-V role and Windows Backup feature. This server is not affected by the problem described in this thread, but the registry SPP folder exists.

    Franz

    Tuesday, January 3, 2012 9:13 AM
  • I have a folder labeled SoftwareProtectionPlatform - is that the correct one?
    Thursday, January 5, 2012 11:07 PM
  • I have a folder labeled SoftwareProtectionPlatform - is that the correct one?


    I'm at the same point like You. I don't have this SPP key in my registry but I have "SoftwareProtectionPlatform" key.
    But I'm thinking this is'nt the needed one.
    So, at the moment I just added " HKey_Local_Maschine\ SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization, New Reg_DWORD, Name "VssRevertTimeout", Value 360 " but this seems to be not enough.

    Errors like below are still there:

    Filter Manager failed to attach to volume '\Device\HarddiskVolume222'. This volume will be unavailable for filtering until a reboot. The final status was 0xc03a001c.

    The shadow copies of volume \\?\Volume{178f0c13-e07c-11e0-8dff-bc305bdd34b1} were aborted during detection because a critical control file could not be opened.


    rgds Sven
    Saturday, January 7, 2012 11:30 AM
  • Anybody else made headway on this issue? 
    Thursday, February 2, 2012 2:47 PM
  • Is not having   HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP     on a 2008 R2 SP1 server with the hyper V role enabled a problem?

    We get two series of errors when backing up a single VHD one is towards the start and the other is close to the end. They are both two filter manager errors and then followed by paging errors. This does not seem to be related to bandwidth/switch fabric/ISCSI to our HP SAN from our HP DL380 G7 server. We are using GRT in back up exec and the disk type in the VM is basic. The VHD is dynamic expanding, but has more than enough room to expand. 

    Is there something that I am missing? Is this related to dynamic expanding VHD's taking a long time to let the VSS created a shadow copy? 

    I am starting to run out of ideas on what else to test to try and pin point this issue. 

     


    Friday, February 3, 2012 5:44 PM
  • I found this KB that may be helpful  http://support.microsoft.com/kb/2013544

    I am going to give it a try  tonight. 

    I also found this thread which pointed me in that direction. http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/ec39f14e-b5fe-44c1-ae5c-157621eac141/

     

    Monday, February 6, 2012 2:33 PM
  • that did not work for me. Still have the errors on backup. 
    Tuesday, February 7, 2012 2:14 PM
  • Same issue here and none of the suggestions have worked.  :(

    I am running Server 2008 R2 SP1 and can't find the HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP path.

    I manually created it (along with the DWORD value) but haven't seen any effect.

    Any other suggestions?

    Monday, February 13, 2012 6:42 AM
  • Still nothing?

    rgds Sven

    Tuesday, February 28, 2012 7:24 AM
  • We've started to see backup failures on a Host machine running W2K8-R2 Enterprise with DPM installed on the host. DPM invokes VSS shadow copies to create its "Recovery Points".

    I initiated my questions under DPM (http://social.technet.microsoft.com/Forums/en-US/dataprotectionmanager/thread/740fd820-2f31-458f-a974-70671d969430). I'm joining the fray in here because this looks more like where the problem is and should be solved. From what I have read so far, it looks like this is a widespread problem that is being ignored by Microsoft. Not happy.

    We are seeing the following on our customer's machines:

    1. volsnap Event ID 64 Volume "\\?\Volume{GUID} is being reverted to the state of a previous shadow copy."
    2. Filter Manager Event ID 3 "Filter Manager failed to attach to volume '\Device\HarddiskVolume48'.  This volume will be unavailable for filtering until a reboot.  The final status was 0xc03a001c."
    3. Disk Event ID 51 "An error was detected on device \Device\Harddisk3\DR11 during a paging operation."

    Sometimes the volsnap events are not there.
    Sometimes the Disk Events are not there.

    Only one customer is experiencing backup issues so far, but I am in the process of reviewing the logs. What we though were user errors in not changing the tapes every day, may have been in fact, in some cases, these errors.

    Looking through the Hyper-V-VMMS event log, I'm seeing Hyper-V-VMMS Event ID 10102 "Failed to create the backup of virtual machine 'xxxxxx'. (Virtual machine ID GUID)". There are no errors for 3 to 9 days after a server reboot. Then, it looks like for one period, one VM is having the problem, then it changes to another for a few times, and then to another. Each period for a machine could last from one instance to seven or eight. The switch-overs are not related to host server reboots.

    Bob.


    • Edited by BobH2 Wednesday, February 29, 2012 1:32 AM
    Wednesday, February 29, 2012 1:30 AM
  • Welcome to the party. I think this is an error in the way hyper-V tries to use VSS, but without MS joining forces with us I am pretty much at a stopping point. I get the same errors you are talking about with backup exec on our hyper-v server so this isn't a certain backup client problem it is a VSS and hyper-v problem on some level. Are your VHD's on the local server or are they on a SAN/NAS? 

    Proactive security not reactive, lets make it happen.

    Tuesday, March 6, 2012 5:29 PM
  • I think I have some improvements regarding this error "The shadow copies of volume \\?\Volume{ba882ffb-e66e-11de-8f15-001517cf3032} were aborted during detection because a critical control file could not be opened."
    I dont have hardware VSS. VHD's resides on two CSV's (I have one fast RAID10 volume and one slower RAID5 volume) residing on DAS (Dell MD3200 connected with servers via SAS).
    For example I have two hosts, each of them contains one guest residing on FAST volume and X guest on both nodes residing on SLOWER volume.

    I have played around with DPM protection groups regarding group members and protection order and for me, it seems, that I have at least found some configurations, wich have lowered the error frequency.

    Maybe it's just a luck or maybe it helps somebody to digg deeper or gives some ideas where to look at.


    rgds Sven

    Tuesday, March 6, 2012 9:34 PM
  • I have the same issue with volsnap event 27 and filtermanager event 3 on a Windows 2008 R2 running Hyper-V.  I added only the DWORD CreateTimeout with a value of 12000000 to HKLM\Software\Microsoft\Windows NT\CurentVersion\SPP, which was existing.  I had to make the same registry change for another client last year on a Windows 2008 SBS server due to a WSB backup error 2155348001 Event 521.  Hopefully this will work for this issue too.
    Wednesday, March 7, 2012 7:37 PM
  • Still had volsnap 27, filtermanager 3 and disk 51 warnings/errors.  Added HKey_Local_Maschine\ SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization, New Reg_DWORD, Name "VssRevertTimeout", Value 360 (decimal) with no difference.

    Thursday, March 8, 2012 6:49 PM
  • seems like this should have been fixed by now, hey MS I am talking to you!

    Proactive security not reactive, lets make it happen.

    Monday, March 12, 2012 5:12 PM
  • Deem... my error "The shadow copies of volume \\?\Volume{178f0c13-e07c-11e0-8dff-bc305bdd34b1} were aborted during detection because a critical control file could not be opened." are back again..

    rgds Sven

    Thursday, March 15, 2012 7:47 AM
  • To: Welcome to the party,

    We are using DAS only.

    Seem to have tracked the problem down to our own app which has its own VSS writer. It has been working fine on other systems for years. On this one particular system, something seems to be messing with it only during DPM backups. This holds up all the other DPM recovery point creation jobs until our process is killed with task manager. Then the DPM backups proceed.

    Its as if the shadow copy manager on the host is blocked by a partial crash in the VM.

    Everything looks good in the VM - plenty of memory and disk space. We are currently waiting to capture more details but the problem is intermittent and may take a few days to a couple of weeks. We don't know yet if our app is causing the problem or is failing due to an OS problem.

    Bob.

    Thursday, March 22, 2012 4:32 PM
  • I narrowed it down to the VM. Our File- and printserver. w2k8 r2 sp1. No other apps. Just blank server for printing and sharing files.


    rgds Sven

    Friday, March 23, 2012 10:26 AM
  • As a followup. I have been experimenting with our app with its own VSS Writer.

    In WIN32 under VC6, if an app dies and has a custom crash handler, a couple of techniques are available to terminate it. One good one is to call FatalAppExit() which displays a message box to the user and terminates the application.

    In WIN64 under VS2010, FatalAppExit() will display the message box, close the windows from the screen but does not terminate the process. It has to be killed using Task Manager.

    During VSS backups, a number of interrupts are sent to an app's VSS writer. If the app does not respond, VSS will hang. The last interrupt is OnBackupShutdown() which tells the app that the backup operation has completed. If the app does not respond, VSS will hang and allow a backup to almost complete: If you are using WBAdmin for example, you can get as far as "The backup of volume... successfully completed" but WBAdmin does not terminate. If you try to start another backup, WBAdmin will report that another operation is in progress.

    So far in my testing, if the app simply dies or is killed, VSS is also released.

    We may be looking for apps or services that have not been updated correctly to WIN64 (or perhaps to WIN32 using VS2010 libraries) and crash without terminating.

    It can be difficult to determine if a service is semi-crashed and not responding to VSS.  Normally, the command "vssadmin list writers" will list all the writers, and if the system is idle, show them as "stable".

    During a deliberate crash in my app, its writer is not listed in vssadmin, but gets listed again when the app is restarted. Looking for a missing writer or a writer with a different State may provide a clue.

    Bob.



    • Edited by BobH2 Sunday, March 25, 2012 11:07 PM
    Sunday, March 25, 2012 10:59 PM
  • Hello chaps,

    Felt I should also weigh in on this.

    We are using a dedicated DPM server to back up 2 HyperV Hosts with 5 VM's running over the 2 hosts. DPM backs the VM's up to local storage on the DPM box.

    The 2 HyperV hosts are identical hardware and OS (Windows Server 2008 R2 SP1)

    Probably important to mention that the DPM backups are completing with no issues at all.

    Host 1 has no errors in the event log, however Host 2 has a collection of errors in the log which only appear during the DPM backup window (while a backup is in progress)

    The errors are:

    • Warning | Disk | evID 51 | An error was detected on device \Device\Harddisk4\DR67 during a paging operation
    • Error | volsnap | evID 27 | The shadow copies of volume \\?\Volume{31c34e48-6b9f-11e1-900f-001a4be5f602} were aborted during detection because a critical control file could not be opened.

    Both host 1 and host 2 show information entries for 'volsnap' while the DPM backup is occurring, examples shown below.

    • Information | volsnap |evID  66 | The reverting of volume \\?\Volume{31c34e48-6b9f-11e1-900f-001a4be5f602} to the state of a previous shadow copy is complete.
    • Information | volsnap | evID 64 | Volume \\?\Volume{31c34e48-6b9f-11e1-900f-001a4be5f602} is being reverted to the state of a previous shadow copy.
    • Information | volsnap | evID 66 | The reverting of volume \\?\Volume{31c34e46-6b9f-11e1-900f-001a4be5f602} to the state of a previous shadow copy is complete.
    • Information | volsnap | evID 64 | Volume \\?\Volume{31c34e46-6b9f-11e1-900f-001a4be5f602} is being reverted to the state of a previous shadow copy.

    I'm concerned about the error and warning events, and it seems as though its a bit of luck and voodoo that is fixing them judging by this thread. I've not applied any fixes for this, I'm in the same situation as others on here, I do not have the SPP registry key.... I'm also cautious about creating new reg keys.

    I have the latest HP Firmware management DVD which I'm planning to run through out of business hours to see if this helps (as suggested on a different thread) Aside of that, I'm stuck.



    • Edited by GSR-UK Wednesday, March 28, 2012 9:26 AM
    Wednesday, March 28, 2012 9:04 AM
  • I am still getting my errors MS seems to have left this party. hahaha

    Proactive security not reactive, lets make it happen.

    Thursday, April 19, 2012 5:40 PM
  • Seeing error 3 on each node roughly a dozen times during DPM backup. \Device\HarddiskVolume# different in each error. The error does not seem to affect the successful completion of backups but worrisome none the less.

    Thursday, May 10, 2012 8:15 PM
  • This thread has gone very quiet, but we are joining the party! We have had this problem for months on our 2008R2 Hyper-V host using BackupAssist 6.4.2. VM's are SBS2011 and 2008R2 (running std SQL 2008). Pausing or stopping VM's allows backup to complete without errors. 

    We are backing up to external USB. Initially had this problem and thought it was related to USB 3 Hard disk - since problem doesn't occur so frequently on backup to USB 2.0 external HDD. Also found we could improve reliability by reformatting target USB to 512 sectors, although not sure now if this is relevant!

    Interestingly without a reboot, if we change the target to a USB 2 device we "sometimes" get a successful backup. But this is not always the case.

    Have not yet tried any registry changes suggested in this thread, but will do so and report back. Any new suggestions?

    Sunday, July 8, 2012 9:07 AM
  • Don't know if this helps but I get these errors at the end of the backup on several servers.  They do not seem to interfere with the backups as some of you report but I have noticed some things that might help.

    1. I do not have Hyper-V installed and have no virtual machines.

    2. During the backup, aparantly a Virtual Hard Drive is created for each Physical Drive.  If the backup completes, these Virtual drives are unmounted or deleted and this seems to trigger the error.  After each backup the Volume number is incremented for each drive untill a reboot.

    3. The Filter Manager Errors all happen a few seconds before the backup is complete.

    4. I am backing up to an external USB 2.0 Drive.

    5. I suspect that in my case the errors can be ignored but I hate to ignore errors.  This is the only error or warning I get every day.  The server is basicly a file server except for the SBS 2011 Essentials box which is also Domain Controller.

    Friday, September 28, 2012 4:14 PM
  • Wednesday, December 5, 2012 11:26 AM
  • Joining the party here, having the exact same issue running Server 2008 R2 with Hyper V backing up to USB 2 hard drives using HP Data Protector Express 6.0.00.

    Backups are reporting as working fine, just getting errors and warnings in the System Event log. The errors and warnings usually occur within 3 minutes of the backup starting.

    Has anyone come across a fix at all?

    Monday, December 10, 2012 1:00 AM
  • Hello

    Same here ! on Windows 2012..

    ????????????????? is it possible there's no fix to this ?

    Regards

    Friday, December 21, 2012 1:09 AM
  • I am not sure if it will help, but I have a similar problem also. (Windows Server Backup fails in my case.)

    • Host: Windows 2008 R2 SP1 with Hyper-V role enabled
    • VMs:
    1. SBS2008 (note AD, Exchange and SQL are present in this server),
    2. Windows 2008 R2 SP1 (file service role),
    3. Windows 2008 (remote desktop services role)

    I have this information to add to the discussion:

    I thought it might be a poorly performing RAID card, so I replaced it and re-installed the host. Windows Server Backup failed again immediately with event ID 27 (volsnap)and 3 (filter manager) present. The failure of Windows Server Backup has now survived a server re-installation. (VMs were recreated using unmodified VHD files on a separate volume after re-installation.)

    I manage other similar environments where this sort of fairly basic virtualization configuration works without issue.

    It is remarkably frustrating that this issue apparently remains unresolved after so much time and so much input from the community.

    -John


    • Edited by JohnW4 Tuesday, January 1, 2013 3:20 PM
    Tuesday, January 1, 2013 3:18 PM
  • Hello, 

    I'm joining your party too.

    Three nodes cluster w2008r2sp1 with almost 80 VM. Only one of them logg this two errors (id3 FilterManager and id 51 Disk source) on node where it runs. They show right 3-6min after DPM starts backup. All vhds are places on FC storage and DPM uses separated lan and backups on local hdds. Also newest firmware applied. This VM which causes mentioned errors has fileserver role (only) and all attached vhds have about 800gb with about 360milions of files (mostly 1kB txt).

    Should I be worried?

    -Anna

    Thursday, January 3, 2013 6:24 AM
  • Hello..

    Since it is a file server, you dont have to be too much worried.. No applications backup no particular vss action in it.. I suppose.. But keep it monitored for further errors eventually.

    Till now my restores are ok even with those errors

    Regards


    Thursday, January 3, 2013 6:39 AM
  • I have been joining this party for quite some while. And it has leaved me puzzling the issue.

    Just today i have found this article: http://www.symantec.com/business/support/index?page=content&id=TECH154720

    The event viewer has provided me with some helpfull add ons. I guess that the snapshot is filling the disc, so when writing the backup; the disc is running out of capacity. I will try a external disc for backups

    Friday, April 12, 2013 8:20 AM
  • Errors are still occurring with Server 2012 Hyper-V host, two 2012 VMs and one Windows 7 VM.

    While the Windows 7 VM is online, I get the same errors when snapshots are taken on the volume that contains that VHDX file. If I take the Windows 7 VM offline I do not get any errors. I have tried to backup just one folder that does not contain the VHDX file and I still get the errors during the snapshot.

    The Server 2012 VMs can be online with no errors. I do have a Server 2012 VHDX OS file on the same volume as the Windows 7 VHDX OS file.

    The errors occur in relation to the Windows 7 VHDX file, when it is mounted on the host for the backup. If you open Disk Management, you can view the different VHDX files as they are mounted while the backup is in progress. This accounts for the Event Viewer specifying disk numbers greater than the number of disks you have in the system.


    KTSaved

    Saturday, October 12, 2013 4:57 PM
  • I've the same issues on Windows 2012 Datacenter with Hyper-V. Due running backup virtual server with DPM 2012 SP1 (RU5) I get sporadic exactly same warnings that is described here.

    Tuesday, April 29, 2014 9:42 AM
  • i have the same problem and i will test these steps
    Sunday, January 4, 2015 12:37 PM
  • Hi There,

    I wonder if this issue might be related to an issue I'm having here:

    https://social.technet.microsoft.com/Forums/en-US/ff320f4f-20ca-4816-9f3e-135a5c5677fb/bug-with-wbadmin-vss-and-vhd-files?forum=winserverhyperv

    From my experimentation with wbadmin and VHDs it seems that the process of detaching a vhd from the OS causes some kind of corruption in the VSS which triggers a volsnap event 27 error (even when I wait for both the Virtual Disk service and the VSS service to stop of their own accord). I have found that if I leave the VHD attached then the error does not occur and even a reboot does not cause the same corruption. 

    Seems like a bug to me


    Olly

    Thursday, December 24, 2015 2:57 PM
  • Hi all,

    further to this, I have been able to resolve my own issues using the Windows Sysinternals Tool "Sync" which can force a flush of the Windows disk cache:

    https://technet.microsoft.com/en-us/sysinternals/bb897438.aspx

    I'm not sure how much relevance it has to your problem but for me this tool prevents VSS images on a VHD disk becoming corrupted when the VHD disk is detached from the host system.

    Hope it help.

    Cheers,

    Olly


    Olly

    Monday, January 18, 2016 12:26 PM