none
Hyper V backup failure with error code 2155348129 and event ID 10111 on Hyper V VMMS

    Question

  • We have a Hyper V server with Windows 2008 R2 SP1.on this server there are 2 VM s one is a VM for Symantec Endpoint manager  and the  other is a SBS 2011 server.

    Windows backup is enabled on the Hyper V server and getting backed up to a  2TB USB 3 drive.

    Intermittently Hyper V backup fails with the following error message

    The backup operation that started at '?2013?-?09?-?20T03:30:10.642000000Z' has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code '2155348129'. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

    When this happens the Hyper V VSS writer displays the following error message

    Writer name: 'Microsoft Hyper-V VSS Writer'

    Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

    Writer Instance Id: {e1f3a9eb-4e83-4998-bd7d-ef5d725dd9e2}

    State: [5] Waiting for completion

    Last error: Unexpected error

    On the Hyper V VMMS admin log entries similar to the following is listed with event ID 10111. Please note that the entry I m posting here is related to the SBS VM I can see entries for the other VM as well.

    Failed to mount the virtual hard disk '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2303\Hyper-V\SBS2011\Virtual Hard Disks\SBS2011.vhd' for virtual machine 'SBS2011' (Virtual machine ID C093E570-411D-41E9-A187-FC4303A57A4A).

    SBS2011.VHD is mounted as the C drive on the SBS VM and this is a fixed size VHD.This is a 200 GB VHD and 102 GB of free disk space remains on the VHD.

    SBS backup is getting written to a physical pass through disk and its getting completed without any issues. SBS backup starts and gets completed few hours before the Hyper V backup starts. All the VSS writers are stable on the SBS VM and the only VSS /SPP related warning message (No Errors) on the SBS is as follows.

    Volume Shadow Copy Service error: Failed resolving account spsearch with status 1376. Check connection to domain controller and VssAccessControl registry key.

    Operation:

       Initializing Writer

    Context:

       Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

       Writer Name: Shadow Copy Optimization Writer

    Error-specific details:

       Error: NetLocalGroupGetMemebers(spsearch), 0x80070560, The specified local group does not exist.

    I don’t think it has any connection because this warning is listed on the days when the Hyper V backup gets successfully completed.

    There are no VSS/SPP errors/warnings on the Hyper V application log.

    If we reboot the Hyper V server (thus rebooting the SBS server) then we can go for about a week or so without any further failures.

    Would really appreciate if someone can assist us on this in order to find a permanent resolution .

    Thanks

    Dhanushka


    • Edited by PCS-Support Friday, September 20, 2013 7:41 AM
    Friday, September 20, 2013 7:40 AM

All replies

  • Hiya, 

    Just to understand what your writing.

    Your taking a VSS type backup of your SBS from the Hyper-V host server. It works fine for a week, but after that it does not.

    On top of that your running a builtin Windows Server backup of the SBS?

    And what is the Endpoint Manager doing?

    Monday, September 23, 2013 8:39 AM
  • Hello Jesper<o:p></o:p>

    Thanks for your reply.<o:p></o:p>

    Well, the backup completion and the success/failure is intermittent .It works for some time (week or two) then it fails.<o:p></o:p>

    Until now we used to reboot the hyper v server immediately we encountered this issue but this time we did not and now for some days backup got completed successfully but on some days it failed. There is no pattern at all.<o:p></o:p>

    For the SBS VM it’s the SBS backup and for the Hyper V it’s the windows backup.<o:p></o:p>

    Symantec endpoint manager is being used to centrally manage all Symantec Antivirus clients.<o:p></o:p>

    Dhanushka<o:p></o:p>


    Monday, September 23, 2013 6:00 PM
  • Hi,

    First of all , my suggestion is checking the service of VSS when the error occur again.

    If the service is ok , please Open Event-Viewer and review at the logs at: Windows Logs\Application where the source is Backup.

    Referring to the following link for details:

    Diagnosing Failures in Windows Server Backup - Part 1 (VSS/SPP Errors)

    http://blogs.technet.com/b/filecab/archive/2009/09/16/diagnosing-failures-in-windows-server-backup-part-1-vss-spp-errors.aspx

    Hope it helps

    Best regards

    Elton ji


    Tuesday, September 24, 2013 12:53 AM
    Moderator
  • Hi Elton

    Thanks for your reply but I have already performed all what you have mentioned but was unable to find any clue.

    Dhanushka

    Tuesday, September 24, 2013 4:40 PM
  • Do you have any performance monitoring on your machines?

    Just had another thread with a Symantec Endpoint Protection eating all resources. Thinking if you sometimes hit into that application performing something requiring intense amount of resources, it will affect your ability to take backup.

    Might be worth to have a look at some performance monitoring around the time your taking backup, to see if something as "simple" as too much I/O is causing your problems.

    Tuesday, September 24, 2013 5:02 PM
  • That is a good suggestion Jesper.

    We do not have any specific resource monitoring tools but I assume windows native resource monitor would be able to get the work done. If not do you any good free tools?

    What do you think that we should be monitoring , disk and CPU?

    Dhanushka

    Wednesday, September 25, 2013 4:29 PM
  • Hiya, 

    I would expect perfmon to be sufficient. And yes, disk and CPU should do it.

    Wednesday, September 25, 2013 5:29 PM
  • Thanks for the update but guess what? Without any modifications backups started to work. Will try to set this up during the weekend and see how it goes. Dhanushka
    Friday, September 27, 2013 9:53 PM
  • Hi,

    I am Chetan Savade from Symantec Technical Support team.

    Please let me know in case you need any help related to Symantec Endpoint Protection.

    Regards,

    Chetan

    Monday, September 30, 2013 4:46 PM
  • Hello Jesper

    Sorry I could not perform the needful during the weekend but the backups are now getting completed without any issues....strange.

    Chetan , thanks for your interest in this issue.

    1.Have you seen / encountered similar issues which SEP is causing the windows backup failures intermittently?

    2.If we are to do a perfmon apart from disk and CPU is there anything else that we need to monitor with regards to SEP?

    Thanks

    Dhanushka

    Tuesday, October 01, 2013 11:16 AM
  • Hi,

    Thanks for the update.

    I could see following fix in SEP 12.1 RU2 fix notes but it's not related to your case.

    ArcServer Backup fails to back up Hyper-V virtual workstations
    Fix ID: 2729418
    Symptom: ArcServer Backup fails to back up Hyper-V virtual workstations with Symantec Endpoint Protection 11.x installed.

    Solution: Updated Auto-Protect to address this issue.

    Reference:http://www.symantec.com/docs/TECH199676

    When the backup fails you see any conflict with SEP related process in eventviewer?

    Regards,

    Chetan


    Wednesday, October 02, 2013 4:17 PM
  • Hi Chetan

    Sorry I could not get back earlier as I was busy with work.

    So far I have not seen any SEP evetnts in the event viewer when the backups fails.Both servers have SEP 12.1.3 installed.

    Dhanushka

    Monday, October 07, 2013 11:31 AM
  • Hi Dhanushka,

    Thanks for the update & I believe problem shouldn't be from SEP end.

    Probably you will have to work with Microsoft support team.

    Regards,

    CHETAN

    Thursday, October 10, 2013 4:20 PM
  • Hello all

    opened up a support ticket with MS partner support portal but its no where near finding a resolution. I was told that this is a known issue but I have been asking for more information for fore that 3 weeks not and not getting a single reply. :(

    As per their advices I disabled the Hyper v VSS writer as per the below listed instructions

    Exclude the Hyper-V write id to fix the issue.

     --------------------------------------------------------------

    a. Beta/Hyper-V writer id matches the one below please use vssadmin list writers. The

    Hyper-V writer id is {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

    b. Please make a backup of the registry key before modifying it.

    c.  Under the registry key "Software\\Microsoft\\Windows NT\\CurrentVersion\\SPP", create a new MULTI_SZ-value with the name “DisabledWriters”.

    d. Enter the writer guid {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} with braces as the

    value.

    e. Shut down wbengine service if it’s running. Use command “net stop wbengine”.

    f.  Restart wbngine service using command “net start wbengine”

     

    Backups are working but when I got to restore the "Application" restore is greyed out on the Hyper V server?

    Will I not be able to do a successful recover of the VM s since its disabled?

    Thanks

    Dhanushka

    Friday, November 01, 2013 2:20 PM
  • Hi PCS-Support,

    "Backups are working but when I got to restore the "Application" restore is greyed out on the Hyper V server?"

    May be you need to register the VSS Writer for windows backup.

    Here is a link refer to "How to Register a Hyper-V VSS Writer for Windows Backup Support":

    http://www.appassure.com/support/KB/4120128/

    RGDS

    Monday, November 04, 2013 1:23 AM
  • Hi Paul

    Hyper V VSS writer was already registered with windows backup.

    Regards

    Dhanushka

    Monday, November 04, 2013 10:53 PM
  • Hi PCS,

    Have you done this before ?

    RGDS

    Tuesday, November 05, 2013 1:18 AM
  • Hi Paul

    Yes Hyper V VSS writer was registered earlier and application recovery option was available untill we did the previously mentioned change.

    Dhanushka

    Wednesday, November 06, 2013 5:37 PM
  • Hi PCS,

    Sorry for the misread of title and what you posted .

    "c.  Under the registry key "Software\\Microsoft\\Windows NT\\CurrentVersion\\SPP", create a new MULTI_SZ-value with the name “DisabledWriters”. "

    this step caused that disabling the vss writer, so the  "Application" restore is greyed.

    But if you do not do that , may be the problem you originally posted will occur again.

    So , please try to reverse the operation of the key value "DisabledWriters" , and try again.

    RGDS

    Monday, November 11, 2013 5:02 AM
  • Hello

    That is not what I want to do at this stage.I do not want to reverse the changes because it was done as a troubleshooting step to get the original problem resolved.

    What I want to find out is if application restoration is disabled if it is possible  for us to recover VM s successfully during a disaster.

    Dhanushka

    Monday, November 11, 2013 8:19 AM
  • Hi Dhanushka,

    Based on my knowledge , If the VSS was not registered , the "windows server backup" can not backup the online VMs .

    So , please try to reverse the changes you posted , and shutdown the SBS2011 VM then use "windows server backup " again for trouble shooting.

    Please refer to following link :

    http://technet.microsoft.com/en-us/library/ee692290(v=ws.10).aspx

    Hope this helps

    Best Regards

    Elton Ji


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    Friday, November 15, 2013 5:36 AM
    Moderator
  • Hi Elton

    Microsoft has replied and based on what was told it appears that VM s can be recovered even without the application recovery option but I too am confused so I requested for more information.

    Its been over a week but received no further replies from Microsoft.

    As I mentioned earlier VSS writer is already registered but disabled.

    I would really appreciate if someone can let me know if we can successfully recover VM s without application recovery option being enabled.

    Dhanushka

    Monday, November 18, 2013 2:37 PM
  • Monday, November 18, 2013 3:40 PM
    Moderator
  • Hi Elton

    Many thanks for your shareing your information on server 2012.

    Hopefully some one will reply with a server 2008 R2 experience as well.

    Unfortunately it not possible to bering down this SBS because the client has multiple Geographical locations which users are accessing data from the SBS most of the time.

    I m sure if we do that backup will work but eventually it will faile , like what has happened originally.Do you think otherwise?

    Dhanushka

    Wednesday, November 20, 2013 6:38 PM