none
VSS issue - System writer fails during system state backup using Symantec Backup Exec 2010

    Question

  • Physical Windows Server 2008 R2 w/SP1 Standard server. Acts as a dedicated media server for Symantec Backup Exec 2010 R3. When performing a system state backup of this media server, the job fails with a VSS error. I have attempted to run the system state backup with Microsoft's backup utility. It also fails. I have a 6 TB LUN connected via iSCSI for a backup target which I did not use for the Microsoft backup test. All other backup jobs run successfully for the physical and virtual environment.

    Backup Exec logs report the following:
    - AOFO: Initialization failure on: "\\bemsvr System?State". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).
    V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

    The application event log reports the following:

    Log Name:      Application
    Source:        VSS
    Date:          9/1/2011 12:21:56 PM
    Event ID:      12293
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      bemsvr
    Description:
    Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details EndPrepareSnapshots({99321340-ad58-44b0-b8b3-34c66d8343ea}) [hr = 0x80070015, The device is not ready.
    ].

    Operation:
       Executing Asynchronous Operation

    Context:
       Current State: DoSnapshotSet
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="VSS" />
        <EventID Qualifiers="0">12293</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2011-09-01T17:21:56.000000000Z" />
        <EventRecordID>26504</EventRecordID>
        <Channel>Application</Channel>
        <Computer>bemsvr</Computer>
        <Security />
      </System>
      <EventData>
        <Data>{b5946137-7b9f-4925-af80-51abd60b20d5}</Data>
        <Data>EndPrepareSnapshots({99321340-ad58-44b0-b8b3-34c66d8343ea})</Data>
        <Data>0x80070015, The device is not ready.
    </Data>
        <Data>

    Operation:
       Executing Asynchronous Operation

    Context:
       Current State: DoSnapshotSet

    I contacted Symantec support and they volunteered to pass this one on to Microsoft, which I have not contacted. I have searched and researched many different variations of VSS issues. Some technotes that I found suggested re-registering the dll's, changing permissions on a couple folders under the \Windows\winsxs\ folder, changing permissions on DWord values, as well as a few others. So far, nothing has worked.

    VSSAdmin reports the following:

    C:\>vssadmin list providers
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.

    Provider name: 'Microsoft Software Shadow Copy provider 1.0'
       Provider type: System
       Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
       Version: 1.0.0.7

    C:\>vssadmin list writers
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.

    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error

    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error

    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error

    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {a9900084-3a02-4d5e-a648-d9ff82ce18f5}
       State: [1] Stable
       Last error: No error

    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {45aba776-c328-4fa8-8532-ad4648a3f663}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'SqlServerWriter'
       Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
       Writer Instance Id: {20b36274-9908-4984-a94b-4641e3fdcf99}
       State: [1] Stable
       Last error: No error

    Writer name: 'Backup Exec Deduplication Storage Writer'
       Writer Id: {55f6d76d-8617-4fc8-81e9-df0de529b555}
       Writer Instance Id: {55bbca04-1f46-4476-b9fa-a2745fcca28f}
       State: [1] Stable
       Last error: No error

    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {2ffe9624-ce3e-4748-a527-c7ef1ebb52c9}
       State: [1] Stable
       Last error: No error

    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {bbe27b77-d304-4ff7-84b6-73dd61527e22}
       State: [1] Stable
       Last error: No error

    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {0e0cc0a8-5eac-4e1d-aed9-d2a7ad9a9d7b}
       State: [1] Stable
       Last error: No error

    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {a11bedf0-f837-449c-bcae-bcf9ad9f91f0}
       State: [1] Stable
       Last error: No error

    Stopping and Restarting the Cryptographic Service will put the System Writer back into a stable state.

    By the way, at one time the system state backup ran successfully on this media server. I don't know what broke it.

    Thursday, September 01, 2011 7:55 PM

All replies

  • Hello,

    for backup questions, ask them here: http://social.technet.microsoft.com/Forums/en-us/windowsbackup/threads

    For BackupExec, ask in Symantec forums.

     


    This posting is provided "AS IS" with no warranties or guarantees , and confers no rights.

    Microsoft Student Partner 2010 / 2011
    Microsoft Certified Professional
    Microsoft Certified Systems Administrator: Security
    Microsoft Certified Systems Engineer: Security
    Microsoft Certified Technology Specialist: Windows Server 2008 Active Directory, Configuration
    Microsoft Certified Technology Specialist: Windows Server 2008 Network Infrastructure, Configuration
    Microsoft Certified Technology Specialist: Windows Server 2008 Applications Infrastructure, Configuration
    Microsoft Certified Technology Specialist: Windows 7, Configuring
    Microsoft Certified IT Professional: Enterprise Administrator
    Microsoft Certified IT Professional: Server Administrator 

    • Proposed as answer by Rich Prescott Thursday, September 01, 2011 11:45 PM
    • Unproposed as answer by Jeff RenModerator Friday, September 02, 2011 2:31 AM
    Thursday, September 01, 2011 9:13 PM
  • Hi,

     

    As per description, I understand that you are encountering VSSwriter issues when you are using Symantec Backup Exec 2010 R3.

     

    Generally speaking, you can take the following steps when encountering VSS writer related issues:

    1.      Retry the backup or restore operation that caused the error condition.

    2.      If this does not resolve the issue, restart the service or process that hosts the writer, and retry the operation.

    3.      If this does not resolve the issue, open Event Viewer as described in the "Open Event Viewer and view events related to VSS" section and look for events related to the service or process that hosts the writer. If necessary, restart the        service or process, and retry the operation.

    4.      If this does not resolve the issue, restart the computer, and retry the operation.

    5.      If restarting the computer does not resolve the issue, provide the Event Viewer information to the vendor whose application is indicated in the event text.

    For more information ,please refer to :

    http://technet.microsoft.com/en-us/library/ee264212(WS.10).aspx

     

    If you does not resolve the issue through above method, you can try to re-register the VSS writers, you can refer to :

    http://www.symantec.com/business/support/index?page=content&id=TECH70486

     

    Best Regards, 

    Jeff Ren


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
    • Proposed as answer by Jeff RenModerator Friday, September 02, 2011 2:37 AM
    • Unproposed as answer by MAlford Friday, September 02, 2011 12:37 PM
    Friday, September 02, 2011 2:36 AM
    Moderator
  • I have done everything suggested in the previous post. The first thing that I done was contacted Symantec. They told me that it was a Microsoft issue. I think that they should take more responsibility toward finding a solution for this issue. I also noticed that there are others in this forum that are in similar situations. That makes me wonder if Microsoft is aware of the issues and may have provided a hot fix.

    I have been working on this myself for for over a week with no success. At this point I feel that I am going to have to open a case with Microsoft on this issue. Before I do though, I am going to take another stabb at Symantec.

    Have anymore suggestions?

     


    Michael Alford
    Friday, September 02, 2011 1:10 PM
  • To Arthur Li and Jeff Ren... exactly why did you move this into a Small
    Business Server forum?
     
    I'm not seeing this is a SBS box.
     
    How about we stop being a bit move happy please Mr. Moderators?
     
    To MA I think you'll have to bite the bullet and put in a call.  I'm not
    seeing Symantec VSS issues in the SBS forum so I'm assuming you are
    seeing them elsewhere.  Forums work for known issues, they suck for
    issues where vendors are finger pointing.  If you are a Technet plus
    subscriber you have two phone call incidents.  If they find out that
    it's a bug and a hotfix is needed they will comp back the call.
     
    Friday, September 02, 2011 3:59 PM
    Moderator
  • "How about we stop being a bit move happy please Mr. Moderators?" Susan r you not one of the moderators that I see often?
    Friday, September 02, 2011 7:47 PM
  • I have a rule.  I don't move posts.  I ask people post in another forum if they post in the wrong place but I think it's rude to move someone's thread.  In this case, it's not even the right forum.
    Friday, September 02, 2011 7:56 PM
    Moderator
  • We are also having issues with this...we spent a long time with Syamantec support only for them to tell us it was Microsofts issue, yet I cannot find anywhere that Microsoft has adopted this problem as thiers.

    I don't care who's problem this is, we just want to get it fixed and its beyond our knowledge base.

    Has anyone found a solution to this problem/bug/feature? I don't like having my systems backup without the system state.

    Please advise and as always thanks for your help in advance

    Thursday, September 15, 2011 6:40 PM
  • We also have this problem and have been spent time with Symantec support with them also telling us it is a Microsoft problem.  We are very frustrated and hope someone can help us out in digging deaper into what the root cause is.
    Saturday, September 24, 2011 3:52 PM
  • One of our clients has the same issue. Backup Exec 2010 R3, Windows Server 2008 SP2, System state backup fails. System writer goes into the failed state. Rebooting sets the writer back to stable but backups just fail again. Would love to know the solution.


    • Edited by Stuart Br Tuesday, September 27, 2011 2:03 PM
    Tuesday, September 27, 2011 2:00 PM
  • Please test the Windows native backup solution first and let us know.
    Tuesday, September 27, 2011 2:03 PM
  • Took some extenstive research this weekend but this fixed it on our server.  For me reading the article it seemed like it really did not apply to us but I found a forum with a person with our extact same problem and thought what do I have to lose.  And it worked!

    http://support.microsoft.com/kb/2013544

     

     

    • Proposed as answer by BTIT4 Tuesday, September 27, 2011 4:50 PM
    Tuesday, September 27, 2011 2:14 PM
  • Hello there,

    I am having the same issue.  Did you eventually solve this problem, MAlford?   If so, how?

    I tried everything listed in this post but nothing helped.

    Thanks in advance for any suggestion.

     

     

    Tuesday, November 01, 2011 6:11 PM
  • Thanks for the replies.

    I still have not found a resoluton to this issue. I did attempt the suggestion from BTTT4's post. After I made the change, I was able to backup the System State with the Microsoft backup utility, but the System State backup continued to fail with Backup Exec 2010. And once I attempted to create the backup with Backup Exec 2010, it would no longer backup with the Microsoft backup utility. Even after rebooting the server. I don't have a clue. Anybody out there got any new information?

    Thanks,


    Michael Alford
    Friday, November 04, 2011 4:51 PM
  • Have you looked at this technote:  http://www.symantec.com/business/support/index?page=content&id=TECH76626

     

    Best Regards,
    Elias AbuGhazaleh
    Symantec Corporation

    Tuesday, November 08, 2011 2:03 PM
  • Yes, I have seen the above technote. Unfortunately, the instructions in the technote did not help in this case. Thanks for the reponse though.
    Michael Alford
    Tuesday, November 08, 2011 6:26 PM
  • Thanks for confirming that Michael.  It does seem that the entry posted here by BTIT4 holds the answer (http://support.microsoft.com/kb/2013544)

    I'll see if I can investigate this further with any contacts at MS about status of their pending resolution as noted.

    Best Regards,
    Elias AbuGhazaleh
    Symantec Corporation

    Tuesday, November 08, 2011 9:21 PM
  • Michael,

    Have you gotten any further with ths issue?

    One other additionl piece of information I was able to get is the following:
    Does your server have a reserved system volume?  If you go into Disk Management, is there a "partition" titled "System Reserved"?  If so, this could be a sporadic issue with VSS and the reserved system volume.

    One thing you can try to do is to assign that "partition" a drive letter.  After doing this, reboot and try the backup of system state again.

    Best Regards,
    Elias AbuGhazaleh
    Symantec Corporation

    Friday, November 18, 2011 12:26 AM
  • No, I haven't made any progress with this issue.

    I do have a remote storage array connected via iSCSI acting as a "backup to disk" target. This volume does have a drive letter assignment. I also have iSCSI connectivity into my VMware SAN which allows the Backup Exec software to have direct SAN access to the volumes for backing up my virtural servers. The BE software only needs RO access to the VMware volumes. The volumes do not have drive letter assignments and they are not titled "System Reserved". VMware does not advise assigning drive letters to these volumes.

    I also would like to add that these volumes are not selected for backup as part of the backup job in question. Only the local disks, SystemState, and Shadow Copy Components are selected. The SystemState is the peice that fails during the backup and leaves the System Writer in a "Failed" state with a "Timed Out" status.

    I have suspected that these volumes might have something to do with the backup job failing, but the backup process needs to maintain connectivity to these volumes.

    I appreciate any suggestions or potential fixes for this issue.

    Thanks.

     

     


    Michael Alford
    Friday, November 18, 2011 9:51 PM
  • Hey Michael.

    I'm working with a top engineer on Backup Exec and I shared with him your latest post.  Would it be possible for you to send me all the XML files that are in the BE Log directory and/or send a screen shot of the disk manager from the server that has the failure for review?  I'm working this issue, and will continue to until we find a resolution for you.
    Please email those (zip) to me at elias_abughazaleh@symantec.com

    Best Regards,
    Elias AbuGhazaleh
    Symantec Corporation

    Saturday, November 19, 2011 9:18 PM
  • Elias,

    I appreciate your interest in finding a resolution for this issue. I have complied with your request for the log files and screenshots. You should be receiving an email with an attached .zip file. If you have questions or need to contact me for anything, my contact information was also included in the email. I will also continue to monitor this thread.

    Thanks.

     


    Michael Alford
    Sunday, November 20, 2011 3:08 AM
  • Michael,

    Glad to have helped you find a work around for the issue.  I'm posting the resolution here, so that others may benefit the suggested work-around, and your success with this.  Just to confirm, our belief is that this is a VSS issue, we around gathering the details around the failure and the work-around to submit to Microsoft.

    Based on the information we asked for from you, which you provided the suggested work around is as follows:

    First, please mount the reserved system volume (\\?\Volume{5183ed87-6279-11e0-9ed8-806e6f6e6963}\) to an available drive letter. 
    The Disk Management plug-in or the MOUNTVOL command line can be used.
    Here is how, from a command prompt:

    MOUTNVOL Example Using the available “B:” volume:
    C:\> MOUNTVOL B: \\?\Volume{5183ed87-6279-11e0-9ed8-806e6f6e6963}\

    DIR

    Directory of B:\

    06/17/2011  09:29 AM    <DIR>          $RECYCLE.BIN
    05/27/2011  09:00 AM    <DIR>          Boot
    11/20/2010  03:40 AM           383,786 bootmgr
    10/21/2010  08:00 PM             8,192 BOOTSECT.BAK
    10/21/2010  12:48 PM    <DIR>          System Volume Information
    11/17/2011  06:55 PM             4,096 VSNAP.IDX
                   3 File(s)        396,074 bytes
                   3 Dir(s)       2,998,272 bytes free

    Then, please verify that the volume is mounted and can be browsed (hidden system files).
    Finally, rerun the previously failing backup job.

    You ran the above steps and confirmed that your backups (full and differential) completed without failure.  Yes, you will need to keep that volume mounted with the drive letter assignment.

    If I gather any further information from our development team, I'll be sure to update you.

    Best Regards,
    Elias AbuGhazaleh
    Symantec Corporation

    Tuesday, November 22, 2011 7:44 PM
  • Confirming above answer, also make sure to set the backup exec advanced open file to use MS VSS

    http://www.routerjanitor.com/2013/02/windows-server-2008-r2-backup-exec-2012.html

    Saturday, February 23, 2013 3:14 PM
  • Just an FYI

    Cannot get BE 2010 to backup a system state with a 4 TB USB drive even connected to the system, not backing up the 4 TB drive, or backing up to the 4TB, backing up to another drive on another server altogether

    Tuesday, June 25, 2013 7:14 PM