none
VSS Error Server 2012 Datacenter / Application: 12289, 12297, 12340, 8194 / System: 9

    Question

  • Hi people!

    I'm using arcserve backup 16.5 to backup my cluster 2012 (datacenter) (2 servers). Physical and virtual machines (15 vm's).
    Now after a while the vss starting to show me some errors and my backup is failing on only both hyperv physical hosts on C drive.
    After much investigating with the support of arcserve they say that they cant do much about this problem and the support redirected me to Microsoft. After much calling and putting me on hold and telling me that they will call me back I'm still sitting here getting old and gray. So hope there is someone out there to help me out with this one.

    The errors are only on the hyper-V hosts. The vm's and other server 2012 are backing up just fine (have not all the updates).

    Here are the errors. (when clicking on some links they show me the CSV attached to my HyperV cluster and C: drive one of them)

    -----------------------------------------------------------------------------------------------------------------------------------

    Volume Shadow Copy Service error: The I/O writes cannot be held during the shadow copy creation period on volume \\?\Volume{1693955f-32a1-4111-a860-e780d6637f1b}\. The volume index in the shadow copy set is 0. Error details: Open[0x00000000, The operation completed successfully.

    ], Flush[0x00000000, The operation completed successfully.

    ], Release[0x00000000, The operation completed successfully.

    ], OnRun[0x80042314, The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.

    ].

    Operation:

       Executing Asynchronous Operation

    Context:

       Current State: DoSnapshotSet

    -------------------------------------------------------------------------------------------------------------------------------------------

    Volume Shadow Copy Service error: The I/O writes cannot be flushed during the shadow copy creation period on volume \\?\Volume{cae23b9e-4d92-11e2-93e7-806e6f6e6963}\. The volume index in the shadow copy set is 2. Error details: Open[0x00000000, The operation completed successfully.

    ], Flush[0x80042313, The shadow copy provider timed out while flushing data to the volume being shadow copied. This is probably due to excessive activity on the volume. Try again later when the volume is not being used so heavily.

    ], Release[0x00000000, The operation completed successfully.

    ], OnRun[0x00000000, The operation completed successfully.

    ].

    Operation:

       Executing Asynchronous Operation

    Context:

       Current State: DoSnapshotSet

    -------------------------------------------------------------------------------------------------------------------------------------------

    Volume Shadow Copy Error: VSS waited more than 40 seconds for all volumes to be flushed.  This caused volume \\?\Volume{cd77626e-893f-4893-b6d3-a1cd5c1da041}\ to timeout while waiting for the release-writes phase of shadow copy creation.  Trying again when disk activity is lower may solve this problem.

    Operation:

       Executing Asynchronous Operation

    Context:

       Current State: flush-and-hold writes

       Volume Name: \\?\Volume{cd77626e-893f-4893-b6d3-a1cd5c1da041}\

    ------------------------------------------------------------------------------------------------------------------------------------------

    The flush and hold writes operation on volume C: timed out while waiting for file system cleanup.

    ------------------------------------------------------------------------------------------------------------------------------------------

    Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.

    . This is often caused by incorrect security settings in either the writer or requestor process.

    Operation:

       Gathering Writer Data

    Context:

       Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}

       Writer Name: System Writer

       Writer Instance ID: {0d3a0201-dcb2-407e-81a6-2482e7d75854}

    ----------------------------------------------------------------------------------------------------------------------------------

    ARCSERVE ERROR: E12502 2014-02-16 13:41:19 The volume shadow service provider has reported a bad state for the operation.

    --------------------------------------------------------------------------------------------------------------

    What I have done so far (without any luck):

    vssadmin list writers are all stable and no errors.
    vssadmin list providers are all Microsoft
    Cleaned up shadow copies
    Installed all updates (server 2012 and arcserve)
    No antivirus installed (removed trendmicro)
    Changed the setting on C: to maximum size: no limit on shadow copies
    Rebooted system many times
    There is enough space on the disks


    Beste Regards, Stephan



    • Edited by Steee-van Wednesday, February 19, 2014 10:00 AM new error
    Sunday, February 16, 2014 1:07 PM

Answers

  • Hi Mandy!

    These were indeed the settings.
    Got a call of arcserve support team yesterday and told them I got a new error.
    They also changed the account of the vss in services.msc to an account that has full permissions.

    At the registery (Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSer\Services\VSS\VssAccessControl
    there a 2 values like this:

    1. REG_DWORD --> DOMAIN\USERFULLPERMISSIONS --> set to 3 (0x0000003)
    2. REG_DWORD --> NT Authority\NetworkService --> set to 3 (0x0000003)

    Now only a warning is there (event 8230), the error is gone:

    Volume Shadow Copy Service error: Failed resolving account administrator with status 1376. Check connection to domain controller and VssAccessControl registry key.

    Thanks!

    P.S. Arcserve support team is still working on it to try to resolve the other issues.

    Regards Stephan.

    Thursday, February 20, 2014 7:59 AM
  • Hi all!

    I have found a "part" of the solution to make 12289, 12297, 12340 an 9 to disappear and run a successfull backup job with arcserve backup 16.5r.
    First of all when selected the whole C:\ drive of both cluster nodes the errors are appearing in the eventvwr and backup fails.
    Now I have deselected one by one the folders on the drive and run a backup job each time. Still the errors are there!
    Now when I deselected the files in the root of C:\ like bootmgr system file and other systems files in the root en selected the folders that I want to backup the error dissappears in the eventvwr. However not all errors are gone but the backup shows now successfull! Now I have only 12292 and 13 to worry about. For the next week I just leave the system files unchecked for backup and see how it will go. I post my results at the end of next week. Thanks people!

    • Marked as answer by Steee-van Friday, February 28, 2014 11:13 AM
    • Edited by Steee-van Friday, February 28, 2014 4:47 PM
    Friday, February 28, 2014 11:12 AM

All replies

  • Hi Stephan, 

    IThis issue may be due to the VSS system files are not registered. Please refer to the article below to re-register VSS dlls:

    Various issues may occur on a Windows Server 2003-based computer that is running the Volume Shadow Copy Service
    http://support.microsoft.com/kb/940032

    Please also refer to the article below to troubleshoot the issue:

    Event ID 12297 — VSS Shadow Copy Creation and Storage Space Availability
    http://technet.microsoft.com/en-us/library/dd364933(v=ws.10).aspx

    Regards, 

    Mandy

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    Monday, February 17, 2014 8:49 AM
    Moderator
  • Hi Mandy!

    Thanks for looking into the problem with vss.
    Will this "2003" commands apply to the server 2012 environment?

    The document says that is for:
    • Microsoft Windows Server 2003, Standard Edition (32-bit x86)
    • Microsoft Windows Server 2003, Enterprise Edition (32-bit x86)
    • Microsoft Windows Server 2003, Datacenter Edition (32-bit x86)
    • Microsoft Windows Server 2003, Standard x64 Edition
    • Microsoft Windows Server 2003, Enterprise x64 Edition
    • Microsoft Windows Server 2003, Datacenter x64 Edition
    • Microsoft Windows Server 2003, Enterprise Edition for Itanium-based Systems
    • Microsoft Windows Server 2003, Datacenter Edition for Itanium-Based Systems
    Thanks!

    Regards, Stephan.
    Monday, February 17, 2014 10:35 AM
  • Hi Stephan, 

    I think the commands mentioned in the article below also apply to server 2012 environment. The commands are common method to troubleshoot VSS issue, please try to run the commands to check the result:

    How to recovery VSS issue
    http://blogs.technet.com/b/csstwplatform/archive/2010/07/21/how-to-recovery-vss-issue.aspx

    Regards, 

    Mandy

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    Tuesday, February 18, 2014 6:16 AM
    Moderator
  • Hi Mandy,

    I tried the above article you have posted without any results.
    Also the recreation of the registery key was not made, so I had put that back of the backup i've made.
    Also have found some articles of fixing vss on 2008 but without luck.

    Can't find any information of vss server 2012 fixes.

    Hope you have some other suggestions for me?

    Beste regards Steph.

    Tuesday, February 18, 2014 9:42 AM
  • Hello,

    I am in the very same boat with you.

    Server 2012 Standard, ARCserve D2D r16.5

    Seems that this started after we added new disk to the physical machine, VSS starting failing with error 12289

    **

    Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{521deea9-4bd1-11e3-93f0-ac162d9ad5fb} - - 00000000000000E8,0x0053c008,000000AB1166A160,0,000000AB1166E1A0,4096,[0]).  hr = 0x80070057, The parameter is incorrect.
    .

    Operation:
       Processing EndPrepareSnapshots

    Context:
       Execution Context: System Provider

    **

    I found this KB that is nearly identical to our situation, applied the hotfix, no change still failing. (KB2914152)

    I've just opened a Partner ticket with MS to fix this as the customer is rather large, and backups are now failing due to VSS unable to operate correctly.  I will post our results as soon as I have them.

    Tuesday, February 18, 2014 5:10 PM
  • Hi Kevin,

    That would be great if you let me know!
    I could not resist and downloaded/installed the KB2914152 to see what will happend but also here no change and still failing.
    I also hope to open a ticket with MS soon to solve this.

    Thanks!

    Regards!

     
    Tuesday, February 18, 2014 7:12 PM
  • A new error appeared:

    Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.

    . This is often caused by incorrect security settings in either the writer or requestor process.

    Operation:

       Gathering Writer Data

    Context:

       Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}

       Writer Name: System Writer

       Writer Instance ID: {0d3a0201-dcb2-407e-81a6-2482e7d75854}

    Wednesday, February 19, 2014 10:01 AM
  • Hi Stephan, 

    For the Event ID: 8194, I suggest you refer to the thread to help troubleshoot the issue:

    VSS error showing in Event Viewer
    http://answers.microsoft.com/en-us/windows/forum/windows_7-system/vss-error-showing-in-event-viewer/34b62c1b-93e6-4848-a9a0-59e3f4019ebb

    Ultimately, the 3rd party vendor is responsible for properly handling the security context of COM callbacks from VSS.

    You may be able to workaround the problem by doing the following: 

    - Start | Run and type dcomcnfg and hit Enter key 
    - Expand Component Services 
    - Expand Computers | My Computer 
    - Right Click on My Computer and choose Properties 
    - Select COM Security 
    - Select Edit Default on Access Permission 
    - Add the Cluster Service Domain account with Local Access and Remote Access allowed 

    If this does not work then add in the following also: 

    - Start | Run and type dcomcnfg and hit Enter key 
    - Expand Component Services 
    - Expand Computers | My Computer | DCOM Config 
    - Select your arcserve client exe 
    - Right click and select Properties 
    - Click on Security tab 
    - Select Customize option for Access Permission and click Edit button 
    - Add the Cluster Service Domain Account to grant access

    Regards, 

    Mandy

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    Thursday, February 20, 2014 2:52 AM
    Moderator
  • Hi Mandy!

    These were indeed the settings.
    Got a call of arcserve support team yesterday and told them I got a new error.
    They also changed the account of the vss in services.msc to an account that has full permissions.

    At the registery (Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSer\Services\VSS\VssAccessControl
    there a 2 values like this:

    1. REG_DWORD --> DOMAIN\USERFULLPERMISSIONS --> set to 3 (0x0000003)
    2. REG_DWORD --> NT Authority\NetworkService --> set to 3 (0x0000003)

    Now only a warning is there (event 8230), the error is gone:

    Volume Shadow Copy Service error: Failed resolving account administrator with status 1376. Check connection to domain controller and VssAccessControl registry key.

    Thanks!

    P.S. Arcserve support team is still working on it to try to resolve the other issues.

    Regards Stephan.

    Thursday, February 20, 2014 7:59 AM
  • I have some more info.

    Our setup is as follows;

    Server 2012 Standard
    128 GB RAM
    2 x 2.9 GHz 8 core CPUs
    Local Disk -
    Disk Array 1 - 60TB
    Disk Array 2 - 75TB

    MS is checking to ensure the VSS limit still exists for Server 2012 at 64TB max for VSS operations.  This is what we are running into. For the single 75TB array as a disk. (http://support.microsoft.com/kb/950141/en-us) This references everything up to 2008 R2, not 2012.  Although I have a feeling it applies as this is the situation we have run into.

    What I did find out was that if I tried to run any VSS attempts agains the large volume, is would mess up a random other volume causes VSS errors.  If I ran a Windows Backup against the volume that had been messed up by the large volume VSS attempt, it would succeed and then all subsequent backup attempts, either from D2D or from Windows Backup, were successful without any VSS errors.

    Looks like we will have to chop that large disk into at least 2 parts to fit under the 64 TB less 8 GB limit for VSS and carry on.

    Still waiting for official word on VSS in Server 2012 though, can't find any official documentation that states the limit for server 2012.  I have found 1 reference to the limit on a blog of a seemingly very knowledgable fellow, but it's not official so can't really use that. (http://workinghardinit.wordpress.com/2012/10/22/windows-server-2012-64tb-volumes-and-the-new-check-disk-approach/) about half way down in the Important Note he menetions the limitation in Server 2012, seems to be more than just VSS as well since the supported volume size is so large,but not all internal workings support the large disks.

    Awaiting official word on the limit, otherwise our situation is determined.

    Thursday, February 20, 2014 4:04 PM
  • Hi all!

    I have found a "part" of the solution to make 12289, 12297, 12340 an 9 to disappear and run a successfull backup job with arcserve backup 16.5r.
    First of all when selected the whole C:\ drive of both cluster nodes the errors are appearing in the eventvwr and backup fails.
    Now I have deselected one by one the folders on the drive and run a backup job each time. Still the errors are there!
    Now when I deselected the files in the root of C:\ like bootmgr system file and other systems files in the root en selected the folders that I want to backup the error dissappears in the eventvwr. However not all errors are gone but the backup shows now successfull! Now I have only 12292 and 13 to worry about. For the next week I just leave the system files unchecked for backup and see how it will go. I post my results at the end of next week. Thanks people!

    • Marked as answer by Steee-van Friday, February 28, 2014 11:13 AM
    • Edited by Steee-van Friday, February 28, 2014 4:47 PM
    Friday, February 28, 2014 11:12 AM
  • Thanks for the info Kevin! When you know more just let it know! Thanks.
    Friday, February 28, 2014 4:34 PM