locked
vss error id 12289 in the event log RRS feed

  • Question

  • I just started to get this error in the event logs on my dpm server. backups have been working fine for months.

    event log reports

    Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{xxxxxxx-bddd-11e2-93f8-157953225e0c} hr = 0x80070005, Access is denied.
    .

    Operation:
       Query diff area for this volume

    Context:
       Volume Name: \\?\Volume{xxxxxxx-bddd-11e2-93f8-157953225e0c}\

    DPM reports


     An unexpected error occurred on DPM server machine during a VSS operation. (ID 207 Details: VssError:The shadow copy provider had an error. Check the System and Application event logs for more information.
     (0x80042306))
     More information
    Recommended action: Retry the operation.
     Synchronize with consistency check.
     Run a synchronization job with consistency check...
    Resolution: To dismiss the alert, click below
     Inactivate

    Any ideas why this is happening?

    thanks

    Wednesday, September 18, 2013 8:44 PM

Answers

  • Hi,

    We have seen a few of these being reported lately, can I ask you to open support cases so we can help find a common denominator and help track the issue.   To be honest it seems the only way out of the condition is to delete all shadow copies for the effected data source replica volume.  We have automated scripts to help if you have many effected volumes.  

    UPDATE: 10-16-13 - The Windows support group investigating this has discovered a change in behavior in Windows 2012 on how it reacts to Disks disappearing and re-appearing when VSS shadow copies are stored on separate volumes, which is the way that DPM configures shadow copies for DPM replica volumes. There is no fix for this issue yet, but the workaround to allow new recovery points (shadow copies) to be created is to delete all shadow copies for the effected replica volume(s). This does delete all DPM recovery points for the effected data sources, but will allow you to start making new ones.   Unsure if there is an actual fix in the works from the Windows group to avoid doing that.   I recommend opening a Windows support ticket to get current status and possible alternate solution.

    UPDATE 1-14-14

    The Windows group has released a preventive fix for this issue. Please install to avoid future outages.

    WINDOWS PREVENTATIVE FIXES
    =====================
    VSS software provider error when you use DPM to back up a Windows 8 or Windows Server 2012-based computer
    http://support.microsoft.com/kb/2914152


    2929881 0x80070005 error when you run DPM jobs in Windows Server 2012 R2
    http://support.microsoft.com/kb/2929881/EN-US


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.




    Wednesday, October 2, 2013 4:58 AM
    Moderator

All replies

  • I've also started getting these for most backups after it had been working well for several months.  No changes have been made.
    Monday, September 30, 2013 11:34 PM
  • Hi,

    We have seen a few of these being reported lately, can I ask you to open support cases so we can help find a common denominator and help track the issue.   To be honest it seems the only way out of the condition is to delete all shadow copies for the effected data source replica volume.  We have automated scripts to help if you have many effected volumes.  

    UPDATE: 10-16-13 - The Windows support group investigating this has discovered a change in behavior in Windows 2012 on how it reacts to Disks disappearing and re-appearing when VSS shadow copies are stored on separate volumes, which is the way that DPM configures shadow copies for DPM replica volumes. There is no fix for this issue yet, but the workaround to allow new recovery points (shadow copies) to be created is to delete all shadow copies for the effected replica volume(s). This does delete all DPM recovery points for the effected data sources, but will allow you to start making new ones.   Unsure if there is an actual fix in the works from the Windows group to avoid doing that.   I recommend opening a Windows support ticket to get current status and possible alternate solution.

    UPDATE 1-14-14

    The Windows group has released a preventive fix for this issue. Please install to avoid future outages.

    WINDOWS PREVENTATIVE FIXES
    =====================
    VSS software provider error when you use DPM to back up a Windows 8 or Windows Server 2012-based computer
    http://support.microsoft.com/kb/2914152


    2929881 0x80070005 error when you run DPM jobs in Windows Server 2012 R2
    http://support.microsoft.com/kb/2929881/EN-US


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.




    Wednesday, October 2, 2013 4:58 AM
    Moderator
  • Has this issue been resolved? Can you please suggest?
    Wednesday, October 16, 2013 5:42 AM
  • Hi,

    As Mike mentionen the only way to solve this is to delete the shodow copies from DPM. I would also recommend you to open a case with MS support.


    Best Regards

    Robert Hedblom

    MVP DPM


    Check out my DPM blog @ http://robertanddpm.blogspot.com

    Wednesday, October 16, 2013 11:11 AM
    Moderator
  • Hi all,

    Microsoft Engg took remote of the DPM & deleted the shadow copies from DPM, now the issue has been resolved. Consistency Chk & replica creation has been started.

    But, it has deleted older Recovery points. Do you have any idea why & how. What can be done to get the recovery point back?

    Thursday, October 17, 2013 4:38 AM
  • Hi,

    Having perhaps the same issues here.  Admittedly had not checked replicas in a while to discover they were not working.  Only solution was the above steps which caused me to lose my recovery points.  I can't present DPM to the business as a credible backup and DR solution if these things happen.  Just today I was asked to restore some Access databases and could not because I had to blow away my replicas.

    At minimum I'm going to look at a different solution for tapes - I can't have all my eggs in the DPM basket at this point.  If anybody is doing mixed backup solution with DPM for disk and something else for tape please chime in.

    -Casey

    Wednesday, November 13, 2013 10:44 PM
  • So if this is the only solution, how do you delete shadow copies from DPM? I've been looking around but cant find any documentation. I am guessing it would need to be done through command line operation?
    Monday, November 18, 2013 6:38 PM
  • So if this is the only solution, how do you delete shadow copies from DPM? I've been looking around but cant find any documentation. I am guessing it would need to be done through command line operation?

    Since it kills your recover points anyway, the easiest thing for me to do was just to remove it from protection, delete recovery points on disk and add it back to protection.  

    I deleted the shadow copies via command line but that didn't seem to fix it for me - maybe I was missing some steps.  

    Sadly with DPM I'm a pro at removing and re-adding things to protection groups :-(

    Monday, November 18, 2013 6:55 PM
  • My main concern in the replica. I have very large replicas and I don't want to have to re-create them. Don't really care about the recovery points.

    I really want DPM to work but if I keep having these issues, I don't think I can stick with it.. It's pointless to backups if you have to keep deleting all the work you do and starting over all the time. Wish Microsoft would make it a bit more reliable.

    Monday, November 18, 2013 7:02 PM
  • My main concern in the replica. I have very large replicas and I don't want to have to re-create them. Don't really care about the recovery points.

    I really want DPM to work but if I keep having these issues, I don't think I can stick with it.. It's pointless to backups if you have to keep deleting all the work you do and starting over all the time. Wish Microsoft would make it a bit more reliable.

    Perhaps they could list step-by-step how to carry this procedure out (if it keeps the replica).  I agree on the product.  I'm looking into other solutions for SQL, Exchange and Sharepoint.  
    Monday, November 18, 2013 7:05 PM
  • I have written a set of batch files that will identify effected DPM replica volumes and delete the associated shadow copies automatically. 

    Make these three batch files.

    REM  SAVE AS 1-GET-DPM2012-VOLS.cmd
    ECHO OFF
    CLS
    COLOR fc
    ECHO.
    ECHO.
    ECHO DPM SQL SERVER\INSTANCE\DATABASE NAMES ARE:
    ECHO.
    reg query "HKLM\SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB" /v SqlServer
    reg query "HKLM\SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB" /v Instancename
    reg query "HKLM\SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB" /v Databasename
    ECHO.
    ECHO If Different than Localhost\MSDPM2012\DPMDB - please modify this line "osql -E -S localhost\MSDPM2012 -d DPMDB -Q" in this file using notepad.
    ECHO.
    ECHO.
    pause
    color
    REM JUST LIST DPM REPLICA VOLUME GUIDS 
    osql -E -S localhost\MSDPM2012 -d DPMDB -Q "select distinct vol.GuidName from tbl_IM_DataSource as ds join tbl_PRM_LogicalReplica as lr on ds.DataSourceId=lr.DataSourceId join tbl_AM_Server as ag on ds.ServerId=ag.ServerId join tbl_SPM_Volume as vol on lr.PhysicalReplicaId=vol.VolumeSetID and vol.Usage in (1) and lr.Validity in (1,2) order by vol.guidname" -o .\delete-me
    
    findstr \\?\ .\delete-me > Replica-vol.txt
    
    del .\delete-me
    
    
    del faulted-vol.txt
    cls
    echo.
    echo.
     FOR /F "Tokens=*" %%n IN ('type replica-vol.txt') DO (vssadmin.cmd %%n)
    
    
    

    REM SAVE AS 2-diskshadow.cmd
    diskshadow.exe /l delete-shadows.txt /s faulted-vol.txt
    

    REM SAVE AS vssadmin.cmd
    echo on
    vssadmin.exe list shadowstorage /for=%1
    if %errorlevel% NEQ 0 echo delete shadows volume %1 >>faulted-vol.txt
    

    Then run them as follows:

    1-GET-DPM2012-VOLS.cmd

    After above finishes run the below cmd file.

    2-diskshadow.cmd

    Supporting batch file that gets called - do not run this individually.

    Vssadmin.cmd

    Once this completes, run a CC and then future recovery points will be able to be made.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    • Proposed as answer by Adam.Morris Tuesday, November 19, 2013 2:52 PM
    Monday, November 18, 2013 7:44 PM
    Moderator
  • I am having the exact same problem, how do I open a ticket without paying $300 if it is a known issue. 15 years of backup exec and I decide to move to Microsoft and this happens.

    Any help without have to wipe out the old copies would be appreciated.

    Thanks 

    Jim Kiddoo

    Saturday, December 21, 2013 5:13 AM
  • Jim, The only solution is to run the above scripts. The Windows hotfix to prevent the error is not yet finished. When it's done, you can download it without cost.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Saturday, December 21, 2013 6:11 AM
    Moderator
  • Hello All,
    Same issue over here. DPM works for like a week orso and then these errors 12289 starts. In my case this doesn't happen on every volume, some DS work some don't. I can recreate the recovery points but after a week same issue.

    Also what the DPM team is investigating is happening here also. I have 2 cases that complete disks where missing from de management tab. After a reboot they where back again. Very weird, looks like something we can't 'work around' until a fix.

     Is there a fix yet?
    Greetz,
    Peter

    • Edited by Peter Stap Tuesday, December 24, 2013 11:55 AM
    Tuesday, December 24, 2013 11:50 AM
  • Hi Peter,

    This condition can only occur if windows looses access to the disks unexpectedly due to connectivity issues. double check your storage subsystem so it's more reliable until the windows fix is available.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 24, 2013 4:09 PM
    Moderator
  • Hi Mike,
    Forgot to mention, it's a DPM guest on Hyper-V (Everything 2012R2) so there's nothing wrong with the storage system. Waiting for the fix.
    Gr,
    Peter


    • Edited by Peter Stap Tuesday, December 24, 2013 6:47 PM
    Tuesday, December 24, 2013 6:46 PM
  • Hi Peter,

    What version of Windows and DPM and what kind of storage (iSCSI, Pass thru, .vhd's) are you using in the DPM Storage pool ?

    The VSS event log ID 12289 seems to be a catch all for a variety of different VSS errors.  Are you certain the details show hr = 0x80070005, Access is denied ?   Also, if you run vssadmin list shadowstorage command, does that also report an error ?    If the fixup scripts resolve the issue, then it most likely is the same problem, but to date, only storage related problems (disks disappearing unexpectedly) can cause that particular issue.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, December 26, 2013 3:11 PM
    Moderator
  • Hello Mike,

    Everything is 2012R2 with all the Windows updates available. I'm using .vhdx for the DPM storage pool. Nothing fancy with pass thru or something it's just a Server 2012R2 host with a DPM2012R2 guest VM. My eventlog error

    Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{c0db736a-6648-11e3-941c-00155d021a1a} - 00000000000002D8,0x0053c06c,0000001BD63FA3A0,0,0000001BD63F9390,4096,[0]).  hr = 0x80070005, Access is denied.

    .

    Operation:

       Query diff area for this volume


    If I run vssadmin list shadowstorage I receive an error and right at that moment the error above. What do you mean by 'fixup scripts' ?

    Friday, December 27, 2013 7:19 AM
  • Hi,

    Further up in this thread there are a set of batch files that you can make that will cleanup the faulted VSS shadowcopy associations so new shadow copies (recovery points) can be created.  


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, December 27, 2013 3:49 PM
    Moderator
  • Hello Mike,

    Ah u mean those scripts. Yes those script resolve the issue temporary but as u are aware this removes the history of the backup so is there any update about a fix yet?


    In my case the scripts removed almost every shadowcopy point so basically the whole backup is gone. Still this is no sollution and the problem will come back soon I guess.
    • Edited by Peter Stap Monday, December 30, 2013 8:27 AM
    Monday, December 30, 2013 8:25 AM
  • Hi,

    If this is a reoccurring problem for you, then I urge you to open ticket with support to help isolate and fix the intermittent storage glitches that places the VSS metadata on the effected volumes in the broken state.  From what I can tell from mail threads, the fix is still in test and barring any problems it should be released early in 2014. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, December 30, 2013 3:29 PM
    Moderator
  • FYI - Windows fix was released.

    WINDOWS PREVENTATIVE FIX
    ====================

    VSS software provider error when you use DPM to back up a Windows 8 or Windows Server 2012-based computer
    http://support.microsoft.com/kb/2914152


    2929881 0x80070005 error when you run DPM jobs in Windows Server 2012 R2
    http://support.microsoft.com/kb/2929881/EN-US


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, February 13, 2014 1:58 AM
    Moderator
  • Thanks for the info Mike, will install it asap....
    Thursday, February 13, 2014 2:07 AM
  • Does this issue also apply to Windows Server 2012 R2? If so, will a similar hotfix be released applicable to 2012 R2?
    I got the same VSS error on a DPM 2012 R2 server running on Windows Server 2012 R2.
    Thursday, February 20, 2014 12:35 PM
  • Hi,

    I don't believe Windows 2012 R2 has this problem.   If you run VSADMIN LIST SHADOWSTORAGE - does that complete without getting a provider error at the end ?


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, February 20, 2014 4:59 PM
    Moderator
  • No, I get this error:

    Error: The shadow copy provider had an error. Please see the system and
    application event logs for more information.

    Friday, February 21, 2014 6:24 AM
  • Hi,

    OK - you seem to have hit the same condition, so the workaround will be the same.  I would open a Windows 2012 R2 support case to have the windows folks investigate why you hit that.

    UPDATE: I did find out that this condition can still occur in Windows server 2012 R2 - a fix is in progress, but don't know when it will be available.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Friday, February 21, 2014 4:52 PM
    Moderator
  • Is there a way we can be notified when a fix is available please or do we just have to randomly keep checking back on this page?

    Backups are one of the most important things for companies so need to make sure I don't miss when a fix/patch is released.
    Tuesday, March 4, 2014 9:00 AM
  • Hi,

    The Windows group is responsible for publishing the fix.  I believe the KB article for Windows 2012 R2 will be 2929881, so keep an eye out for that.

    UPDATE:  This is the Windows 2012 R2 fix

    2929881 0x80070005 error when you run DPM jobs in Windows Server 2012 R2
    http://support.microsoft.com/kb/2929881/EN-US


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Tuesday, March 4, 2014 3:42 PM
    Moderator
  • Mike, when I click the link I only get a hotfix for Win8 RTM. (470825_intl_i386_zip.exe). I had this problem hit with lost iscsi targets and used your batch files to clean up the snapshots (thanks for that). Am running DPM 2012 on 2012 std. Thanks. 
    Thursday, March 6, 2014 6:15 PM
  • Hi,

    Windows-8 = Desktop operating system.
    Windows 2012 = Server version of same operating system

    Same fix applies to both.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, March 6, 2014 6:25 PM
    Moderator
  • Thanks for the reply, but I understand that the fix can be for both win8 and 2012 but when i attempt to install i got message that it was incompatible with my installation. and since it had i386 in the file-name, i wondered if there was a missing x64 version of the fix. we've looked multiple times but never see more than the win8rtm fix file. 

    Thursday, March 6, 2014 6:36 PM
  • Hi,

    You need to click on the "Show hotfixes for all platforms and languages (2)" link to show you the x64 file to download.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, March 6, 2014 6:46 PM
    Moderator
  • OOPS! didn't notice x64. My eyes only saw 8RTM! Again, thanks.
    Thursday, March 6, 2014 6:51 PM