none
RESOLVED - Backup failing on Server 2008 R2 - system writer missing

    General discussion

  •  

    System is non production, running server 2008 enterprise (localized in Dutch).

    "Bare Metal Recovery" selected, backups scheduled daily using Windows server back-up.

    Backup destination is a local disk, with plenty of disk space still free.

    backups running fine up and until 6 june.

    At that date we installed SQL server 2008 R2 and Advantage ADS (other database server).

    I recently noticed that the backups have been failing since 6/6.

    Investigation performed so far :

    -> error message in backup wizard : "the system writer is not found in the backup"

    -> confirmed that system writer is not in the writer list when doing vssadmin list writers in a cmd box.

    -> found a gazillion articles on google indicating wrong permissions on %windir%\winsxs\Filemaps\*

    reference article :  http://support.microsoft.com/kb/2009272

    most of these also mention problems with CAPI2 (cryptographic provider) but I see no evidence of that in my application log, only event ID 517 is present, indicating error 2155347997.

    I follow the article instructions anyway and update file permissions but system writer is still not listed.

    Then the article mentions that Event ID 8213, indicating permission issues, might be showing, but it's NOT.

    registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl is already on 1

     

    -> NO log is created under C:\Windows\Logs\WindowsServerBackup\ during the backup attempt.

    -> I was able to restore the winsxs\filemaps folder from the last successfull backup without any problem.
    log file was created during that restore.

    -> I'm not a big hero with process monitor but quickly comparing outputs to a working machine yields no obvious culprit. It's too verbose to really dig in without advanced knowledge of what you're looking for

    -> using vsstrace.exe I get this output repeating over and over.
    There are error messages in there but I have no idea what they mean and searching the net doesn't give me anything :

    ERROR 1 CONTEXT (repeaded approx 300 times)

     

    [10:14:23.075 P:150C T:1388 INCTYPEH(0256) GEN] ENTER[CVssAutomaticLock2::CVssAutomaticLock2]
    [10:14:23.075 P:150C T:1388 INCTYPEH(0197) GEN] ENTER[CVssCriticalSection::Lock]
    [10:14:23.075 P:150C T:1388 INCTYPEH(0197) GEN] EXIT[CVssCriticalSection::Lock] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:23.075 P:150C T:1388 INCTYPEH(0256) GEN] EXIT[CVssAutomaticLock2::CVssAutomaticLock2] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:23.075 P:150C T:1388 CORPRVMC(0363) COORD] ENTER[CVssProviderManager::GetProviderItfArrayInternal]
    [10:14:23.075 P:150C T:1388 CORPRVMC(0363) COORD] EXIT[CVssProviderManager::GetProviderItfArrayInternal] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:23.075 P:150C T:1388 CORPRVMC(0468) COORD] ENTER[CVssProviderManager::GetProviderInterfaceInternal]
    [10:14:23.075 P:150C T:1388 CORPRVMC(0470) COORD] CONTEXT[CVssProviderManager::GetProviderInterfaceInternal] Adding context: 'Aanroepbare interface voor deze provider verkrijgen' (0)
    [10:14:23.075 P:150C T:1388 CORPRVMC(0471) COORD] CONTEXT[CVssProviderManager::GetProviderInterfaceInternal] Adding context: 'Provider-id' = '{b5946137-7b9f-4925-af80-51abd60b20d5}' (2)
    [10:14:23.076 P:150C T:1388 CORPRVMC(0472) COORD] CONTEXT[CVssProviderManager::GetProviderInterfaceInternal] Adding context: 'Klasse-id' = '{00000000-0000-0000-0000-000000000000}' (2)
    [10:14:23.076 P:150C T:1388 CORPRVMC(0473) COORD] CONTEXT[CVssProviderManager::GetProviderInterfaceInternal] Adding context: 'Context van snapshot' = '-1' (2)
    [10:14:23.076 P:150C T:1388 CORSOFTC(0155) COORD] ENTER[CVssSoftwareProviderWrapper::AddRef]
    [10:14:23.076 P:150C T:1388 CORSOFTC(0156) COORD] Provider Wrapper AddRef(00000000004006D0) 2 --> 3
    [10:14:23.076 P:150C T:1388 CORSOFTC(0155) COORD] EXIT[CVssSoftwareProviderWrapper::AddRef] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:23.076 P:150C T:1388 CORSOFTC(0163) COORD] ENTER[CVssSoftwareProviderWrapper::Release]
    [10:14:23.076 P:150C T:1388 CORSOFTC(0164) COORD] Provider Wrapper Release(00000000004006D0) 3 --> 2
    [10:14:23.076 P:150C T:1388 CORSOFTC(0163) COORD] EXIT[CVssSoftwareProviderWrapper::Release] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:23.076 P:150C T:1388 CORSOFTC(0123) COORD] ENTER[CVssSoftwareProviderWrapper::QueryInternalInterface]
    [10:14:23.076 P:150C T:1388 CORSOFTC(0128) COORD] Error while obtaining an interface interface 0x80004002
    [10:14:23.076 P:150C T:1388 CORSOFTC(0123) COORD] EXIT[CVssSoftwareProviderWrapper::QueryInternalInterface] Time spent: 00:00:00-0000; total: 0; HRESULT: 0x80004002
    [10:14:23.076 P:150C T:1388 CORSOFTC(0155) COORD] ENTER[CVssSoftwareProviderWrapper::AddRef]
    [10:14:23.076 P:150C T:1388 CORSOFTC(0156) COORD] Provider Wrapper AddRef(00000000004006D0) 2 --> 3
    [10:14:23.076 P:150C T:1388 CORSOFTC(0155) COORD] EXIT[CVssSoftwareProviderWrapper::AddRef] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:23.076 P:150C T:1388 CORPRVMC(0468) COORD] EXIT[CVssProviderManager::GetProviderInterfaceInternal] Time spent: 00:00:00-0000; total: 0; HRESULT: 0x80004002
    

     

    Error message 1 :

    [10:14:23.076 P:150C T:1388 CORSOFTC(0123) COORD] ENTER[CVssSoftwareProviderWrapper::QueryInternalInterface] 
    [10:14:23.076 P:150C T:1388 CORSOFTC(0128) COORD] Error while obtaining an interface interface 0x80004002

     

     

    ERROR 2 CONTEXT (repeaded 7 times) :

     

    [10:14:33.503 P:04A0 T:14E4 WRTWRTIC(2274) WRITER] EXIT[CVssWriterImpl::RequestWriterInfoInternal] Time spent: 00:00:00-0015; total: 0xf; HRESULT: 0
    [10:14:33.505 P:150C T:1388 WRTASRWC(0183) WRITER] EXIT[CVssAsrWriterBackup::_AddComponents] Time spent: 00:00:00-0015; total: 0xf; HRESULT: 0
    [10:14:33.505 P:150C T:1388 WRTASRWC(0627) WRITER] ENTER[CVssAsrWriterBackup::TranslateWriterError] 
    [10:14:33.505 P:150C T:1388 WRTASRWC(0628) WRITER] TranslateWriterError - 0x00000000
    [10:14:33.505 P:150C T:1388 WRTASRWC(0633) WRITER] TranslateWriterError - 0x00000000
    [10:14:33.505 P:150C T:1388 WRTASRWC(0627) WRITER] EXIT[CVssAsrWriterBackup::TranslateWriterError] Time spent: 00:00:00-0000; total: 0; HRESULT: 0
    [10:14:33.505 P:150C T:1388 WRTASRWC(0146) WRITER] EXIT[CVssAsrWriterBackup::OnIdentify] Time spent: 00:00:00-0015; total: 0xf; HRESULT: 0

     

    -> vssadmin list writers output :
    vssadmin 1.1 - Opdrachtregelbeheerprogramma voor Volume Shadow Copy-service
    (C) Copyright 2001-2005 Microsoft Corp.
    
    Naam schrijver: 'Task Scheduler Writer'
     Schrijver-id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
     Id van schrijverinstantie: {1bddd48e-5052-49db-9b07-b96f96727e6b}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'VSS Metadata Store Writer'
     Schrijver-id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
     Id van schrijverinstantie: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'Performance Counters Writer'
     Schrijver-id: {0bada1de-01a9-4625-8278-69e735f39dd2}
     Id van schrijverinstantie: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'ASR Writer'
     Schrijver-id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
     Id van schrijverinstantie: {7ddb5279-28d6-4f25-af34-535b05c33460}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'Shadow Copy Optimization Writer'
     Schrijver-id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
     Id van schrijverinstantie: {5c4742bd-a15c-4769-9e11-7422623c4f35}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'SqlServerWriter'
     Schrijver-id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Id van schrijverinstantie: {6b99c086-1f34-4cb8-bea5-887846f86a75}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'Registry Writer'
     Schrijver-id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
     Id van schrijverinstantie: {7a76da99-a6f1-4b4b-85f4-5040079f6487}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'COM+ REGDB Writer'
     Schrijver-id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
     Id van schrijverinstantie: {93839243-d67d-49d2-856e-49a1ec4dc87b}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    Naam schrijver: 'WMI Writer'
     Schrijver-id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
     Id van schrijverinstantie: {2ae99e39-7c2c-4fbb-809a-9d42b4aba1ff}
     Status: [1] Stabiel
     Laatste fout: Geen fout
    
    

     

    notice that also the BITS writer is missing, but the others seem just fine ...

    -> followed instructions in this article : http://briandrab.spaces.live.com/blog/cns!B69C8C4D664E3F01!252.entry?sa=609120164

    moved all files under filemaps to a different folder but that didn't work either so i just restored them.

    -> I probably could restore the backup from 6/6 completely and reinstall sql + ads, but I want to find out what is happening before I do that, I find not much info on the web over what mechanism makes the writers visible exactly.
    This would be a lot of work with no guarantee that things don't break again.

    At this point I welcome any suggestion, but i'd like to see some arguments as to why it would be a good idea, not just try this or try that, as I'm trying to understand the whole vss setup a little better.

    UPDATE : BITS writer came back in stable state after reboot, so only system writer is still missing.

    • Changed type Christine Fok Thursday, September 2, 2010 12:55 AM tracked in separate support case
    • Edited by chimf Monday, September 6, 2010 12:17 PM
    Thursday, August 5, 2010 9:12 AM

All replies

  • will it be possible to raise a support case and share the SR ID?
    ------- this information is provided as-is without any warranties, implicit or explicit.
    Friday, August 6, 2010 4:01 AM
  •  

    OK I'll try that next week and post feedback here.

    cheers

    Friday, August 6, 2010 10:30 AM
  • support call has been opened 110081649056099

    I'll do my best to update this page for future reference.

    Monday, August 16, 2010 1:40 PM
  • looks like we got somewhere, kudos to the MS support engineer,

    turns out this was in fact due to incorrect file permissions, namely on this folder :

    %windir%\winsxs\temp\PendingRenames

    I already had these groups & permissions :

    -SYSTEM : read & execute, list folder contents, read
    -Administrators ([hostname]\Administrators) : read & execute, list folder contents, read
    -Users ([hostname]\Users) : read & execute, list folder contents, read
    -TrustedInstaller : full control

    After adding these groups :

    -Domain Admins : read & execute, list folder contents, read
    -Domain Users : read & execute, list folder contents, read

    and restarting cryptsvc the backup started working and the vss system writer showed up again.

    I'm happy my backup is working but I'm not sure why though, and I did ask the support engineer if he could try to find out :

    I am logged on as a domain user, so it makes sense to add read permissions for that group, but on the other hand this domain user is a member of the local admin group, which already had the same read permissions as the ones we added for those other groups.

    So why didn't he pick up the local admin group permissions to which this user belongs ?

    My knowledge of the permission precedence mechanism is not deep enough to be able to explain this, but I am curious to find out so if someone can shed some light on this be my guest.

    As a side note, I found a lot of similar cases via google where users complained that permissions where reset after reboot so i tried that but mine are still fine.

    second side note, I mentioned in the original post that I saw no CAPI2 errors, that changed later on, but they did not show up when taking a backup, only when restarting cryptsvc, and even then not consistently. so even if you are not seeing those errors in your output you should still consider this a valid troubleshooting option.

     

     

     

    Thursday, September 2, 2010 10:10 AM
  • Hello,

    I have tried all solutions from here and other places for this issue, but nothing helped me till I have found some short notice about using sfc /scannow command for dealing with some corrupted files in system files.

    I have tried it, it found some corrupted files, fixed most automatically and it's working now.

    Hope it will work for You all too.

    Best Regards,

    Przemek

    Monday, December 13, 2010 7:26 PM
  • @chimf,

    I had the same issue, and performing the following helped me as well:

    After adding these groups to %windir%\winsxs\temp\PendingRenames:

    -Domain Admins : read & execute, list folder contents, read
    -Domain Users : read & execute, list folder contents, read

    Monday, March 5, 2012 8:19 PM
  • Hi Brian,

    Thanks for the suggestion. but it did not work for me.

    Sunday, May 13, 2012 3:14 PM
  • Hi chimf,

    I tried many ways from many places, but non of them could fix this issue on my server ONLY except yours. Thanks very much for your solution. However, I still have some questions on this issue.

    1. I think this is not the root case because the security settings of the folder %windir%\winsxs\temp\PendingRenames on my others servers don't have Domain Admins or Domain Users listed, but they are all working fine.
    2. There were no problems in the past for this server. And I didn't change any permission settings on the folder %windir%\winsxs\temp\PendingRenames.

    Thanks,
    高麻雀

    • Edited by 高麻雀 Thursday, June 20, 2013 3:53 PM missed one point
    Thursday, June 20, 2013 3:11 PM
  • Hi,

    I had a similar problem but the above solution does not work for me. Any ideas

    Shailender

    Tuesday, December 10, 2013 11:58 AM
  • same here I had Windows 2008 R2 server which is suddenly failing on backup after doing root cause came to know VSS service is not showing under services.

    when I do run VSSADMIN List writers got following error.

    Error: A Volume Shadow Copy Service component encountered an unexpected error.
    Check the Application event log for more information.

     

    Tuesday, January 14, 2014 6:43 AM
  • Hi,

    I had a similar issue, in my case 'System Writer' was not present when I run 'vssadmin list writers' - I tried numerous solutions, but it didn't help. Even this one, from above.

    I bumped on one post here on technet, where they said there is a hotfix.

    This one hotfix helped me solve my issue: http://support.microsoft.com/kb/2807849

    System Writer is showing when I run vssadmin list writers command.


    If this helped you resolve your issue, please mark it Answered, thanks! Kind regards

    Monday, June 23, 2014 11:30 AM
  • was getting error when running "vssadmin list writers":

    Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

    Details:

    AddCoreCsiFiles : GetNextFileMapContent() failed.

    System Error:

    The parameter is incorrect.

    --

    Tried permissions and several other issues, did not fix problem. I believe I just resolved.

    Appears to have been corrupted files in winsxs/filemaps.

    Running process monitor sysinternals tool, correlating the timestamp of the event and look at the file Winsxs being accessed by the last time before and after the event time.  (before is file read, after is file close).  Jump to that file and remove it from winsxs directory, these appear to be corrupted files.  Had to repeat several times and found 5 corrupted files.  Opening one of these files showed it as containing a string of null bytes.  Others appeared to have data from a casual glance but I am guessing they were also corrupted -

    program_files_reference_assemblies_microsoft_framework_v3.0_44577d982216c291.cdf-ms

    program_files_windows_nt_accessories_156d2b9b22040474.cdf-ms

    program_files_x86_common_files_system_en-us_514b68afe0bd6701.cdf-ms

    program_files_x86_common_files_system_msadc_607f0693c9effa29.cdf-ms

    program_files_x86_reference_assemblies_microsoft_framework_v3.0_1dfad1527dc1078c.cdf-ms

    After removing these files vssadmin list writers shows system writer and no more CAPI2 events.

    Tuesday, July 22, 2014 4:54 PM
  • Estou com este mesmo problema na empresa no final como foi resolvido?
    Thursday, March 5, 2015 6:40 PM
  • Yes !

    This KB helps really well, at least in a Windows Server 2008 R2 SP1 version !

    Regards.

    FB

    Wednesday, July 29, 2015 1:59 PM
  • I solved this error repairing the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\Diag\System Writer. I export/import by another server and reboot system.
    Thursday, December 31, 2015 1:38 PM
  • I have  " System writer "  missing issue for windows 2008 Standard   SP2 server.

    Due to this Disaster recovery backup is failing.

    Tried all above steps .

    Re re-registered the backup writers and rebooted the server.

    Executed the permission script 

    Still System writer is missing .  anyone can help this issue.

    Friday, October 20, 2017 9:22 AM
  • Hello,

    I have come across System Writer missing issue multiple times. The following steps have ALWAYS fixed the issue for me(Works on Windows 2008 R2 SP1 & Windows 2012 R2).

    1. Ran cmd to reset the eventsystem permission - (Copy paste and run the whole thing between the quotes in CMD)

    "  Sc.exe sdset eventsystem D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)  "
    2. Restarted Com + Event System 
    3. Restarted Cryptographic server.
    4. Restarted Volume Shadow Copy service
    5. Ran VSS admin list writers. System writer is back!

    Safe to run in production and did not cause any issues, at least in our environment. No reboot needed
    • Edited by Rahul9911 Wednesday, November 22, 2017 2:48 PM
    Wednesday, November 22, 2017 2:36 PM
  • ran your cmd to reset eventsysten and got:

    ---------------------

    [SC] OpenService FAILED 5:

    Access is denied.

    --------------------------------------

    verified im local admin on the box.

    Friday, December 1, 2017 11:54 PM
  • Thanks lot for your commands.  my issue resolved.
    Thursday, February 1, 2018 12:23 AM
  • Stop the "Cryptographic Services"

     

    Copy the following files in 'inf' folder from other working same OS version server and same service pack.

     

    "setupapi.ev1"

    "setupapi.ev2"

    "setupapi.ev3"

     

    Start the "Cryptographic Services"

     

    Restart the computer if needed.


    Monday, May 7, 2018 7:50 PM
  • Thanks   Jack-Jaga.

    It is nice way and good solution. many times it resolves my issues.  Great thanks. Saved my time.

    Stop the "Cryptographic Services"

     

    Copy the following files in 'inf' folder from other working same OS version server and same service pack.

     

    "setupapi.ev1"

    "setupapi.ev2"

    "setupapi.ev3"

     

    Start the "Cryptographic Services"

     

    Restart the computer if needed.



    Friday, October 12, 2018 6:34 PM