DPM 2007 Server 2008 System State Backup System Writer missing
Hello,
I am working with a client and we are running into issues backing up the Windows Server 2008 System State.
We are running DPM 2007 SP1. We have Windows Server Backup installed on the Server 2008 system, and what we are seeing is that in DPM the System State Backup of the server, the status of the Protection State is "Replica is inconsistent".
The alert in DPM reads the following
Affected area: Computer\SystemState
Occurred since: 3/3/2009 5:31:19 PM
Description: The replica of System State Computer\SystemState on <servername> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent (ID 3106)
DPM failed to create the system state backup. If you are trying to create the system state of a Windows 2008 Server operating system, verify that the Windows Server Backup (WSB) is installed, and that there is enough free disk space on the protected server to store the system state. (ID 30214 Details: Internal error code: 0x809909FB)
WSB is installed on the protected system and it has plenty of space.
When I look at the logs on the Server 2008 system, I see the following errors:
Log Name: Microsoft-Windows-Backup
Source: Microsoft-Windows-Backup
Date: 3/3/2009 5:11:30 PM
Event ID: 5
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: <servername>
Description:
Backup started at '3/3/2009 4:11:26 PM' failed with following error code '2155348226'.Log Name: Application
Source: Microsoft-Windows-CAPI2
Date: 3/3/2009 8:46:23 PM
Event ID: 513
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: <servername>
Description:
Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.
Details:
AddCoreCsiFiles : BeginFileEnumeration() failed.
System Error:
Access is denied.Log Name: Application
Source: Microsoft-Windows-Backup
Date: 3/3/2009 5:12:58 PM
Event ID: 517
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: <servername>
Description:
Backup started at '3/3/2009 4:12:55 PM' failed with following error code '2155348226' (System writer is not found in the backup.). Please rerun backup once issue is resolved.
I followed the instructions of the following article to make sure that the COM permissions are correct.
http://technet.microsoft.com/en-us/library/cc734021.aspx
Everything is correct, except that when I look at the List of Writers (vssadmin list writers), I do not see the "System Writer" which is what WSB/DPM 2007 is trying to use in the System State Backup.
My question then is, how do I add the System Writer into the subscriber list for VSS?
Answers
Hi all,
Please try the steps below:
This problem occurs if NTFS permissions have been removed to the
C;\WINDOWS\WINSXS\FILEMAPS directory and / or files.
Did a process monitor and found that we're receiving an access is denied on
$$.cdf-ms (this is the first file) other files can be the cause of the problem
too.
To resolve this issue please run the following command from an administrative CMD
prompt:
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users: (RX)
System Writer will now show up and you can take a backup
regards,
Lauwrensz- Proposed As Answer byLauwrensz Bakker [MSFT] Friday, March 20, 2009 7:48 AM
- Marked As Answer byavaz2000 Thursday, April 30, 2009 2:07 PM
- Our System State backup was working fine until we installed about 15 of the most recent Windows Updates last week. I checked the Effective Permissions on the the %windir%\winsxs\filemaps folder and they were correct. However I decided to try Lauwrensz's permissions fix and it worked. These were my steps:
- Overwrote permissions as follows:
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)- Stopped and started Cryptographic service
net stop cryptsvc
net start cryptsvc- Verified “System Writer” showed up in list of writers using "vssadmin list writers"
- Successfully tested the system state backup using the DPM console
There must have been a file/files down deeper that had incorrect permissions. Backup is now working again. Thanks Lauwrensz and Sushil!- Marked As Answer byavaz2000 Thursday, April 30, 2009 2:07 PM
All Replies
- Thanks for reporting the issue.
We have seen similar issue being reported earlier also. Please try granting read access to the Authenticated Users and rerun the backup to check if it works.
Thanks
Sandeep
MSFT - Hello Sandeep,
Thank you for the prompt reply. I tried your suggestions and I am still
getting the same issue where the System Writer is not being subscribed, thus
I get the Access Denied error in the CAPI2.
I noticed that when certain services startup, new writers will get added
into the list shown in "vssadmin list writers". What DLL and or service
launches the System Writer?
What I am wondering is this, assuming that the Writer list is composed of
the approved "accounts" that can interface with VSS, then the Windows Server
Backup is using an "account" that has not been approved to access VSS, and
that is why it is being denied by the Cryptography service.
~avaz2000 - System Writer is launched as a part of the Cryptography service (cryptsvc). If that service fails to launch, System Writer will not show up in the 'vssadmin list writers' list. The problem here is not about Windows Sever Backup not being able to interface with VSS, but the fact that Syste-Writer is missing in the list of writers reported by VSS, so Windows Server Backup aborts the backup, as it is an essential component for System-State.
Can you try running the following commands in that order and let us know if the event (ID: 12289) is generated? Or does the Cryptographic service starts normally? If it fails, what is the error in the eventlog?
1. net stop cryptsvc
2. net start cryptsvc
Also, could you tell us if this is an AD server on which you are trying the backup?
Thanks
Sandeep
MSFT Hello avaz2000 and Sandeep,
I'm looking at the exact same problem on one of our costumers Exchange Servers. Protection of the storage groups works fine, but the system state is still not backing up because the VSS system writer is unavailable. I have checked the COM permissions and also added the authenticated users to the AdminSDholder ACL with read permissions. Still no luck. Do you have any further suggestions we can do to fix this problem?
Thanks,
MarkHi all,
Please try the steps below:
This problem occurs if NTFS permissions have been removed to the
C;\WINDOWS\WINSXS\FILEMAPS directory and / or files.
Did a process monitor and found that we're receiving an access is denied on
$$.cdf-ms (this is the first file) other files can be the cause of the problem
too.
To resolve this issue please run the following command from an administrative CMD
prompt:
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users: (RX)
System Writer will now show up and you can take a backup
regards,
Lauwrensz- Proposed As Answer byLauwrensz Bakker [MSFT] Friday, March 20, 2009 7:48 AM
- Marked As Answer byavaz2000 Thursday, April 30, 2009 2:07 PM
Did the solution proposed by Lauwrensz work for you?
------- this information is provided as-is without any warranties, implicit or explicit.- I am having the same problem with backup, but on Vista Ultimate 64bit. Incredible, but I found your replies in Google almost in realtime (only 20 hours passed).
I did not have System Writer too. After following Lauwrensz' instructions and restarting cryptsvc I can see System Writer in the list. By the way, you have an error in the last icacls commend (extra space after : makes the command invalid).
I then restarted the system and no System Writer in the list again! I executed the same commands again and restarted cryptsvc and got System Writer back again. Strange. Will see what happens after next restart.
Anyway, even with the System Writer running I still ge the same error. I get the following sequence of errors actually when running Vista backup:
Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object. Details: AddCoreCsiFiles : BeginFileEnumeration() failed. System Error: Access is denied.
Then
two times within 3 minites of each other while backup is running.
Then
Warning: File backup was cancelled by the user. (This is of course bulls...t, I did not cancel it).
File backup failed. The error is: The system cannot find the file specified. (0x80070002). And then
The shadow copies of volume C: were aborted because of an IO failure on volume C:. source: volsnap
- Our System State backup was working fine until we installed about 15 of the most recent Windows Updates last week. I checked the Effective Permissions on the the %windir%\winsxs\filemaps folder and they were correct. However I decided to try Lauwrensz's permissions fix and it worked. These were my steps:
- Overwrote permissions as follows:
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)- Stopped and started Cryptographic service
net stop cryptsvc
net start cryptsvc- Verified “System Writer” showed up in list of writers using "vssadmin list writers"
- Successfully tested the system state backup using the DPM console
There must have been a file/files down deeper that had incorrect permissions. Backup is now working again. Thanks Lauwrensz and Sushil!- Marked As Answer byavaz2000 Thursday, April 30, 2009 2:07 PM
- This is strange. I restarted the server and the System Writer is no longer available. I first tried net stop cryptsvc and net start cryptsvc but that did not fix it. I then overwrote the permissions again, restarted the Cryptographic services and System Writer is now available. Any thoughts on what is happening?
Additional note - I tried another restart and again System Writer was no longer available. Restarting the Cryptographic services by itself worked this time. - 1. is the issue still reproducable on your machine with a restart as state above? if so - please revert.
2. if there are some unique things about the configuration of the mahcine and software that is running on the machine - do provide the data.
------- this information is provided as-is without any warranties, implicit or explicit.- Edited bySushil.Baid [MSFT]MSFT, OwnerFriday, April 03, 2009 5:28 AMadded more questions
- My System Writer lives again! THANK YOU for posting this!
Promotcho Bad Self! http://promotely.com - These answers worked for me too, thanks for sharing the solution!
- These answers worked for me too, thanks for sharing the solution! I have not restarted the server yet but the backup worked after implementing the changes.
Thank you very much Lauwrensz Bakker!!! This has corrected my issue.
-avaz2000- Did anyone from Microsoft submit a defect to the team supporting the VSS System Writer? I doubt if this issue should prevent the System Writer from completely not working. If it does, then there need to be more warnings .. i.e. event log entries. The one event log entry reported is for the cryptographic service and that doesn't imply the VSS System Writer to most people.
By the way, I also had this issue on a system. Thanks for the workaround to get backups working correctly again. - Since I first used Lauwrensz Bakker's fix I've had three reboots. Two of them caused the System State backups to start failing again. One reboot (after an update to the DPM agent) did not cause the System State backups to fail but they did start failing 3 weeks later on their own. LB's fix worked to correct the problem each time.
The server's only function is running Exchange 2007 - and primarily as a backup to another identical server running Exchange 2007 that is not having the problem. I followed the steps that Lauwrensz had recommend, but I'm still getting the same error.
After undertaking the steps I type in "vssadmin list writers" I see the "System Writer":
State: [1] Stable
Last error: No error
When I try to Perform a consistency check at the DPM admin console and it still failed whit the same error.
Is there another solutions to solve this problem?- We are using an other Backup solution but looks like we got the same problem. CAPI2 Event Log Entry
failing backups and missing system writer.
Problem is system writer dissappears after every reboot. the security settings from $$.cdf-ms are resetted after every reboot. Besides that backup doesn't work even after changing the security settings on the filemaps folder.
Any info from MS what caused this or even more hox to fix it?
EDIT: Today noticed that if I open the Device Manager and try to recieve volume information from the devices explorer crashes and I recieve an device driver error. Disconnected all USB devices, started backup and it worked. I will check if after a reboot, which resets the security settings inside %windir%\winsxs\filemaps\ , if backups still work or not without changing the settings.
I don't know why but know the security settings aren't reset anymore after a reboot. No Error in the Eventlog anymore, backup works. And USB-HDD definitly had a problem. Windows doesn't even recognise it anymore as a HDD.
So I don't know if it actually only was a HDD problem or a combination.- Edited byDeatheye Friday, June 19, 2009 12:09 PM
- Well HDD trown away, backup problem still there after reboot till I set the security settings again -.-
Still wonder how that started cause the backup was running for months. Our System State backup was working fine until we installed about 15 of the most recent Windows Updates last week. I checked the Effective Permissions on the the %windir%\winsxs\filemaps folder and they were correct. However I decided to try Lauwrensz's permissions fix and it worked. These were my steps:
- Overwrote permissions as follows:
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)- Stopped and started Cryptographic service
net stop cryptsvc
net start cryptsvc- Verified “System Writer” showed up in list of writers using "vssadmin list writers"
- Successfully tested the system state backup using the DPM console
There must have been a file/files down deeper that had incorrect permissions. Backup is now working again. Thanks Lauwrensz and Sushil!
Did the abowe and it worked...THXThis is strange. I restarted the server and the System Writer is no longer available. I first tried net stop cryptsvc and net start cryptsvc but that did not fix it. I then overwrote the permissions again, restarted the Cryptographic services and System Writer is now available. Any thoughts on what is happening?
Same problem over here. After each reboot, the system writer is gone...
Additional note - I tried another restart and again System Writer was no longer available. Restarting the Cryptographic services by itself worked this time.Also after some running time it's suddenly gone and backups will fail :(At this moment I've put the takeown and icalcs commands into my backup script.This is a workaround but isn't a fix.- Proposed As Answer byaalbany Wednesday, July 01, 2009 10:06 AM
- Sorry about the above "propose as answer".
What I mean to say is that we have had similar issues in a Windows 2008 domain in which we have removed the rights for "Authenticated Users' from the "AdminSDHolder" object and replaced them with similar rights for the "SELF" account. To fix issue of the Cryptographic service not starting the VSS "System Writer" we found we needed to add rights for the "NETWORK SERVICE" account to the AdminSDHolder object (rights are "List contents", "Read all properties" and "Read Permissions" for "This object only"). Then needed to wait for an hour to ensure the changes to the AdminSDHolder rights had been applied. Then restarting the Cryptographic service was no longer getting any errors (it had been getting a VSS Event ID 12289 error followed by a CAPI2 Event Id 512 error). - My last post on an other thread regarding this:
For us it looks like the cause was lying in this two foldes and files / subfolders:
winsxs\filempas
winsxs\temp
The wrong NTFS rights inside winsxs\temp seem to somehow overwrite the rights inside winsxs\filemaps. So as long as the error exists inside the temp folder it will reset the right on the filemaps folder.
Did the following inside command shell
Takeown /f %windir%\winsxs\filemaps /a
icacls %windir%\winsxs\filemaps /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps /grant "BUILTIN\Users:(RX)"
icacls %windir%\winsxs\filemaps /grant "Administratoren:(RX)"
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant "BUILTIN\Users:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "Administrators:(RX)"
Takeown /f %windir%\winsxs\temp\PendingRenames /a
icacls %windir%\winsxs\temp\PendingRenames /grant "Administrators:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames /grant "BUILTIN\Users:(RX)"
Takeown /f %windir%\winsxs\temp\PendingRenames\*.* /a
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "Administrators:(RX)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "BUILTIN\Users:(RX)"
As a side note:
I think this can also cause severe problems in case you want to upgrade a system with this error.
winsxs\temp had no NTFS right. There was just no account inside the ntfs rights register. So no access for local system, trusted installer or anything at all.
Tried to install SP2 on that machine.. totaly fucked it up. Now it even tells me to install SP1! on a windows server 2008.
Not sure if this really was the cause but it's definitly not a good thing. So for me this means I check for this error before I install any updates... - Thanks, this fixed the issue for me, but I don't understand why it would be messed up on a brand-new, clean install of Server 2008 plus the DPM client.
- Thank you very much Lauwrensz Bakker!!! This has corrected my issue.
- Thanks guys, this System writer fix script works for me too. However, I have to run the script every few weeks on both of my clustered 2008 file servers. I wonder why this keeps breaking?Thanks,
Gary
Gary Evans - Hi Gary,
If this issue repros for you, I suggest you open a support case by following instructions in this link:
https://support.microsoft.com/oas/default.aspx?gprid=14134&st=1
Our support team would get in touch with you to help you resolve this issue.
We have not been able to isolate this issue in our local testing so far. It would be great if you could help us get to the root of the issue so that many other customers running into this issue get benefitted.
Thanks,
Sriram [MSFT] - Hi,
I also have this problem on some servers. The fix work. But, after installing a new hotfix (.MSU) package, the problems are back.
Therefore, the Update Installer may cause these problems.
I had rewrite the script to
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "BUILTIN\Administrators:(F)"
icacls %windir%\winsxs\filemaps\*.* /setowner "NT Service\trustedinstaller"
icacls %windir%\winsxs\filemaps\*.* /reset
This change the owner to "trustedinstaller". But this also won't help.
I had currently open a supoort case about this. - Now, I think, I had found the problematic upate:
KB951847 x64. This is reproducible, if this update is installed using WSUS "at shutdown". If this update installed in
interactive mode using Windows Update, all works fine.
This update may set a wrong permission to "winsxs\temp\PendingRenames". These wrong permission are moved after reboot to "filemaps". In interactive mode only wrong permission are set to files in "winsxs\temp\PendingRenames".
The case is still open. Hi again,
now, it becames diffcult. I understand, why this case is difficult to reproduce.
The problem does not happen, if KB951847 installed stand alone, without any other fix.
The update must be installed together with other fixes.
If the directory "winsxs\temp\PendingRenames" monitored remotely using c$ share, there may happens following:
(with WSUS 3.0 SP2)
KB951847 alone:
- files created in winsxs\temp\PendingRenames and removed (moved ?) quickly. The ACl of PendingRenames leave unchanged
KB951847 with KB973687 at once
- KB 973687 is installed at 1th and leave files in PendingRenames for the next reboot. ACL was unchanged.
But now, KB951847 leave also it files in PendingRenames. At the end of the installation, i lose acces to PendingRenames, the ACL has been changed to SYSTEM account only.
If, after the next reboot, a 3rd update installed, files in PendingRenames inherited the wrong ACl from this dir. These files are moved to "filemaps" with the wrong ACL.
This cause the "systen writer" fail.
Make sure, all updates must be installed "at shutdown".
Hope this help. I'am now out of business of for 2 weeks...
PS: A quick way to reproduce: Use a fresh 2008 x64 SP2 installation make it member to a domain an connect it to a WSUS 3.0 SP2. Now permit all available upates (including non security and KB951647) and install it at shutdown.
The 2nd: Make the same on a other fresh installation (or, on ESX, revert to a snapshot ;) ) and install all updates, except KB951847.- HeyAny news about this problem? Hotfix or Patch available?We have exactly the same on our Exhange 2007 server and this bug is existing since months.Regards
I combined multiple scripts, because it seems that something is reseting ACLs and breaking them in multiple folders...
REM*************************
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "BUILTIN\Administrators:(F)"
icacls %windir%\winsxs\filemaps\*.* /setowner "NT Service\trustedinstaller"
icacls %windir%\winsxs\filemaps\*.* /reset
Takeown /f %windir%\winsxs\Temp\PendingRenames\* /a
icacls %windir%\winsxs\Temp\PendingRenames\*.* /grant "BUILTIN\Administrators:(F)"
icacls %windir%\winsxs\Temp\PendingRenames\*.* /setowner "NT Service\trustedinstaller"
icacls %windir%\winsxs\Temp\PendingRenames\*.* /reset
Takeown /f %windir%\winsxs\Temp\PendingDeletes\* /a
icacls %windir%\winsxs\Temp\PendingDeletes\*.* /grant "BUILTIN\Administrators:(F)"
icacls %windir%\winsxs\Temp\PendingDeletes\*.* /setowner "NT Service\trustedinstaller"
icacls %windir%\winsxs\Temp\PendingDeletes\*.* /reset
Takeown /f %windir%\winsxs\Temp\* /a
icacls %windir%\winsxs\Temp\*.* /grant "BUILTIN\Administrators:(F)"
icacls %windir%\winsxs\Temp\*.* /setowner "NT Service\trustedinstaller"
icacls %windir%\winsxs\Temp\*.* /reset
PauseREM*************************************
Try this batch file and see if it helps, i hope it will on my end; I hope it will not return, as well as i hope i didnt miss any other folders...
