Answered by:
DPM2010 System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1

Question
-
DPM2010 Version (3.0.7707.0) System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1
I'm trying to utilize BMR protection on two DPM Servers but it fails with DPM based protection or Windows Serverbackup protection. Both DPM based and Windows Server Backup protection work successfully when I only select System
State protection. I had BMR DPM based protection on one of the servers until 10 days ago when it all-of-a-sudden
stopped working. The failures seem to be related to or caused by ASR Writer retryable errors. I found the ASR /
BMR VSS System State protection hotfix (Article ID: 983425) but it does not apply to 2008 R2. I have the same
problem in R2. Does anyone know of a work-a-round or fix for R2 BMR based protection on servers with 400 plus
volumes? I have included error and log information from DPM, Event Viewer, Windows Server Backup and vssadmin list
writers output. Is there another hotfix that I'm not finding so far? Any assistance would be greatly appreciated.
Thank you.
DPM2010 Error message failures:
Affected area: Computer\System Protection
Occurred since: 5/4/2012 2:09:18 PM
Description: The replica of System Protection Computer\System Protection on PHQDPM04.ppd.ci.phoenix.az.us isinconsistent with the protected data source. All protection activities for data source will fail until the replica
is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points
cannot be created until the replica is consistent.
For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup
inconsistent databases, run a consistency check on the farm. (ID 3106)
DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered anerror (WSB Event ID: 521, WSB Error Code: 0x807800A1). (ID 30229 Details: Internal error code: 0x809909FB)
More information
Recommended action: For resolution actions and more information on the WSB error, go tohttp://technet.microsoft.com/en-us/library/cc734488(WS.10).aspx.
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution: To dismiss the alert, click below
Inactivate alert
Protected server event viewer logs:
Log Name: Application
Source: Microsoft-Windows-Backup
Date: 5/3/2012 7:53:34 AM
Event ID: 753
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: PHQDPM04.ppd.ci.phoenix.az.us
Description:
The Block Level Backup Engine service has successfully started.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
<EventID>753</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>0</Task>
<Opcode>1</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2012-05-03T14:53:34.951101000Z" />
<EventRecordID>24238</EventRecordID>
<Correlation />
<Execution ProcessID="10600" ThreadID="2812" />
<Channel>Application</Channel>
<Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
</EventData>
</Event>Log Name: Application
Source: VSS
Date: 5/3/2012 7:54:57 AM
Event ID: 8204
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: PHQDPM04.ppd.ci.phoenix.az.us
Description:
Volume Shadow Copy Service error: An invalid XML document was returned from writer{8c69af6d-69ce-4a38-9a51-8ae9da693d7e}.
Operation:
Writer Modifying Modifying Backup DocumentContext:
Execution Context: Requestor
Writer Instance ID: {8C69AF6D-69CE-4A38-9A51-8AE9DA693D7E}
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="VSS" />
<EventID Qualifiers="0">8204</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2012-05-03T14:54:57.000000000Z" />
<EventRecordID>24239</EventRecordID>
<Channel>Application</Channel>
<Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
<Security />
</System>
<EventData>
<Data>{8c69af6d-69ce-4a38-9a51-8ae9da693d7e}</Data>
<Data>Operation:
Writer Modifying Modifying Backup DocumentContext:
Execution Context: RequestorWriter Instance ID: {8C69AF6D-69CE-4A38-9A51-8AE9DA693D7E}</Data>
<Binary>2D20436F64653A2042554543584D4C4330303030303137312D2043616C6C3A2042554543584D4C4330303030303135362D205049443A
202030303031303630302D205449443A202030303031303932382D20434D443A202022433A5C57696E646F77735C73797374656D33325C776265
6E67696E652E657865222020202020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D31
2D352D313820</Binary>
</EventData>
</Event>Log Name: Application
Source: Microsoft-Windows-Backup
Date: 5/3/2012 7:56:19 AM
Event ID: 521
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: PHQDPM04.ppd.ci.phoenix.az.us
Description:
The backup operation that started at '2012-05-03T14:54:38.690000000Z' has failed because the Volume Shadow CopyService operation to create a shadow copy of the volumes being backed up failed with following error code
'2155348129'. Please review the event details for a solution, and then rerun the backup operation once the issue is
resolved.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
<EventID>521</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2012-05-03T14:56:19.116768600Z" />
<EventRecordID>24244</EventRecordID>
<Correlation />
<Execution ProcessID="10600" ThreadID="9364" />
<Channel>Application</Channel>
<Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data Name="BackupTime">2012-05-03T14:54:38.690000000Z</Data>
<Data Name="ErrorCode">2155348129</Data>
<Data Name="ErrorMessage">%%2155348129</Data>
</EventData>
</Event>Log Name: Application
Source: Microsoft-Windows-Backup
Date: 5/3/2012 8:03:34 AM
Event ID: 754
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: PHQDPM04.ppd.ci.phoenix.az.us
Description:
The Block Level Backup Engine service has stopped.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
<EventID>754</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>0</Task>
<Opcode>2</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2012-05-03T15:03:34.882768600Z" />
<EventRecordID>24246</EventRecordID>
<Correlation />
<Execution ProcessID="10600" ThreadID="2812" />
<Channel>Application</Channel>
<Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
</EventData>
</Event>Protected Server WSB status message:
A Volume Shadow Copy Service operation failed. Detailed error: The volume shadow copy operation failed with error
0x800423F3. View the event log for more information.
Error:
The operation to backup the volume was stopped before the operation started running.
vssadmin list writers output:
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No errorWriter name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No errorWriter name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No errorWriter name: 'DPM Writer'
Writer Id: {0074a271-4f24-43e0-9ec7-28a44189cb80}
Writer Instance Id: {6d12bab2-02c7-4ffa-91f6-6e7c86c6255f}
State: [1] Stable
Last error: No errorWriter name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {1d195d62-bca4-4e70-a90a-18a75552c45a}
State: [7] Failed
Last error: Retryable errorWriter name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {ca86c26b-6789-45be-8c65-388df4d64585}
State: [7] Failed
Last error: Timed outWriter name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {0068c611-4c35-48b6-ae52-e58ab8e16ba9}
State: [7] Failed
Last error: Timed outWriter name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {b85f6194-90e8-4f7c-ba42-e1915229a1d5}
State: [7] Failed
Last error: Timed outWriter name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {29aa7f07-b0a6-4b68-9382-5957f65de7a0}
State: [7] Failed
Last error: Timed outWriter name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {c25547c2-59f5-4dd3-bd9a-07d0050514fe}
State: [7] Failed
Last error: Timed outWriter name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {c18765ff-7202-4e6a-b5aa-8dda26346ef1}
State: [7] Failed
Last error: Timed outWriter name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {98b8968b-e6ce-4a12-bb19-81577c79e55c}
State: [7] Failed
Last error: Timed outWriter name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {ffdcf8f7-5f75-42d1-84f9-ef938fca3d82}
State: [7] Failed
Last error: Timed out- Changed type Mike JacquetMicrosoft employee Tuesday, May 8, 2012 11:42 PM
Tuesday, May 8, 2012 7:26 PM
Answers
-
Hi,
This is a Windows Server ASR writer Limitation so BMR backup will fail if you exceed the volume limitation.
POSSIBLE WORKAROUND
=================See if this works for you.
You can modify BmrBackup.cmd file on the protect server to only backup systemstate, C: and the system partition. That is equivilent to a BMR backup without using the ASR Writer.
Step 1 - you need to get the volume GUID for the System Partition using mountvol.exe so we can add it to the backup. It will be the one without a mountpoint as shown in the example below:
C:\Windows\system32>mountvol.exe
Creates, deletes, or lists a volume mount point.MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /Epath Specifies the existing NTFS directory where the mount
point will reside.
VolumeName Specifies the volume name that is the target of the mount
point.
/D Removes the volume mount point from the specified directory.
/L Lists the mounted volume name for the specified directory.
/P Removes the volume mount point from the specified directory,
dismounts the volume, and makes the volume not mountable.
You can make the volume mountable again by creating a volume
mount point.
/R Removes volume mount point directories and registry settings
for volumes that are no longer in the system.
/N Disables automatic mounting of new volumes.
/E Re-enables automatic mounting of new volumes.Possible values for VolumeName along with current mount points are:
\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\
*** NO MOUNT POINTS ***Step-2 - Make a copy of the MBRbackup.cmd file and then make the following modification.
Replace this line:
start /WAIT %SystemRoot%\system32\wbadmin.exe start backup -allcritical -quiet -backuptarget:%1With this line:
start /WAIT %SystemRoot%\system32\wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ -backuptarget:%1Now when you kick off a BMR backup with DPM it should work since the ASR writer is not invoked.
To verify the backup is good for a BMR restore, restore the BMR backup to a volume, Share out the folder called "C-Volume" in the restored RP as BMRBACKUP which contains the WindowsImagebackup folder.
Then run the following to verify Bare Metal Recovery is listed.
C:\>wbadmin get versions -backuptarget:\\Server\BMRSHARE
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.Backup time: 2/21/2012 5:13 PM
Backup location: Network Share labeled \\Server\BMRSHARE
Version identifier: 02/21/2012-23:13
Can recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System StatePlease remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
- Proposed as answer by Mike JacquetMicrosoft employee Tuesday, May 8, 2012 11:43 PM
- Marked as answer by Yegor Startsev Friday, October 26, 2012 9:40 AM
Tuesday, May 8, 2012 11:39 PM
All replies
-
Hi,
This is a Windows Server ASR writer Limitation so BMR backup will fail if you exceed the volume limitation.
POSSIBLE WORKAROUND
=================See if this works for you.
You can modify BmrBackup.cmd file on the protect server to only backup systemstate, C: and the system partition. That is equivilent to a BMR backup without using the ASR Writer.
Step 1 - you need to get the volume GUID for the System Partition using mountvol.exe so we can add it to the backup. It will be the one without a mountpoint as shown in the example below:
C:\Windows\system32>mountvol.exe
Creates, deletes, or lists a volume mount point.MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /Epath Specifies the existing NTFS directory where the mount
point will reside.
VolumeName Specifies the volume name that is the target of the mount
point.
/D Removes the volume mount point from the specified directory.
/L Lists the mounted volume name for the specified directory.
/P Removes the volume mount point from the specified directory,
dismounts the volume, and makes the volume not mountable.
You can make the volume mountable again by creating a volume
mount point.
/R Removes volume mount point directories and registry settings
for volumes that are no longer in the system.
/N Disables automatic mounting of new volumes.
/E Re-enables automatic mounting of new volumes.Possible values for VolumeName along with current mount points are:
\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\
*** NO MOUNT POINTS ***Step-2 - Make a copy of the MBRbackup.cmd file and then make the following modification.
Replace this line:
start /WAIT %SystemRoot%\system32\wbadmin.exe start backup -allcritical -quiet -backuptarget:%1With this line:
start /WAIT %SystemRoot%\system32\wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ -backuptarget:%1Now when you kick off a BMR backup with DPM it should work since the ASR writer is not invoked.
To verify the backup is good for a BMR restore, restore the BMR backup to a volume, Share out the folder called "C-Volume" in the restored RP as BMRBACKUP which contains the WindowsImagebackup folder.
Then run the following to verify Bare Metal Recovery is listed.
C:\>wbadmin get versions -backuptarget:\\Server\BMRSHARE
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.Backup time: 2/21/2012 5:13 PM
Backup location: Network Share labeled \\Server\BMRSHARE
Version identifier: 02/21/2012-23:13
Can recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System StatePlease remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
- Proposed as answer by Mike JacquetMicrosoft employee Tuesday, May 8, 2012 11:43 PM
- Marked as answer by Yegor Startsev Friday, October 26, 2012 9:40 AM
Tuesday, May 8, 2012 11:39 PM -
Mike Jacjuet,
Thank you for the quick response and for taking the time to offer a possible workaround. I have not been able to get the workaround to work for me at this point. I modified the BmrBackup.cmd file and tried the DPM System State / BMR protection again and the job does not fail but runs and runs without transfering any data. Windows Server Backup did not show any activity at all. I tried to run just this part of the command: wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ (with my volume GUID info. of course) and Windows Server Backup does attempt to run but the operation stops with a detailed error: The parameter is incorrect. Now I did use the C: volume GUID for the system partition volume since our system, boot and C: are all on the same volume. I did not have a separate option for system volume when I ran mountvol.exe. Do I need to do this step differently? I also have a D: application volume that I would like to include in BMR protection. I sure wish their was a ASR Writer Hotfix.
Thursday, May 10, 2012 10:36 PM -
Hi,
OK, if the system, boot partition are both C: then you don't need to include the volume guid. You can add D: and that should take care of you.
To be clear, this should work: wbadmin start backup -systemstate -include:c:,d:
The ASR bug fix in Windows 2008 you are referring to was to remove ASR writer from a systemstate backup. Prior to that fix, Systemstate backup would fail for the same reason that BMR fails today.
let me know how you make out.
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
- Edited by Mike JacquetMicrosoft employee Thursday, May 10, 2012 10:45 PM
Thursday, May 10, 2012 10:44 PM -
I too am having the same issues. MS has to many work arounds. Just issue a hotfix. If I would have known all this I would have stayed with netvault.
Shawn
Friday, June 22, 2012 4:34 PM -
Mike Jacquet,
I did try the work-around with the following details:
start /WAIT %SystemRoot%\system32\wbadmin.exe start backup -systemstate -include:c:,d: -backuptarget:%1
I modified the BmrBackup.cmd file and tried the DPM System State / BMR protection again and the job does not fail but runs and runs without transfering any data. Windows Server Backup did not show any activity at all. I end up having to just cancel the job.
Tuesday, December 4, 2012 5:55 PM -
Hi,
You will not see any data transferred by DPM agent because Windows backup is writing to the DPM replica volume directly, so DPM agent has no way of showing you the amount that Windows backup is writing. You can open Resource Monitor and view disk activity see that wbengine.exe is doing on the protected server.
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
Tuesday, December 4, 2012 6:04 PM -
Mike Jacquet,
Sorry I guess I didn't describe this very well. I only cancelled the BMR replica creation jobs during my testing and troubleshooting after 17, 25, 30 hours of no success or failure. I have 60 DPM protected servers and even when BMR replica creation or recovery point creation is going slow on those servers they never run longer than 90 minutes....2 hours max before completing. (Failures usually occur much faster and normally much less than one hour) I figured out that normally within 10 minutes of starting a consisistency check or replica creation job I can logon to the protected server and view the Windows Server Backup interface and see that a backup is running. But after applying this work-a-round I do not ever see a backup job start up on the affected remote host server. And on the DPM server the job just keeps on going.
Michael
Tuesday, December 4, 2012 8:51 PM