Remove From My Forums

DPM2010 System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1

Question
0

Sign in to vote
DPM2010 Version (3.0.7707.0) System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1

I'm trying to utilize BMR protection on two DPM Servers but it fails with DPM based protection or Windows Server

backup protection. Both DPM based and Windows Server Backup protection work successfully when I only select System

State protection. I had BMR DPM based protection on one of the servers until 10 days ago when it all-of-a-sudden

stopped working. The failures seem to be related to or caused by ASR Writer retryable errors. I found the ASR /

BMR VSS System State protection hotfix (Article ID: 983425) but it does not apply to 2008 R2. I have the same

problem in R2. Does anyone know of a work-a-round or fix for R2 BMR based protection on servers with 400 plus

volumes? I have included error and log information from DPM, Event Viewer, Windows Server Backup and vssadmin list

writers output. Is there another hotfix that I'm not finding so far? Any assistance would be greatly appreciated.

Thank you.

DPM2010 Error message failures:

Affected area: Computer\System Protection
Occurred since: 5/4/2012 2:09:18 PM
Description: The replica of System Protection Computer\System Protection on PHQDPM04.ppd.ci.phoenix.az.us is

inconsistent with the protected data source. All protection activities for data source will fail until the replica

is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points

cannot be created until the replica is consistent.

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup

inconsistent databases, run a consistency check on the farm. (ID 3106)
DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an

error (WSB Event ID: 521, WSB Error Code: 0x807800A1). (ID 30229 Details: Internal error code: 0x809909FB)
More information
Recommended action: For resolution actions and more information on the WSB error, go to

http://technet.microsoft.com/en-us/library/cc734488(WS.10).aspx.
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution: To dismiss the alert, click below
Inactivate alert

Protected server event viewer logs:

Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          5/3/2012 7:53:34 AM
Event ID:      753
Task Category: None
Level:         Information
Keywords:
User:          SYSTEM
Computer:      PHQDPM04.ppd.ci.phoenix.az.us
Description:
The Block Level Backup Engine service has successfully started.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
    <EventID>753</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>1</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2012-05-03T14:53:34.951101000Z" />
    <EventRecordID>24238</EventRecordID>
    <Correlation />
    <Execution ProcessID="10600" ThreadID="2812" />
    <Channel>Application</Channel>
    <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
    <Security UserID="S-1-5-18" />
</System>
<EventData>
</EventData>
</Event>

Log Name:      Application
Source:        VSS
Date:          5/3/2012 7:54:57 AM
Event ID:      8204
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PHQDPM04.ppd.ci.phoenix.az.us
Description:
Volume Shadow Copy Service error: An invalid XML document was returned from writer

{8c69af6d-69ce-4a38-9a51-8ae9da693d7e}.

Operation:
   Writer Modifying Modifying Backup Document

Context:

Execution Context: Requestor
   Writer Instance ID: {8C69AF6D-69CE-4A38-9A51-8AE9DA693D7E}
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="VSS" />
    <EventID Qualifiers="0">8204</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-05-03T14:54:57.000000000Z" />
    <EventRecordID>24239</EventRecordID>
    <Channel>Application</Channel>
    <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
    <Security />
</System>
<EventData>
    <Data>{8c69af6d-69ce-4a38-9a51-8ae9da693d7e}</Data>
    <Data>

Operation:
   Writer Modifying Modifying Backup Document

Context:
   Execution Context: Requestor

Writer Instance ID: {8C69AF6D-69CE-4A38-9A51-8AE9DA693D7E}</Data>

<Binary>2D20436F64653A2042554543584D4C4330303030303137312D2043616C6C3A2042554543584D4C4330303030303135362D205049443A

202030303031303630302D205449443A202030303031303932382D20434D443A202022433A5C57696E646F77735C73797374656D33325C776265

6E67696E652E657865222020202020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D31

2D352D313820</Binary>
</EventData>
</Event>

Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          5/3/2012 7:56:19 AM
Event ID:      521
Task Category: None
Level:         Error
Keywords:
User:          SYSTEM
Computer:      PHQDPM04.ppd.ci.phoenix.az.us
Description:
The backup operation that started at '‎2012‎-‎05‎-‎03T14:54:38.690000000Z' has failed because the Volume Shadow Copy

Service operation to create a shadow copy of the volumes being backed up failed with following error code

'2155348129'. Please review the event details for a solution, and then rerun the backup operation once the issue is

resolved.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
    <EventID>521</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2012-05-03T14:56:19.116768600Z" />
    <EventRecordID>24244</EventRecordID>
    <Correlation />
    <Execution ProcessID="10600" ThreadID="9364" />
    <Channel>Application</Channel>
    <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
    <Security UserID="S-1-5-18" />
</System>
<EventData>
    <Data Name="BackupTime">2012-05-03T14:54:38.690000000Z</Data>
    <Data Name="ErrorCode">2155348129</Data>
    <Data Name="ErrorMessage">%%2155348129</Data>
</EventData>
</Event>

Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          5/3/2012 8:03:34 AM
Event ID:      754
Task Category: None
Level:         Information
Keywords:
User:          SYSTEM
Computer:      PHQDPM04.ppd.ci.phoenix.az.us
Description:
The Block Level Backup Engine service has stopped.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
    <EventID>754</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>2</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2012-05-03T15:03:34.882768600Z" />
    <EventRecordID>24246</EventRecordID>
    <Correlation />
    <Execution ProcessID="10600" ThreadID="2812" />
    <Channel>Application</Channel>
    <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
    <Security UserID="S-1-5-18" />
</System>
<EventData>
</EventData>
</Event>

Protected Server WSB status message:

A Volume Shadow Copy Service operation failed. Detailed error: The volume shadow copy operation failed with error

0x800423F3. View the event log for more information.

Error:

The operation to backup the volume was stopped before the operation started running.

vssadmin list writers output:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'DPM Writer'
   Writer Id: {0074a271-4f24-43e0-9ec7-28a44189cb80}
   Writer Instance Id: {6d12bab2-02c7-4ffa-91f6-6e7c86c6255f}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {1d195d62-bca4-4e70-a90a-18a75552c45a}
   State: [7] Failed
   Last error: Retryable error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {ca86c26b-6789-45be-8c65-388df4d64585}
   State: [7] Failed
   Last error: Timed out

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {0068c611-4c35-48b6-ae52-e58ab8e16ba9}
   State: [7] Failed
   Last error: Timed out

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {b85f6194-90e8-4f7c-ba42-e1915229a1d5}
   State: [7] Failed
   Last error: Timed out

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {29aa7f07-b0a6-4b68-9382-5957f65de7a0}
   State: [7] Failed
   Last error: Timed out

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {c25547c2-59f5-4dd3-bd9a-07d0050514fe}
   State: [7] Failed
   Last error: Timed out

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {c18765ff-7202-4e6a-b5aa-8dda26346ef1}
   State: [7] Failed
   Last error: Timed out

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {98b8968b-e6ce-4a12-bb19-81577c79e55c}
   State: [7] Failed
   Last error: Timed out

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {ffdcf8f7-5f75-42d1-84f9-ef938fca3d82}
   State: [7] Failed
   Last error: Timed out
- Changed type Mike JacquetMicrosoft employee, Moderator Tuesday, May 8, 2012 11:42 PM
Tuesday, May 8, 2012 7:26 PM

Reply

|

Quote

Answers

1

Sign in to vote
Hi,

This is a Windows Server ASR writer Limitation so BMR backup will fail if you exceed the volume limitation.

POSSIBLE WORKAROUND
=================

See if this works for you.

You can modify BmrBackup.cmd file on the protect server to only backup systemstate, C: and the system partition. That is equivilent to a BMR backup without using the ASR Writer.

Step 1 - you need to get the volume GUID for the System Partition using mountvol.exe so we can add it to the backup. It will be the one without a mountpoint as shown in the example below:

C:\Windows\system32>mountvol.exe
Creates, deletes, or lists a volume mount point.

MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /E

    path        Specifies the existing NTFS directory where the mount
                point will reside.
    VolumeName Specifies the volume name that is the target of the mount
                point.
    /D          Removes the volume mount point from the specified directory.
    /L          Lists the mounted volume name for the specified directory.
    /P          Removes the volume mount point from the specified directory,
                dismounts the volume, and makes the volume not mountable.
                You can make the volume mountable again by creating a volume
                mount point.
    /R          Removes volume mount point directories and registry settings
                for volumes that are no longer in the system.
    /N          Disables automatic mounting of new volumes.
    /E          Re-enables automatic mounting of new volumes.

Possible values for VolumeName along with current mount points are:

    \\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\
        *** NO MOUNT POINTS ***

Step-2 - Make a copy of the MBRbackup.cmd file and then make the following modification.

Replace this line:
      start /WAIT %SystemRoot%\system32\wbadmin.exe start backup -allcritical -quiet -backuptarget:%1

With this line:
      start /WAIT %SystemRoot%\system32\wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ -backuptarget:%1

Now when you kick off a BMR backup with DPM it should work since the ASR writer is not invoked.

To verify the backup is good for a BMR restore, restore the BMR backup to a volume, Share out the folder called "C-Volume" in the restored RP as BMRBACKUP which contains the WindowsImagebackup folder.

Then run the following to verify Bare Metal Recovery is listed.

C:\>wbadmin get versions -backuptarget:\\Server\BMRSHARE
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.

Backup time: 2/21/2012 5:13 PM
Backup location: Network Share labeled \\Server\BMRSHARE
Version identifier: 02/21/2012-23:13
Can recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System State

Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
- Proposed as answer by Mike JacquetMicrosoft employee, Moderator Tuesday, May 8, 2012 11:43 PM
- Marked as answer by Yegor StartsevModerator Friday, October 26, 2012 9:40 AM
Tuesday, May 8, 2012 11:39 PM

Reply

|

Quote

Moderator

All replies

1

Sign in to vote
Hi,

This is a Windows Server ASR writer Limitation so BMR backup will fail if you exceed the volume limitation.

POSSIBLE WORKAROUND
=================

See if this works for you.

You can modify BmrBackup.cmd file on the protect server to only backup systemstate, C: and the system partition. That is equivilent to a BMR backup without using the ASR Writer.

Step 1 - you need to get the volume GUID for the System Partition using mountvol.exe so we can add it to the backup. It will be the one without a mountpoint as shown in the example below:

C:\Windows\system32>mountvol.exe
Creates, deletes, or lists a volume mount point.

MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /E

    path        Specifies the existing NTFS directory where the mount
                point will reside.
    VolumeName Specifies the volume name that is the target of the mount
                point.
    /D          Removes the volume mount point from the specified directory.
    /L          Lists the mounted volume name for the specified directory.
    /P          Removes the volume mount point from the specified directory,
                dismounts the volume, and makes the volume not mountable.
                You can make the volume mountable again by creating a volume
                mount point.
    /R          Removes volume mount point directories and registry settings
                for volumes that are no longer in the system.
    /N          Disables automatic mounting of new volumes.
    /E          Re-enables automatic mounting of new volumes.

Possible values for VolumeName along with current mount points are:

    \\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\
        *** NO MOUNT POINTS ***

Step-2 - Make a copy of the MBRbackup.cmd file and then make the following modification.

Replace this line:
      start /WAIT %SystemRoot%\system32\wbadmin.exe start backup -allcritical -quiet -backuptarget:%1

With this line:
      start /WAIT %SystemRoot%\system32\wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ -backuptarget:%1

Now when you kick off a BMR backup with DPM it should work since the ASR writer is not invoked.

To verify the backup is good for a BMR restore, restore the BMR backup to a volume, Share out the folder called "C-Volume" in the restored RP as BMRBACKUP which contains the WindowsImagebackup folder.

Then run the following to verify Bare Metal Recovery is listed.

C:\>wbadmin get versions -backuptarget:\\Server\BMRSHARE
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.

Backup time: 2/21/2012 5:13 PM
Backup location: Network Share labeled \\Server\BMRSHARE
Version identifier: 02/21/2012-23:13
Can recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System State

Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
- Proposed as answer by Mike JacquetMicrosoft employee, Moderator Tuesday, May 8, 2012 11:43 PM
- Marked as answer by Yegor StartsevModerator Friday, October 26, 2012 9:40 AM
Tuesday, May 8, 2012 11:39 PM

Reply

|

Quote

Moderator
0

Sign in to vote

Mike Jacjuet,

Thank you for the quick response and for taking the time to offer a possible workaround. I have not been able to get the workaround to work for me at this point. I modified the BmrBackup.cmd file and tried the DPM System State / BMR protection again and the job does not fail but runs and runs without transfering any data. Windows Server Backup did not show any activity at all. I tried to run just this part of the command: wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ (with my volume GUID info. of course) and Windows Server Backup does attempt to run but the operation stops with a detailed error: The parameter is incorrect. Now I did use the C: volume GUID for the system partition volume since our system, boot and C: are all on the same volume. I did not have a separate option for system volume when I ran mountvol.exe. Do I need to do this step differently? I also have a D: application volume that I would like to include in BMR protection. I sure wish their was a ASR Writer Hotfix.

Thursday, May 10, 2012 10:36 PM

Reply

|

Quote
0

Sign in to vote
Hi,

OK, if the system, boot partition are both C: then you don't need to include the volume guid. You can add D: and that should take care of you.

To be clear, this should work: wbadmin start backup -systemstate -include:c:,d:

The ASR bug fix in Windows 2008 you are referring to was to remove ASR writer from a systemstate backup. Prior to that fix, Systemstate backup would fail for the same reason that BMR fails today.

let me know how you make out.

Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
- Edited by Mike JacquetMicrosoft employee, Moderator Thursday, May 10, 2012 10:45 PM
Thursday, May 10, 2012 10:44 PM

Reply

|

Quote

Moderator
0

Sign in to vote

I too am having the same issues. MS has to many work arounds. Just issue a hotfix. If I would have known all this I would have stayed with netvault.
Shawn

Friday, June 22, 2012 4:34 PM

Reply

|

Quote
0

Sign in to vote

Mike Jacquet,

I did try the work-around with the following details:

start /WAIT %SystemRoot%\system32\wbadmin.exe start backup -systemstate -include:c:,d: -backuptarget:%1

I modified the BmrBackup.cmd file and tried the DPM System State / BMR protection again and the job does not fail but runs and runs without transfering any data. Windows Server Backup did not show any activity at all. I end up having to just cancel the job.

Tuesday, December 4, 2012 5:55 PM

Reply

|

Quote
0

Sign in to vote

Hi,

You will not see any data transferred by DPM agent because Windows backup is writing to the DPM replica volume directly, so DPM agent has no way of showing you the amount that Windows backup is writing. You can open Resource Monitor and view disk activity see that wbengine.exe is doing on the protected server.
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

Tuesday, December 4, 2012 6:04 PM

Reply

|

Quote

Moderator
0

Sign in to vote

Mike Jacquet,

Sorry I guess I didn't describe this very well. I only cancelled the BMR replica creation jobs during my testing and troubleshooting after 17, 25, 30 hours of no success or failure. I have 60 DPM protected servers and even when BMR replica creation or recovery point creation is going slow on those servers they never run longer than 90 minutes....2 hours max before completing. (Failures usually occur much faster and normally much less than one hour) I figured out that normally within 10 minutes of starting a consisistency check or replica creation job I can logon to the protected server and view the Windows Server Backup interface and see that a backup is running. But after applying this work-a-round I do not ever see a backup job start up on the affected remote host server. And on the DPM server the job just keeps on going.

Michael

Tuesday, December 4, 2012 8:51 PM

Reply

|

Quote

Products

Resources

Solutions

Updates

Trials

Related Sites

Training

Certifications

Other resources

Support options

More support

Not an IT pro?

Answered by:

DPM2010 System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1

Question

Answers

All replies