none
DPM2010 System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1 RRS feed

  • Question

  • DPM2010 Version (3.0.7707.0) System State and BMR backup protection failing for Windows Server 2008 R2 Ent. SP1


    I'm trying to utilize BMR protection on two DPM Servers but it fails with DPM based protection or Windows Server

    backup protection.  Both DPM based and Windows Server Backup protection work successfully when I only select System

    State protection.  I had BMR DPM based protection on one of the servers until 10 days ago when it all-of-a-sudden

    stopped working.  The failures seem to be related to or caused by ASR Writer retryable errors.  I found the ASR /

    BMR VSS System State protection hotfix (Article ID: 983425) but it does not apply to 2008 R2.  I have the same

    problem in R2. Does anyone know of a work-a-round or fix for R2 BMR based protection on servers with 400 plus

    volumes?  I have included error and log information from DPM, Event Viewer, Windows Server Backup and vssadmin list

    writers output.  Is there another hotfix that I'm not finding so far?  Any assistance would be greatly appreciated. 

    Thank you. 

     

    DPM2010 Error message failures:

    Affected area: Computer\System Protection
    Occurred since: 5/4/2012 2:09:18 PM
    Description: The replica of System Protection Computer\System Protection on PHQDPM04.ppd.ci.phoenix.az.us is

    inconsistent with the protected data source. All protection activities for data source will fail until the replica

    is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points

    cannot be created until the replica is consistent.

    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup

    inconsistent databases, run a consistency check on the farm. (ID 3106)
     DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an

    error (WSB Event ID: 521, WSB Error Code:  0x807800A1). (ID 30229 Details: Internal error code: 0x809909FB)
     More information
    Recommended action: For resolution actions and more information on the WSB error, go to

    http://technet.microsoft.com/en-us/library/cc734488(WS.10).aspx.
     Synchronize with consistency check.
     Run a synchronization job with consistency check...
    Resolution: To dismiss the alert, click below
     Inactivate alert

     


    Protected server event viewer logs:


    Log Name:      Application
    Source:        Microsoft-Windows-Backup
    Date:          5/3/2012 7:53:34 AM
    Event ID:      753
    Task Category: None
    Level:         Information
    Keywords:     
    User:          SYSTEM
    Computer:      PHQDPM04.ppd.ci.phoenix.az.us
    Description:
    The Block Level Backup Engine service has successfully started.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
        <EventID>753</EventID>
        <Version>0</Version>
        <Level>4</Level>
        <Task>0</Task>
        <Opcode>1</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2012-05-03T14:53:34.951101000Z" />
        <EventRecordID>24238</EventRecordID>
        <Correlation />
        <Execution ProcessID="10600" ThreadID="2812" />
        <Channel>Application</Channel>
        <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        VSS
    Date:          5/3/2012 7:54:57 AM
    Event ID:      8204
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      PHQDPM04.ppd.ci.phoenix.az.us
    Description:
    Volume Shadow Copy Service error: An invalid XML document was returned from writer

    {8c69af6d-69ce-4a38-9a51-8ae9da693d7e}.

    Operation:
       Writer Modifying Modifying Backup Document

    Context:

    Execution Context: Requestor
       Writer Instance ID: {8C69AF6D-69CE-4A38-9A51-8AE9DA693D7E}
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="VSS" />
        <EventID Qualifiers="0">8204</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2012-05-03T14:54:57.000000000Z" />
        <EventRecordID>24239</EventRecordID>
        <Channel>Application</Channel>
        <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
        <Security />
      </System>
      <EventData>
        <Data>{8c69af6d-69ce-4a38-9a51-8ae9da693d7e}</Data>
        <Data>

    Operation:
       Writer Modifying Modifying Backup Document

    Context:
       Execution Context: Requestor

    Writer Instance ID: {8C69AF6D-69CE-4A38-9A51-8AE9DA693D7E}</Data>

    <Binary>2D20436F64653A2042554543584D4C4330303030303137312D2043616C6C3A2042554543584D4C4330303030303135362D205049443A

    202030303031303630302D205449443A202030303031303932382D20434D443A202022433A5C57696E646F77735C73797374656D33325C776265

    6E67696E652E657865222020202020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D31

    2D352D313820</Binary>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        Microsoft-Windows-Backup
    Date:          5/3/2012 7:56:19 AM
    Event ID:      521
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      PHQDPM04.ppd.ci.phoenix.az.us
    Description:
    The backup operation that started at '‎2012‎-‎05‎-‎03T14:54:38.690000000Z' has failed because the Volume Shadow Copy

    Service operation to create a shadow copy of the volumes being backed up failed with following error code

    '2155348129'. Please review the event details for a solution, and then rerun the backup operation once the issue is

    resolved.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
        <EventID>521</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2012-05-03T14:56:19.116768600Z" />
        <EventRecordID>24244</EventRecordID>
        <Correlation />
        <Execution ProcessID="10600" ThreadID="9364" />
        <Channel>Application</Channel>
        <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="BackupTime">2012-05-03T14:54:38.690000000Z</Data>
        <Data Name="ErrorCode">2155348129</Data>
        <Data Name="ErrorMessage">%%2155348129</Data>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        Microsoft-Windows-Backup
    Date:          5/3/2012 8:03:34 AM
    Event ID:      754
    Task Category: None
    Level:         Information
    Keywords:     
    User:          SYSTEM
    Computer:      PHQDPM04.ppd.ci.phoenix.az.us
    Description:
    The Block Level Backup Engine service has stopped.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
        <EventID>754</EventID>
        <Version>0</Version>
        <Level>4</Level>
        <Task>0</Task>
        <Opcode>2</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2012-05-03T15:03:34.882768600Z" />
        <EventRecordID>24246</EventRecordID>
        <Correlation />
        <Execution ProcessID="10600" ThreadID="2812" />
        <Channel>Application</Channel>
        <Computer>PHQDPM04.ppd.ci.phoenix.az.us</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
      </EventData>
    </Event>

     

     

    Protected Server WSB status message:

    A Volume Shadow Copy Service operation failed. Detailed error: The volume shadow copy operation failed with error

    0x800423F3. View the event log for more information.

    Error:

    The operation to backup the volume was stopped before the operation started running.

    vssadmin list writers output:


    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.

    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error

    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error

    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error

    Writer name: 'DPM Writer'
       Writer Id: {0074a271-4f24-43e0-9ec7-28a44189cb80}
       Writer Instance Id: {6d12bab2-02c7-4ffa-91f6-6e7c86c6255f}
       State: [1] Stable
       Last error: No error

    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {1d195d62-bca4-4e70-a90a-18a75552c45a}
       State: [7] Failed
       Last error: Retryable error

    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {ca86c26b-6789-45be-8c65-388df4d64585}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'SqlServerWriter'
       Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
       Writer Instance Id: {0068c611-4c35-48b6-ae52-e58ab8e16ba9}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {b85f6194-90e8-4f7c-ba42-e1915229a1d5}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {29aa7f07-b0a6-4b68-9382-5957f65de7a0}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {c25547c2-59f5-4dd3-bd9a-07d0050514fe}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'IIS Config Writer'
       Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
       Writer Instance Id: {c18765ff-7202-4e6a-b5aa-8dda26346ef1}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {98b8968b-e6ce-4a12-bb19-81577c79e55c}
       State: [7] Failed
       Last error: Timed out

    Writer name: 'IIS Metabase Writer'
       Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
       Writer Instance Id: {ffdcf8f7-5f75-42d1-84f9-ef938fca3d82}
       State: [7] Failed
       Last error: Timed out

    Tuesday, May 8, 2012 7:26 PM

Answers

  • Hi,

    This is a Windows Server ASR writer Limitation so BMR backup will fail if you exceed the volume limitation. 

    POSSIBLE WORKAROUND
    =================

    See if this works for you.

    You can modify BmrBackup.cmd file on the protect server to only backup systemstate, C: and the system partition.  That is equivilent to a BMR backup without using the ASR Writer.

    Step 1 - you need to get the volume GUID for the System Partition using mountvol.exe so we can add it to the backup.  It will be the one without a mountpoint as shown in the example below:

    C:\Windows\system32>mountvol.exe
    Creates, deletes, or lists a volume mount point.

    MOUNTVOL [drive:]path VolumeName
    MOUNTVOL [drive:]path /D
    MOUNTVOL [drive:]path /L
    MOUNTVOL [drive:]path /P
    MOUNTVOL /R
    MOUNTVOL /N
    MOUNTVOL /E

        path        Specifies the existing NTFS directory where the mount
                    point will reside.
        VolumeName  Specifies the volume name that is the target of the mount
                    point.
        /D          Removes the volume mount point from the specified directory.
        /L          Lists the mounted volume name for the specified directory.
        /P          Removes the volume mount point from the specified directory,
                    dismounts the volume, and makes the volume not mountable.
                    You can make the volume mountable again by creating a volume
                    mount point.
        /R          Removes volume mount point directories and registry settings
                    for volumes that are no longer in the system.
        /N          Disables automatic mounting of new volumes.
        /E          Re-enables automatic mounting of new volumes.

    Possible values for VolumeName along with current mount points are:

        \\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\
            *** NO MOUNT POINTS ***

    Step-2 - Make a copy of the MBRbackup.cmd file and then make the following modification.

    Replace this line: 
          start  /WAIT %SystemRoot%\system32\wbadmin.exe start backup -allcritical -quiet -backuptarget:%1

    With this line:
          start  /WAIT %SystemRoot%\system32\wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ -backuptarget:%1

    Now when you kick off a BMR backup with DPM it should work since the ASR writer is not invoked.

    To verify the backup is good for a BMR restore, restore the BMR backup to a volume,  Share out the folder called "C-Volume" in the restored RP as BMRBACKUP which contains the WindowsImagebackup folder.

    Then run the following to verify Bare Metal Recovery is listed.

    C:\>wbadmin get versions -backuptarget:\\Server\BMRSHARE
    wbadmin 1.0 - Backup command-line tool
    (C) Copyright 2004 Microsoft Corp.

    Backup time: 2/21/2012 5:13 PM
    Backup location: Network Share labeled \\Server\BMRSHARE
    Version identifier: 02/21/2012-23:13
    Can recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System State


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, May 8, 2012 11:39 PM
    Moderator

All replies

  • Hi,

    This is a Windows Server ASR writer Limitation so BMR backup will fail if you exceed the volume limitation. 

    POSSIBLE WORKAROUND
    =================

    See if this works for you.

    You can modify BmrBackup.cmd file on the protect server to only backup systemstate, C: and the system partition.  That is equivilent to a BMR backup without using the ASR Writer.

    Step 1 - you need to get the volume GUID for the System Partition using mountvol.exe so we can add it to the backup.  It will be the one without a mountpoint as shown in the example below:

    C:\Windows\system32>mountvol.exe
    Creates, deletes, or lists a volume mount point.

    MOUNTVOL [drive:]path VolumeName
    MOUNTVOL [drive:]path /D
    MOUNTVOL [drive:]path /L
    MOUNTVOL [drive:]path /P
    MOUNTVOL /R
    MOUNTVOL /N
    MOUNTVOL /E

        path        Specifies the existing NTFS directory where the mount
                    point will reside.
        VolumeName  Specifies the volume name that is the target of the mount
                    point.
        /D          Removes the volume mount point from the specified directory.
        /L          Lists the mounted volume name for the specified directory.
        /P          Removes the volume mount point from the specified directory,
                    dismounts the volume, and makes the volume not mountable.
                    You can make the volume mountable again by creating a volume
                    mount point.
        /R          Removes volume mount point directories and registry settings
                    for volumes that are no longer in the system.
        /N          Disables automatic mounting of new volumes.
        /E          Re-enables automatic mounting of new volumes.

    Possible values for VolumeName along with current mount points are:

        \\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\
            *** NO MOUNT POINTS ***

    Step-2 - Make a copy of the MBRbackup.cmd file and then make the following modification.

    Replace this line: 
          start  /WAIT %SystemRoot%\system32\wbadmin.exe start backup -allcritical -quiet -backuptarget:%1

    With this line:
          start  /WAIT %SystemRoot%\system32\wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ -backuptarget:%1

    Now when you kick off a BMR backup with DPM it should work since the ASR writer is not invoked.

    To verify the backup is good for a BMR restore, restore the BMR backup to a volume,  Share out the folder called "C-Volume" in the restored RP as BMRBACKUP which contains the WindowsImagebackup folder.

    Then run the following to verify Bare Metal Recovery is listed.

    C:\>wbadmin get versions -backuptarget:\\Server\BMRSHARE
    wbadmin 1.0 - Backup command-line tool
    (C) Copyright 2004 Microsoft Corp.

    Backup time: 2/21/2012 5:13 PM
    Backup location: Network Share labeled \\Server\BMRSHARE
    Version identifier: 02/21/2012-23:13
    Can recover: Volume(s), File(s), Application(s), Bare Metal Recovery, System State


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, May 8, 2012 11:39 PM
    Moderator
  • Mike Jacjuet,

    Thank you for the quick response and for taking the time to offer a possible workaround.  I have not been able to get the workaround to work for me at this point.  I modified the BmrBackup.cmd file and tried the DPM System State / BMR protection again and the job does not fail but runs and runs without transfering any data.   Windows Server Backup did not show any activity at all.  I tried to run just this part of the command:  wbadmin start backup -systemstate -include:c:,\\?\Volume{05abb0d4-759d-11e0-a684-806e6f6e6963}\ (with my volume GUID info. of course) and Windows Server Backup does attempt to run but the operation stops with a detailed error:  The parameter is incorrect.  Now I did use the C: volume GUID for the system partition volume since our system, boot and C: are all on the same volume.  I did not have a separate option for system volume when I ran mountvol.exe.  Do I need to do this step differently?  I also have a D: application volume that I would like to include in BMR protection.  I sure wish their was a ASR Writer Hotfix.

    Thursday, May 10, 2012 10:36 PM
  • Hi,

    OK, if the system, boot partition are both C: then you don't need to include the volume guid.  You can add D: and that should take care of you.

    To be clear, this should work:  wbadmin start backup -systemstate -include:c:,d:

    The ASR bug fix in Windows 2008 you are referring to was to remove ASR writer from a systemstate backup.  Prior to that fix, Systemstate backup would fail for the same reason that BMR fails today. 

    let me know how you make out.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, May 10, 2012 10:44 PM
    Moderator
  • I too am having the same issues. MS has to many work arounds. Just issue a hotfix. If I would have known all this I would have stayed with netvault.

    Shawn

    Friday, June 22, 2012 4:34 PM
  • Mike Jacquet,

    I did try the work-around with the following details:

    start /WAIT  %SystemRoot%\system32\wbadmin.exe start backup -systemstate -include:c:,d: -backuptarget:%1

    I modified the BmrBackup.cmd file and tried the DPM System State / BMR protection again and the job does not fail but runs and runs without transfering any data.   Windows Server Backup did not show any activity at all.  I end up having to just cancel the job.

    Tuesday, December 4, 2012 5:55 PM
  • Hi,

    You will not see any data transferred by DPM agent because Windows backup is writing to the DPM replica volume directly, so DPM agent has no way of showing you the amount that Windows backup is writing.  You can open Resource Monitor and view disk activity see that wbengine.exe is doing on the protected server.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 4, 2012 6:04 PM
    Moderator
  • Mike Jacquet,

    Sorry I guess I didn't describe this very well.  I only cancelled the BMR replica creation jobs during my testing and troubleshooting after 17, 25, 30 hours of no success or failure.  I have 60 DPM protected servers and even when BMR replica creation or recovery point creation is going slow on those servers they never run longer than 90 minutes....2 hours max before completing.  (Failures usually occur much faster and normally much less than one hour)  I figured out that normally within 10 minutes of starting a consisistency check or replica creation job I can logon to the protected server and view the Windows Server Backup interface and see that a backup is running.  But after applying this work-a-round I do not ever see a backup job start up on the affected remote host server.  And on the DPM server the job just keeps on going.

    Michael

    Tuesday, December 4, 2012 8:51 PM