none
DPM 2010 Secondary Site Protection Groups Inconsistent - Fail with Error 0x800423F4 RRS feed

  • Question

  • Every replica in my Secondary Protection group is suddenly failing with "Replica is inconsistent". Every replica on my Primary Server is working fine. No inconsistencies, and backups to both disk and tape are successful on the Primary Server. Both servers are 2008 R2 Enterprise 64-bit SP1. DPM 2010 is updated with the most recent QFE on both servers.

    Below is the DPM error from an example replica (the errors are the same for all of my protected data, this particular data is a small SQL database for Backup Exec on 'Server2'):

    Affected area:    Server2\BKUPEXEC\BEDB(server2.domain)
    Occurred since:    07/06/2011 7:42:31 AM
    Description:    The replica of SQL Server 2005 database Server2\BKUPEXEC\BEDB(server2.domain) on server.domain is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.

    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
        Synchronization for replica of Server2\BKUPEXEC\BEDB(server2.domain) on server.domain failed because the replica is not in a valid state or is in an inactive state. (ID 30300 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried,
    the error is likely to reoccur.
     (0x800423F4))
        More information
    Recommended action:    Review the application event log on the protected computer for errors from the DPM writer service. Take appropriate action and retry the operation.
        Synchronize with consistency check.
        Run a synchronization job with consistency check...
    Resolution:    To dismiss the alert, click below
        Inactivate alert

    The Application Eventlog on the Primary Server shows the following event for the above Replica:

    DPM has run out of free recovery point space and will fail snapshots for Server2\BKUPEXEC\BEDB in order to prevent existing recovery points from getting recycled.  1) Increase the space allocated for the recovery point volume for Server2\BKUPEXEC\BEDB.
    2) Retry the operation after increasing the space for the recovery point volume.

    NOTE: I have verified that the Replica in question has plenty of free space in both the Replica and Recovery point volumes on both the primary and secondary servers.

    When I run vssadmin list writers on the Primary server I get the following for the DPM Writer:

    Writer name: 'DPM Writer'
       Writer Id: {0074a271-4f24-43e0-9ec7-28a44189cb80}
       Writer Instance Id: {ea038d23-cd78-4d2a-9171-bfd76f52a264}
       State: [7] Failed
       Last error: Non-retryable error

    Steps I've done to try and resolve this:

    1. Consistency Checks (on both Primary and Secondary. Primary works. Secondary fails every time)
    2. Restarting both Primary and Secondary Servers
    3. Restarting the server protected by the Primary Server
    4. Removing protection for a particular Replica on the Secondary Server and then adding the Replica back into the Protection Group (including removing the backup data entirely)

    Suggestions?

    -Thanks

    Tuesday, June 7, 2011 3:22 PM

Answers

  • Hi,

     

    Another possibility is that the mountpoint for that data sources replica or diffarea is missing / broken.  Run mountvol.exe and check for volumes that don't have a mountpoint.

    You can get the link to the mountpoint by clicking on the "Path to Replica" under the details of the data souce, then check to be sure that pointpoint is healthy and that you can get to the replica and diffarea volumes from an administrative command prompt.  If it's missing, you need to recreate it using mountvol.exe or you can run dpmsync -sync command however the later will mark ALL the replica inconsistent and you will need to run CC againt all your data sources.

    Once the mountpoint(s) are fixed, then secondary protection should resume.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by TheWaker Tuesday, June 7, 2011 9:32 PM
    Tuesday, June 7, 2011 4:28 PM
    Moderator

All replies

  •  

    Hi,

    1) From an administrative command prompt, Please run:  vaadsmin list shadowstorage

    Shadow Copy Storage association
       For volume: (C:\Program Files\Microsoft DPM\DPM\Volumes\Replica\File System\vol_203279b9-1f5c-4613-844b-fcd0bf338780\)\\?\Volume{f6bb9dc8-7e61-11e0-ba8e-001c23cb7375}\
       Shadow Copy Storage volume: (C:\Program Files\Microsoft DPM\DPM\Volumes\DiffArea\File System\vol_203279b9-1f5c-4613-844b-fcd0bf338780\)\\?\Volume{f6bb9dce-7e61-11e0-ba8e-001c23cb7375}\
       Used Shadow Copy Storage space: 20.453 MB (0%)
       Allocated Shadow Copy Storage space: 627.344 MB (15%)
       Maximum Shadow Copy Storage space: UNBOUNDED (100%)    <--- MAKE SURE THIS SAYS UNBOUNDED

    2) Make sure Maximum shadow storage space for the effected data source shows UNBOUNDED - if not, run this command to make it unbounded - be sure to use the proper volumes as taken from the shadowstorage output:

    C:\>vssadmin resize shadowstorage /for=\\?\Volume{f6bb9dc8-7e61-11e0-ba8e-001c23cb7375} /on=\\?\Volume{f6bb9dce-7e61-11e0-ba8e-001c23cb7375} /maxsize=unbounded
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.

    Successfully resized the shadow copy storage association

    3) Make a new RP on the primary, then secondary.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Tuesday, June 7, 2011 4:00 PM
    Moderator
  • Both Primary and Secondary Servers list all data sources as UNBOUNDED (100%).
    Tuesday, June 7, 2011 4:07 PM
  • Hi,

     

    Another possibility is that the mountpoint for that data sources replica or diffarea is missing / broken.  Run mountvol.exe and check for volumes that don't have a mountpoint.

    You can get the link to the mountpoint by clicking on the "Path to Replica" under the details of the data souce, then check to be sure that pointpoint is healthy and that you can get to the replica and diffarea volumes from an administrative command prompt.  If it's missing, you need to recreate it using mountvol.exe or you can run dpmsync -sync command however the later will mark ALL the replica inconsistent and you will need to run CC againt all your data sources.

    Once the mountpoint(s) are fixed, then secondary protection should resume.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by TheWaker Tuesday, June 7, 2011 9:32 PM
    Tuesday, June 7, 2011 4:28 PM
    Moderator
  • I ran Mountvol and I appear to have quite a few replica volumes without Mount Points:

      \\?\Volume{ba1b8b49-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b4f-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b56-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b5d-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b65-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b6b-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b75-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b7c-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b8c-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8b92-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8ba3-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8ccd-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8cd3-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8cd9-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{ba1b8cdf-8bd9-11e0-862b-001018413716}\
          *** NO MOUNT POINTS ***

      \\?\Volume{c0375318-6b85-11e0-9a94-001018413716}\
          *** NO MOUNT POINTS ***

    I'll run DPMSync and clear any inconsistencies and let you know what happens.

    Tuesday, June 7, 2011 4:52 PM
  • Hi,

    Yes, this seems to be the problem - did you recently perform an operating system upgrade ?   An upgrade does not maintain mountpoints and will cause this issue to occur.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Tuesday, June 7, 2011 5:11 PM
    Moderator
  • Yes SP1 refused to install so I did an in-place upgrade to Server 2008 R2 RTM, then installed SP1 afterwards.

    Is there anything else that may be 'broken' in DPM due to an in-place upgrade? I did not realize that mountpoints were removed. Good to know.

    Tuesday, June 7, 2011 5:15 PM
  • Hi,

    Not that I am aware of.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Tuesday, June 7, 2011 5:45 PM
    Moderator
  • I had the same problem...secondary server showing replica inconsistent. I fixed it by installing windows server backup feature in the primary dpm server.
    Tuesday, July 24, 2012 6:03 PM