none
DPM 2010 "recovery point creation failed" for SQL Data RRS feed

  • Question

  • I am learning to use System Center and performed a fresh install this week of DPM 2010 on a Hyper-V host server. Using the Protection Wizard, I successfully created a protection group for a Windows 7 VM and it backed up overnight successfully. However, though I successfully created a protection group for an SQL VM on another Hyper-V host, it failed overnight to create the recovery point with only the error message "recovery point error message failed" -- no drill down available on the error message.

    1) The SQL Server VSS Writer is started & running on the SQL Server. SQL 2008 Standard Edition is the only application running on this dedicated SQL server. The SQL databases are located on a SAN attached to the HBA card on the SQL server.

    2) Both the DPM 2010 server and the SQL Server are logged in using the Domain Administrator account.

    3) In the DPM Management console's Agents tab, the Agent status is "OK" for the SQL server.

    4) Reviewing my choices made during the Create Protection Group Wizard: Group Type is Server, Members has the path to the two databases in the SQL instance on the SQL server, Protection media is Short Term - Disk, Total disk size is "Co-locate data in DPM Storage Pool, automatically grow volume," the disk is 2TB with 80% free and the Windows 7 VM successfully saved to it last night, Retention Range is the default 5 days, Synchroniztion is the default 15 minutes, Express Full Backup is 7pm weekdays.

    I do not see what different choices I could have made in the protection group creation, and the error message is too vague to know what step is blocking the replica creation.

    Suggestions?

    Jeffrey, New Orleans

    MCITP Enterprise Administrator, Virtualization Administrator

    Wednesday, October 26, 2011 2:45 PM

Answers

  • Hi Jeffrey,

    Here is the code you need:

    $pg = @(Get-ProtectionGroup (&hostname))
    $ds = @(Get-Datasource $pg[<N1>])
    New-RecoveryPoint -Datasource $ds[<N2>] -disk -backuptype expressfull
    

    Example:

    # I want to backup database msdb1 that is on protection group SQL.

    # $pg will contain all protection groups I have on my DPM server. On $pg output below, SQL group was the fourth in the list. As the list starts from 0, SQL # protection Group ID will be 3 <N1>.


    PS C:\Program Files\Microsoft DPM\DPM\bin> $pg = @(Get-ProtectionGroup (&hostname))
    PS C:\Program Files\Microsoft DPM\DPM\bin> $pg | ft -AutoSize

    Name                           Protection method
    ----                              -----------------
    WILSON-R2 - Folder      Short-term using disk
    Exchange Mailbox          Short-term using disk
    Bare Metal Protection     Short-term using disk
    SQL                              Short-term using disk

    # $ds is the list of data sources within a protection group. From the list below, msdb1 is the first on the list. Because the list starts from 0, if I want select msdb1, its number is 0 <N2>....

    PS C:\Program Files\Microsoft DPM\DPM\bin> $ds = @(Get-Datasource $pg[3])
    PS C:\Program Files\Microsoft DPM\DPM\bin> $ds | ft -AutoSize

    Computer               Name                                                          Type
    --------                   ----                                                             ----
    WILSON-SQLVIVEK WILSON-SQLCLUS1\SQLINSTANCE\msdb1      SQL Server 2008 database
    WILSON-R2            WILSON-R2\MSDPM2010\VMM                       SQL Server 2008 database
    WILSON-R2            WILSON-R2\MSDPM2010\test                        SQL Server 2008 database

    # This is the command that will fire the Recoery Point creation for msdb1

    PS C:\Program Files\Microsoft DPM\DPM\bin> New-RecoveryPoint -Datasource $ds[0] -disk -backuptype expressfull

    JobCategory     Status
    -----------         ------
    ShadowCopy    InProgress

     

    You can check the status of the job via DPM 2010 Administrator Console, Monitoring, Jobs tab.


    Thanks, Wilson Souza - MSFT This posting is provided "AS IS" with no warranties, and confers no rights
    Tuesday, November 1, 2011 9:33 PM
    Moderator

All replies

  • Hi Jeffrey,

    Where did you look for the error message "recovery point error message failed"?

    You can go to to DPM Console, Monitoring/Jobs and create a filter for the failed Database.

    To create the filter, do the following:

    1. Go to Monitoring/Jobs and click on Create Filter on the right hand side

    2. Change the highlighted information accordingly. Then click on Protection tab and just select the database you are having problem with. Once you are done, click on the Preview button.

    3. On group by, select Computer. Then Highlight the first error you find after the last successfull backup. Under Details Pane you have the actuall error message. Copy and paste here in the forum.


    Thanks, Wilson Souza - MSFT This posting is provided "AS IS" with no warranties, and confers no rights
    Friday, October 28, 2011 6:04 AM
    Moderator
  • Dear Wilson:

    Thanks for providing steps on how to get more detail on the error message. Since I have just installed DPM 2010 this week and this is the first protection group I have created for a SQL Server, I don't have an example of a successful DPM SQL backup from which to copy settings. The error message "replica creation point failed" appears when I open the DPM console's Protection tab and browse to Protection group 2 and the LAKEWAY1-SQL computer.

    Protection > Protection Group 2 > Computer > LAKEWAY1-SQL

    Status: Recovery point creation failed
    Replica path: Click to view details
    Latest recovery point: 10/27/2011 9:00:02 PM
    Oldest recovery point: 10/25/2011 6:03:31 PM
    Total recovery points: 31
    Disk allocation: Replica volume (co-located): 10.00 GB allocated, 6.70 GB used
    Recovery point volume (co-located): 7.58 GB allocated, 712.34 MB used
    Parent instance: LAKEWAY1-SQL\JURISSQL

    Following your steps, here is the more detailed message:

    Monitoring > Jobs > Failed jobs
    Type: Recovery point
    Status: Failed

    Description: DPM tried to do a SQL log backup, either as part of a backup job or a recovery to latest point in time job. The SQL log backup job has detected a discontinuity in the SQL log chain for  SQL Server 2008 database database LAKEWAY1-SQL\JURISSQL\Juris8083000 since the last backup. All incremental backup jobs will fail until an express full backup runs. (ID 30140 Details: Internal error code: 0x80990D11)

    More information
    End time: 10/28/2011 11:15:12 AM
    Start time: 10/28/2011 11:15:02 AM
    Time elapsed: 00:00:09
    Data transferred: 0 MB
    Cluster node -
    Recovery Point Type Incremental Sync
    Source details:  LAKEWAY1-SQL\JURISSQL\Juris8083000
    Protection group: Protection Group 2 - SQL Juris databases

    Looking at Server Manager > Disk Management on the DPM 2010 server, I can see where space on the empty 2TB unformatted NTFS drive has been successfully allocated to the DPM; both the 10GB replica volume and 7.58GB replica point volume are listed as "Healthy".

    Suspecting permissions on the LAKEWAY1-SQL virtual machine, I have browsed to its Local Users & Groups and see the groups DPMRADCPMTrustedMachines and DPMRADmTrustedMachines both have the computer account of the DPM 2010 server as a member. The built-in Adminsitrators group has Domain Admins and the computer account of the DPM Server as members. I logged into the DPM 2010 server as using the Domain Administrator account then installed DPM 2010. I am still logged into the server using the Domain Administrators account.

    So, in summary, the two DPM volumes have been created and are "healthy," the DPM 2010 server's computer account and the Domain Administrator seem to have permissions on the LAKEWAY1-SQL server, and I have created a generic protection group, type server, and pointed it to two databases on the LAKEWAY1-SQL server, but it fails to create the initial replica point.

    I hope to learn how to configure the DPM 2010 software correctly, then persuade my clients in the financial, medical, and legal fields to shift to this [better] backup methodology as a first step toward using the integrated System Center products. I moved them this year to Virtual Machines and the VMM Manager.


    Jeffrey - New Orleans MCITP Enterprise Administrator, Virtualization Administrator
    Friday, October 28, 2011 4:55 PM
  • Hi, Jeffrey, It looks like protection is working 'Ok' as the last successfull recovery point from the output you posted occurred yesterday @ 9 PM. The reason for the error you are getting ("The SQL log backup job has detected a discontinuity in the SQL log chain") means that there is another backup running besides DPM. This could be another backup solution backing up this database or having a SQL maintenance job that does backup on this database. This means that this other backup is truncating SQL logs and when it is time for DPM to run its backup it notices that some info in the log went missing. The only time DPM will allow a backup to complete when you have SQL log discontinuity is when an express full backup takes place and in your scenario your express full runs @ 9 PM. To avoid this from happening you will need to ensure DPM is the only backup program that touches this database. Hope this helps.
    Thanks, Wilson Souza - MSFT This posting is provided "AS IS" with no warranties, and confers no rights
    • Proposed as answer by Simone_Bennett Friday, September 30, 2016 1:35 AM
    Friday, October 28, 2011 6:45 PM
    Moderator
  • Dear Wilson:

    Thank you for reviewing the logs -- I have tested your sugggestion that another backup task is causing the "discontinuity in the SQL log chain."

    After manually creating a [Full Express] replica on Saturday morning, the DPM 2010 incremental backups ran successfully until 9pm, which coincidentally is when the Microsoft SQL Management Studio runs its daily maintenance task, creating an SQL dump file that is copied (by a simple Microsoft Server 2008 scheduled task) to the disaster site file server.

    On Sunday morning, I again manually created a [Full Express] replica and the DPM incremental backups ran until 9pm, so I believe this is sufficient evidence that you are correct -- the built-in SQL task leaves the DPM incremental backup unable to execute indefinitely, not merely during the 5 minute interval that the Microsoft SQL task creates the SQL dump file. It is unfortunate that the two Microsoft products (SQL, DPM) interfere with each other -- perhaps that will be addressed in a later version of DPM.

    Up until now, I have been using a SonicWALL CDP appliance that had an SQL incremental agent that was not disrupted by the Microsoft maintenance task, enabling this law firm to have two independent backup systems -- no single point of failure. I completely uninstalled the SonicWALL software before installing the DPM software to replace it. I would like to avoid disabling the Microsoft SQL maintenance task and relying solely on the DPM software -- a single point of failure -- this would be a step backward in the robustness of the current backup system.

    As a work-around, is there a way to create a second DPM job that creates a Full Express replica once a day at 9:15pm (after the Microsoft SQL maintenance task jas created the SQL dump file)? Would this second DPM backup job also interfere & disable the DPM incremental backup jobs or would the next incremental backup (at 9:30pm) run successfully?

    Jeffrey, New Orleans

    MCITP Enterprise/Virtualization Administrator, CISSP-ISSAP, CISA


    Jeffrey - New Orleans MCITP Enterprise Administrator, Virtualization Administrator
    Monday, October 31, 2011 2:01 PM
  • Hi Jeffrey,

    An option would be to create a PowerShell script to run an Express Full backup around 9:15 PM, 9:30 PM.

    And yes, if the express full is running while it is time for a Sync job to start, the Sync job will fail saying that it can't run because there is an ongoing job.....

    Another option is to do change your backup configuration from the SQL side to run with NO_TRUNCATE or COPY_ONLY option....

    http://msdn.microsoft.com/en-us/library/ms186865(v=SQL.105).aspx

     

     

     


    Thanks, Wilson Souza - MSFT This posting is provided "AS IS" with no warranties, and confers no rights
    Monday, October 31, 2011 8:37 PM
    Moderator
  • Dear Wilson:

    Thanks you for your help in better understanding how the DPM 2010 product interacts with SQL software.

    Since some of my clients are banks that receive ATM transaction 24x7, I worry about the warning that the NO_TRUNCATE option might capture the database in an inconsistent state; however, since the SQL Maintenance Task is a once a night full backup (to SQL dump file), I will inquire with the application vendor about the consequences of modifying their SQL Maintenance Task to include the COPY_ONLY option.

    In the meantime, I have modified the DPM Protection Group settings so that the Express Full Replica runs daily at 9:30pm (after the SQL Maintenance Task runs from 9:00pm to 9:07pm). Though the DPM incremental backup will fail at 9:15pm and 9:30pm, it should then succeed at 9:45pm because an Express Full Replica was just created at 9:30pm. I will check the logs tomorrow and see if this succeeds.

    If you can point me to documentation examples of a PowerShell script that creates a DPM Express Full backup, I would appreciate it -- I could not find any searching the web & manuals. Helpfully, some of the other System Center products allow one to View & Save the script from each Action Pane item, perhaps a future feature for DPM.

    Jeffrey

    New Orleans

     

     


    Jeffrey - New Orleans MCITP Enterprise Administrator, Virtualization Administrator
    Tuesday, November 1, 2011 12:56 PM
  • Hi Jeffrey,

    Here is the code you need:

    $pg = @(Get-ProtectionGroup (&hostname))
    $ds = @(Get-Datasource $pg[<N1>])
    New-RecoveryPoint -Datasource $ds[<N2>] -disk -backuptype expressfull
    

    Example:

    # I want to backup database msdb1 that is on protection group SQL.

    # $pg will contain all protection groups I have on my DPM server. On $pg output below, SQL group was the fourth in the list. As the list starts from 0, SQL # protection Group ID will be 3 <N1>.


    PS C:\Program Files\Microsoft DPM\DPM\bin> $pg = @(Get-ProtectionGroup (&hostname))
    PS C:\Program Files\Microsoft DPM\DPM\bin> $pg | ft -AutoSize

    Name                           Protection method
    ----                              -----------------
    WILSON-R2 - Folder      Short-term using disk
    Exchange Mailbox          Short-term using disk
    Bare Metal Protection     Short-term using disk
    SQL                              Short-term using disk

    # $ds is the list of data sources within a protection group. From the list below, msdb1 is the first on the list. Because the list starts from 0, if I want select msdb1, its number is 0 <N2>....

    PS C:\Program Files\Microsoft DPM\DPM\bin> $ds = @(Get-Datasource $pg[3])
    PS C:\Program Files\Microsoft DPM\DPM\bin> $ds | ft -AutoSize

    Computer               Name                                                          Type
    --------                   ----                                                             ----
    WILSON-SQLVIVEK WILSON-SQLCLUS1\SQLINSTANCE\msdb1      SQL Server 2008 database
    WILSON-R2            WILSON-R2\MSDPM2010\VMM                       SQL Server 2008 database
    WILSON-R2            WILSON-R2\MSDPM2010\test                        SQL Server 2008 database

    # This is the command that will fire the Recoery Point creation for msdb1

    PS C:\Program Files\Microsoft DPM\DPM\bin> New-RecoveryPoint -Datasource $ds[0] -disk -backuptype expressfull

    JobCategory     Status
    -----------         ------
    ShadowCopy    InProgress

     

    You can check the status of the job via DPM 2010 Administrator Console, Monitoring, Jobs tab.


    Thanks, Wilson Souza - MSFT This posting is provided "AS IS" with no warranties, and confers no rights
    Tuesday, November 1, 2011 9:33 PM
    Moderator