none
What is the recommended way to perform tape verification? RRS feed

  • Question

  • I currently have 12 protection groups with a total of about 30 protected members.  I have the "Check backup for data integrity (time consuming operation)" option enabled for all jobs.  The problem is with the way that DPM 2012R2 performs verification.  Here is the chronology of backing up an SCCM server's SQL databases that I just witnessed:

    The summary of what DPM did is as follows:

    • write and verify approximately 5.6 GB of data
    • unload and load the same tape 8 times
    • elapsed time: 43 minutes, 6 seconds
    • average data rate: 2.2 megabytes per second

    When doing verification DPM unloads and loads the same tape once for each protected member.  Obviously this doesn't scale.  Furthermore, these unnecessary cycles of the tape loading mechanism will reduce the life of the tape library because the mechanism has a mean time before failure measured in tape load cycles.  So the question is, what is the currently recommend practice for achieving tape verification with DPM 2012R2? 

    I have read here that "Tape verify jobs should be scheduled to start after all the tape backups jobs finish."  Is the corollary to this statement to "disable Check backup for data integrity" on all protection groups?  Also, if this is indeed the recommended practice, then how, exactly, do you "schedule a tape verify job to start after all the tape backup jobs finish"?

    Thanks for your help,

    Alex



    • Edited by alx9r Tuesday, August 26, 2014 6:16 PM
    Tuesday, August 26, 2014 4:55 PM

All replies

  • Hi,

    The screen shot above shows that all the tape backups completed, then the tape verification jobs ran.  I think your only concern is that each verify operation is totally independent and DPM does not keep the tape mounted when the next verification job requires the same tape.

    That is a valid concern for all the reasons stated.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, August 26, 2014 5:36 PM
    Moderator
  • Is there some other way to get DPM to perform tape verification of all of the jobs on a single tape without unloading and loading between each protected member?  Does recovery confirm data integrity?  Is there a way to initiate a recovery of the contents of an entire tape without reloading? 
    Tuesday, August 26, 2014 6:14 PM
  • Hi,

    Unfortunately, the only way to test tape data is to use the below powershell command, but again, that performs the verification for each data source, not the entire tape.

    Test-DPMTapeData
    http://technet.microsoft.com/en-nz/library/ff631948(en-us).aspx


    PS C:\Program Files\Microsoft DPM\DPM\bin> get-help Test-DPMTapeData -examples

    NAME
        Test-DPMTapeData

    SYNOPSIS
        Verifies the data set for a recovery point.

        EXAMPLE 1

        $pg = Get-ProtectionGroup -DPMServerName TestingServer
        $ds = Get-Datasource -ProtectionGroup $pg[x]
        $rp = Get-RecoveryPoint -Datasource $ds[x]
        $rsl = Get-RecoveryPointLocation -RecoveryPoint $rp[1]
        Test-DPMTapeData -RecoveryPoint $rp[1] -RecoveryPointLocation $rsl


        This command verifies the recovery point at the specified location.

     

    Yes, restoring from the tape is a valid verification that the backup is good.  Any good backup admin will perform test restores periodically to verify backups working and are restorable.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, August 26, 2014 6:23 PM
    Moderator
  • Ugh.  This is looking pretty awkward.  Here are the facts as I see them.  If you want to verify that the entire contents of a tape was correctly written you have three options:

    • enable "Check backup for data integrity (time consuming operation)" for each PG that is written to the tape
    • run Test-DPMTapeData for each recovery point on the tape
    • recover each recovery point on the tape

    Each and every one of these options results in a minimum of one tape load and unload cycle per recovery point.  I am using a Dell TL2000 tape library and DPM 2012R2.  Based on the example in my original post, DPM was able to load, verify, and unload 6 recovery points in 32 minutes for a rate of approximately 11 recovery points per hour.  If you are doing daily tape backups, then the absolute highest number of recovery points you can ever expect to verify is 11*24=264 recovery points.  This is so even if all of those recovery points are on a single tape and is a best-case scenario assuming 100% of the duty cycle of the tape library is dedicated to verification, which of course would never be the case in real life.

    If I have made a factual error here please correct me.  Assuming I have these facts correct, we can conclude the following:

    Using DPM 2012R2 there is no possible method to comprehensively verify the contents of daily tape backups if there are more than approximately 250 recovery points per day.

    Above that limit, the most verification you could hope for is spot-checking.  Furthermore, the life expectancy of a tape library is likely to be reduced to months from years if it is performing 250 tape load cycles every day.  This is rather an unacceptable result for an enterprise-class backup system.  The solution is straightforward: DPM should provide a means of verifying, copying, or recovering all recovery points on a single tape in a single load/read/unload cycle.

    Am I missing something here?  I just don't see how any form of substantial tape backup verification can work using DPM in its current form at scale.

    Alex

    Tuesday, August 26, 2014 8:36 PM
  • Hi,

    The amount of time to verify a dataset depends on how large that dataset is.  As noted, the more datasets on the tape, the more load / unload operations performed which does waste time.  I will run this issue by the DPM product group and see if we can improve upon current design.

    FYI - With modern LTO tape technology - the importance of tape verification done by the application is reduced.  LTO tape drive technology verifies data immediately after write and will rewrite the data if it needs to.  See this LTO primer.  However, with that said, application verification is an added insurance policy and is still a recommended best practice.  Better to be safe than sorry.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, August 26, 2014 9:59 PM
    Moderator
  • Hi Mike,

    Thank you for escalating this.  I am otherwise quite pleased with DPM 2012 R2 so far.  But not being able to verify tapes is (and should be for all tape users) a showstopper. 

    I have not come across the rewrite guarantee you mentioned in the product literature for my tape library.  It is LTO6.  I will do some research to determine how reliable the writes for my particular hardware are.  Regardless of how reliable the writes are, however, we will all encounter the same problem when attempting to verify the entire contents of a tape during off-site retrieval audits.  If you have thousands of recovery points on a tape, it will take weeks to verify the contents of that tape.

    Thanks for your help,

    Alex

    Tuesday, August 26, 2014 10:14 PM
  • Hi Mike,

    I'm wondering if you have any news that might allow me to reduce the number of tape load cycles when performing tape verification.  I investigated the efficacy of the LTO verify-on-write scheme and found that we cannot rely on that scheme alone because there doesn't seem to be any method supported by either Microsoft or Dell for testing verify-on-write failures.  So I can't tell how either the tape drive or DPM will behave if there are verify-on-write failures.

    Alex

    Monday, September 29, 2014 4:53 PM