none
Old recovery points seems to not being cleaned up RRS feed

  • Question

  • I'm running a Windows Server 2012 server with DPM 2012 SP1, acting as a secondary DPM server for a couple of primary servers. However, the last 5-6 weeks it has begun to behave very strange. Suddenly, I get a lot of "Recovery Point volume threshold exceeded", "DPM does not have sufficient storage space available on the recovery point volume to create new recovery Points" and "The used disk space on the computer running DPM for the recovery point volume of SQL Server 2008 database XXXXX\DB(servername.domain.com) has exceeded the threshold value of 90% (DPM accounts 600 MB for internal usage in addition to free space available). If you do not allocate more disk space, synchronization jobs may fail due to insufficient disk space. (ID 3169).

    All of these alerts seem to have a common source - disk space of course, but there is currently 8 TB free in the DPM disk pool. However, I have a feeling that all of this started when we added another DPM disk to the storage pool. Could it be that DPM doesn't clean up expired disk data correctly any longer?

    /Amir

    Tuesday, July 9, 2013 7:10 PM

Answers

  • Case finally closed! MS provided us with an updated version of "objectmodel.dll" (will be part of QFE4), and after replacing this .dll in the BIN directory all old recovery points are now cleaned up correctly every night.

    /Amir

    • Marked as answer by Jaan Meijer Wednesday, October 9, 2013 12:03 PM
    Wednesday, October 9, 2013 12:02 PM

All replies

  • Hi,

    DPM removes expired recovery points every night at midnight by executing the pruneshadowcopiesdpm2010.ps1 powershell script.  Does the datasource in question have more days with recovery points than what the retention range is set for ?  If not, then pruning is working.

    You can get more information about what the script is doing by running these commands in DPM powershell.

    Type the following and see if it completes.

    $VerbosePreference = "Continue"
    pruneshadowcopiesDpm2010.ps1

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, July 10, 2013 5:00 AM
    Moderator
  • Hello Mike,

    I have run the script and gets a lot of following in the DPM Management Shell:

    VERBOSE: Incrementing distinct days
    VERBOSE: Recovery point time = 2013-06-03 00:20:53
    VERBOSE: Incrementing distinct days
    VERBOSE: Recovery point time = 2013-06-02 00:58:42
    VERBOSE: Incrementing distinct days

    However, it now seems to be stuck on the same line - it has remained without action for the latest hour. How do I see if it's completed or not? Do I have to reboot the server after completion of the script? If I check the number of the recovery points, it varies a lot, but it's not in the line with the retention time set in the PG. For instance, there is recovery points from January this year, even though I've set the retention time to 30 days for one of the groups.

    I also found this on the net, could it be something? The NetBIOS name of the DPM server is exactly 15 characters.

    http://blogs.technet.com/b/dpm/archive/2011/05/04/easy-dpm-2010-fix-disk-based-recovery-points-are-not-deleted-as-per-retention-goals.aspx

    /Amir


    • Edited by Jaan Meijer Wednesday, July 10, 2013 3:01 PM found more info on the internet
    Wednesday, July 10, 2013 12:40 PM
  • We had the same issue with our DPM 2012 box. You can run the command .\show-pruneshadowcopies.ps1 to see if there are recovery points that should be deleted.

    I ended up having to contact Microsoft to fix our issue. It took them several hours digging through SQL to locate the issue.

    Wednesday, July 10, 2013 6:16 PM
  • Thank you both for your replies. I ran the script show-pruneshadowcopies.ps1 and found thousands of recovery points that would be deleted (according to the result of the script), but for some reason they don't. Something is obviously preventing the pruneshadowcopiesdpm2010.ps1 from finishing too, or perhaps it just needs to be run several times as of the great amount of recovery points which are waiting for deletion.

    Wednesday, July 10, 2013 7:01 PM
  • Hi,

    If the pruneshadowcopiesdpm201.ps1 is not completing, hangs, or crashes, then that needs to be addressed as that will definitely cause storage usage problems.

    In the meantime you can use this powershell script to delete old recovery points to help free disk space.  It will prompt to select a datasource, then a date to delete all recovery points made before that time.

    #Author	: Ruud Baars
    #Date	: 11/09/2008
    #Edited : 11/15/2012 By: Wilson S.
    #edited : 11:27:2012 By: Mike J.
    
    # NOTE: Update script to only remove recovery points on Disk. Recovery points removed will be from the oldest one up to the date
    #       entered by the user while the script is running
    #deletes all recovery points before 'now' on selected data source.
    $version="V4.7" 
    $ErrorActionPreference = "silentlycontinue"
    add-pssnapin sqlservercmdletsnapin100
    Add-PSSnapin -Name Microsoft.DataProtectionManager.PowerShell
    #display RP's to delete and ask to continue. 
    #Check & wait data source to be idle else removal may fail (in Mojito filter on 'intent' to see the error)
    #Fixed prune default and logfile name and some logging lines (concatenate question + answer)
    #Check dependent recovery points do not pass BEFORE date and adjust selection to not select those ($reselect)
    #--- Fixed reselect logic to keep adjusting reselect for as long as older than BEFORE date
    #--- Fixed post removal rechecking logic to match what is done so far (was still geared to old logic)
    #--- Modified to remove making RP and ask for pruning, fixed logic for removal rechecking logic
    
    $MB=1024*1024
    $logfile="DPMdeleteRP.LOG"
    $wait=10	#seconds
    $confirmpreference = "None" 
    
    function Show_help
    {
    	cls
    	$l="=" * 79
    	write-host $l -foregroundcolor magenta
    	write-host -nonewline "`t<<<" -foregroundcolor white
    	write-host -nonewline " DANGEROUS :: MAY DELETE MANY RECOVERY POINTS " -foregroundcolor red
    	write-host ">>>" -foregroundcolor white
    	write-host $l -foregroundcolor magenta
    	write-host "Version: $version" -foregroundcolor cyan
        	write-host "A: User Selects data source to remove recovery points for" -foregroundcolor green
    	write-host "B: User enters date / time (using 24hr clock) to Delete recovery points" -foregroundcolor green
    	write-host "C: User Confirms deletion after list of recovery points to be deleted is displayed." -foregroundcolor green
    	write-host "Appending to log file $logfile`n" -foregroundcolor white
    	write-host "User Accepts all responsibilities by entering a data source or just pressing [Enter] " -foregroundcolor white -backgroundcolor blue
    
    }
    
    
    "**********************************" >> $logfile
    "Version $version" >> $logfile
    get-date >> $logfile
    show_help
    
    $DPMservername=&"hostname"
    "Selected DPM server = $DPMservername" >> $logfile
    write-host "`nConnnecting to DPM server retrieving data source list...`n" -foregroundcolor green
    $pglist = @(Get-ProtectionGroup $DPMservername) # WILSON - Created PGlist as array in case we have a single protection group.
    $ds=@()
    $tapes=$null
    $count = 0
    $dscount = 0
    foreach ($count in 0..($pglist.count - 1)) 
    {
        # write-host $pglist[$count].friendlyname
        $ds += @(get-datasource $pglist[$count]) # WILSON - Created DS as array in case we have a single protection group.
        # write-host $ds
        # write-host $count -foreground yellow 
    } 
    if ( Get-Datasource $DPMservername -inactive) {$ds += Get-Datasource $DPMservername -inactive}
    
    $i=0
    write-host "Index Protection Group     Computer             Path"
    write-host "---------------------------------------------------------------------------------"
    foreach ($l in $ds) 
    {
    	"[{0,3}] {1,-20} {2,-20} {3}" -f $i, $l.ProtectionGroupName, $l.psinfo.netbiosname, $l.logicalpath
    	$i++
    }
    $DSname=read-host "`nEnter a data source index from list above - Note co-located datasources on same replica will be effected"
    
    if (!$DSname) 
    {
    	write-host "No datasource selected `n" -foregroundcolor yellow
    	"Aborted on Datasource name" >> $logfile
    	exit 0
    }
    $DSselected=$ds[$DSname]
    if (!$DSselected) 
    {
    	write-host "No datasource selected `n" -foregroundcolor yellow
    	"Aborted on Datasource name" >> $logfile
    	exit 0
    }
    $rp=get-recoverypoint $DS[$dsname]
    $rp
    # $DoTape=read-host "`nDo you want to remove when recovery points are on tape ? [y/N]"
    # "Remove tape recovery point = $DoTape" >> $logfile
    
    write-host "`nCollecting recoverypoint information for datasource $DSselected.name" -foregroundcolor green
    if ($DSselected.ShadowCopyUsedspace -gt 0)
    {
        while ($DSSelected.TotalRecoveryPoints -eq 0)
        { # "still 0"
        }
    
    	#this is on disk
    	$oldShadowUsage=[math]::round($DSselected.ShadowCopyUsedspace/$MB,1)
    	$line=("Total recoverypoint usage {0} MB on DISK in {1} recovery points" -f $oldShadowUsage ,$DSselected.TotalRecoveryPoints  ) 
    	$line >> $logfile
    	write-host $line`n -foregroundcolor white
    }
    
    #this is on tape
    #$trptot=0
    #$tp= Get-RecoveryPoint($dsselected) | where {($_.Datalocation -eq "Media")}
    #foreach ($trp in $tp) {$trptot += $trp.size }
    #if ($trptot -gt 0 )
    #{
    #	$line=("Total recoverypoint usage {0} MB on TAPE in {1} recovery points" -f ($trptot/$MB) ,$DSselected.TotalRecoveryPoints  ) 
    #	$line >> $logfile
    #	write-host $line`n -foregroundcolor white		
    #}
    
    [datetime]$afterdate="1/1/1980"
    #$answer=read-host "`nDo you want to delete recovery points from the beginning [Y/n]" 
    #if ($answer -eq "n" )
    #{
    #	[datetime]$afterdate=read-host "Delete recovery points AFTER date [MM/DD/YYYY hh:mm]"
    #}
    [datetime]$enddate=read-host "Delete ALL Disk based recovery points BEFORE and Including date/time entered [MM/DD/YYYY hh:mm]" 
    "Deleting recovery points until $enddate" >>$logfile
    write-host "Deleting recovery points until and $enddate" -foregroundcolor yellow
    $rp=get-recoverypoint $DSselected
    
    if ($DoTape -ne "y" )
    {
    	$RPselected=$rp | where {($_.representedpointintime -le $enddate) -and ($_.Isincremental -eq $FALSE)-and ($_.DataLocation -eq "Disk")}
    }
    else
    {
    	$RPselected=$rp | where {($_.representedpointintime -le $enddate) -and ($_.Isincremental -eq $FALSE)}
    }
    
    if (!$RPselected) 
    {
    	write-host "No recovery points found!" -foregroundcolor yellow
    	"No recovery points found, aborting...!" >> $logfile
    	exit 0
    }
    $reselect = $enddate
    $adjustflag = $false
    foreach ($onerp in $RPselected)
    {
    	$rtime=[string]$onerp.representedpointintime
    	$rsize=[math]::round(($onerp.size/$MB),1)
    	$line= "Found {0}, RP size= {1} MB (If 0 MB, co-located datasource cannot be computed), Incremental={2} "-f $rtime, $rsize,$onerp.Isincremental
            $line >> $logfile
    	write-host "$line" -foregroundcolor yellow
    	#
    	#Get dependent rp's for data source
    	#
    	$allRPtbd=$DSselected.GetAllRecoveryPointsToBeDeleted($onerp)
    	foreach ($oneDrp in $allRPtbd)
    	{
    		if ($oneDrp.IsIncremental -eq $FALSE) {continue}
    		$rtime=[string]$oneDrp.representedpointintime
    		$rsize=[math]::round(($oneDrp.size/$MB),1)
    		$line= ("`t...is dependancy for {0} size {1} `tIncremental={2}" -f $rtime, $rsize, $oneDrp.Isincremental) 
    		$line >> $logfile
    		if ($oneDrp.representedpointintime -ge $enddate)
    		{
    			#stick to latest full ($oneDrp = dependents, $onerp = full)
    			$adjustflag = $true
    			$reselect = $onerp.representedpointintime
    			"<< Dependents newer than BEFORE date >>>" >> $logfile
    			Write-Host -nonewline "`t <<< later than BEFORE date >>>" -foregroundcolor white -backgroundcolor red
    			write-host "$line" -foregroundcolor yellow
    		}
    		else
    		{
    			#Ok, include current latest incremental
    			$reselect =  $oneDrp.representedpointintime
            	write-host "$line" -foregroundcolor yellow
    		}
    	}
    }
    if ($reselect -lt $oneDrp.representedpointintime) 
    {
    	#we adjusted further backward than latest incremental within selection
            $reselect =  $rtime 
    	$line =  "Adjusted BEFORE date to be $reselect to include dependents to $enddate"
    	$line >> $logfile
    	Write-Host $line -foregroundcolor white -backgroundcolor blue
    }
    $line="`n<<< SECOND TO LAST CHANCE TO ABORT - ONE MORE PROMPT TO CONFIRM. >>>"
    write-host $line  -foregroundcolor white -backgroundcolor blue
    $line >> $logfile
    $line="Above recovery points within adjusted range will be permanently deleted !!!"
    write-host $line -foregroundcolor red
    $line >> $logfile
    $line="These RP's include dependent recovery points and may contain co-located datasource(s)"
    write-host $line -foregroundcolor red 
    $line >> $logfile
    $line="Data source activity = " + $DSselected.Activity
    $line >> $logfile
    write-host $line -foregroundcolor white
    $DoDelete=""
    while (($DoDelete -ne "N" ) -and ($DoDelete -ne "Y"))
    {
    	$line="Continue with deletion (must answer) Y/N? "
    	write-host $line -foregroundcolor white
    	$DoDelete=read-host
    	$line = $line + $DoDelete
    	$line >> $logfile
    }
    
    if (!$DSselected.Activity -eq "Idle") 
    {
    	$line="Data source not idle, do you want to wait Y/N ? "
    	write-host $line -foregroundcolor yellow
    	$Y=read-host
    	$line = $line + $Y
    	$line >> $logfile
    	if ($Y -ieq "Y") 
    	{
    		Write-Host "Waiting for data source to become idle..."  -foregroundcolor green
    		while ($DSselected.Activity -ne "Idle")
    		{
    			("Waiting {0} seconds" -f $wait) >>$logfile
    			Write-Host -NoNewline "..." -ForegroundColor blue
    			start-sleep -s $wait
    		}
    	}
    }
    
    if ($DoDelete -eq "Y")
    {
       foreach ($onerp in $RPselected)
    	{
    		#reselect is adjusted to safe range relative to what was requested
    		#--- if adjustflag not set then all up to including else only older because we must keep the full
    		if ((($onerp.representedpointintime -le $reselect) -and ($adjustflag -eq $false)) -or ($onerp.representedpointintime -lt $reselect))
    		{
    			$rtime=[string]$onerp.representedpointintime
    			write-host `n$line -foregroundcolor red
    			$line >>$logfile
    			if (($onerp ) -and ($onerp.IsIncremental -eq $FALSE)) { remove-recoverypoint -RecoveryPoint $onerp -confirm:$True} # >> $logfile}
    			$line =("---`nDeleting recoverypoint -> " + $rtime)
    			$line >>$logfile
    		}
    	}
    }
    "All Done!" >> $logfile
    write-host "`nAll Done!`n`n" -foregroundcolor white
    $line="Do you want to View DPMdeleteRP.LOG file Y/N ? "
    	write-host $line -foregroundcolor white
    	$Y=read-host
    	$line = $line + $Y
    	$line >> $logfile
    	if ($Y -ieq "Y") 
    	{		
    	Notepad DPMdeleteRP.LOG
    	}

      


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, July 10, 2013 8:42 PM
    Moderator
  • Thanks Mike

    I have run that script to clear a lot of RPs for loads of datasources, but as the pruneshadowcopiesdpm2010.ps1 script still hangs after a while, I have opened up a MS support case in order to find the root cause of the problem.

    I'll post here as soon as we've found a solution.

    /Amir

    Friday, July 12, 2013 7:35 AM
  • Sorry to hijack old thread but did MS resolve your issue Amir.

    I am about to open a case with MS if we get no other resolution for the same problem.

    Dave

    Friday, August 2, 2013 1:09 PM
  • Sorry for the late Reply Dave, but believe it or not - the case is still active. MS responded last week confirming that this issue would be fixed in the next QFE, and that they are currently working with the Product team in order to get a private fix while waiting for it.

    Will post here again as soon as the issue is resolved.

    /Amir

    Monday, September 9, 2013 3:12 PM
  • Thanks for the update, Amir! We too have a case open with Microsoft Germany about the same issue for 4 weeks now. Interesting to hear that a fix is coming.

    Tuesday, September 10, 2013 8:41 AM
  • Case finally closed! MS provided us with an updated version of "objectmodel.dll" (will be part of QFE4), and after replacing this .dll in the BIN directory all old recovery points are now cleaned up correctly every night.

    /Amir

    • Marked as answer by Jaan Meijer Wednesday, October 9, 2013 12:03 PM
    Wednesday, October 9, 2013 12:02 PM
  • Case finally closed! MS provided us with an updated version of "objectmodel.dll" (will be part of QFE4), and after replacing this .dll in the BIN directory all old recovery points are now cleaned up correctly every night.

    /Amir

    Hey Amir, 

    Does this work for DPM 2012 R2 also? i'm also stuck in the same position as you are but im using 2012 R2


    M.Juma

    Friday, October 31, 2014 10:14 AM
  • Case finally closed! MS provided us with an updated version of "objectmodel.dll" (will be part of QFE4), and after replacing this .dll in the BIN directory all old recovery points are now cleaned up correctly every night.

    /Amir

    Hey Amir, 

    Does this work for DPM 2012 R2 also? i'm also stuck in the same position as you are but im using 2012 R2


    M.Juma

    same situation here with DPM 2012 R2

    Tuesday, July 12, 2016 8:58 PM