Incremental SQL backup failures RRS feed

  • Question

  • Have a DPM 2010 installation protecting a SQL 2008 server. The server has about a dozen databases. I am using colocation - and DPM decided to break up the colocation into 4 groups. 1 of the groups that protects about 6 small db's on the server seems to be having problems. Recovery points for any db in that group fail with the following error:

    Affected area: SQLSERVER\model
    Occurred since: 8/7/2010 11:02:10 AM
    Description: Recovery point creation jobs for SQL Server 2008 database SQLSERVER\model on have been failing. The number of failed recovery point creation jobs = 2.
     If the datasource protected is SharePoint, then click on the Error Details to view the list of databases for which recovery point creation failed. (ID 3114)
     An unexpected error occurred while the job was running. (ID 104 Details: Unspecified error (0x80004005))
     More information
    Recommended action: Retry the operation.
     Create a recovery point...
    Resolution: To dismiss the alert, click below
     Inactivate alert

    I checked Event Viewer on SQLSERVER and there's no indication that an attempt was even made to back up the database. Looking at the error logs on the DPM server, I see the following:

    0D7C 1F28 08/07 15:01:14.608 27 FsmBlock.cs(178)  49684505-8BEF-486D-A67F-4754C7A9043B WARNING Backup.ReplicaPreBackupBlock : RAPreBackup, StatusReason = Error (StatusCode = -2147467259, ErrorCode = GenericAgentFailure, workitem = ea337c0c-2225-4429-912c-c981ae00bbd5)
    0D7C 1F28 08/07 15:01:14.608 27 FsmBlock.cs(178)  49684505-8BEF-486D-A67F-4754C7A9043B WARNING Response: <?xml version="1.0"?>
    0D7C 1F28 08/07 15:01:14.608 27 FsmBlock.cs(178)  49684505-8BEF-486D-A67F-4754C7A9043B WARNING <Status xmlns="" StatusCode="-2147467259" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="c6f57fbe-5cda-4aa9-baaa-73b4dc84579a" GuidWorkItem="ea337c0c-2225-4429-912c-c981ae00bbd5" TETaskInstanceID="49684505-8bef-486d-a67f-4754c7a9043b"><ErrorInfo xmlns="" ErrorCode="998" DetailedCode="-2147467259" DetailedSource="2"/><RAStatus><RAPreBackup xmlns=""><BackupTime>129256668735970000</BackupTime><DSStatus><ComponentName>shipTalk</ComponentName><LogicalPath>AIRDB01</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    0D7C 13E4 08/07 15:01:14.734 27 FsmBlock.cs(178)  6E03CAF2-A6E0-468E-AF1F-B52803BAAB75 WARNING Backup.ReplicaPreBackupBlock : RAPreBackup, StatusReason = Error (StatusCode = -2147467259, ErrorCode = GenericAgentFailure, workitem = 57d8d566-9c92-48d2-b948-0e526821423f)
    0D7C 13E4 08/07 15:01:14.734 27 FsmBlock.cs(178)  6E03CAF2-A6E0-468E-AF1F-B52803BAAB75 WARNING Response: <?xml version="1.0"?>
    0D7C 13E4 08/07 15:01:14.734 27 FsmBlock.cs(178)  6E03CAF2-A6E0-468E-AF1F-B52803BAAB75 WARNING <Status xmlns="" StatusCode="-2147467259" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="5eaefc21-a0df-4523-8b36-fd9f2c1df8df" GuidWorkItem="57d8d566-9c92-48d2-b948-0e526821423f" TETaskInstanceID="6e03caf2-a6e0-468e-af1f-b52803baab75"><ErrorInfo xmlns="" ErrorCode="998" DetailedCode="-2147467259" DetailedSource="2"/><RAStatus><RAPreBackup xmlns=""><BackupTime>129256668736680000</BackupTime><DSStatus><ComponentName>distribution</ComponentName><LogicalPath>AIRDB01</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    0D7C 1E94 08/07 15:01:14.770 27 FsmBlock.cs(130)  017485BD-9BC3-444B-8EA0-19273D9C3EC1 WARNING Backup.ReplicaPreBackupBlock : <-- Exited FSM block with FAILURE (errorCode = RmGenericError)
    0D7C 11FC 08/07 15:01:14.791 27 FsmBlock.cs(178)  7027129B-3E61-4D1B-B3D4-F09FD255BE21 WARNING Backup.ReplicaPreBackupBlock : RAPreBackup, StatusReason = Error (StatusCode = -2147467259, ErrorCode = GenericAgentFailure, workitem = 0e8ae96b-ea84-4634-954a-c3885125fba9)
    0D7C 11FC 08/07 15:01:14.791 27 FsmBlock.cs(178)  7027129B-3E61-4D1B-B3D4-F09FD255BE21 WARNING Response: <?xml version="1.0"?>
    0D7C 11FC 08/07 15:01:14.791 27 FsmBlock.cs(178)  7027129B-3E61-4D1B-B3D4-F09FD255BE21 WARNING <Status xmlns="" StatusCode="-2147467259" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="32bc8ae2-d269-4f45-9024-8e6119f39244" GuidWorkItem="0e8ae96b-ea84-4634-954a-c3885125fba9" TETaskInstanceID="7027129b-3e61-4d1b-b3d4-f09fd255be21"><ErrorInfo xmlns="" ErrorCode="998" DetailedCode="-2147467259" DetailedSource="2"/><RAStatus><RAPreBackup xmlns=""><BackupTime>129256668739500000</BackupTime><DSStatus><ComponentName>otp_extranet_reporting</ComponentName><LogicalPath>AIRDB01</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    0D7C 1F28 08/07 15:01:14.847 27 CommonErrorHandler.cs(165)  49684505-8BEF-486D-A67F-4754C7A9043B WARNING AgentStatus[RAForRead] - (CommandID=RAPreBackup, StatusReason=Error) failed with HRESULT 0x80004005, error -2147467259.
    0D7C 1F28 08/07 15:01:14.847 27 CommonErrorHandler.cs(265)  49684505-8BEF-486D-A67F-4754C7A9043B NORMAL Unmapped agent error code = GenericAgentFailure
    0D7C 13E4 08/07 15:01:14.852 27 CommonErrorHandler.cs(165)  6E03CAF2-A6E0-468E-AF1F-B52803BAAB75 WARNING AgentStatus[RAForRead] - (CommandID=RAPreBackup, StatusReason=Error) failed with HRESULT 0x80004005, error -2147467259.
    0D7C 13E4 08/07 15:01:14.853 27 CommonErrorHandler.cs(265)  6E03CAF2-A6E0-468E-AF1F-B52803BAAB75 NORMAL Unmapped agent error code = GenericAgentFailure

    If I manually run a consistency check, the alert will clear until the next incremental backup. If I run an express backup, the alert will clear until the next incremental backup. All of the db's being backed up are in the same protection group, the only thing these db's that are failing have in common, is that they are all colocated on the same volume. Any ideas?

    Jeff Graves, ORCS Web, Inc.
    Saturday, August 7, 2010 4:05 PM


  • Tx Danny. It looks like a reboot ended up fixing this. We had a maintenance window to install patches on 9/18 and since then, I have not seen this error. Chalk it up to "ghosts" in the server.

    Jeff Graves, ORCS Web, Inc.
    Thursday, September 30, 2010 8:20 PM

All replies

  • Hi Jeff,

    Are those databases that are running only the Express Full and not the incremental backups set to a recovery mode of simple on the SQL server? Or were they in the past set to advanced recovery mode and are now simple recovery mode?

    From what I understand with DPM 2010 and SQL 2008 is that you can only perform an express full on databases that are in simple recovery mode.

    Friday, August 13, 2010 2:30 AM
  • Thanks for the follow-up Danny. Looking at the databases in question, all of them are set to Full recovery mode. As I'm not the DBA, I can't say if they were set to simple at one point, but that's certainly a possibility. The strange thing is that this seems to be failing before the VSS writer is even called on the protected server. Also, I tried deleting the colocated replica and recreating it but am seeing the same results.

    Jeff Graves, ORCS Web, Inc.
    Friday, August 13, 2010 2:53 PM
  • Given that the SQL Consistency Check and Express Full succeed this means DPM is able to transfer changes to the database files themselves but not the database log files. Since an incremental backup is essentially copying just the log files it could be an issue with the DPM server accessing the volume that the database log files reside on or the configuration of the SQL transaction logs. At one of the error messages indicates "If the database transaction log unexpectedly becomes full on a computer that is running SQL Server, your synchronization jobs will start to fail."

    Maybe check with the DBA to see whether there is transaction log space available and if manual backup processes on those SQL transaction logs will succeed.

    Monday, August 16, 2010 5:00 AM
  • Looking at each of the DB configurations, it definitely seems to be something to do with the DB recovery model. All of these DB's that are failing are set to Full - any that are set to simple work properly. It seems to just be a coincidence that all of those set to full are also colocated in the same volume.

    This protection group has disk resources as well. Is it possible to change all of the SQL objects to just daily express full backups without changing their recovery model to simple or creating a separate protection group for them? Some of the other db's are already in simple mode and DPM automatically detected that and does express fulls only. Can I manually set the rest that way via powershell? We don't need hourly sync's for the DB's, but do for disk. These have tape protection too, but co-location is disabled because of other issues with other protection groups and different retention periods - thus the reason all of these objects are in the same protection group.

    Jeff Graves, ORCS Web, Inc.
    Monday, August 16, 2010 8:41 PM
  • Hi Jeff - can you confirm with your dbadmin that the actual transaction logs options (for the databases set to full recovery mode) do not have a limit to their size set - not the amount of space on the SQL disk or on the DPM disk?

    From what I understand when the database recovery mode is in full, even if you set a recovery point to take place immediately before an express full it will still try to transfer the SQL transaction log files across to the DPM server. I'm not aware of any powershell to configure those protected data sources to only do an express full.

    You could check the security permissions to the database log files (if in a different location to the databases). The DPMRA service runs by default under the context of local system so you could verify that the local system account on the SQL server has access to the log files.

    Can you also check the ScriptingConfig.xml file on the protected server in the install path\Microsoft Data Protection Manager\DPM\Scripting folder to make sure you don't have anything trying to call a script on the SQL server prior to the DPM protection running.

    Hope this helps

    Thursday, August 19, 2010 7:04 AM
  • I checked and there is plenty of space on the partition holding the logs (73GB free). I also checked the log configuration in SQL and all of the databases are configured to let their logs grow without a limit. I also checked the security permissions on the log folder directory, and they are correct. The ScriptingConfig.xml file contains just the header informaiont - no scripts are configured to run.

    As I mentioned initially, looking at EV and the protected server, there's not even any indication a backup is trying to take place during these incremental log backups.

    Jeff Graves, ORCS Web, Inc.
    Thursday, August 19, 2010 2:05 PM
  • Sorry for the gap there ......

    I'm running out of ideas on this one - given the express full and consistency check is succeeding but the incrementals are failing - are any SQL backups taking place I.e. through a SQL task is the SQL Administrator doing any maintenance tasks to backup and truncate the SQL databases on a regular basis or manually is this being done on a regular basis? Is any transaction log shipping taking place?

    Also if you run a consistency check or express full manually then immediately try to run an incremental does it fail?


    Friday, September 17, 2010 5:10 AM
  • Tx Danny. It looks like a reboot ended up fixing this. We had a maintenance window to install patches on 9/18 and since then, I have not seen this error. Chalk it up to "ghosts" in the server.

    Jeff Graves, ORCS Web, Inc.
    Thursday, September 30, 2010 8:20 PM