none
DPM 2010, error ID910 SQL deadlock RRS feed

  • Question

  • I am running dpm 2010.

    I am getting this error over the past few days and a reboot won't fix it.

    It has happened in the past but is now pretty consistent,

    Below is the message in DPM, the event in the application log, and the sql log.

    Can someone make any suggestions on how to fix.

    Any assistance would be greatly apreciated.

     

     

     

    pp

    The DPM service terminated unexpectedly during completion of the job. The termination may have been caused by a system reboot. (ID 910)

    In the application log this is displayed

    The description for Event ID 945 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    Unable to connect to the DPM database because of a general database failure. Make sure that SQL Server is running and that it is configured correctly.

    Problem Details:

    <FatalServiceError><__System><ID>19</ID><Seq>11558</Seq><TimeCreated>1/09/2010 6:30:07 PM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>NonFatalDbException</ExceptionType><ExceptionMessage>exception</ExceptionMessage><ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.DB.NonFatalDbException: exception ---&gt; System.Data.SqlClient.SqlException: Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)

    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)

    at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)

    at System.Data.SqlClient.SqlDataReader.HasMoreRows()

    at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)

    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryReader.Read()

    --- End of inner exception stack trace ---

    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryReader.Read()

    at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.Replica.ReplicaTrail.GetActiveTasks(DbContext ctx, Guid replicaId, Guid verbId, BackupTypeType backupType)

    at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.PrmCatalog.GetActiveTasksForReplicaTrail(Guid replicaId, Guid verbId, BackupTypeType backupType)

    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ProtectionRecoveryTaskExecutor.FailIfSimilarActiveTaskFound(Guid taskId, Guid datasourceId, Guid replicaId, Guid verbId, BackupTypeType backupType)

    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ProtectionRecoveryTaskExecutor.InitializeBackupTask(Guid jobId, Guid verbId, Guid taskId, BackupTaskDefinitionType taskDef)

    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ProtectionRecoveryTaskExecutor.GetBackupTypeTaskXml(Guid verbId, Guid jobId, TaskDefType taskDef, Guid taskId)

    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ProtectionRecoveryTaskExecutor.GenerateWorkplan(Guid jobId, WorkPlanFlags flags, TaskDefType taskDef, String jobDef, Guid taskId, Boolean&amp; isSerial)

    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskExecutor.GenerateWorkplan(Guid jobID, Guid taskID, String jobDef, String taskDef, WorkPlanFlags flags, GenerateWorkplanCallback callback)

    at Microsoft.Internal.EnterpriseStorage.Dls.JobManager.ActiveJob.DispatchRequestCallback(Object state)

    at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.DpmThreadPool.Function(Object state)

    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)

    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)

    *** Mojito error was: DatabaseNonFatalError; 0; None

    --- SqlException details -----------------

    System.Data.SqlClient.SqlException: Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)

    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)

    at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)

    at System.Data.SqlClient.SqlDataReader.HasMoreRows()

    at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)

    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryReader.Read()

    Error = 1205

    Index #0

    Source: .Net SqlClient Data Provider

    Number: 1205

    State: 51

    Class: 13

    Server: DPM\MSDPM2010

    Message: Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

    Procedure: prc_RM_ReplicaTrail_GetActiveTasks

    Line: 16

    --- Caller StackTrace -----------------

    FileName:; Method:Read(); lineNo:0; ilOffset:2.

    FileName:; Method:GetActiveTasks(); lineNo:0; ilOffset:127.

    FileName:; Method:GetActiveTasksForReplicaTrail(); lineNo:0; ilOffset:95.

    FileName:; Method:FailIfSimilarActiveTaskFound(); lineNo:0; ilOffset:0.

    FileName:; Method:InitializeBackupTask(); lineNo:0; ilOffset:2165.

    FileName:; Method:GetBackupTypeTaskXml(); lineNo:0; ilOffset:162.

    FileName:; Method:GenerateWorkplan(); lineNo:0; ilOffset:281.

    FileName:; Method:GenerateWorkplan(); lineNo:0; ilOffset:212.

    FileName:; Method:DispatchRequestCallback(); lineNo:0; ilOffset:445.

    FileName:; Method:Function(); lineNo:0; ilOffset:64.

    FileName:; Method:Run(); lineNo:0; ilOffset:95.

    FileName:; Method:PerformWaitCallbackInternal(); lineNo:0; ilOffset:28.

    FileName:; Method:PerformWaitCallback(); lineNo:0; ilOffset:40.

    </ExceptionDetails></FatalServiceError>

    ??

    the message resource is present but the message is not found in the string/message table

    and Finally from SQL server log a sample of the deadlock in the log is this

    09/02/2010 08:00:07,spid18s,Unknown,waiter id=processb1d708 mode=IX requestType=convert
    09/02/2010 08:00:07,spid18s,Unknown,waiter-list
    09/02/2010 08:00:07,spid18s,Unknown,owner id=processb03b88 mode=S
    09/02/2010 08:00:07,spid18s,Unknown,owner-list
    09/02/2010 08:00:07,spid18s,Unknown,pagelock fileid=1 pageid=72880 dbid=7 objectname=DPMDB.dbo.tbl_TE_TaskTrail id=locke49ce100 mode=SIU associatedObjectId=72057594062700544
    09/02/2010 08:00:07,spid18s,Unknown,waiter id=processb03b88 mode=S requestType=wait
    09/02/2010 08:00:07,spid18s,Unknown,waiter-list
    09/02/2010 08:00:07,spid18s,Unknown,owner id=processb1d708 mode=X
    09/02/2010 08:00:07,spid18s,Unknown,owner-list
    09/02/2010 08:00:07,spid18s,Unknown,keylock hobtid=72057594044743680 dbid=7 objectname=DPMDB.dbo.tbl_TE_TaskTrail indexname=ix_TE_TaskTrail_VerbID_TaskID id=lock84bcd000 mode=X associatedObjectId=72057594044743680
    09/02/2010 08:00:07,spid18s,Unknown,resource-list
    09/02/2010 08:00:07,spid18s,Unknown,Proc [Database Id = 7 Object Id = 2068202418]
    09/02/2010 08:00:07,spid18s,Unknown,inputbuf
    09/02/2010 08:00:07,spid18s,Unknown,WHERE TaskID = @TaskID
    09/02/2010 08:00:07,spid18s,Unknown,StartedDateTime = @StartedDateTime
    09/02/2010 08:00:07,spid18s,Unknown,LastStateName = @LastStateName<c/>
    09/02/2010 08:00:07,spid18s,Unknown,SET ExecutionState = @ExecutionState<c/>
    09/02/2010 08:00:07,spid18s,Unknown,UPDATE dbo.tbl_TE_TaskTrail
    09/02/2010 08:00:07,spid18s,Unknown,frame procname=DPMDB.dbo.prc_TE_TaskTrail_Start line=18 stmtstart=536 stmtend=876 sqlhandle=0x03000700b243467be47eaf00829d00000100000000000000
    09/02/2010 08:00:07,spid18s,Unknown,executionStack
    09/02/2010 08:00:07,spid18s,Unknown,process id=processb1d708 taskpriority=0 logused=304 waitresource=PAGE: 7:1:72880 waittime=3805 ownerId=77524465 transactionname=UPDATE lasttranstarted=2010-09-02T08:00:04.070 XDES=0x80f68e90 lockMode=IX schedulerid=4 kpid=4776 status=suspended spid=56 sbid=0 ecid=0 priority=0 trancount=2 lastbatchstarted=2010-09-02T08:00:04.070 lastbatchcompleted=2010-09-02T08:00:04.067 clientapp=MSDPM hostname=DPM hostpid=6696 loginname=NT AUTHORITY\SYSTEM isolationlevel=read committed (2) xactid=77524465 currentdb=7 lockTimeout=4294967295 clientoption1=538968096 clientoption2=128024
    09/02/2010 08:00:07,spid18s,Unknown,Proc [Database Id = 7 Object Id = 678293476]
    09/02/2010 08:00:07,spid18s,Unknown,inputbuf
    09/02/2010 08:00:07,spid18s,Unknown,AND replicaTrail.IsGCed = 0
    09/02/2010 08:00:07,spid18s,Unknown,AND taskTrail.ErrorCode = 0
    09/02/2010 08:00:07,spid18s,Unknown,AND taskTrail.ExecutionState not in (2<c/>3) -- 3 = Failed and 2 = Success
    09/02/2010 08:00:07,spid18s,Unknown,AND replicaTrail.BackupType = @BackupType
    09/02/2010 08:00:07,spid18s,Unknown,AND taskTrail.VerbId = @VerbId
    09/02/2010 08:00:07,spid18s,Unknown,WHERE replicaTrail.ReplicaId = @ReplicaId
    09/02/2010 08:00:07,spid18s,Unknown,ON replicaTrail.TaskId = taskTrail.TaskId
    09/02/2010 08:00:07,spid18s,Unknown,JOIN dbo.tbl_TE_TaskTrail taskTrail
    09/02/2010 08:00:07,spid18s,Unknown,FROM dbo.tbl_RM_ReplicaTrail replicaTrail
    09/02/2010 08:00:07,spid18s,Unknown,SELECT replicaTrail.TaskId
    09/02/2010 08:00:07,spid18s,Unknown,frame procname=DPMDB.dbo.prc_RM_ReplicaTrail_GetActiveTasks line=16 stmtstart=550 stmtend=1432 sqlhandle=0x03000700e4ef6d284085af00829d00000100000000000000
    09/02/2010 08:00:07,spid18s,Unknown,executionStack
    09/02/2010 08:00:07,spid18s,Unknown,process id=processb03b88 taskpriority=0 logused=0 waitresource=KEY: 7:72057594044743680 (9d047b315048) waittime=3798 ownerId=77524443 transactionname=SELECT lasttranstarted=2010-09-02T08:00:04.030 XDES=0xcd8e1a90 lockMode=S schedulerid=3 kpid=3012 status=suspended spid=61 sbid=0 ecid=0 priority=0 trancount=0 lastbatchstarted=2010-09-02T08:00:04.030 lastbatchcompleted=2010-09-02T08:00:04.030 clientapp=MSDPM hostname=DPM hostpid=6696 loginname=NT AUTHORITY\SYSTEM isolationlevel=read committed (2) xactid=77524443 currentdb=7 lockTimeout=4294967295 clientoption1=538968096 clientoption2=128056
    09/02/2010 08:00:07,spid18s,Unknown,process-list
    09/02/2010 08:00:07,spid18s,Unknown,deadlock victim=processb03b88
    09/02/2010 08:00:07,spid18s,Unknown,deadlock-list
    09/02/2010 08:00:07,spid4s,Unknown,ResType:LockOwner Stype:'OR'Xdes:0x00000000CD8E1A90 Mode: S SPID:61 BatchID:0 ECID:0 TaskProxy:(0x00000000CD81A538) Value:0xd0534280 Cost:(0/0)
    09/02/2010 08:00:07,spid4s,Unknown,Victim Resource Owner:
    09/02/2010 08:00:07,spid4s,Unknown,Log Viewer could not read information for this log entry. Cause: Data is Null. This method or property cannot be called on Null values.. Content:
    09/02/2010 08:00:07,spid4s,Unknown,ResType:LockOwner Stype:'OR'Xdes:0x0000000080F68E90 Mode: IX SPID:56 BatchID:0 ECID:0 TaskProxy:(0x00000000CD880538) Value:0x85a14bc0 Cost:(0/304)
    09/02/2010 08:00:07,spid4s,Unknown,Requested by:
    09/02/2010 08:00:07,spid4s,Unknown,Grant List 3:
    09/02/2010 08:00:07,spid4s,Unknown,Input Buf: RPC Event: Proc [Database Id = 7 Object Id = 678293476]
    09/02/2010 08:00:07,spid4s,Unknown,SPID: 61 ECID: 0 Statement Type: SELECT Line #: 16
    09/02/2010 08:00:07,spid4s,Unknown,Owner:0x00000000A4CCFD80 Mode: S Flg:0x40 Ref:0 Life:00000001 SPID:61 ECID:0 XactLockInfo: 0x00000000CD8E1AD0
    09/02/2010 08:00:07,spid4s,Unknown,Grant List 2:
    09/02/2010 08:00:07,spid4s,Unknown,PAGE: 7:1:72880 CleanCnt:2 Mode:SIU Flags: 0x3
    09/02/2010 08:00:07,spid4s,Unknown,Node:2
    09/02/2010 08:00:07,spid4s,Unknown,Log Viewer could not read information for this log entry. Cause: Data is Null. This method or property cannot be called on Null values.. Content:
    09/02/2010 08:00:07,spid4s,Unknown,ResType:LockOwner Stype:'OR'Xdes:0x00000000CD8E1A90 Mode: S SPID:61 BatchID:0 ECID:0 TaskProxy:(0x00000000CD81A538) Value:0xd0534280 Cost:(0/0)
    09/02/2010 08:00:07,spid4s,Unknown,Requested by:
    09/02/2010 08:00:07,spid4s,Unknown,Input Buf: RPC Event: Proc [Database Id = 7 Object Id = 2068202418]
    09/02/2010 08:00:07,spid4s,Unknown,SPID: 56 ECID: 0 Statement Type: UPDATE Line #: 18
    09/02/2010 08:00:07,spid4s,Unknown,Owner:0x00000000A4C85D40 Mode: X Flg:0x40 Ref:1 Life:02000000 SPID:56 ECID:0 XactLockInfo: 0x0000000080F68ED0
    09/02/2010 08:00:07,spid4s,Unknown,Grant List 3:
    09/02/2010 08:00:07,spid4s,Unknown,KEY: 7:72057594044743680 (9d047b315048) CleanCnt:2 Mode:X Flags: 0x1
    09/02/2010 08:00:07,spid4s,Unknown,Node:1
    09/02/2010 08:00:07,spid4s,Unknown,Log Viewer could not read information for this log entry. Cause: Data is Null. This method or property cannot be called on Null values.. Content:
    09/02/2010 08:00:07,spid4s,Unknown,Wait-for graph
    09/02/2010 08:00:07,spid4s,Unknown,Deadlock encountered .... Printing deadlock information

     

    ??

    • Edited by PP1234 Thursday, September 2, 2010 11:51 AM
    Wednesday, September 1, 2010 11:21 PM

Answers

  • Hello,

    There is a QFE that will be released soon to address crashes due to SQL deadlock issues. As of now, there is no specified time as to when it will be released.


    Thanks
    Shane

    Thursday, September 16, 2010 3:34 PM

All replies

  • What version of DPM do you have?

    This used to be a known bug,

    See here: http://support.microsoft.com/kb/970868 (item 5)

    Just my 2 cents,

    Cheers,

    Mike Resseler


    Visit System Center User Group Belgium @ http://scug.be and http://scug.be/blogs/scdpm
    Thursday, September 2, 2010 6:40 AM
    Moderator
  • Oh Sorry I should have said.

    Its DPM 2010.

    i'll edit the original message


    pp
    Thursday, September 2, 2010 11:49 AM
  • If I read it correctly, then it is the DPM database itself that gives you the trouble.  So basically, your DPM server service crashes due to the deadlock.

    Question: Are there maintenance plans, other backup products or Anti-Virus things that have been configured on that database by a DBA admin or so?  (I know DBA admins don't like the fact that they can't touch it :-)) If not, I would advise to open a support call with MS.  If it is a bug, then you don't need to pay for it.  But I think it is not a bug, but something else that is playing with the database outside of DPM.

    Sorry I can't help you more

    Cheers,

    Mike Resseler


    Visit System Center User Group Belgium @ http://scug.be and http://scug.be/blogs/scdpm
    Thursday, September 2, 2010 11:55 AM
    Moderator
  • Same problem here,

    get a deadlock,

    1ED8    1F9C    09/09    01:15:06.862    08    DpmThreadPool.cs(121)            WARNING    Procedure: prc_RM_ReplicaTrail_GetActiveTasks

    1ED8    1F9C    09/09    01:15:06.862    08    DpmThreadPool.cs(121)            WARNING    Line: 16

    CRITICAL    Exception Message = exception of type Microsoft.Internal.EnterpriseStorage.Dls.DB.NonFatalDbException, process will terminate after generating dump

    PP1234, did you upgrade DPM2010 from DPM2010 RC?

    Marc

     

    Thursday, September 9, 2010 9:34 AM
  • Hi

    Do you got the DPM database local or on a remote SQL?



    Check out my DPM blog @ http://robertanddpm.blogspot.com

    Thursday, September 9, 2010 1:13 PM
    Moderator
  • Hi

    It's an local installation of the SQL2008SP1 Database.

    Sqlservr.exe Version 2007.100.2531.0

    Marc

    Thursday, September 9, 2010 1:22 PM
  • Hi

    Does your DPM server protect a big enviroment? Like 1000 databases etc. etc.

    There are som limitations. Have you check out Mikes note regarding a known bug? If so open a case with MS.

     

    BR

    Robert Hedblom

     



    Check out my DPM blog @ http://robertanddpm.blogspot.com

    Thursday, September 9, 2010 1:31 PM
    Moderator
  • Nop, i think 30 SQL Databases. Total backup Volume 1TB
    Yes i checked this, but this is DPM 2010. The artikel points at DPM 2007

     

    Thursday, September 9, 2010 1:42 PM
  • Hi All,

    Update on this SQL deadlock is its stopped for now, that's what makes it so strange(and annoying).

    dpm sqlserver is local to the machinew, it is a bare metal install of dpm2010.

    with sql server 2008

     


    pp
    Monday, September 13, 2010 4:46 AM
  • Hello,

    There is a QFE that will be released soon to address crashes due to SQL deadlock issues. As of now, there is no specified time as to when it will be released.


    Thanks
    Shane

    Thursday, September 16, 2010 3:34 PM
  • Hey ShaneB.

    Thanks for the information, keep us posted when it is ready...

    Thanks

    Mike Resseler


    Visit System Center User Group Belgium @ http://scug.be and http://scug.be/blogs/scdpm
    Thursday, September 16, 2010 4:48 PM
    Moderator
  • That would be great, thanks for the info.

    Marc

    Thursday, September 16, 2010 7:05 PM
  • This thread is a month old now, so has a fix been release?

    Do I need to open a support case to get the fix?

     

    greg

    Friday, October 15, 2010 7:33 PM
  • Has anyone heard of a fix for this?

    This issue is getting a bit annoying - I need to manually shut down the 15 syncronization from all my protection groups every weekend, in order to be able to finish my tape backups! And it's beennearly two months since this issue was noticed...

    Antti


    Antti Laatikainen IT Security Manager Santen Europe
    Monday, November 22, 2010 7:38 PM
  • Fixed in KB2250444? I will test the rollup this week.

    http://support.microsoft.com/kb/2250444/

     

    Marc
    Wednesday, November 24, 2010 9:43 AM