none
DPM 2012 SP1 RU2 crashes when creating PG for DPMDB (short time to tape) RRS feed

  • Question

  • Hi everybody,

    I recently rebuilt my single DPM server that was running DPM 2012 SP1 RU2 on Windows Server 2008/SP2. The reason for rebuild was to replace harddrive (C:\)  with a bigger one and update OS to Windows Server 2012. I went with the recommended procedure, but when restoring DPMDB (upgraded from DPM 2007->DPM 2010->DPM 2012->DPM 2012 SP1 RU2) it failed like this (just for info, but i'm curious why. Read on):
    --------------------------
    C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin>DpmSync.exe -RestoreDb -DbLoc E:\DPMDB-backup\DPM2012-SP1-RU2\DPMDB.bak
    DpmSync 2.0 - DPM database synchronization command-line tool
    Copyright (c) 2012 Microsoft Corporation. All rights reserved.

    Unhandled Exception: Microsoft.SqlServer.Management.Smo.FailedOperationException: Restore failed for Server 'JULIET\MSDPM2012'.  ---> Microsoft.SqlServer.Management.Common.ExecutionFailureException: An exception occurred while executing a Transact-SQL statement or batch. --->System.Data.SqlClient.SqlException: Directory lookup for the file "c:\Program Files\Microsoft DPM\DPM\DPMDB\MSDPM2012$DPMDB.mdf" failed with the operating system error 3(failed to retrieve text for this error. Reason: 15100).
    File 'DPMDB2007RTM_dat' cannot be restored to 'c:\Program Files\Microsoft DPM\DPM\DPMDB\MSDPM2012$DPMDB.mdf'. Use WITH MOVE to identify a valid location for the file.
    Directory lookup for the file "c:\Program Files\Microsoft DPM\DPM\DPMDB\MSDPM2012$DPMDB_log.ldf" failed with the operating system error 3(failed to retrieve text for this error. Reason: 15105).
    File 'DPMDB2007RTMLog_dat' cannot be restored to 'c:\Program Files\Microsoft DPM\DPM\DPMDB\MSDPM2012$DPMDB_log.ldf'. Use WITH MOVE to identify a valid locationfor the file.
    Problems were identified while planning for the RESTORE statement. Previous messages provide details.
    RESTORE DATABASE is terminating abnormally.
       at Microsoft.SqlServer.Management.Common.ConnectionManager.ExecuteTSql(ExecuteTSqlAction action, Object execObject, DataSet fillDataSet, Boolean catchException)
       at Microsoft.SqlServer.Management.Common.ServerConnection.ExecuteNonQuery(String sqlCommand, ExecutionTypes executionType)
       --- End of inner exception stack trace ---
       at Microsoft.SqlServer.Management.Common.ServerConnection.ExecuteNonQuery(String sqlCommand, ExecutionTypes executionType)
       at Microsoft.SqlServer.Management.Common.ServerConnection.ExecuteNonQuery(StringCollection sqlCommands, ExecutionTypes executionType)
       at Microsoft.SqlServer.Management.Smo.ExecutionManager.ExecuteNonQuery(StringCollection queries)
       at Microsoft.SqlServer.Management.Smo.BackupRestoreBase.ExecuteSql(Server server, StringCollection queries)
       at Microsoft.SqlServer.Management.Smo.Restore.SqlRestore(Server srv)
       --- End of inner exception stack trace ---
       at Microsoft.SqlServer.Management.Smo.Restore.SqlRestore(Server srv)
       at Microsoft.Internal.EnterpriseStorage.Dls.RestoreDbSync.RestoreDBHelper.RestoreFromBackupFile(String dbLocation)
       at Microsoft.Internal.EnterpriseStorage.Dls.RestoreDbSync.RestoreDBHelper.RestoreDb(String dbLocation)
       at Microsoft.Internal.EnterpriseStorage.Dls.RestoreDbSync.RestoreDbSync.Main(String[] args)

    ------------------

    I was able to restore DPMDB database using SSMS (and Forum posts, thanks), and now everything is working except for protecting DPMDB to tape (short-term protection). Finalizing the PG wizard ends with a crash in MSDPM throwing the following error in DPM Management Console:
    -----------
    Connection to the DPM service has been lost.

    Review the application event log for information about a possible service shutdown. Verify that the following services are not disabled:
                DPM
                DPM Replication Agent
                SQLAgent$MSDPM2012
                MSSQL$MSDPM2012
                Virtual Disk Service
                Volume Shadow Copy

    ID: 917
    ------------

    The Application event log error is as follows:

    The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    An unexpected error caused a failure for process 'msdpm'.  Restart the DPM process 'msdpm'.

    Problem Details:

    <FatalServiceError><__System><ID>19</ID><Seq>348</Seq><TimeCreated>30.04.2013 08:59:53</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>NullReferenceException</ExceptionType><ExceptionMessage>Object reference not set to an instance of an object.</ExceptionMessage><ExceptionDetails>System.NullReferenceException: Object reference not set to an instance of an object.

       at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.BaseWriterHelper.InvalidateReplicaAndForceHeavyWeightValidation(Message msg, Replica dbReplica, String traceFormatMessage)

       at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SqlWriterHelper.HandleUsnIdChangeForSqlType(Message msg)

       at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SqlWriterHelper.HandleUsnIdChangesOnConfigureProtectionSuccess(Message msg)

       at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.BaseWriterHelper.OnConfigureDeltaTrackingSuccess(Message msg)

       at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ConfigureProtectionMachine.ConfigureDeltaTrackingSucceeded(Message msg)

       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Transition.Execute(Message msg)

       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg)

       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy)

       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj)

       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)

       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</ExceptionDetails></FatalServiceError>

     

    the message resource is present but the message is not found in the string/message table

    MSDPMCurr.errlog.2013-04-30_08-59-53.Crash info (last rows):

    ----------------------
    11A8 0F8C 04/30 08:59:53.396 01 TaskExecutor.cs(692)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 NORMAL Setting up Fsm: verb[85d11aaa-bbbc-427d-9932-eeb3fd4759f5]
    11A8 0F8C 04/30 08:59:53.412 01 TaskExecutor.cs(487)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 NORMAL ExecuteTask(job=b63aaebe-27e6-4893-9001-f184a04845f2) - Starting task instance
    11A8 0C20 04/30 08:59:53.412 03 timer.cpp(490)   ACTIVITY CTimerMgr::TimerThread
    11A8 0C20 04/30 08:59:53.412 03 timer.cpp(503) [000000001A531978]  ACTIVITY CTimerMgr::TimerThreadInternal
    11A8 0F8C 04/30 08:59:53.412 02 EventManager.cs(98)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 NORMAL Publishing event from TaskInstance.cs(406): TaskStart, [TaskID=f7a771c8-9509-4613-af3f-8d4a0d4eefa3]
    11A8 0F8C 04/30 08:59:53.427 01 TaskInstance.cs(442)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 NORMAL Task: Initial transition enqueued, startState='ConfigureProtection.StartState'
    11A8 0F8C 04/30 08:59:53.427 27 FindPrincipalServerBlock.cs(245)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 NORMAL Found Principal server for DatasourceId = 21216256-25f3-4e35-b551-66cdbed2de28 Server= JULIET.RI.Reslab.no
    11A8 0F4C 04/30 08:59:53.615 09 AppAssert.cs(126)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING ASSERT: (FileName:SqlWriterHelper.cs; LineNumber:250)
    11A8 0F4C 04/30 08:59:53.615 09 AppAssert.cs(126)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING The parameter 'Not a valid replica for protection' is null.
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING Caught unhandled exception : System.NullReferenceException: Object reference not set to an instance of an object.
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.BaseWriterHelper.InvalidateReplicaAndForceHeavyWeightValidation(Message msg, Replica dbReplica, String traceFormatMessage)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SqlWriterHelper.HandleUsnIdChangeForSqlType(Message msg)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SqlWriterHelper.HandleUsnIdChangesOnConfigureProtectionSuccess(Message msg)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.BaseWriterHelper.OnConfigureDeltaTrackingSuccess(Message msg)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ConfigureProtectionMachine.ConfigureDeltaTrackingSucceeded(Message msg)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Transition.Execute(Message msg)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
    11A8 0F4C 04/30 08:59:53.615 68 DpmThreadPool.cs(121)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
    11A8 0F4C 04/30 08:59:53.630 68 DpmThreadPool.cs(149)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING Attempting to write error to the event log Object reference not set to an instance of an object.
    11A8 0F4C 04/30 08:59:53.630 02 EventManager.cs(143)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 NORMAL Logging event to NT log from DpmThreadPool.cs(163): FatalServiceError
    11A8 0F4C 04/30 08:59:53.630 68 DpmThreadPool.cs(170)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 WARNING Attempting to invoke Watson
    11A8 0F4C 04/30 08:59:53.646 09 everettexception.cpp(761)  F7A771C8-9509-4613-AF3F-8D4A0D4EEFA3 CRITICAL Exception Message = Object reference not set to an instance of an object. of type System.NullReferenceException, process will terminate after generating dump
    ----------------------

    Final Words: I can successfully backup DPMDB using 'DPMbackup -db' and I can easily create PG to protect a folder on the DPM server hard disk (C:\something), and create recover points to both disk and tape. 

    Any idea whats wrong?

    Thanks in advance :-)

    Tuesday, April 30, 2013 10:40 AM

All replies