none
DPM crashed after SAN outage RRS feed

  • Question

  • We had a fibre switch fail and knock our entire Hyper-V Failover cluster offline, we got everythign back online without any data loss but during the outage our DPM server became corrupt.  The DPM server doesn't connect to fibre or anything, it didn't reboot or lose power or go offline in anyway but somehow the service no longer works.

    Does anybody have any suggestions?

    -----------------------------------------------

    The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event: 

    An unexpected error caused a failure for process 'msdpm'.  Restart the DPM process 'msdpm'.

    Problem Details:
    <FatalServiceError><__System><ID>19</ID><Seq>0</Seq><TimeCreated>4/20/2012 9:24:41 PM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>NullReferenceException</ExceptionType><ExceptionMessage>Object reference not set to an instance of an object.</ExceptionMessage><ExceptionDetails>System.NullReferenceException: Object reference not set to an instance of an object.
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.BackupEventIntegration.WriteBackupEvent(BackupEventEntry backupEventEntry)
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.BackupEventIntegration.WriteNonLoggedBackupEntries()
       at Microsoft.Internal.EnterpriseStorage.Dls.Prm.PRMHealthProvider.Initialize()
       at Microsoft.Internal.EnterpriseStorage.Dls.JobManager.JobManager.Initialize()
       at Microsoft.Internal.EnterpriseStorage.Dls.JobManager.JobManager.InitializeIfNecessary(Object state)
       at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.DpmThreadPool.Function(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</ExceptionDetails></FatalServiceError>


    the message resource is present but the message is not found in the string/message table

    -----------------------------------------------

    The description for Event ID 0 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event: 

    Service started successfully.

    the message resource is present but the message is not found in the string/message table


    Lync/Asterisk blog: www.andrewparisio.com


    Friday, April 20, 2012 9:47 PM

All replies

  • I opened a ticket and after a few hours they took our logs and decided we had probably found a bug in DPM.  They took our SQL DB to try and see what happened and have suggested we restore our DB from two days before.  Imagine if our SAN had actually failed, we would have been hurting...  DPM now scares me knowing how easily it can corrupt itself during the time you need it most.

    I wrote a post about it:

    http://www.andrewparisio.com/2012/04/data-protection-manager-2010-crash_23.html


    Lync/Asterisk blog: www.andrewparisio.com

    Monday, April 23, 2012 9:13 PM
  • Hi,

    Just an FYI - we have not seen this particular crash before, so until we can figure out what the cause is we can't draw any conclusions.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, April 23, 2012 10:59 PM
    Moderator
  • Andrew, did you get anywhere at all with this? Was using a backup of the DPM database the only recovery mechanic?

    We are currently suffering the same symptoms, which is MSDPM Event 999, with the details being precisely the same as what you have posted above (i.e. error 19, Object reference not set to an instance of an object).

    We are using the cohabited SQL installation and hadn't gotten around to setting up local backups or maintenance plans, so we're without the option of recovering it. The only option I have is to see what can be traced from an SQL Server Profiler session, which is going to be tedious without any guarantee of success.

    Cheers,
    Lain

    Wednesday, May 23, 2012 6:22 AM
  • Actually, I have to clarify the above. The error and line numbers are indeed the same - which is as far as I had bothered to make the comparison, however, the referenced .NET modules are indeed different. So, disregard the above post and I'll re-post this as a new topic.

    For posterity, this is the error I get and it's from trying to access the Recovery tab within the MMC.

    <fatalserviceerror><__System><id>19</id><seq>0</seq><timecreated>23/05/2012 6:41:27 AM</timecreated>DpmThreadPool.cs<line>163</line><haserror>True</haserror><exceptiontype>NullReferenceException</exceptiontype><exceptionmessage>Object reference not set to an instance of an object.</exceptionmessage><exceptiondetails>System.NullReferenceException: Object reference not set to an instance of an object.
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.RecoveryPage.RecoveryTreeNodeFactory.CleanupEmptyNodes(TreeView treeView, TreeNode startNode)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.RecoveryPage.RecoveryBrowseTab.RenderTreeView()
       at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)</exceptiondetails></fatalserviceerror>

    Cheers,
    Lain

    Wednesday, May 23, 2012 7:10 AM