none
DPM 2012 R2 RU2 - KB2958100 - DPM AccessManager Service crash

    Question

  • We've just installed KB2958100 and it's broken our primary DPM server.

    http://support.microsoft.com/kb/2958100/en-gb

    The DPM AccessManager Service crashes moments after starting.  We see the following series of events in the event log:

    • DPMAccessManager / EventID 0 / Service started sucessfully
    • MSDPM / EventID 999 /

    An unexpected error caused a failure for process 'DPMAMService'.  Restart the DPM process 'DPMAMService'.

    Problem Details:
    <FatalServiceError><__System><ID>19</ID><Seq>0</Seq><TimeCreated>23/04/2014 09:29:15</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>DlsException</ExceptionType><ExceptionMessage>exception</ExceptionMessage><ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.Utils.DlsException: exception ---&gt; System.IndexOutOfRangeException: Index was outside the bounds of the array.
       at System.Data.SqlClient.SqlDataReader.CheckDataIsReady(Int32 columnIndex, Boolean allowPartiallyReadColumn, Boolean permitAsync, String methodName)
       at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
       at System.Data.SqlClient.SqlDataReader.ReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
       at System.Data.SqlClient.SqlDataReader.GetString(Int32 i)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.ProtectableObject..ctor(DpmServer server, OMSqlRetryReader reader, DpmObjectType type)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.Datasource..ctor(DpmServer server, DpmObjectType objectType, OMSqlRetryReader reader)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.FileSystem.FileObjectFactory.CreateDataSource(OMSqlRetryReader reader, DpmObjectType objectType, DpmServer server)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmHelper.ConstructDatasource(OMSqlRetryReader reader, Guid writerId, String componentType, DpmServer dpmServer)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.ReadExternalDatasourceFromDb(Guid datasourceId)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.InitializeExternalDatasourceCache()
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.SubscribeToDlsEvents()
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.Connect(AsyncOperation asyncOperation)
       --- End of inner exception stack trace ---
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.Connect(AsyncOperation asyncOperation)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.GetDpmServerObject(String serverName, AsyncOperation asyncOperation, DpmServerScope dpmServerScope)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServerFactory.GetServer(String serverName, AsyncOperation asyncOperation, DpmServerScope dpmServerScope)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.AutoHeal.AutoHeal.DpmStarted()
       at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.DpmThreadPool.Function(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
    *** Mojito error was: ConnectionToServerFailed; 0; None</ExceptionDetails></FatalServiceError>

    • Followed by several Windows Error Reporting / EventID 1001's

    We have rebooted the several times and tried reinstalling the update.  The dpm database is operational.

    Anyone had the same problem?  We are about to open a support case with MS


    • Edited by DJL Wednesday, April 23, 2014 9:40 AM
    Wednesday, April 23, 2014 9:38 AM

Answers

  • Hello All,

    We have seen this error and we are researching the issue internally - when we have a solution I will post it hear.

    UPDATE: 4-23-14

    The problem seems to happen if you have imported tapes and re-cataloged them which adds entries under the External DPM tapes tree under the recovery tab. 

    The below SQL Script will remove the external DPM tape entries and allow the service to start and the console will open.   However, re-cataloging another imported tape gets you in the same crash loop, so we are investigating and will provide a permanent fix.

    delete from dbo.tbl_MM_MediaMap Where DatasetID in (select DatasetId from dbo.tbl_MM_PhysicalDataset
    where ExternalDatasetInformationId IS NOT NULL) delete from dbo.tbl_MM_PhysicalDataset

    where ExternalDatasetInformationId IS NOT NULL Delete from dbo.tbl_MM_ExternalDatasetInformation where dependantdatasetid IS NULL



    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.





    Wednesday, April 23, 2014 8:24 PM
    Moderator

All replies

  • Hi, we have the same problem (except the EventIDs 1001s)...
    Wednesday, April 23, 2014 11:43 AM
  • We have the same problem. Any word from Microsoft?
    Wednesday, April 23, 2014 12:32 PM
  • No luck with Microsoft yet. They have taken some logs and they are analysing them at the moment - they should phone us back in about 30 mins. 

    I'll post the fix once/if we get one

    Wednesday, April 23, 2014 12:38 PM
  • I have the same problem
    Wednesday, April 23, 2014 1:06 PM
  • I have the same problem too after installing KB2958100.

    Peter

    Wednesday, April 23, 2014 1:07 PM
  • It is good to know, that we are all not alone ;-)

    DJL, thanks for helping us in this matter!

    Wednesday, April 23, 2014 1:11 PM
  • I have the same problem with KB2958100...
    Wednesday, April 23, 2014 2:38 PM
  • Just to say we're still working on this with MS.  No solution so far...
    Wednesday, April 23, 2014 4:27 PM
  • OK so...

    Microsoft have fixed our server! :)

    However unfortunately the solution does not appear trivial at the moment.

    As our case is the first they have seen they don't yet have enough information to put together a generic fix.  They are going to attempt to recreate our setup in their lab, but what they really need is more people to call them regarding the problem so they can narrow down the issue and put together a generic fix.

    Our case number is 114042311384618 and our engineer was Satish Pandey.

    They have recommended that we don't update our other DPM servers until they have sorted out this issue!

    Sorry I can't give you a solution at the moment


    • Edited by DJL Thursday, April 24, 2014 8:03 AM
    Wednesday, April 23, 2014 5:09 PM
  • Hello All,

    We have seen this error and we are researching the issue internally - when we have a solution I will post it hear.

    UPDATE: 4-23-14

    The problem seems to happen if you have imported tapes and re-cataloged them which adds entries under the External DPM tapes tree under the recovery tab. 

    The below SQL Script will remove the external DPM tape entries and allow the service to start and the console will open.   However, re-cataloging another imported tape gets you in the same crash loop, so we are investigating and will provide a permanent fix.

    delete from dbo.tbl_MM_MediaMap Where DatasetID in (select DatasetId from dbo.tbl_MM_PhysicalDataset
    where ExternalDatasetInformationId IS NOT NULL) delete from dbo.tbl_MM_PhysicalDataset

    where ExternalDatasetInformationId IS NOT NULL Delete from dbo.tbl_MM_ExternalDatasetInformation where dependantdatasetid IS NULL



    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.





    Wednesday, April 23, 2014 8:24 PM
    Moderator
  • Hello,

    the SQL-Script worked and solved our problem. Thanks DJL and Mike for sharing the solution!

    First we thought the script would not help us because we couldn't remember to have imported tapes. Then we checked database with script  - "select" instead "delete" - and there were some imported tapes from 2011...hard to believe, but an old action three years ago stopped our DPM in 2014 ;-)

    Thanks again!


    • Edited by trcc Thursday, April 24, 2014 6:08 AM Typo
    Thursday, April 24, 2014 6:07 AM
  • The SQL-script solved the issue for me as well.

    Thanks Mike!

    Will you update this post when the root cause is resolved as well. I need to do a restore from an older tape (external) so I'm holding off until a fix is released.

    Marc

    Thursday, April 24, 2014 6:24 AM
  • SQL script solved our problem too.

    Thanks!


    Peter

    Thursday, April 24, 2014 8:34 AM
  • Exactly the same issue here. Relieved I'm not all alone.

    :-(


    Andrew France - http://andrewsprivatecloud.wordpress.com

    Thursday, April 24, 2014 10:48 AM
  • Thank You Mike for the SQL script.

    (I change the script a bit to suit our database:)

    delete from dbo.tbl_MM_PhysicalDataset

    Thursday, April 24, 2014 11:00 AM
  • Thanks for the script, it also fixed our DPM server that got broken by this update.
    Thursday, April 24, 2014 11:49 AM
  • Thanks,

    The same thing happed here too.

    I don't normally install rollups till after several weeks pass and no one complains but this one slipped by.

    Hopefully MS has pulled this rollup from distribution. 

    Thursday, April 24, 2014 8:48 PM
  • yes, Update is pulled:

    Update Rollup 2 for Microsoft System Center 2012 R2 Data Protection Manager is currently unavailable for download because of issues that are related to tape backup and recovery. We have determined the cause of these issues and will release an updated version of this update rollup soon.

    Friday, April 25, 2014 5:22 AM
  • UPDATE: 4-23-14

    The problem seems to happen if you have imported tapes and re-cataloged them which adds entries under the External DPM tapes tree under the recovery tab. 

    The below SQL Script will remove the external DPM tape entries and allow the service to start and the console will open.   However, re-cataloging another imported tape gets you in the same crash loop, so we are investigating and will provide a permanent fix.

    delete from dbo.tbl_MM_MediaMap Where DatasetID in (select DatasetId from dbo.tbl_MM_PhysicalDataset
    where ExternalDatasetInformationId IS NOT NULL) delete from dbo.tbl_MM_PhysicalDataset

    where ExternalDatasetInformationId IS NOT NULL Delete from dbo.tbl_MM_ExternalDatasetInformation where dependantdatasetid IS NULL


    I am running into the same issue.  I can confirm Mike's TSQL did temporarily allow me to get into DPM Console and work, but it is a temporary fix.  I hope that Microsoft will have a perm solution this coming week.

    Cameron Mergel


    Cameron

    Sunday, April 27, 2014 3:10 AM
  • yes, Update is pulled:

    Update Rollup 2 for Microsoft System Center 2012 R2 Data Protection Manager is currently unavailable for download because of issues that are related to tape backup and recovery. We have determined the cause of these issues and will release an updated version of this update rollup soon.

    Anyone have any news on when the updated version will be released?

    Our primary DPM server got the original update but it was pulled before the secondary DPM got it so now the secondary server wont sync from the primary as the agent is an older version.

    Wednesday, April 30, 2014 10:38 AM
  • Hello,

    We are not using tapes yet with DPM, so we did not see the crashing issue described above, however after installing RU2, and when going to the RECOVERY tab, expand the protection groups, expand a Hyper-V group it lists the VMs but as soon as we try to click on one to recover or explore, the entire protection group tree collapses and nothing happens. not sure if this is related to RU2, but it did started happening the minute we installed the roll up update 2.

    Thanks

    Wednesday, April 30, 2014 9:59 PM
  • Hi all,

    We've just gone to restore a database - the first time since installing RU2 - and we can't restore anything now! 

    The console crashes with a SQL error:

    Exception Name: System.IndexOutOfRangeException
      Exception Point: System.Data.SqlClient.SqlDataReader.CheckHeaderIsReady

    If you haven't already tried restoring something since RU2 I would suggest you try it to make sure its working.  I'm about to open a support ticket.

    Thursday, May 22, 2014 7:52 AM
  • The update was rereased yesterday: Description of Update Rollup 2 for System Center 2012 R2 Data Protection Manager

    Notice

    This update rollup was re-released on May 20, 2014 to resolve a "DPMAMService" crash that occurred if the original update (KB 2958100) was applied.
    If you already installed Update Rollup 2 on your Windows Server 2003 servers, you do not have to reinstall the Update Rollup 2 agent that is required for Windows Server 2003 communication for this re-release. Additional updates after this re-release will install and work as expected.

    • Proposed as answer by Sven Bürger Thursday, May 22, 2014 11:25 AM
    Thursday, May 22, 2014 11:25 AM