none
DPM Server OS in place upgrade problems RRS feed

  • Question

  • Hello,

    I'm experiencing a fairly significant problem since performing an in place upgrade on my primary DPM box.

    Last night I upgraded DPM from 2012 R2 UR 10 on both primary and secondary servers and this morning I upgraded Windows Server 2012 on the primary server to Windows Server 2016. Since then DPM has been unavailable as the DPM AccessManager Service wont start.

    Attempting to start the service gives me the following in the application event log:

    The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

     

    If the event originated on another computer, the display information had to be saved with the event.

     

    The following information was included with the event:

     

    An unexpected error caused a failure for process 'DPMAMService'.  Restart the DPM process 'DPMAMService'.

     

    Problem Details:

    <FatalServiceError><__System><ID>19</ID><Seq>0</Seq><TimeCreated>10/31/2016 11:13:56 PM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>DlsException</ExceptionType><ExceptionMessage>exception</ExceptionMessage><ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.Utils.DlsException: exception ---&gt; System.Runtime.InteropServices.COMException: The group or resource is not in the correct state to perform the requested operation. (Exception from HRESULT: 0x8007139F)

     

    Server stack trace:

       at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)

       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CProxyUtils.HandleErrors(Int32 hr, tagSAFEARRAY* exceptionResult)

       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.EngineServicesProxy.CheckForPendingReboot()

       at Microsoft.Internal.EnterpriseStorage.Dls.EngineProxyWrapper.EngineServiceProxyWrapper.CheckForPendingReboot()

       at Microsoft.Internal.EnterpriseStorage.Dls.EngineProxyWrapper.EngineServiceProxyWrapper.ConnectAsAdmin(String dpmServerName, AsyncOperation asyncOperation)

       at Microsoft.Internal.EnterpriseStorage.Dls.EngineProxyWrapper.EngineServiceProxyWrapper.GetInstance(String dpmServerName, AsyncOperation asyncOperation)

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.ReadRegistryKeyOnDPMServer(String registryKeyPath, String registryKeyName, RegistryValueKind registryValueKind)

       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]&amp; outArgs)

       at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

     

    Exception rethrown at [0]:

       at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)

       at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData&amp; msgData)

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.ReadRegistryKeyAsync.EndInvoke(IAsyncResult result)

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.ReadRegistryKey(String registryKeyPath, String registryKeyName, RegistryValueKind registryValueKind)

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.InitializeIgnorableSqlErrorNumbersList()

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServer.GetDpmServerObject(String serverName, AsyncOperation asyncOperation, DpmServerScope dpmServerScope)

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServerFactory.GetServer(String serverName, AsyncOperation asyncOperation, DpmServerScope dpmServerScope)

       --- End of inner exception stack trace ---

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.DpmServerFactory.GetServer(String serverName, AsyncOperation asyncOperation, DpmServerScope dpmServerScope)

       at Microsoft.Internal.EnterpriseStorage.Dls.UI.AutoHeal.AutoHeal.DpmStarted()

       at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.DpmThreadPool.Function(Object state)

       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

       at System.Threading.ThreadPoolWorkQueue.Dispatch()

    *** Mojito error was: ConnectionToServerFailed; 0; None</ExceptionDetails></FatalServiceError>

     

     

    the message resource is present but the message is not found in the string/message table

    I've uninstalled it, removed the DB, reinstalled it, restored the DB from the secondary, when I try to run DPMSYNC -sync  on the box I get this the first time:

    PS C:\Program Files\Microsoft System Center 2016\DPM\DPM\bin> dpmsync -sync
    DpmSync 2.0 - DPM database synchronization command-line tool
    Copyright (c) 2013 Microsoft Corporation. All rights reserved.

    Note: The DPM role configuration of this server will also be rolled back during this operation.

    Error ID: 454
    DpmSync failed to connect to the specified SQL Server instance. Make sure that you have specified a valid SQL server instance associated with DPM and you are logged on as a user with administrator privileges.

    Detailed Error: The DELETE statement conflicted with the REFERENCE constraint "fk__Replica_Alerts__AHP_Alerts". The conflict occurred in database "DPMDB_MADEADPM00153594a9e_1c4c_4d5c_8725_84f33cece1af", table "dbo.tbl_PRM_ReplicaAlerts", column 'AlertId'.
    The statement has been terminated.

    (I am definitely running it as an admin)

    Subsequent executions result in:

    PS C:\Program Files\Microsoft System Center 2016\DPM\DPM\bin> dpmsync -sync
    DpmSync 2.0 - DPM database synchronization command-line tool
    Copyright (c) 2013 Microsoft Corporation. All rights reserved.

    Note: The DPM role configuration of this server will also be rolled back during this operation.

    Unhandled Exception: System.Runtime.InteropServices.COMException: The remote procedure call failed. (Exception from HRESULT: 0x800706BE)
       at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CProxyUtils.HandleErrors(Int32 hr, tagSAFEARRAY* exceptionResult)
       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.RestoreDbSyncServicesProxy.RestoreDbSynchronize(Boolean reallocateReplica, Boolean* isExternalVolumeMissing)
       at Microsoft.Internal.EnterpriseStorage.Dls.RestoreDbSync.RestoreDbSync.SyncFunction(Boolean reallocateReplica, Boolean& isExternalVolumeMissing)
       at Microsoft.Internal.EnterpriseStorage.Dls.RestoreDbSync.RestoreDbSync.Main(String[] args)

    I am fairly desperate to get this back up and running (there are 56TB of SQL backups on there) so I'm grateful for any suggestions.

    Steve

    Monday, October 31, 2016 11:20 PM

All replies

  • From what i've been able to gather, upgrading the OS on a DPM 2012R2 isn't supported. DPM 2012R2 running on Windows Server 2016 also isn't supported.

    If you want to upgrade from DPM 2012R2 on Windows Server 2012R2 to DPM and Windows 2016 you have to first upgrade DPM 2016. Upgrade the operating system after you are done with the DPM upgrade.

    Wednesday, November 2, 2016 4:20 PM
  • In place OS upgrade from 2012 R2 to 2016 post DPM 2012 to 2016 upgrade is a supported scenario.
    Thursday, November 3, 2016 1:04 PM
  • OP upgraded Windows Server, but not DPM.
    Thursday, November 3, 2016 4:09 PM
  • Hi Stephen, DPM 2016 is supported on WS 2012 R2. So you've to upgrade DPM from 2012 R2 (UR11) to DPM 2016 (UR1) and than (in place) upgrade your Windows Server 2012 R2 to 2016.

    In case you've already upgrade Windows without upgrading DPM to 2016 you can try to upgrade DPM now on WS2016 - not sure if possible.

    Friday, November 4, 2016 8:51 AM
  • OP upgraded Windows Server, but not DPM.

    Incorrect

    "Last night I upgraded DPM from 2012 R2 UR 10 on both primary and secondary servers"

    I appreciate you trying to help, but please, at least read and understand the actual problem.

    Friday, November 4, 2016 9:22 AM
  • hi Heyko,

    Thanks for your post however, as you can see in my original post:

    "Last night I upgraded DPM from 2012 R2 UR 10 on both primary and secondary servers"

    This is the upgrade path, its fairly clear and on the face of it straightforward:

    • Upgrade DPM 2012 R2 to DPM 2012 R2 Update Rollup 10. You can obtain the Update Rollups from Windows Update.
    • Upgrade DPM 2012 R2 Update Rollup 10 to DPM 2016.
    • Update the agents on the protected servers.
    • Upgrade Windows Server 2012 R2 to Windows Server 2016.

    this is from https://technet.microsoft.com/en-us/system-center-docs/dpm/get-started/upgrade-to-dpm-2016?f=255&MSPPError=-2147217396

    For what its worth I gave up trying to fix the error, rebuilt the server WS2012R2 with DPM 2012 R2 UR10, restored the DB, then upgraded to DPM 2016.
    Friday, November 4, 2016 9:24 AM
  • Ok, missed that you've upgraded DPM, too. Maybe it's already to late for you. In case you're not able to start 'DPMAMService' you can check if your DPMDB is in Recovery Mode.

    1. Connect to the SQL Server with SQL Management Studio
    2. Expand Databases -->DPMDB -->Tables -->dbo.tbl_DLS_GlobalSetting
    3. Click on Open / Edit Table
    4. Check if DbRecovery is set to 1 (set to 0)

    The Recovery Mode is a possible reason that you're unable to start the DPMAMService.

    • Proposed as answer by Marcel_H Wednesday, November 16, 2016 9:59 AM
    Friday, November 4, 2016 9:34 AM
  • Hi Heyko,

    Thanks for your post, unfortunately I've already rebuilt the box so I cant give your suggestion a try.

    I appreciate your effort though.

    Steve

    Friday, November 4, 2016 9:37 AM