none
MSDPM Process crashes since installing UR9 for DMP2012 R2 (Event ID 943) RRS feed

  • Question

  • Hi,

    Right after installing UR9 for DPM 2012R2 my DPM server started to behave badly.
    Frequent crashes for MSDPM Process. I don't see this happening on particular jobs. At least it's not related to the startup of jobs. Maybe I'm correct if I say that this happends during jobs. Some backup job's are completed succesfully despite this error and some are "rerun". PG's shows all green. Any ideas?

    The description for Event ID 943 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    Unable to connect to the DPM database because the database is in an inconsistent state.

    Problem Details:
    <FatalServiceError><__System><ID>19</ID><Seq>5089</Seq><TimeCreated>1.02.2016 17:46:10</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>DBCorruptionException</ExceptionType><ExceptionMessage>DLS error: AMServerInfo: dbo.prc_AM_Server_Get SP='dbo.prc_AM_Server_Get' Error=50001</ExceptionMessage><ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error: AMServerInfo: dbo.prc_AM_Server_Get SP='dbo.prc_AM_Server_Get' Error=50001
       at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ValidateReturnValue(String errorTitle)
       at Microsoft.Internal.EnterpriseStorage.Dls.AgentManager.DB.AMServerInfo.GetServerDetails(SqlContext ctx, String serverName)
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.PrmCatalog.GetInstalledAgentVersionNumber(String AgentMachineName)
       at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.IsEngineForceFixupRequired(Guid datasourceId, Message msg)
       at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.RAPreBackupSendCommon(Message msg, Guid volumeBitmapId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.SendPrebackupCommand(Message msg)
       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.ConnectionPoint.Execute(Message msg)
       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg)
       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy)
       at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()</ExceptionDetails></FatalServiceError>


    the message resource is present but the message is not found in the string/message table


    rgds Sven

    Tuesday, February 2, 2016 7:03 AM

Answers

  • We are able to root cause the issue, it will happen only if you are protecting Client machines using DPM and are trying to backup the client machine when it is connected via VPN.

    In case you are hitting this issue please follow the below steps on your DPM Server to permanently fix the issue.

    1. Open Microsoft SQL Server Management Studio

    2. Connect to SQL instance where DPM database is hosted

    3.  Right click on the DPM database and select New Query. Make sure DPM database is selected from available databases drop down.

    4. Paste the below SQL procedure in the query windows and click on Execute


    IF EXISTS (SELECT * FROM dbo.sysobjects

               WHERE id = OBJECT_ID(N'dbo.prc_AM_AgentInstalledOnServer_Get')

               AND OBJECTPROPERTY(id, N'IsProcedure') = 1)

    DROP PROCEDURE dbo.prc_AM_AgentInstalledOnServer_Get

    GO

     

    CREATE PROCEDURE dbo.prc_AM_AgentInstalledOnServer_Get

    (

      @ServerName nvarchar(400),

      @DataSourceType smallint

    )

    AS

    DECLARE @error int,

            @rowCount int

     

    SET @error = 0

    SET @rowCount = 0

     

    SET NOCOUNT ON

     

     

    SELECT InstallId,

              AgentId,

              OSType,   

              dbo.tbl_AM_Server.ServerName,

              RebootRequired,

              DomainName,

              NetbiosName,

              PatchId

    FROM dbo.tbl_AM_InstalledAgent

    JOIN dbo.tbl_AM_Server ON dbo.tbl_AM_InstalledAgent.ServerId = dbo.tbl_AM_Server.ServerId

    WHERE (dbo.tbl_AM_Server.ServerName = @ServerName OR (NetbiosName is not null and @ServerName is not null and NetbiosName = @ServerName))

    AND Deleted = 0

    AND AgentID in (SELECT DISTINCT a.AgentID

                    FROM dbo.tbl_AM_Agent a, dbo.tbl_AM_AgentType t

                    WHERE a.TypeID = t.TypeID and t.DataSourceType = @DataSourceType

                    )

       

    SELECT @error = @@ERROR, @rowCount = @@ROWCOUNT

     

    IF (@error = 0 AND @rowCount > 1)

        SET @error = 50001

     

    SET NOCOUNT OFF

     

    RETURN @error

    GO

     

    GRANT EXECUTE ON  dbo.prc_AM_AgentInstalledOnServer_Get  TO MSDPMExecRole

    GO

    If you are not sure of the DPM SQL instance or DPMDB you can fetch the information using below steps: 

    1. Login to DPM Server

    2. Click on About DPM 

    3. DPM's SQL Server field will contain this information in following format sqlservername\sqlinstancename\dpmdbname

    Friday, February 5, 2016 2:45 PM
    Moderator

All replies

  • I had the same problem.

    1. open all Protection Group, no change, next, next ... finish.

    2. Run consistency check all protection data

    this fix my problem, MSDPM proces no crash, no Event ID 943 

    Wednesday, February 3, 2016 9:02 AM
  • Run stored procedure dbo.prc_AM_Server_Get in DPM database. What result is? 
    Wednesday, February 3, 2016 9:04 AM
  • Running this procedure  is out of my expertise :(

    At least I was able to run "dbcc checkdb" agains DPM database using management studio and this gives me no error.
    As of today, I'v restarted DPM server several times and all PG's have been run trough at least once. Error are gone. So I don't know what to think about it. I'll wait for some time and if everything looks normal I think I may close the case/thread..



    rgds Sven

    Wednesday, February 3, 2016 2:16 PM
  • Thanks for reporting the issue, is it possible to send the DPM logs from the below location as well a copy of DPMDB so we can investigate the issue.

    <DPM Installation drive>\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Temp

    Sven- Seems like you already have a support case opened for this issue? If yes can you please give us the support number so we can track it and get all relevant details. 

    Wednesday, February 3, 2016 4:26 PM
    Moderator
  • I don't have a support case.

    Yes, I can send my entire Temp but it's about 1.5GB in size..DB+ldf is about 1GB.
    If I backup DB as a bak file, I don't know the output size.
    I suppouse that ZIP'ing helps a bit.
    Anyway, if You think that my log's will help You, let me know.


    rgds Sven

    Wednesday, February 3, 2016 5:31 PM
  • We are able to root cause the issue, it will happen only if you are protecting Client machines using DPM and are trying to backup the client machine when it is connected via VPN.

    In case you are hitting this issue please follow the below steps on your DPM Server to permanently fix the issue.

    1. Open Microsoft SQL Server Management Studio

    2. Connect to SQL instance where DPM database is hosted

    3.  Right click on the DPM database and select New Query. Make sure DPM database is selected from available databases drop down.

    4. Paste the below SQL procedure in the query windows and click on Execute


    IF EXISTS (SELECT * FROM dbo.sysobjects

               WHERE id = OBJECT_ID(N'dbo.prc_AM_AgentInstalledOnServer_Get')

               AND OBJECTPROPERTY(id, N'IsProcedure') = 1)

    DROP PROCEDURE dbo.prc_AM_AgentInstalledOnServer_Get

    GO

     

    CREATE PROCEDURE dbo.prc_AM_AgentInstalledOnServer_Get

    (

      @ServerName nvarchar(400),

      @DataSourceType smallint

    )

    AS

    DECLARE @error int,

            @rowCount int

     

    SET @error = 0

    SET @rowCount = 0

     

    SET NOCOUNT ON

     

     

    SELECT InstallId,

              AgentId,

              OSType,   

              dbo.tbl_AM_Server.ServerName,

              RebootRequired,

              DomainName,

              NetbiosName,

              PatchId

    FROM dbo.tbl_AM_InstalledAgent

    JOIN dbo.tbl_AM_Server ON dbo.tbl_AM_InstalledAgent.ServerId = dbo.tbl_AM_Server.ServerId

    WHERE (dbo.tbl_AM_Server.ServerName = @ServerName OR (NetbiosName is not null and @ServerName is not null and NetbiosName = @ServerName))

    AND Deleted = 0

    AND AgentID in (SELECT DISTINCT a.AgentID

                    FROM dbo.tbl_AM_Agent a, dbo.tbl_AM_AgentType t

                    WHERE a.TypeID = t.TypeID and t.DataSourceType = @DataSourceType

                    )

       

    SELECT @error = @@ERROR, @rowCount = @@ROWCOUNT

     

    IF (@error = 0 AND @rowCount > 1)

        SET @error = 50001

     

    SET NOCOUNT OFF

     

    RETURN @error

    GO

     

    GRANT EXECUTE ON  dbo.prc_AM_AgentInstalledOnServer_Get  TO MSDPMExecRole

    GO

    If you are not sure of the DPM SQL instance or DPMDB you can fetch the information using below steps: 

    1. Login to DPM Server

    2. Click on About DPM 

    3. DPM's SQL Server field will contain this information in following format sqlservername\sqlinstancename\dpmdbname

    Friday, February 5, 2016 2:45 PM
    Moderator
  • Is this fix included in some future Update Rollup?

    Sven

    Friday, February 5, 2016 4:54 PM
  • Friday, February 19, 2016 6:06 AM
    Moderator
  • I tried to run Fix, but gives errors PowerShell.
    It is also trying to restore the database from a backup, but get error - base buse but before it stopped all services PDM.

    Александр Гавриленков

    Thursday, April 7, 2016 1:51 PM