none
Recovery from imported tape media fails and causes DPM management console to crash with ID: 917 RRS feed

  • Question

  • Hi.

    I have a newly installed DPM 2012 R2 on a Windows 2012 R2 with a standalone dedicated SQL server which is causing me a bit of a headache right now.

    I'm about to restore some files from an imported tape media, the tape is from our previous DPM server (DPM 2010). I have identified the media and cataloged it and i'm able to browse it to choose wich files to restore. But as soon as i try to finish up a restore job it causes the console to crash.

    My server is running just fine with backups and restores from i.e tapes that is backed up by this server - no problem there - but imported is not working.

    Below is the message from console when it crashes.

    Connection to the DPM service has been lost.

    Review the application event log for information about a possible service shutdown. Verify that the following services are not disabled:
                DPM
                DPM Replication Agent
                SQLAgent$MSSQLSERVER
                MSSQL$MSSQLSERVER
                Virtual Disk Service
                Volume Shadow Copy

    ID: 917

    Any ideas?


    //Dan



    • Edited by Dannenyo Monday, March 9, 2015 8:42 AM
    Monday, March 9, 2015 7:38 AM

Answers

  • Wow,

    Excuse the tardiness of reply - wrong procedure guys.  Was the following one....

    prc_ARM_DataSource_GetServerName

    USE [DPMDB]
    GO
    /****** Object:  StoredProcedure [dbo].[prc_ARM_DataSource_GetServerName]    Script Date: 07/17/2015 14:35:52 ******/
    SET ANSI_NULLS OFF
    GO
    SET QUOTED_IDENTIFIER OFF
    GO
     
    ALTER PROCEDURE [dbo].[prc_ARM_DataSource_GetServerName]
    (
        @DatasourceId GUID,
        @ServerName   nvarchar(64) OUT
    )
    AS
    DECLARE @error int
        SET @error = 0
     
        SET NOCOUNT ON
     
        SELECT @ServerName = AM.ServerName
        FROM dbo.tbl_IM_Datasource AS DS WITH (NOLOCK)
        JOIN dbo.tbl_AM_Server as AM
            ON DS.ServerId = AM.ServerId
        WHERE DS.DatasourceId = @DatasourceId
        AND DS.IsGCed = 0
     
         ----- BEGIN CHANGE: Proposed Change: Use the Local DPM server as the servername in case the query returns NULL
         if @ServerName is null
         begin
               select @ServerName = servername from tbl_AM_Server where serverid = DPMServerId
         end
         else
         begin
             SELECT @ServerName = AM.ServerName
               FROM dbo.tbl_IM_Datasource AS DS WITH (NOLOCK)
               JOIN dbo.tbl_AM_Server as AM
                    ON DS.ServerId = AM.ServerId
               WHERE DS.DatasourceId = @DatasourceId
               AND DS.IsGCed = 0
         end
         ----- END CHANGE
     
     
        SELECT @error = dbo.udf_DPS_CheckRowCount(1)
     
    DONE:
        SET NOCOUNT OFF
        RETURN @error
    Note the code block referencing the local dpm server name if null....
    • Proposed as answer by Braham20 Friday, July 17, 2015 1:59 PM
    • Marked as answer by Dannenyo Saturday, July 18, 2015 11:26 AM
    Friday, July 17, 2015 1:38 PM

All replies

  • Please confirm if you have installed the Update rollup 05 for DPM 2012 R2. If yes, does the console seems to hang or frozen for some time before it crashes or does it crashes immediately?

    This might be helpful:

    http://www.microsoft.com/en-us/download/details.aspx?id=45914

    If this is not applicable, please provide the latest crash log file from the TEMP folder in the DPM bin folder.

    Tuesday, March 10, 2015 6:11 PM
  • Hi.

    Well, we have the Update Rollup 05 and the patch (KB3040326) released a few days ago installed as well.

    Now it seems that maby there's something wrong with the tape. The import (identifying and recatolog) runs without any errors and the content shows up in recovery area (External DPM Tapes), and i'm able to browse for the data i wish to recover, but when finishing up for the job it results in the error described ni the first post.

    Yesterday i tryed with another tape from the exakt same source and with that one a restore completes without any error. The differens i can see is that the tape i'm using when the job is failing is that it's not showing any numbers in the "Data written" field (in the libraries area) after the recatolog job is done, but on the tape from where the job is running fine it shows numbers in the field after the recatalog is done.

    However the tapes are from the same DPM server, and both jobs ran without any issues so maby the failing tape have been damaged when where handlining it outside the autoloader.

    So my conclusion is that maybe there is something wrong the tape media.


    //Dan



    • Marked as answer by Dannenyo Wednesday, March 11, 2015 8:06 AM
    • Unmarked as answer by Dannenyo Wednesday, March 11, 2015 12:18 PM
    • Edited by Dannenyo Wednesday, March 11, 2015 12:19 PM
    Wednesday, March 11, 2015 8:06 AM
  • Hi.

    Well, we have the Update Rollup 05 and the patch (KB3040326) released a few days ago installed as well.

    Now it seems that maby there's something wrong with the tape. The import (identifying and recatolog) runs without any errors and the content shows up in recovery area (External DPM Tapes), and i'm able to browse for the data i wish to recover, but when finishing up for the job it results in the error described ni the first post.

    Yesterday i tryed with another tape from the exakt same source and with that one a restore completes without any error. The differens i can see is that the tape i'm using when the job is failing is that it's not showing any numbers in the "Data written" field (in the libraries area) after the recatolog job is done, but on the tape from where the job is running fine it shows numbers in the field after the recatalog is done.

    However the tapes are from the same DPM server, and both jobs ran without any issues so maby the failing tape have been damaged when where handlining it outside the autoloader.

    So my conclusion is that maybe there is something wrong the tape media.


    //Dan



    New conclusion: Tapes that contains data from any clustered resources seems to be the problem. No numbers in the "Written Data" field after "Recoverable items recatalog" these tapes.
    Tapes that only contains data from "real" machines seems to work fine and the numbers in the "Written Data" field appears as expected.

    So maybe it's the "Recoverable items recatalog" that is the real problem?

    //Dan

    Wednesday, March 11, 2015 12:29 PM
  • Hi Dan,

    I'm seeing *exactly* the same problem with *exactly* the same scenario; new install last week on a DPM 2012 R2 server with a local SQL 2012 and Windows 2012 R2, all patched to the most recent release (Roll Up 5 for DPM).  HP MSL 4048 tape library.

    This server is meant to be used just for the restoration of tapes from our other offices, DPM 2010 created backups.  

    I can recatalog the tapes successfully, I can drill down in recovery screen and select files on the External Tapes to restore but as soon as I hit go DPM crashes with the ID:917 error.

    I checked through the DPM error logs and would be interested if you're seeing the following. If you check the DPM install directory there is a temp folder with the crash dump logs, on mine it points to an error 943 which means the problem is in the DPM SQL back end database.  I think I've discovered that the stored procedure dbo.prc_ARM_DataSource_GetServerName is returning a null value because it can't find the server ID associated with the datasource ID DPM wants to restore, this returns a NULL value for the @Serverid variable resulting in a crash (poor error handling?).  If you look at the stored proc you can see it runs a query against tbl_IM_Datasource (my table only has entries for the DPM server itself I believe, I looked at our other DMP servers and the protected servers are in here also) and tbl_IM_DataSource (I believe this stores information about protected resources) if I plug in the variables in the log I can see it returns empty row as it can't determine the server ID for the datasource.  I'm wondering if the recatalog isn't getting all the data needed for a granular file level restore.

    All well and good I know, but we still don't know how to fix this issue (which is pretty serious as I need to be able to recover from any of 2000 tapes for the last 5 years)

    I've found that SQL data restores DO seem to work as expected, I can recover them as .mdf and .ldf files to my DPM server from tapes without it crashing.  Any other file so far causes the DPM GUi crash though. I've tried half a dozen different tapes from different DPM servers in our offices.

    There is one method that does work for restore; instead of going to the recovery screen to do the restore if you go to the tapes themselves in the library screen and right click them directly you can select 'view contents' and then select an entire volume for restore.  This does appear to work succcessfully, no crashes, but it isn't really viable if I want to recover a single word document but have to recover 1.5TB of files to get that.


    I'd be interested to see how you're getting on with this problem.

    Below is the entry in my DPM crash dump log that points to this:

    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1315)                                   NORMAL          Caught exception 'Unable to cast object of type 'System.DBNull' to type 'System.String'.' while reading SqlParameter value for @ServerName, defaulting to DBNULL
    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1435)                                   WARNING         SqlRetryCommand diag details - SqlCommandText  => Name=dbo.prc_ARM_DataSource_GetServerName, CommandType=StoredProcedure
    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1435)                                   WARNING         CommandDiagInfo => CanRetry=True, CommandTimeout=3600
    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1435)                                   WARNING         CommandParams   => Count=3, InTx=False
    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1435)                                   WARNING              Param[0]   => ParameterName=@DatasourceId | Value=edf86a20-87ac-4c0c-9760-b5ad451bc481 | Size=0 | DbType=UniqueIdentifier | Direction=Input | IsNullable=False
    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1435)                                   WARNING              Param[1]   => ParameterName=@ServerName | Value=[DBNull] | Size=64 | DbType=NVarChar | Direction=Output | IsNullable=False
    0D04    0A10     03/16    14:52:05.141     09         Catalog.cs(1435)                                   WARNING              Param[2]   => ParameterName=@RETURN_VALUE | Value=50001 | Size=0 | DbType=Int | Direction=ReturnValue | IsNullable=False
    0D04    0A10     03/16    14:52:05.141     17         RecoveryTranslator.cs(695)                                WARNING         Error in StartRecovery
     
     

    Wednesday, March 18, 2015 9:19 PM
  • Hi.

    I do se the same in my logs as you describe, so the scenario is the same with the NULL in return. Right now i tried as you explained it, to do a "copy job" and it starts to restore data. This solves my problem for now but it's not good at all in the long run. In this case i have to restore 1,3tb just to get a few mb's back.

    Thank you for your answer, as said it solved it for now.



    //Dan

    Wednesday, March 18, 2015 9:37 PM
  • Darn, I was hoping you were going to post back saying you resolved this issue :). I'm glad that workaround is sufficient for the moment but I agree that this is not really an acceptable solution.  Hopefully a DPM guru might be able to make another suggestion.  At least in the meantime we can get *something* off these tapes in the interim if we get in a tight spot.
    Wednesday, March 18, 2015 10:29 PM
  • Hello Dannenyo & D-Tilley,

    I'm currently experiencing the same issue with 2 of my DPM servers in our Environment.

    I've escalated the case to a DPM MS Specialist a few weeks ago, they are troubleshooting the issue.

    I will now try your solution to recovery directly from the tape to a disk and see if it works for me as well.

    Wednesday, April 1, 2015 8:26 AM
  • Hello,

    I am having the exact same problem when trying to restore from tape on our secondary DPM server.  I checked the event viewer and found event ID 943 (even though the console crash shows ID 917).  The event viewer error is below:

    Log Name:      Application
    Source:        MSDPM
    Date:          4/2/2015 9:02:55 AM
    Event ID:      943
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      DPM_SERVER_NAME
    Description:
    The description for Event ID 943 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    Unable to connect to the DPM database because the database is in an inconsistent state.

    Problem Details:
    <FatalServiceError><__System><ID>19</ID><Seq>98909</Seq><TimeCreated>4/2/2015 2:02:55 PM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>DBCorruptionException</ExceptionType><ExceptionMessage>DLS error:  SP='dbo.prc_ARM_DataSource_GetServerName' Error=50001</ExceptionMessage><ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error:  SP='dbo.prc_ARM_DataSource_GetServerName' Error=50001
       at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ValidateReturnValue(String errorTitle)
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.DatasourceHelper.GetDatasourceServerName(SqlContext sqlCtx, Guid datasourceId)
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.PrmCatalog.GetDatasourceServerName(Guid datasourceId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.FillAppSpecificPropertiesForFS(ProtectionConfigType protectionConfig)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.SetDestinationConfig(IMCatalogDB imCatalog, DatasetType dataset, RecoveryInformationType recoveryInformation, RecoveryTaskDefinitionType recoveryTaskDefinition)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.AddRecoveryTaskDefinitionForDataset(IMCatalogDB imCatalog, JobTranslator jobTranslator, DatasetType dataset, RecoveryInformationType recoveryInformation, Boolean additionalRestore, Boolean isSerial, Guid parentTaskDefId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.GenerateRecoveryTaskDefinitions(IMCatalogDB imCatalog, JobTranslator jobTranslator, RecoveryInformationType recoveryInformation, Guid parentTaskDefId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.GenerateRecoveryTaskDefinitionsRecursively(IMCatalogDB imCatalog, JobTranslator jobTranslator, RecoveryInformationType recoveryInformation, Guid parentTaskDefId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.GenerateRecoveryTaskDefinitions(IMCatalogDB imCatalog, RecoveryInformations recoveryInformations)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.StartRecovery(Guid jobDefinitionId, Guid jobId, RecoveryInformations recoveryInformations, AdhocJobsContext jobContext)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.StartRecovery(String recoveryXml, Guid jobDefinitionId, Guid jobId, AdhocJobsContext jobContext)
       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CIntentServices.TriggerRecovery(UInt16* bstrIMCatalogXml, _GUID jobDefinitionId, _GUID jobId, AdhocJobsContext jobContext)
       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CCoreServices.TriggerRecovery(CCoreServices* , UInt16* bstrRecoveryInformationXml, _GUID jobDefinitionId, _GUID jobId, UInt16* bstrJobContextXml, tagSAFEARRAY** exceptionResult)</ExceptionDetails></FatalServiceError>


    the message resource is present but the message is not found in the string/message table

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSDPM" />
        <EventID Qualifiers="0">943</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2015-04-02T14:02:55.000000000Z" />
        <EventRecordID>17544</EventRecordID>
        <Channel>Application</Channel>
        <Computer>DPM_SERVER_NAME</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Unable to connect to the DPM database because the database is in an inconsistent state.

    Problem Details:
    &lt;FatalServiceError&gt;&lt;__System&gt;&lt;ID&gt;19&lt;/ID&gt;&lt;Seq&gt;98909&lt;/Seq&gt;&lt;TimeCreated&gt;4/2/2015 2:02:55 PM&lt;/TimeCreated&gt;&lt;Source&gt;DpmThreadPool.cs&lt;/Source&gt;&lt;Line&gt;163&lt;/Line&gt;&lt;HasError&gt;True&lt;/HasError&gt;&lt;/__System&gt;&lt;ExceptionType&gt;DBCorruptionException&lt;/ExceptionType&gt;&lt;ExceptionMessage&gt;DLS error:  SP='dbo.prc_ARM_DataSource_GetServerName' Error=50001&lt;/ExceptionMessage&gt;&lt;ExceptionDetails&gt;Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error:  SP='dbo.prc_ARM_DataSource_GetServerName' Error=50001
       at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ValidateReturnValue(String errorTitle)
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.DatasourceHelper.GetDatasourceServerName(SqlContext sqlCtx, Guid datasourceId)
       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.PrmCatalog.GetDatasourceServerName(Guid datasourceId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.FillAppSpecificPropertiesForFS(ProtectionConfigType protectionConfig)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.SetDestinationConfig(IMCatalogDB imCatalog, DatasetType dataset, RecoveryInformationType recoveryInformation, RecoveryTaskDefinitionType recoveryTaskDefinition)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.AddRecoveryTaskDefinitionForDataset(IMCatalogDB imCatalog, JobTranslator jobTranslator, DatasetType dataset, RecoveryInformationType recoveryInformation, Boolean additionalRestore, Boolean isSerial, Guid parentTaskDefId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.GenerateRecoveryTaskDefinitions(IMCatalogDB imCatalog, JobTranslator jobTranslator, RecoveryInformationType recoveryInformation, Guid parentTaskDefId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.GenerateRecoveryTaskDefinitionsRecursively(IMCatalogDB imCatalog, JobTranslator jobTranslator, RecoveryInformationType recoveryInformation, Guid parentTaskDefId)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.GenerateRecoveryTaskDefinitions(IMCatalogDB imCatalog, RecoveryInformations recoveryInformations)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.StartRecovery(Guid jobDefinitionId, Guid jobId, RecoveryInformations recoveryInformations, AdhocJobsContext jobContext)
       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.StartRecovery(String recoveryXml, Guid jobDefinitionId, Guid jobId, AdhocJobsContext jobContext)
       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CIntentServices.TriggerRecovery(UInt16* bstrIMCatalogXml, _GUID jobDefinitionId, _GUID jobId, AdhocJobsContext jobContext)
       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CCoreServices.TriggerRecovery(CCoreServices* , UInt16* bstrRecoveryInformationXml, _GUID jobDefinitionId, _GUID jobId, UInt16* bstrJobContextXml, tagSAFEARRAY** exceptionResult)&lt;/ExceptionDetails&gt;&lt;/FatalServiceError&gt;
    </Data>
        </EventData>
    </Event>

    I'm guessing if you checked your event viewer, you would have the same error after the crash.  I'm hoping Microsoft has a fix for this besides restoring the database...

    Thursday, April 2, 2015 5:52 PM
  • Hi there,

    We're seeing a very similar issue.  Rebuilt DPM 2012 R2 update 5 server (2008R2 sp1 standard OS), imported backup tape set (encrypted with cert); cert used for backups imported with private key to dpmrestorestore; catalogue the tape set fine, but when trying to recover a single file / folder, dpm console crashes with error:

    Connection to the DPM service has been lost.

    Review the application event log for information about a possible service shutdown. Verify that the following services are not disabled:
                DPM
                DPM Replication Agent
                SQLAgent$MSDPM2012
                MSSQL$MSDPM2012
                Virtual Disk Service
                Volume Shadow Copy

    ID: 917

    Application Event log errors as follows:

    The description for Event ID 943 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    Unable to connect to the DPM database because the database is in an inconsistent state.

    Problem Details:

    <FatalServiceError><__System><ID>19</ID><Seq>32405</Seq><TimeCreated>4/13/2015 8:51:53 AM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>DBCorruptionException</ExceptionType><ExceptionMessage>DLS error:  SP='dbo.prc_ARM_DataSource_GetServerName' Error=50001</ExceptionMessage><ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error:  SP='dbo.prc_ARM_DataSource_GetServerName' Error=50001

       at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ValidateReturnValue(String errorTitle)

       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.DatasourceHelper.GetDatasourceServerName(SqlContext sqlCtx, Guid datasourceId)

       at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.PrmCatalog.GetDatasourceServerName(Guid datasourceId)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.FillAppSpecificPropertiesForFS(ProtectionConfigType protectionConfig)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.SetDestinationConfig(IMCatalogDB imCatalog, DatasetType dataset, RecoveryInformationType recoveryInformation, RecoveryTaskDefinitionType recoveryTaskDefinition)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.AddRecoveryTaskDefinitionForDataset(IMCatalogDB imCatalog, JobTranslator jobTranslator, DatasetType dataset, RecoveryInformationType recoveryInformation, Boolean additionalRestore, Boolean isSerial, Guid parentTaskDefId)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.Datasource.GenerateRecoveryTaskDefinitions(IMCatalogDB imCatalog, JobTranslator jobTranslator, RecoveryInformationType recoveryInformation, Guid parentTaskDefId)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.GenerateRecoveryTaskDefinitionsRecursively(IMCatalogDB imCatalog, JobTranslator jobTranslator, RecoveryInformationType recoveryInformation, Guid parentTaskDefId)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.GenerateRecoveryTaskDefinitions(IMCatalogDB imCatalog, RecoveryInformations recoveryInformations)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.StartRecovery(Guid jobDefinitionId, Guid jobId, RecoveryInformations recoveryInformations, AdhocJobsContext jobContext)

       at Microsoft.Internal.EnterpriseStorage.Dls.Intent.RecoveryTranslator.StartRecovery(String recoveryXml, Guid jobDefinitionId, Guid jobId, AdhocJobsContext jobContext)

       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CIntentServices.TriggerRecovery(UInt16* bstrIMCatalogXml, _GUID jobDefinitionId, _GUID jobId, AdhocJobsContext jobContext)

       at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CCoreServices.TriggerRecovery(CCoreServices* , UInt16* bstrRecoveryInformationXml, _GUID jobDefinitionId, _GUID jobId, UInt16* bstrJobContextXml, tagSAFEARRAY** exceptionResult)</ExceptionDetails></FatalServiceError>

     

    the message resource is present but the message is not found in the string/message table

    Did you get any further with MS Support?  We have 24/7 web support looking at it, looking to escalate to premier support this morning to get it fixed asap.

    Jonthan

    Monday, April 13, 2015 8:54 AM
  • I rebuilt our secondary DPM server with a fresh install of Windows Server 2012 R2 and SQL 2012 with SP2.  I then installed DPM 2012 R2.  Without any DPM updates installed, I was able to restore from tape at the file level.  I found that we had UR3 for DPM 2012 R2 so I updated to UR3 and again, I was able to restore from tape at the file level.  I installed UR5 and the console crashed when trying to restore from tape at the file level, so I installed the MS Hotfix KB3040326.  The console crashed when trying to restore from tape at the file level.  The last thing I did was install UR6, but the issue remains that the console crashes when attempting to restore from tape at the file level.

    I made sure to reboot the server in between all UR updates and after the Hotfix as well.  I even went as far as to rebuild the server again with a fresh install of Server 2012 R2 with SQL 2012 SP2.  Once that was all installed, I installed DPM 2012 R2 and went directly to UR6.  The console crashed when attempting to restore from tape at the file level.

    I'm not sure what else to do.  My primary DPM server is at UR5, so I can't leave my backup server at UR3 - and I haven't had the chance to see if the problem lies with that server too or not as I haven't had a chance to restore from tape.

    Has anyone heard anything from MS about this?

    Thursday, May 14, 2015 5:23 PM
  • I went through the server rebuild process again and installed Server 2012 R2 with SQL Server 2012 SP2 and DPM 2012 R2.  I tested restoring from tape from our primary DPM server at the file level and the restore worked.  I then downloaded and installed UR4 and tested.  I was again able to restore from tape from our primary DPM server at the file level.  I installed UR5, the UR5 hotfix and UR6 - all of which failed, so there is something wrong with the Update Rollouts after UR4.

    Apparently Microsoft does not have an answer for this.

    Thursday, May 28, 2015 8:45 PM
  • Sorry to keep bringing this up but I am still experiencing this issue.  Has anyone found a fix?  UR6 did not solve the problem I am experiencing and restoring an entire tape to get a single file isn't really an answer for me as the server will need several TB available in order to restore a tape.

    What I don't understand is why everything works fine up to UR4, but I have the problem with UR5 (with and without the hotfix) and UR6.

    Thanks in advance to any information anyone can provide.

    Wednesday, June 17, 2015 3:08 PM
  • We have exact same issue as above running on the UR6.

    We see this issue occur when we try and recover data from a re-cataloged tape (tape expired).

    This causes console to crash with the above error.

    Wednesday, July 8, 2015 10:30 AM
  • Hi all,

    We got this resolved. I opened a case with MS Support using our web based SA capabilities.  Turns out the issue was, for us at least, to do with DPM not being able to identify / retrieve the correct name dfor the server that originally owned the backups.

     

    To caveat, our system was originally a DPM 2010, went through the entire upgrade path from 2010 to 2012 to R2 to latest update; sql was also updated I there along the road.

    To fix it, MS Engineering had us alter the following stored procedure:

     

    dbo.prc_AM_GetPrimaryDPMServerName

     

    the following code could be copyright - watch out, but I'll post in case anyone needs a hand.  This is now our stored procedure (the one listed above):

     
    USE [DPMDB]
    GO
    /****** Object:  StoredProcedure [dbo].[prc_AM_GetPrimaryDPMServerName]    Script Date: 7/8/2015 2:09:52 PM ******/
    SET ANSI_NULLS OFF
    GO
    SET QUOTED_IDENTIFIER OFF
    GO
    
    ALTER  PROCEDURE [dbo].[prc_AM_GetPrimaryDPMServerName]
    (
        @DatasourceId GUID
    )
    AS
    
    DECLARE @error int,
            @rowCount int
    
    SET @error = 0
    SET @rowCount = 0 
    
    SET NOCOUNT ON
    
    SELECT 
        AM.ServerName
    FROM dbo.tbl_AM_Server AM
    JOIN dbo.tbl_IM_DPMDatasourceReplicaProperties DPMDS ON
    DPMDS.ServerId = AM.ServerId
    WHERE DPMDS.DataSourceId = @DatasourceId
    AND DPMDS.IsGCed = 0 AND AM.MarkedForDeletion = 0
        
    SELECT @error = dbo.udf_DPS_CheckRowCount(1)
    
    
    SET NOCOUNT OFF
    
    RETURN @error

    Hope this helps you out.

    Regards,

    jonathan

    Wednesday, July 8, 2015 1:14 PM
  • Hi Jonathan! This sounds like great news and i will try it out ASAP and i'l reply after it. \\Dan

    //Dan


    • Edited by Dannenyo Wednesday, July 8, 2015 7:47 PM
    Wednesday, July 8, 2015 7:46 PM
  • This code is exactly what is already in the stored procedure
    Thursday, July 16, 2015 3:48 PM
  • Yeah that's definitely the same one... I felt a flutter of hope for a second, we're having the exact same problem described in this thread.
    Friday, July 17, 2015 11:04 AM
  • Wow,

    Excuse the tardiness of reply - wrong procedure guys.  Was the following one....

    prc_ARM_DataSource_GetServerName

    USE [DPMDB]
    GO
    /****** Object:  StoredProcedure [dbo].[prc_ARM_DataSource_GetServerName]    Script Date: 07/17/2015 14:35:52 ******/
    SET ANSI_NULLS OFF
    GO
    SET QUOTED_IDENTIFIER OFF
    GO
     
    ALTER PROCEDURE [dbo].[prc_ARM_DataSource_GetServerName]
    (
        @DatasourceId GUID,
        @ServerName   nvarchar(64) OUT
    )
    AS
    DECLARE @error int
        SET @error = 0
     
        SET NOCOUNT ON
     
        SELECT @ServerName = AM.ServerName
        FROM dbo.tbl_IM_Datasource AS DS WITH (NOLOCK)
        JOIN dbo.tbl_AM_Server as AM
            ON DS.ServerId = AM.ServerId
        WHERE DS.DatasourceId = @DatasourceId
        AND DS.IsGCed = 0
     
         ----- BEGIN CHANGE: Proposed Change: Use the Local DPM server as the servername in case the query returns NULL
         if @ServerName is null
         begin
               select @ServerName = servername from tbl_AM_Server where serverid = DPMServerId
         end
         else
         begin
             SELECT @ServerName = AM.ServerName
               FROM dbo.tbl_IM_Datasource AS DS WITH (NOLOCK)
               JOIN dbo.tbl_AM_Server as AM
                    ON DS.ServerId = AM.ServerId
               WHERE DS.DatasourceId = @DatasourceId
               AND DS.IsGCed = 0
         end
         ----- END CHANGE
     
     
        SELECT @error = dbo.udf_DPS_CheckRowCount(1)
     
    DONE:
        SET NOCOUNT OFF
        RETURN @error
    Note the code block referencing the local dpm server name if null....
    • Proposed as answer by Braham20 Friday, July 17, 2015 1:59 PM
    • Marked as answer by Dannenyo Saturday, July 18, 2015 11:26 AM
    Friday, July 17, 2015 1:38 PM
  • You sir, are a genius! That worked a treat - thank you :)
    Friday, July 17, 2015 2:03 PM
  • You're welcome - hope this helps everyone else out too. 

    Jonathan

    Friday, July 17, 2015 2:43 PM
  • That worked a treat - thank you Jonathan. (y)

    //Dan

    Saturday, July 18, 2015 11:28 AM
  • Worked for me as well, thanks for posting your fix.
    Friday, August 28, 2015 6:39 AM
  • Great stuff! Thanks!
    Thursday, November 12, 2015 11:54 AM
  • I'm still getting the 917 error even after applying this Stored Procedure modification.

    I am restoring data from a datasource that no longer physically exists, but I assume that it's a legitimate operation because it prompts for an alternate restore location.

    I'm performing a Copy direct from the tape inventory dialog and although the job starts it crashes the console shortly afterwards.

    Any suggestions welcome.

    Windows Server 2012 R2
    DPM 2012 R2 v4.2.1553.0

    Here's the error in case it means anything to anyone:

    <FatalServiceError>
    <__System>
    <ID>19</ID>
    <Seq>1197</Seq>
    <TimeCreated>01/12/2016 11:24:06</TimeCreated>
    <Source>DpmThreadPool.cs</Source>
    <Line>163</Line>
    <HasError>True</HasError>
    </__System>
    <ExceptionType>DBCorruptionException</ExceptionType>
    <ExceptionMessage>Couldn't access DataSourceInfo '00000000-0000-0000-0000-000000000000' in the database</ExceptionMessage>
    <ExceptionDetails>Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: Couldn't access DataSourceInfo '00000000-0000-0000-0000-000000000000' in the database... etc.

    • Edited by AndyChips Thursday, December 1, 2016 11:30 AM
    Thursday, December 1, 2016 11:12 AM
  • Have you found any solution for this issue yet? 

    I got exactly same problem.

    Please let me know.

    Thursday, January 26, 2017 11:56 PM