none
DPM 2019 UR1 Error during VMWare restore RRS feed

  • Domanda

  • We have a DPM Protection Group which consist of VCenter and one virtual machine (linux based) for a backup. Backup was sucsesfull.

    However after we delete the VM on VCenter we can't restore. There was two scenarios that we tested:

    • During recovery procedure select Recover to original instance.

    Error we got was:

    The recovery jobs for rix3-midpoint02 that started at Thursday, March 5, 2020 11:15:40 AM, with the destination of rix3-vcenter01.mgmt.evolutiongaming.com, have completed. Most or all jobs failed to recover the requested data. (ID: 3111)
    DPM encounterd error from VMware server rix3-vcenter01.mgmt.evolutiongaming.com with Fault - VMNotFound (ID: 33614)

    • During recovery procedure select Recovery as virtual machine on any host.

    During select ESXi Host (via Browse button) - console just dies and we got error:

    The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event: 

    An unexpected error caused a failure for process 'mmc'.  Restart the DPM process 'mmc'.

    Problem Details:
    <FatalServiceError><__System><ID>19</ID><Seq>0</Seq><TimeCreated>03/05/20 11:19:40</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>ArgumentOutOfRangeException</ExceptionType><ExceptionMessage>Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index</ExceptionMessage><ExceptionDetails>System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index
       at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.RecoveryPage.VMWareVMAlternateTarget.PopulateHostInquiryResult(TreeNode parentNode, ProductionServer productionServer, VMWareInventory inventory)
       at Microsoft.Internal.EnterpriseStorage.Dls.UI.RecoveryPage.VMWareVMAlternateTarget.vmwareInventory_InquiryComplete(Object sender, VMWareInventoryObjectDetectedEventArgs e)</ExceptionDetails></FatalServiceError>


    The message resource is present but the message was not found in the message table

    Does some one faced same errors?

    Does any one has ideas how we can fix that?

    giovedì 5 marzo 2020 11:20

Tutte le risposte

  • Hello Pjotrs,

    Could you give us some details about your DPM environment?

    So you are currently running DPM 2019 with Update Rollup 1 installed, could you also provide the following information:

    • Which Windows Server version is the DPM server running on?
    • Which SQL Server version is the DPM database using?
    • Which version of vCenter are you using?
    • Have you tried restoring a Windows virtual machine?

    Note the following:

    Original location recovery (OLR) - Use OLR to restore a protected VM to its original location. You can restore a VM to its original location only if no disks have been added or deleted, since the back up occurred. If disks have been added or deleted, you must use alternate location recovery.


    Alternate location recovery (ALR) - When the original VM is missing, or you don't want to disturb the original VM, recover the VM to an alternate location. To recover a VM to an alternate location, you must provide the location of an ESXi host, resource pool, folder, and the storage datastore and path. To help differentiate the restored VM from the original VM, DPM appends "-Recovered" to the name of the VM.

    ----------------------------------------------------------------------------------------------------------------

    If you've done accordingly to the above, then I would suggest you to check the DPM log for any more clues:

    DPM server

    • %ProgramFiles%\Microsoft System Center\DPM\DPM\Temp\MSDMCurr.errlog

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    giovedì 5 marzo 2020 13:04
    • Which Windows Server version is the DPM server running on? Windows Server 2019 (Build 17763.1075)
    • Which SQL Server version is the DPM database using? SQL Server 2017 14.0.2027.2
    • Which version of vCenter are you using? vCenter 6.7
    • Have you tried restoring a Windows virtual machine? No

    Log output:

    127C 04AC 03/05 11:19:37.299 03 service.cpp(147) [00000048D927F870] ACTIVITY CService::ServiceHandlerExInternal
    127C 04AC 03/05 11:19:37.299 03 service.cpp(147) [00000048D927F870] ACTIVITY CService::ServiceHandlerExInternal
    127C 2AB0 03/05 11:19:38.080 05 command.cpp(286) [0000019C10EE5EA0] NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 2074 03/05 11:19:38.080 03 cworkitem.cpp(429) [0000019C10E10778] B7E9FD4E-8456-4833-B33B-80598D5C335B ACTIVITY Idle Timer created with timeout = 390000
    127C 2074 03/05 11:19:38.096 05 command.cpp(286) [0000019C12ED69C0] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 2074 03/05 11:19:38.096 61 inquirysubtask.cpp(1030) [0000019C12EF7910] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL ==>Tasks::CInquirySubTask::ExecuteInquiry
    127C 2074 03/05 11:19:38.096 61 nevmwareserver.cpp(313) [0000019C12D20600] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL ==>CNEVMWareServer::EnumerateChildren
    127C 2074 03/05 11:19:38.096 61 nevmwareserver.cpp(464) [0000019C12D20600] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL ==>CNEVMWareServer::EnumerateInvetoryObjectChildren
    127C 2074 03/05 11:19:38.096 61 nevmwareserver.cpp(422) [0000019C12D20600] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL ==>CNEVMWareServer::GetVMWareConnectionInfo
    127C 2074 03/05 11:19:38.096 61 nevmwareserver.cpp(422) [0000019C12D20600] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL <--CNEVMWareServer::GetVMWareConnectionInfo
    127C 2074 03/05 11:19:38.096 22 VMWareServer.cs(237) NORMAL Calling VMWareAPI:RetrieveServiceContent with argument serviceReferenceMOR:MOR = type:ServiceInstance, value:ServiceInstance and URL:https://vcenter01:443/sdk
    127C 2AB0 03/05 11:19:38.096 05 command.cpp(286) [0000019C1186E570] NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 2074 03/05 11:19:38.455 22 VMWareServer.cs(244) NORMAL Calling VMWareAPI:Login with arguments sessionManager:MOR = type:SessionManager, value:SessionManager, username:mgmt\adm-pjotrs
    127C 2074 03/05 11:19:38.564 22 VMWareServer.cs(253) NORMAL Connected to VMWareServer vcenter01:443
    127C 2074 03/05 11:19:38.564 22 VMWareServer.cs(2591) NORMAL GetInventoryObjects() Called 
    127C 1318 03/05 11:19:39.096 05 command.cpp(286) [0000019C11780E20] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 2AB0 03/05 11:19:39.096 05 command.cpp(286) [0000019C11780E20] NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 2074 03/05 11:19:39.392 22 VMWareServer.cs(357) NORMAL Disconnected to VMwareServer vcenter01:443
    127C 2074 03/05 11:19:39.392 61 nevmwareserver.cpp(464) [0000019C12D20600] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL <--CNEVMWareServer::EnumerateInvetoryObjectChildren
    127C 2074 03/05 11:19:39.392 61 nevmwareserver.cpp(313) [0000019C12D20600] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL <--CNEVMWareServer::EnumerateChildren
    127C 2074 03/05 11:19:39.392 61 inquirysubtask.cpp(1030) [0000019C12EF7910] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL <--Tasks::CInquirySubTask::ExecuteInquiry
    127C 1318 03/05 11:19:39.392 61 inquirysubtask.cpp(457) [0000019C12EF7910] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL Sending final response with 107 records
    127C 1318 03/05 11:19:39.424 05 command.cpp(286) [0000019C12EE4510] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 1318 03/05 11:19:39.424 61 inquirysubtask.cpp(989) [0000019C12EF7910] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL CInquirySubTask::Inquiry finished with status [0000000000]
    127C 1318 03/05 11:19:39.424 05 command.cpp(286) [0000019C2CDA24E0] B7E9FD4E-8456-4833-B33B-80598D5C335B NORMAL CCommand::GetCommandTimeout, registry for command timeout not found, setting default timeout
    127C 1318 03/05 11:19:39.424 03 cworkitem.cpp(23) [0000019C10E10700] B7E9FD4E-8456-4833-B33B-80598D5C335B ACTIVITY WorkItem stopping


    giovedì 5 marzo 2020 13:38
  • Note the following:

    Original location recovery (OLR) - Use OLR to restore a protected VM to its original location. You can restore a VM to its original location only if no disks have been added or deleted, since the back up occurred. If disks have been added or deleted, you must use alternate location recovery.


    Alternate location recovery (ALR) - When the original VM is missing, or you don't want to disturb the original VM, recover the VM to an alternate location. To recover a VM to an alternate location, you must provide the location of an ESXi host, resource pool, folder, and the storage datastore and path. To help differentiate the restored VM from the original VM, DPM appends "-Recovered" to the name of the VM.

    Did you have a look at the above?

    Nothing out of the ordinary in the log part that you've attached, but the log should be bigger, you could upload it to for example Microsoft OneDrive or Google Drive and share the link here.


    Blog: https://thesystemcenterblog.com LinkedIn:

    giovedì 5 marzo 2020 14:45
  • Leon, as I rad what you posted above. As I mentioned in my first post - while trying to recover VM In to Alternative Location while browsing VMWare Server - Console just shuts down/closes.

    I've copied here info from logs related to exact time when we tried to restore VMWare VM.

    venerdì 6 marzo 2020 08:09
  • Thanks for the detailed GIF, ok this is definitely not normal...

    What have you tried so far, have you tried rebooting the DPM server?

    Can you try to perform this on any other virtual machine, just to see if the behavior is the same?


    Blog: https://thesystemcenterblog.com LinkedIn:

    venerdì 6 marzo 2020 10:42
  • What we tested so far:

    Created Windows Server VM, created recovery point

    • Tried to recover to Alternative Location - same console behavior - just closes.
    • Tried to recover single file to Network folder - restore was successful
    • Tried to recover whole VM to same location (while VM object is still on VMWare Server) - restore was successful

    Any ideas what we else we can check/test?



    venerdì 6 marzo 2020 14:50
  • This phenomenon is documented in the following Microsoft blog post from the System Center team:


    Reference:
    https://techcommunity.microsoft.com/t5/system-center-blog/implementing-microsoft-dpm-host-level-protection-of-vmware-vms/ba-p/351733


    Blog: https://thesystemcenterblog.com LinkedIn:

    venerdì 6 marzo 2020 15:42
  • Don't get me wrong, but let's not call "bug" as a "phenomenon".
    Is there any chance that this bug will be fixed?
    lunedì 9 marzo 2020 08:00
  • I don't have an answer for that, but if it is a bug, then you should raise a ticket directly to Microsoft and they may be able to provide a private fix.

    https://support.microsoft.com/en-us/hub/4343728/support-for-business


    Blog: https://thesystemcenterblog.com LinkedIn:

    lunedì 9 marzo 2020 08:05