none
cannot backup VM - The DPM service was unable to communicate with the protection agent RRS feed

  • Question

  • Hello.

    New day, new DPM problem:

    Cannot protect one particular VM using host-level protection. It's the only VM having this problem. The error message (redacted names):

    Affected area: RCT\<VMNAME>
    Occurred since: 24.05.2019 15:52:46
    Description: The replica of Microsoft Hyper-V RCT\<VMNAME> on <HVHOST> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with
     consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 
    
    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
    The DPM service was unable to communicate with the protection agent on <HVHOST>. (ID 52 Details: The remote computer refused the network connection (0x800704C9))
    More information
    Recommended action: 
    1) Restart the DPM Replication Agent service on <HVHOST>.
    2) If <HVHOST> is configured using certificates, ensure that the DPM CPWrapper service is running on DPM Server and aof0078.ad.apponfly.com. Also ensure that certificates used by both the computers are valid.
    Synchronize with consistency check.
    Run a synchronization job with consistency check...
    Resolution: To dismiss the alert, click below

    Can backup all other VMs on the Hyper-V host without issues. Hyper-V 2016 Server protected by DPM 2019. Previously protected by DPM 2016 UR7, where backup of this particular VM worked. HV hosts and DPM are in the same domain.

    What I've tried:

     - CC job - no good

     - removed the protected DS and re-add it to DPM - no good. Actually the initial backup 'finishes' without a problem, but the backup has ~100MB (instead of ~40GB). RP count 0. CC job doesn't help.

     - migrated the VM to different host - same issue

     - 'reset' RCT (by shutting down the VM and deleting the .RCT .MRT files) - no good

     - tried monitoring the DPMRA log during the backup attempt: this error comes out, however doesn't makes it any clearer:

    1250 16A4 
    05/24 13:51:13.117
    03 cbackupsubtask.cpp(653)
    [000001F7BE491E40] C4FBC1EF-FB64-41D5-9F10-5A44C6B4D7B0
    WARNING <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147023671" Reason="Error" CommandID="RAReadDatasetFixup" CommandInstanceID="044cde5b-e0d9-4fd3-8c37-a787048583e2"
     GuidWorkItem="e8436d7b-8204-4910-b7a6-d08637ec1f45" TETaskInstanceID="c4fbc1ef-fb64-41d5-9f10-5a44c6b4d7b0"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147023671"
     DetailedSource="2"/><RAStatus><RAReadDatasetFixup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><LWVStatus BytesTransferred="1272" NumberOfFilesTransferred="7" NumberOfFilesFailed="0"
     DataCorruptionDetected="false"/><FixupStatus BytesTransferred="0" NumberOfFilesTransferred="0" NumberOfFilesFailed="0" DataCorruptionDetected="false"/></RAReadDatasetFixup></RAStatus></Status>

    Any ideas are welcome


    • Edited by MarkosP Friday, May 24, 2019 2:06 PM
    Friday, May 24, 2019 2:04 PM

Answers

  • My original suspicion was probably correct - the VM name length seemed to be the culprit. I shortened the VM name (and also the VHDX name) and I can now successfully backup the VM. Also tried moving the VM to other host, backup there, move back and backup, it's working for now.

    If the situation changes, I'll update this thread.

    If the name length is indeed a problem for DPM, it should be state in the what DPM can/cannot backup in the documentation. Feeling like a MS beta-tester half of the time...

    • Marked as answer by MarkosP Thursday, May 30, 2019 9:54 AM
    Thursday, May 30, 2019 9:54 AM

All replies

  • Hello Markos,

    Can backup all other VMs on the Hyper-V host without issues. Hyper-V 2016 Server protected by DPM 2019. Previously protected by DPM 2016 UR7, where backup of this particular VM worked. HV hosts and DPM are in the same domain.

    Is this a Hyper-V Server 2016 or Windows Server 2016 with the Hyper-V role installed?

    What Hyper-V configuration version is the VM running? Any other differences in these VMs that are working and this that does not work?

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Friday, May 24, 2019 2:29 PM
  • Hyper-V Server 2016 as all our hosts, config version 8 as all the other VMs.

    No significant differences I can think of. The only thing is that the VM name is somewhat long - 48 chars, but no special ones, so that shouldn't be an issue IMO and DPM/RA doesn't indicate that as a problem anywhere.

    Checked VSS writers inside the VM, all good there. All ICs enabled.

    Hyper-V creates a checkpoint successfully, job in DPM runs for about a minute, but doesn't transfer any data, the checkpoint is removed correctly, DPM job fails.

    Friday, May 24, 2019 3:00 PM
  • Affected area: RCT\<VMNAME>
    Occurred since: 24.05.2019 15:52:46
    Description: The replica of Microsoft Hyper-V RCT\<VMNAME> on <HVHOST> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with
     consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 
    
    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
    The DPM service was unable to communicate with the protection agent on <HVHOST>. (ID 52 Details: The remote computer refused the network connection (0x800704C9))
    More information
    Recommended action: 
    1) Restart the DPM Replication Agent service on <HVHOST>.
    2) If <HVHOST> is configured using certificates, ensure that the DPM CPWrapper service is running on DPM Server and aof0078.ad.apponfly.com. Also ensure that certificates used by both the computers are valid.
    Synchronize with consistency check.
    Run a synchronization job with consistency check...
    Resolution: To dismiss the alert, click below

    This does indicate that something is blocking the connection, a firewall perhaps, you could try using the SetDPMServer.exe command from the virtual machine that is failing, to add the firewall exceptions:

    SetDpmServer.exe -dpmServerName <DPMServerName.domain.com>


    You could also try unchecking the Backup (volume shadow copy) in the Integration Services settings of the virtual machine settings?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Friday, May 24, 2019 3:17 PM
  • Disabling Backup IC doesn't help, not sure what you mean with running setdpmserver.exe from the VM OS - it doesn't have DPM agent installed.
    Saturday, May 25, 2019 8:41 AM
  • I misread, thought it was a guest-level backup.

    There is something blocking the communication to the Hyper-V host during the backup for this VM, do you have the firewall enabled on the Hyper-V host?

    Could you monitor the firewall logs when trying to perform the backup of the VM?

    Make sure you’re not blocking any of the dynamic ports:

    https://docs.microsoft.com/en-us/system-center/dpm/configure-firewall-settings-for-dpm?view=sc-dpm-2019

    Do you have any antivirus software on the Hyper-V host? 


    Blog: https://thesystemcenterblog.com LinkedIn:

    Saturday, May 25, 2019 5:28 PM
  • My original suspicion was probably correct - the VM name length seemed to be the culprit. I shortened the VM name (and also the VHDX name) and I can now successfully backup the VM. Also tried moving the VM to other host, backup there, move back and backup, it's working for now.

    If the situation changes, I'll update this thread.

    If the name length is indeed a problem for DPM, it should be state in the what DPM can/cannot backup in the documentation. Feeling like a MS beta-tester half of the time...

    • Marked as answer by MarkosP Thursday, May 30, 2019 9:54 AM
    Thursday, May 30, 2019 9:54 AM
  • Thanks for sharing Markos!

    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, May 30, 2019 9:55 AM