none
Occasional error backing up VMs, "DPM was unable to establish a connection with the Virtual Machine Manager (VMM) Server" RRS feed

  • Question

  • I've seen this error come up several times since upgrading DPM and setting up the VMM integration.  

    Affected area:	\Backup Using Child Partition Snapshot\SERVER1
    Occurred since:	2/10/2013 7:06:27 PM
    Description:	Recovery point creation jobs for Microsoft Hyper-V \Backup Using Child Partition Snapshot\SERVER1 on HOST.DOMAIN.com have been failing. The number of failed recovery point creation jobs = 1.
     If the data source protected has some dependent data sources (like a SharePoint Farm), then click on the Error Details to view the list of dependent data sources for which recovery point creation failed. (ID 3114)
    	DPM was unable to establish a connection with the Virtual Machine Manager (VMM) server.
    Server name: VMMSERVER.DOMAIN.com.
    Exception Message: Type: System.TimeoutException, Message: This request operation sent to net.tcp://DPMSERVER.DOMAIN.com:6070/VmmHelperService/TcpEndpoint did not receive a reply within the configured timeout (00:01:00).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client. (ID 33400)
    	More information
    Recommended action:	1) Verify that VMM Console features are installed on the DPM Server.
    2) Verify that the VMM server is online. If you need to reconfigure the VMM server in DPM, use the Set-DPMGlobalProperty cmdlet as follows:
    Set-DPMGlobalProperty -DpmServerName <DPMServerName> -KnownVMMServers <VMMServerName>.
    3) Verify that the VMM server has been configured to accept requests from this DPM server.
    	Create a recovery point...
    Resolution:	To dismiss the alert, click below
    	Inactivate
    

    When this occurs, it seems that all of my backups of VM's, across multiple hosts, fail with the same error.  However, later jobs for the same VM's appear to be completing successfully.  I just checked one of the VM's that was affected by this error over the weekend, and it currently shows the latest recovery point was earlier this morning. 

    The first time this happened was right after the upgrade, and I found the DPMVMMHelperService was not running.  Since then, when I see this error I check the service and have always found it running.  I don't see anything related in the event logs on the VMM server.  On the DPM server, in the "VM Manager" log, I do have several entries regarding a "Refresh Performance Data" job that failed to completed, but that appears to be an error that is logged periodically throughout the day, not matching up with the time of the backup issue.  

    What would be causing the dpm server to intermittenly run into this issue?

    Monday, February 11, 2013 8:34 PM

All replies

  • I have seen this happen every often and sometimes it was caused by issues with SCVMM. Restarting the SCVMM service did the trick. This was with the RC of SCVMM 2012 SP1 RC...
    Tuesday, February 12, 2013 11:17 AM
  • I did try restarting just the service, and later the vmm and dpm servers.  Unfortunately, since last week, I have started seeing a slightly different error with these jobs, and this one is coming up regularly.  

    The error is still related to communicating with the VMM server, but now it appears that the issue is with 'get-scvmmserver' not being a recognized cmdlet.  I've confirmed that the vmm console is installed on the dpm server, and is the same version as is installed on the vmm server.  Both have VMM 2012 SP1 UR1 installed.  The DPM Server has had two versions of UR1 for DPM 2012 sp1  (KB2791508 and kb2802095) applied.

    If I launch the VMM Management shell, I'm able to run get-scvmmserver successfully. 


    Monday, February 18, 2013 4:22 PM
  • Hi, I have been having the same issue.  I find that it tends to follow a cluster name DNS registration issue that is always solved by taking the cluster name offline in Failover Cluster manager and then clicking repair.  Then I generally see this VMM server timeout error on our DPM/SCVMM server.  I can't be sure they're related though.  Either way, to fix the VM backups on DPM I restart the "DPM-VMM Helper Service" and run consistency checks, and it's good.  So not sure of the cause but prob just some bugs in the Failover Cluster/DPM/VMM interactions.  BTW this is all 2012.

    -M

    Wednesday, February 27, 2013 4:43 PM
  • In my case it either seems to be the VMM Helper service not responding which means I have to forcibly stop it or it seems to be an issue on the SCVMM server where a restart of the service on the SCVMM server fixes the issue but strange enough the SCVMM console is still working but DPM backups are failing...
    Wednesday, February 27, 2013 4:46 PM
  • Hi Guys

    Did you find any solution ?

    Tks

    Alex

    Friday, April 19, 2013 9:43 PM
  • Hello

    I have the same problem with Etablished connections from DPM to VMM. I have 4 DPM-Servers 2012 SP1 & 1 VMM Virtual Server. DPM Server opens >70 Etablished conecctions on Port 8100. 4 DPM's > 200 open connections. In VMM-Server that More than 200 Etablished connections, no Connect with VMM Admin Console. Restart DPM-VMM Helper on 1 DPM-Server, closes the connections from 1 DPM-Server and VMM Admin Console works.

    Can you check in VMM-Server with netstat -a how many connections to Port 8100 are open? Resrart 1 DPM-VMM Helper Service, check Ports and check the connection VMM Admin Console.

    I don't know why the Helper Service the Ports or connections not closed. I think in the next UR from VMM solved the Problem. I hope...

    Greetings   Reto 


    Friday, April 26, 2013 2:37 PM
  • Presently, thare are no connections listed from the dpm server to the vmm server.  The only established connection on port 8100 was from the vmm server back to itself on port 49394.  I'll check it again later and see if that changes
    Friday, April 26, 2013 4:38 PM
  • No solutions yet. Other than restart DPM - VMM Helper service occasionally.  Here's another thread with the same discussion.
    Friday, April 26, 2013 4:52 PM
  • We get the same error. Anyone an idea for an fix?

    Exception Message: Type: System.Management.Automation.CmdletInvocationException, Message: Unable to connect to the VMM management server. The Virtual Machine Manager service on that server did not respond. (Error ID: 1602).

    When we restart the DPM-VMM Helper Service on our DPM Server the connection to VMM is fine again.


    Certifications: MCSA 2003|MCSE 2003|MCTS(5*)| MCTIP:SA | MCSA:2012


    Sunday, April 28, 2013 11:27 AM
  • How often do the backups crash for you all?

    I get these errors every day or every other day and on rare occasions it runs for 3-4 days without crashing.

    Thursday, May 2, 2013 1:35 PM
  • Same for me, after reboot, i am able to perform backup but after 1 or 2 backups, i still get the same error.

    I have just updated through windows update and a new version of DPM Agent was there. I made the update. Let's wait and see in a few days

    Alex

    Thursday, May 2, 2013 2:42 PM
  • That sounds about right.  Every day or two, occasionally longer.

    Recently we started having more issues with our backups, including CSV failures during backups and HyperV VSS writer failures on our 2012 Cluster.  Those may not be related at all to the dpmvmm helper service issue, but they have distracted me from it for the time being.  I have a ticket open with MS Support for the CSV failures, hopefully they can find a solution.  

    Tuesday, May 7, 2013 6:30 PM
  • Yes, i think there are 2 problems :

    - DPM VMM Helper service : for me, right now, i don't have anymore issue about not finding the "Get-SCVMMServer" command but it is quite recent ...

    - VSS Writer failure on 2012 cluster : i have allowed only 1 VM backup at a time through registry key (as of DPM 2010) and it seems to work fine. In fact, with multiple VM at the same time, i think HyperV is trying to do VSS quicence exactly at the same time on the same node and it is generating the "non ready" or timeout issue on the HyperV VSS writer.

    Perhaps MS should have a look to the new parallel backup across cluster. The issue is only happening when 2 VMs backups are launched exactly at the same time, on the same node which is the case when you have multiple VMs inside a protection group.

    Alex

    Wednesday, May 8, 2013 10:27 AM
  • Hi guys, Since my last post, communication error between DPM and VMM (helper service) came back ! I really don ´t know what CAN be the pb. If i restart the service, next backup is successfull Since i allowed one VM max through registry i don ´t have anymore issue with VSS but it is not a clean solution ... Any news from your side ? RGDS Alex
    Sunday, May 19, 2013 12:16 PM
  • I just wanted to chime in and say that I have the same issue.

    Backing up a Server 2012 Hyper-V Cluster stops working. VMMHelperServiceHost.exe is currently using a little over 5 GB of RAM. Restarting the service fails so I have to kill it. After it starts up again everything is working fine for a little while and backups are running. I am also only allowing a single VM to be backed up at a time.

    Wednesday, May 22, 2013 7:36 PM
  • Could you please tell me on which account de DPM-VMM Helper service is running ? Mine runs with System Local.

    I have scheduled a planned task to stop and start this service 15 minutes before job begins. I have no problem.

    I will open a case at MS tomorrow and keep you updated

    Alex

    Wednesday, May 29, 2013 8:15 PM
  • Hi,

    mine is running with Network Service.

    I'd like to add that the VMMHelper Service seems to crash the Virtual Machine Manager service on the VMM Server. After restarting the VMM Service, things usually start working again. Sometimes I also have to restart the VMM Helper service on the DPM server.

    • Edited by Nordland Wednesday, May 29, 2013 8:43 PM added more info
    Wednesday, May 29, 2013 8:35 PM
  • Alex,

    Any luck with MS Support?

    Monday, June 3, 2013 3:18 PM
  • I didn't reach support yet because of a problem with our corporate credential here in EMEA

    BUT

    it seems that microsoft released a fixe similar to my issue with VSS writer busy

    Have a look to http://support.microsoft.com/kb/2838669/en-us

    Issue 6
    Assume that you create a snapshot for a CSV volume on a Windows Server 2012-based failover cluster. When the snapshot creation is still in progress, another snapshot creation is requested on the same CSV volume. In this situation, the snapshot creation fails and all later snapshot creation attempts on the CSV volume fail.

    Note You cannot create a snapshot for the CSV volume until the volume fails over or the volume goes offline and then back online.

    I'l keep you updated as soon as i have contacted support

    Rgds

    Alex

    Tuesday, June 11, 2013 7:33 AM
  • Alex, thanks for the update. My issue is slightly different and has more to do with the VMMService crashing when backing up the cluster. I'll open a case with MS and see what they can come up with.

    Tuesday, June 11, 2013 1:28 PM
  • Hi

    When restarting the Helper service just before backup begins, i have no more error with the helper service. Let's say that it will have to be troubleshooted anyway.

    Regarding VSS error and sometimes VMM backup failing because of VSS error, i have submit a case at MS. With have the same problem with another customer with Comm Vault / Simpana backing up a 2012 cluster, but VM sometimes stops. (It is not related to CSV offline)

    I'll keep you updated

    Alex

    Tuesday, June 18, 2013 8:12 AM
  • The DPM-VMM Helper Service has a serious problem. We have the same issues as stated above and have applied all the hot fixes. I have to restart the service almost everyday or DPM will miss backups if any changes were made to the protection groups. In addition, today I woke up to find 42 backup issue with from Standalone Hyper-V servers that stated the following.

    Exception Message: Type:
    System.TimeoutException, Message: This request operation sent to

    net.tcp://dpm01.domain.local:6070/VmmHelperService/TcpEndpoint did not receive a reply within the configured timeout (00:01:00).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

    For more information, open DPM Administrator Console and review the alert details in the Monitoring task area.

    I have been trying to implement a Hyper-V Cluster for the last 2 weeks and every time I think I have it working, more bugs come up. Come on Microsoft!

    Tuesday, July 2, 2013 2:44 PM
  • Just to finish this thread, Microsoft has fixed this issue with the release of SP1 Cumulative update 3.

    http://blogs.technet.com/b/dpm/archive/2013/08/21/update-rollup-3-for-system-center-2012-data-protection-manager-service-pack-1-has-been-re-released.aspx

    http://blogs.technet.com/b/dpm/archive/2013/07/30/important-update-on-dpm-2012-sp1-update-rollup-3-issues-and-workarounds.aspx

    • Proposed as answer by m3koval Thursday, September 26, 2013 8:51 PM
    Thursday, September 26, 2013 6:14 PM
  • Thanks, I'll be applying the rollup tonight, it's good to know that it fixes the issue!
    Thursday, September 26, 2013 6:52 PM
  • Here is the link to get the update: http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2877075

    Note:

    1. Open in IE

    2. Don't forget to update Agents

    3. After the Update set VMM Helper Service to Automatic start and Start it. as it gets disabled after the update.

    • Proposed as answer by m3koval Thursday, September 26, 2013 8:51 PM
    Thursday, September 26, 2013 8:49 PM