locked
SCVMM 2008 RTM: Virtual Server 2005 R2 SP1: Connection Status - Not Responding RRS feed

  • Question

  • Hi,

     

    I'm running SCVMM 2008 RTM (2.0.3444.0) on dedicated VMM server.
    I can manage Hyper-V hosts fine using SCVMM.
    I'm trying to add a 'Virtual Server 2005 R2 SP1' running on Server 2003 SP2 x64 and receiving this error every time:

    ------------------------------
    Error (2923)
    A malformed response was received trying to contact the agent on vs02.mydom.local server.
     (The operation completed successfully (0x0))

    Recommended Action
    Ensure the agent is installed and running. If the agent is running, restart the agent and verify that WS-Management is installed correctly.

    ------------------------------


    In the VMM / Debugview trace I can see the following:

     

    RunningTaskDbWriter.cs(464): MarkTaskComplete --> TaskResult: HostAgentInvalidResponseFromServer (2923); HR: 0x0|TaskID=69867398-3800-4E18-A7BD-5A3936C7D9C7

     

    This can be 100% reproduced from Powershell by invoking the Refresh-VMHost command.

     

    Under: Host / Properties / Status, we have:

    Connection Status: Not Responding

    Agent Status: Responding

    Agent Version: up-to-date

    Virtualization Service Status: Running

    Virtualization Service Version: Up-to-date

     

     

    I have tried a number of workarounds including removing/readding host and also installing the VMM agent locally first.

    Have also followed the suggestions from Cheng's blog:
    (some of these aren't relevant to Virtual Server, although slight modifications can be made to achieve equivalent behaviour e.g. WMI namespace)

    http://blogs.technet.com/chengw/archive/2008/05/02/what-to-do-when-i-have-host-in-needs-attention-state-or-my-connection-status-is-not-responding.aspx
    http://blogs.technet.com/chengw/archive/2008/07/30/what-else-can-i-do-when-my-host-is-in-not-responding-state.aspx

    Also,

    WS-Management is listening correctly, and can be connected across the network.
    Windows Firewall is disabled on both VMM server and VS2005 host.

    There are no USB decides attached to either host or VMM server.

     

    Any clues?

    Wednesday, November 5, 2008 5:56 PM

Answers

  • Peter,

     

    Could you please check to see if the VMM agent and the VMM server are at the exact same version? The WinRM error below seems to indicate a mismatch agent/server pair, or a possibly corrupted WinRM on the host.

     

    00000065             9.31293297          [4616] 1208.1718::11/07-12:09:35.502#18Tongue TiederverConnection.cs(1125): Unknown exception received in PerformOperation  
    00000066             9.31367397          [4616] 1208.1718::11/07-12:09:35.502#18Tongue TiederverConnection.cs(1125): Microsoft.VirtualManager.Utils.CarmineException: A malformed response was received trying to contact the agent on 1eprdvs02.1e.local server.          
    00000067             9.31367397          [4616] Ensure the agent is installed and running. If the agent is running, restart the agent and verify that WS-Management is installed correctly.  
    00000068             9.31367397          [4616]    at Microsoft.SystemCenter.VirtualMachineManager.TaskHelper.ThrowOnFailure()                
    00000069             9.31367397          [4616]    at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdlet`1.Process()       
    00000070             9.31367397          [4616]    at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.CmdletBase.ProcessRecord()            
    00000071             9.31367397          [4616] *** Carmine error was: HostAgentInvalidResponseFromServer (2923); HR: 0x0   
    00000072             9.31367397          [4616] *** 2003 member in domain (linked to 2003 AD DC) ** 1eprdvs02.1e.local ** D:\ISO\OpenManage5.40.iso **              

     

    Thanks!
    Cheng

    http://blogs.technet.com/chengw/

     

     

     

    Thursday, November 13, 2008 6:44 AM
  • Update:  Our issue was resolved on two hosts so far by updating the Broadcom Management Suite to a new version.  The old one was dated 2004.  So far so good.  

    It took hours to get any hint of a solution but after removing the Broadcom software, Iwas able to complete a refresh of the host at 100%.

    Mike
    Thursday, December 4, 2008 9:55 PM

All replies

  • Hi Peter,

     

    Can you please try reproducing this host refresh error, collect the complete VMM Server traces and send it to me?

     

    Thanks!

    Cheng

    http://blogs.technet.com/chengw

     

    Friday, November 7, 2008 1:13 AM
  • Cheng, you should have the results in your mailbox.

     

    -Peter

    Monday, November 10, 2008 9:50 AM
  • Peter,

     

    Could you please check to see if the VMM agent and the VMM server are at the exact same version? The WinRM error below seems to indicate a mismatch agent/server pair, or a possibly corrupted WinRM on the host.

     

    00000065             9.31293297          [4616] 1208.1718::11/07-12:09:35.502#18Tongue TiederverConnection.cs(1125): Unknown exception received in PerformOperation  
    00000066             9.31367397          [4616] 1208.1718::11/07-12:09:35.502#18Tongue TiederverConnection.cs(1125): Microsoft.VirtualManager.Utils.CarmineException: A malformed response was received trying to contact the agent on 1eprdvs02.1e.local server.          
    00000067             9.31367397          [4616] Ensure the agent is installed and running. If the agent is running, restart the agent and verify that WS-Management is installed correctly.  
    00000068             9.31367397          [4616]    at Microsoft.SystemCenter.VirtualMachineManager.TaskHelper.ThrowOnFailure()                
    00000069             9.31367397          [4616]    at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdlet`1.Process()       
    00000070             9.31367397          [4616]    at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.CmdletBase.ProcessRecord()            
    00000071             9.31367397          [4616] *** Carmine error was: HostAgentInvalidResponseFromServer (2923); HR: 0x0   
    00000072             9.31367397          [4616] *** 2003 member in domain (linked to 2003 AD DC) ** 1eprdvs02.1e.local ** D:\ISO\OpenManage5.40.iso **              

     

    Thanks!
    Cheng

    http://blogs.technet.com/chengw/

     

     

     

    Thursday, November 13, 2008 6:44 AM
  •  

    Cheng,

     

    The versions of the server and agent match exactly (2.0.3444.0).

    I will configure WinRM tracing and send the results to you.

     

    -Peter

    Thursday, November 13, 2008 5:50 PM
  • I am having basically the same problem on a 2008 Core Hyper-V host.  The guests show up in VMM, but the status shows as "Host Not Responding".  When I try to refresh the host, the following error is generated:

     

    Error (2923)

    A malformed response was received trying to contact the agent on bgbvirtual8.bg.brasfieldgorrie.com server.

    (The operation completed successfully (0x0))

    Recommended Action

    Ensure the agent is installed and running. If the agent is running, restart the agent and verify that WS-Management is installed correctly.

     

    So my question is this:  How do I verify WS-Management is installed correctly, and how is it that it would be installed incorrectly or corrupted?

     

    Thanks for any insight or explanation.

     

    Michael

    Thursday, November 13, 2008 8:58 PM
  • Cheng,

     

    I have mailed WinRM traces to you, taken from both VMM server and VM host.

     

    -Peter

    Friday, November 14, 2008 11:38 AM
  • Michael,

     

    Could you please also check and verify that your agent and VMM server are running at the same version? The error message is from WinRM and generic. It essentially means that VMM server received a WinRM response in an unsupported format. We have seen this error the most in agent/server version mismatch scenarios.

     

    If you want to, you can simply uninstall the agent from the Server Core host(s) and re-add the hosts on the VMM 2008 server.

     

    Also, when you re-add the hosts, please turn on the traces before you retry adding them back, so that we have all the needed details to help you diagnose, in case there is any issue with your "Add Hosts" job.

     

    Lastly, when the host is in "Needs Attention" state, it'd be helpful if you provide other sub-statuses (connection status, agent status, agent version, virtualization service status, virtualization service version).

     

     

    Hope this helps,

    Cheng

    http://blogs.technet.com/chengw

    Wednesday, November 19, 2008 6:47 PM
  • Peter -

     

    Yes, the new traces you sent on Nov 17 contain some useful information and we're reviewing your WinRM traces with the WinRM team. I'll email you or post update back when we have some new information.

     

    Thanks for your patience!

    Cheng

    http://blogs.technet.com/chengw

     

    Wednesday, November 19, 2008 6:52 PM
  •  

    Has any solution to this been discovered yet?  I am having the same problem.  I'm running VMM on a Windows 2008 Enterprise server with Hyper-V and have no problems managing the local system as a host.  However, when I try to add a Windows 2003 x64 Enterprise SP2 system running Virtual Server 2005 R2 host, I receive the same connectivity error message 2923 about a malformed response when trying to contact the agent.  I have installed, restarted both the WinRM service and the VMM agent (even trying to install the VMM agent from disk), but the VMM admin console still shows no connectivity to the host.

     

    Thursday, November 20, 2008 8:44 PM
  • I am having the same issue with a couple of my Windows 2003 x64 Enterprise R2 SP2 servers running Virtual Server SP1.  What I have discovered is that VMM 2008 reports that the Virtualization Service Version is up-to-date on the Status tab under the properties of the host, when it really isn't.  The option to update the virtual server host in VMM is also greyed out.  My servers that are running correctly have a Virtual Server version of 1.1.629.0 EE R2 SP1.  The ones that are experiencing the errors being reported in this thread are 1.1.603.0 EE R2 SP1.  These are the values reported though the VS 2005 Administration website.  I will try to manually update these during my next maintenance cycle to see if this is the cause.  Is anyone else seeing this?

     

     

    Update:  I was able to apply KB948515 to the server which brought the version number to 1.1.629.0 EE R2 SP1.  Unfortunately, i am still getting the same error.  Any update on this? 

    Monday, November 24, 2008 7:32 PM
  • The agent version I'm running on my 2003 Servers are 2.0.3444.0.  Still see an error in Connection Status as Not Responding but everything else is good.  Apparently Connection Status is not needed to view CPU Average or Available Memory since those values are actually viewable in the VMM console. 

    Tuesday, November 25, 2008 7:28 PM
  • We are having the same problems on our VS 2005 Windows 2003 servers as well.  All three servers are up to date with MS updates and still give the Error 2923 error message when attempting to add them to SCVMM RTM.  They worked fine in the beta version.  Removed the agent from both and installed the RTM version so they would match, still same error.

    mike
    Monday, December 1, 2008 2:38 PM
  • This should update the date of the message:

     

    Update:  I was able to apply KB948515 to the server which brought the version number to 1.1.629.0 EE R2 SP1.  Unfortunately, i am still getting the same error.  Any update on this? 

     

    Thursday, December 4, 2008 9:10 PM
  • Update:  Our issue was resolved on two hosts so far by updating the Broadcom Management Suite to a new version.  The old one was dated 2004.  So far so good.  

    It took hours to get any hint of a solution but after removing the Broadcom software, Iwas able to complete a refresh of the host at 100%.

    Mike
    Thursday, December 4, 2008 9:55 PM
  • Thats very interesting. Our issue has also recently been cleared.

    This coincided with installation of a new Broadcom network interface (5709) for which I think we updated the BACS software and rebooted the system.

    The status in VMM is now all clear.

    BACS is at version: 11.3.14.0
    Monday, December 8, 2008 3:31 PM
  • The Broadcom update fixed my issues as well.  Thanks.

     

    Wednesday, December 10, 2008 3:13 PM