none
CentOS 6.5 / Linux VM KVP IP can't be shown on Hyper-V Manager after querying its KVP/IP information on Hyper-V host several times

    Question

  • Hello

    I am using a CentOS 6.5 VM (Linux kernel 2.6.32-431) with Hyper-V KVP daemon installed, and I periodically query its IP (via using WMI or Powershell to query its KVP information) to manage it.

    However, after querying its IP (KVP) for several times, its IP can’t be queried or shown on Hyper-V Manager anymore.

    I attached a simple KVP query Powershell script as the follows, and this problem can be reproduced in couple minutes if you run two instances with this script at the same time.

    $VMName = $args[0]
    write-host "$VMName"
    
    filter Import-CimXml
    {
        $CimXml = [Xml]$_
        $CimObj = New-Object -TypeName System.Object
        foreach ($CimProperty in $CimXml.SelectNodes("/INSTANCE/PROPERTY"))
        {
            if ($CimProperty.Name -eq "Name" -or $CimProperty.Name -eq "Data")
            {
                $CimObj | Add-Member -MemberType NoteProperty -Name $CimProperty.NAME -Value $CimProperty.VALUE
            }        
        }
        $CimObj
        $CimObj = $null
    }
    
    for ($i=1 ; $i -le 10000 ; $i++) {    
        $a = Get-Date
        write-host "$i - Time: " $a.ToLocalTime()
        $vm = Get-WmiObject -Namespace root\virtualization\v2 -Query "Select * From Msvm_ComputerSystem Where ElementName='$VMName'"
        $vm.ElementName
        $vmkvp = Get-WmiObject -Namespace root\virtualization\v2 -Query "Associators of {$vm} Where AssocClass=Msvm_SystemDevice ResultClass=Msvm_KvpExchangeComponent"
        $vmkvp.GuestIntrinsicExchangeItems | Import-CimXml
    }

    By the way, I found the following patches and gave them a try, but the problem still remains (And the generation 2 Ubuntu 14.04 with Linux kernel 3.13 also has this problem).

     - Patch "Drivers: hv: util: Fix a bug in the KVP code" has been added to the 3.14-stable tree

     - Drivers: hv: vmbus: Fix a bug in the channel callback dispatch code

    P.S. Generation 2 CentOS 7 VM (Linux kernel 3.10.0-123) also has this problem, but Windows VM is ok

    Thanks,

    Paul

    Monday, August 25, 2014 12:22 PM

All replies

  • It seems that (generation 1) Ubuntu 14.04 VM with Linux kernel 3.17 doesn't encounter this problem after querying its KVP/IP information on Hyper-V host several times.

    Does anyone know what changes in Linux kernel 3.17 fix this issue?

    Thanks,

    Paul

    Tuesday, August 26, 2014 8:02 AM
  • I'm not able to reproduce the same result. 

    I have a CentOS 6.5 fresh install - all defaults - desktop - static RAM, static MAC

    Linux kernel is 2.6.32

    Linux Integration Service (LIS) is 3.1

    I ran your script, and it ran 10k times with no issues:

    Can you run this on the host and post the results:

    (Get-VM -Name v-centos65-01).vmIntegrationService
    
    (Get-VM -Name v-centos65-01).vmIntegrationService | FL | Out-File .\v-Centos65-01a.txt
    notepad .\v-Centos65-01a.txt

    substitute v-centos65-01 with your VM name of course..


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)

    Tuesday, August 26, 2014 10:14 AM
  • I upgraded the LIS on the CentOS 65 VM to v3.5

     You can see the difference in services available before and after:

    I'm running the same script now, at about 2k iterations now, with no signs of a problem. Showing more info from KVP than with LIS 3.1..


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)



    • Edited by Sam Boutros Tuesday, August 26, 2014 12:41 PM
    Tuesday, August 26, 2014 10:47 AM
  • Well, it's showing same result now - after about 2,500 iterations:

    The odd thing is I'm still seeing LIS 3.1 on the VM. I expect to see LIS 3.5 after a successful upgrade:

    However, this command shows LIS 3.5 on the VM:

    [root@v-CentOS65-01 ~]# rpm -qa | grep hyper-v
    kmod-microsoft-hyper-v-rhel63.3.5-20131212.x86_64
    microsoft-hyper-v-rhel63.3.5-20131212.x86_64
    

    Same exact versions for all 5 mods:

    Same exact thing after upgrade to LIS 3.5!!??
    modinfo hv_vmbus => 3.1
    modinfo hv_netvsc => 3.1
    modinfo hv_storvsc => 3.1
    modinfo hv_blkvsc => not found
    modinfo hv_utils => 3.1


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)


    • Edited by Sam Boutros Tuesday, August 26, 2014 11:48 AM
    Tuesday, August 26, 2014 11:04 AM
  • Well, according to Michael Kelley [MSFT] in this post, we should not upgrade LIS 3.1 to 3.5..

    "The LIS 3.5 package downloadable from Microsoft is intended ONLY for use with the older Linux distros as listed in the documentation grid.  It should not be applied to Linux distros that have the LIS built-in."

    That's CentOS 6.3 and older, not 6.5..

    The obvious issue is LIS 3.1 that comes with the CentOS 6.5 distro does not provide any KVP as shown above..

    BTW, this forum may be a best fit for this question..

    The documentation says:

    "13. In Windows Server 2012 R2, the key/value pair (KVP) infrastructure might not function correctly without a Linux software update. Contact your distribution vendor to obtain the software update in case you see problems with this feature."

    oh well, back to CentOS..


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)



    • Edited by Sam Boutros Tuesday, August 26, 2014 11:40 AM
    Tuesday, August 26, 2014 11:27 AM
  • Hi Sam,

    Thanks for your replies and time, and it seems that you also reproduced this issue in your environment.

    By the way, according to your first results (only shows 4 keys from the 1st time) and  PrimaryStatusDescript" of  "Key Value Pair Exchange" shows "No Contact" (with built-in LIS 3.1), I believe that you didn't manually install the KVP daemon on the CentOS VM (by default, KVP daemon is not installed and you need to "yum install hypervkvpd" for it).

    Actually, if your CentOS VM (has LIS) installed with KVP daemon running well, my test script will show more than 4 keys (include NetworkAddressIPv4 or NetworkAddressIPv6 keys).
    However, while the KVP daemon becomes problematic, it will only show a few keys (ex. 4~6 keys) and at this moment, Hyper-V Manager also can't show IP address of it anymore and you may need to reboot the CentOS VM to recover it.

    For the odd things you mentioned that you still saw LIS 3.1 after installing LIS 3.5, it is because LIS doesn't update the version definition in the source codes after version 3.1 and the version is just for the references.

    Thanks for your help, and I will ask for the help of this issue in the Linux Integration Services for Microsoft Hyper-V forum.


    Tuesday, August 26, 2014 11:55 AM
  • You're welcome. 

    Frankly Microsoft needs to step up and provide upgraded LIS (Linux Integration Services) drivers for CentOS/RedHat versions 6.5 and above.

    I've seen numerous questions and posts about issues with CentOS/RedHat 6.5 and above on Hyper-V 2012 R2, all stemming from outdated integration service. These issues include ability to use dynamic memory, ability to use dynamic MAC address, ability to backup VM without having to down it first or use an in-guest agent.

    Without these basic features, Linux would not be a viable guest OS option in enterprise environment. Will enterprise users abandon Linux as a guest OS option, or will they abandon Hyper-V as a hypervisor option? Some may consider this a bad bet


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)

    Tuesday, August 26, 2014 12:52 PM
  • I'll change the case type to Discussion as it is a Hyper-V for Linux related topic. Thank you for your understanding. 

    If you have any feedback on our support, please send to tnfsl@microsoft.com.

    Thursday, September 11, 2014 10:24 AM
    Moderator