none
Dynamic Memory on Linux VM

    Question

  • Hello!

    Hyper-V 3.0 is great! After it will be released, I think it will become the most popular hypervisor. But it remains a major drawback.

    Nowhere announced support for dynamic memory for Linux VM on Hyper-V.

    Planned at least in some perspective to implement this functionality?

    Now we have to use two different hypervisors, as Hyper-V does not meet all the requirements of our customers.

    Mark Tepterev

    Oversun

    • Moved by BrianEhMVP Wednesday, March 14, 2012 6:44 PM (From:Hyper-V)
    Wednesday, March 14, 2012 6:00 PM

Answers

  • We're definitely aware of the demand for Dynamic Memory for Linux guests. :)


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, March 16, 2012 5:08 AM
    Owner

All replies

  • To my knowledge from outside, the developers have not been pushing changes to upstream that would  add this feature (yet), but lots of stability and bugfixes. With Linux 3.3 most, with 3.4 all Hyper-V drivers will be part of the kernel.org upstream. And they seem to be in a very good shape now, maturing even further.

    Most of the other nice features like available to Window guests like vCPU limitations (beyond 4 vCPUs), SCSI hot add/remove are now also part of the Linux drivers, but not dynamic memory.

    Thursday, March 15, 2012 1:38 PM
  • We're definitely aware of the demand for Dynamic Memory for Linux guests. :)


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, March 16, 2012 5:08 AM
    Owner
  • Support for dynamic memory for Linux VM is coming: http://lkml.indiana.edu/hypermail/linux/kernel/1210.1/01654.html

    "Add a basic balloon driver to take advantage of the dynamic memory management functionality supported on Windows hosts. Windows requires the guests to support both memory hot add as well as ballooning. In this patch we are adding the basic balloon driver. Memory hot add will be added in asubsequent patch and this requires some changes yet to be made on the host side."


    Uellington Santos MCSE Private Cloud, MCSA Windows Server 2012


    Thursday, November 22, 2012 5:30 PM
  • ~

    ~

    ==

    Support for dynamic memory for Linux VM is coming: http://lkml.indiana.edu/hypermail/linux/kernel/1210.1/01654.html

    "Add a basic balloon driver to take advantage of the dynamic memory management functionality supported on Windows hosts. Windows requires the guests to support both memory hot add as well as ballooning. In this patch we are adding the basic balloon driver. Memory hot add will be added in asubsequent patch and this requires some changes yet to be made on the host side."

    ==

    ~

    Dynamic memory hot add work

    Details:

    http://vvm.blog.tut.by/2013/09/11/linux-hyper-v-dynamic-memory-hot-add/

    ==

    ===
    The set of six new patches enhances their memory balloon driver to add in support for memory hot-add. System memory to Linux guests is dynamically managed at run-time and now implemented for the Windows Dynamic Memory protocol, which is a combination of ballooning and hot-add for the dynamic balanacing of available memory across competing virtual machines.
    ===

    Dynamic memory hot add work ( . . . )

    =

    SLES_v11.3_x64 Dynamic Memory 598Mb Uptime 00-55-15

    =

     

    ==

    ~


    Wednesday, September 11, 2013 3:53 PM
  • I try add comment on page

    http://www.aidanfinn.com/?p=15458

    but:
    ==
    Sorry, comments are closed for this item.
    ==

    See text of this Comment

    http://www.aidanfinn.com/?p=15458&cpage=1#comment-75033

    ====

      [ VVM: i skip info waht is dubl of info from my prev. post ]

    Dynamic memory hot add work

    Details:
    ==
    Image with SLES 11.3
    Startup Memory 512MB
    Memory 512MB
    Assigned Memory 598MB

    ==

    ====

    [Reply] ( http://www.aidanfinn.com/?p=15458&replytocom=74985#respond )

    Artenmonk said: 2013.09.10 14:22

    Aidan, can show system load ‘w command’? When I enable dynamic memory, I see load 1. When the system doesn not work, load should be 0. I think here is a bug.

    [Reply] ( http://www.aidanfinn.com/?p=15458&replytocom=75004#respond )

    John Spade said: 2013.09.11 01:50

    I believe what Artenmonk meant was something I experienced on WS 2012 (R1) and Centos 6.4. I noticed that dynamic memory was functioning and it seemed to adjust reasonably well. However, when you looked at an idle system, the load via ‘w’ or ‘uptime’ was always 1.00 rather than 0.00, but only when dynamic memory was enabled. Inside the WS2012 host, there did not appear to be any CPU load. I also experienced a few crashes of that VM. Disabling dynamic memory made it function normally and stable. I hope to re-test it with R2 soon.


    More details:

    http://social.technet.microsoft.com/Forums/windowsserver/en-US/c8a06a23-2ad7-4a47-ad24-a7eec8438c06/2012-r2-dynamic-memory-for-linux-guests-system-load-reporting-error-100-instead-of-000-for-idle

    ==

    2012 R2 Dynamic Memory for Linux guests system load reporting error 1.00 instead of 0.00 for idle

    For non Unix/Linux folk, an equivalent would be looking at taskman and seeing 25% CPU all the time on a quad core system, yet no processes at 25%.

    Dynamic Memory started working in R1 and CentOS 6.4 but there was a 'bug' where an idle system would report 1.00 for system load instead of 0.00 when it was turned on. With R2 dynamic memory is now working for Ubuntu 13.04 and Centos 6.4 but both have this same issue where an idle system reports 1.00 as baseline.

    It appears it may be a 'cosmetic' issue as I don't see a process in guest or host spinning. What concerns me (besides SNMP showing a erroneous system load) is that processes that are waiting for 'idle' to do maintenance or accept more connections may behave incorrectly.

    More background:

    This appears to happen after memory trimming starts.

    It does not appear to be a true load, no CPU core is spinning on any process I can see.

    You can see the load with 'w' or 'top' or 'uptime'. Top shows 1.00 but no process with any CPU load over 0%.

    I have not tried it with a single 'cpu' assigned.

    When dynamic memory is unchecked and guest started the idle CPU remains 0.00

    Host CPU type does not appear to matter, tested on Q9550s, Xeon 5300,5400, 5500 series.

    For CentOS happened in 2012 R1 and R2

    Latest kernel updates tried for both guest operating systems

    -John Spade (Connect)

    ==

    Thursday, September 12, 2013 8:14 AM
  • ~

    ~

    ==

    http://www.aidanfinn.com/?p=15458

    ====

    [Reply] ( http://www.aidanfinn.com/?p=15458&replytocom=74985#respond )

    Artenmonk said: 2013.09.10 14:22

    Aidan, can show system load ‘w command’? When I enable dynamic memory, I see load 1. When the system doesn not work, load should be 0. I think here is a bug.

    [Reply] ( http://www.aidanfinn.com/?p=15458&replytocom=75004#respond )

    John Spade said: 2013.09.11 01:50

    I believe what Artenmonk meant was something I experienced on WS 2012 (R1) and Centos 6.4. I noticed that dynamic memory was functioning and it seemed to adjust reasonably well. However, when you looked at an idle system, the load via ‘w’ or ‘uptime’ was always 1.00 rather than 0.00, but only when dynamic memory was enabled. Inside the WS2012 host, there did not appear to be any CPU load. I also experienced a few crashes of that VM. Disabling dynamic memory made it function normally and stable. I hope to re-test it with R2 soon.


    More details:

    http://social.technet.microsoft.com/Forums/windowsserver/en-US/c8a06a23-2ad7-4a47-ad24-a7eec8438c06/2012-r2-dynamic-memory-for-linux-guests-system-load-reporting-error-100-instead-of-000-for-idle

    ==

    2012 R2 Dynamic Memory for Linux guests system load reporting error 1.00 instead of 0.00 for idle

    For non Unix/Linux folk, an equivalent would be looking at taskman and seeing 25% CPU all the time on a quad core system, yet no processes at 25%.

    Dynamic Memory started working in R1 and CentOS 6.4 but there was a 'bug' where an idle system would report 1.00 for system load instead of 0.00 when it was turned on. With R2 dynamic memory is now working for Ubuntu 13.04 and Centos 6.4 but both have this same issue where an idle system reports 1.00 as baseline.

    It appears it may be a 'cosmetic' issue as I don't see a process in guest or host spinning. What concerns me (besides SNMP showing a erroneous system load) is that processes that are waiting for 'idle' to do maintenance or accept more connections may behave incorrectly.

    More background:

    This appears to happen after memory trimming starts.

    It does not appear to be a true load, no CPU core is spinning on any process I can see.

    You can see the load with 'w' or 'top' or 'uptime'. Top shows 1.00 but no process with any CPU load over 0%.

    I have not tried it with a single 'cpu' assigned.

    When dynamic memory is unchecked and guest started the idle CPU remains 0.00

     . . .

    -John Spade (Connect)

    ==

    ~

    https://bugzilla.redhat.com/show_bug.cgi?id=979257

    Bug 979257 - [Hyper-V][RHEL6.5][RFE]in-kernel online support for memory hot-add

    ==

    K. Y. Srinivasan 2013-09-12 15:27:34 EDT
    Created attachment 796988 [details]
    Patch to support in-kernel onlining The current machinery for hot-adding memory requires having user level to bring the memory segments online. Export the necessary functionality to bring the memory segment online without involving user space code.
    Comment 22 K. Y. Srinivasan 2013-09-12 15:28:46 EDT
    Created attachment 796989 [details]
    patch to online memory in the balloon driver Leverage the newly exported functionality to bring memory online without involving user level code.

    ==

    ~





    Tuesday, September 24, 2013 6:13 AM
  • {{{ VVM  public 2013-12-13:

     I asked this Q and by e-mail msg:

    -----Original Message-----
    From: Victor Miasnikov
    Sent: Tuesday, December 10, 2013 11:37 PM     (  10:37 AM  TZ Nairoby)
    To: Abhishek Gupta (LIS)

    }}}

    SUSE virtual machines feature distribution map
    http://technet.microsoft.com/en-us/library/dn531027.aspx

     and

    CentOS, Oracle Linux, Red Hat Enterprise Linux virtual machines feature distribution map
    http://technet.microsoft.com/en-us/library/dn531026.aspx

    ==
     . . .
       If you are using Dynamic Memory on Windows Server 2008 R2 operating systems, specify Startup memory, Minimum
    memory, and Maximum memory parameters in multiples of 128 megabytes (MB). Failure to do so can lead to Hot-Add failures,
    and you may not see any memory increase in a guest operating system.
     . . .
    ==

    Q:

     only on Windows Server 2008 R2 ?

     Or and on Windows Server 2012 R2 ( 2012 R0 ) operating systems to?

     I.e. may be

    ==

    on Windows Server 2008 R2  operating systems,
    specify Minimum memory, and Maximum memory parameters in multiples of 128 megabytes (MB).

    on Windows Server 2012 R2 ( 2012 R0 ) operating systems,
    specify Startup memory, Minimum memory, and Maximum memory parameters in multiples of 128 megabytes (MB).

    ==

    ~

    ~

    (  VVM: small remark

     Windows Server 2008 R2  not have field "Startup memory"  only  "Minimum memory" and "Maximum memory"

    )

    {{  2013-12-12: or as say Michael Kelley:

    WS 2008 R2 does not show a "Startup memory" field in the UI

    }}

    {{  VVM say on 2013-12-12:

     "Startup memory" , on practic, is ammount of RAM used VM before memory ballon driver is started) 

    }}

    {{{ VVM say on 2013-12-12 ( and move this text fragment 2013-12-13 to this post ) :

     In  "GUI-centric" :-) look on the world:

    in Win 2008 R2

    Startup memory ( on practic, ammount of RAM used VM before memory ballon driver is started) 

    always set equal to Minimum Memory

    }}}

    ~

    ~







    • Edited by Victor Miasnikov Wednesday, February 26, 2014 7:01 AM Small edit -- Add "C" to praCtic
    Wednesday, December 11, 2013 7:34 AM
  • The "multiple of 128 Mbytes" requirement really does apply only to WS 2008 R2, and not to WS 2012 or WS 2012 R2.  You are correct that WS 2008 R2 does not show a "Startup memory" field in the UI, but it exists in a hidden fashion such that Startup memory is always set equal to Minimum Memory unless you make changes via the WMI interface.  See

    http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/09/15/scripting-dynamic-memory-part-5-changing-minimum-memory.aspx


    Michael Kelley, Lead Program Manager, Open Source Technology Center


    Wednesday, December 11, 2013 11:36 PM
    Moderator
  • ~~

    ~~

     I asked this Q and by e-mail msg:

    -----Original Message-----
    From: Victor Miasnikov
    Sent: Tuesday, December 10, 2013 11:37 PM     (  10:37 AM  TZ Nairoby)
    To: Abhishek Gupta (LIS)

     and receive msg with answer:

    ----- Original Message -----
    From: "Abhishek Gupta (LIS)"
    To: "Victor Miasnikov"
    Sent: Thursday, December 12, 2013  23:40 TZ=Nairoby  ( Or Date: Thu, 12 Dec 2013 20:40:17 +0000 )

    Hi Victor,

    {{

    ~~

    Michael Kelley say:

    ~~

     The "multiple of 128 Mbytes" requirement really does apply only to WS 2008 R2,
    and not to WS 2012 or WS 2012 R2. 


    ~~

    That comment is for Windows 2008 R2 only. Only for Windows 2008 R2 you have to use 128MB aligned values.

    }}

    {{

    ~~

    Michael Kelley say:

    ~~

    You ( i.e. V.V.Miasnikov ) are correct
    that WS 2008 R2 does not show a "Startup memory" field in the UI,
    but it exists in a hidden fashion such that Startup memory is always set equal to Minimum Memory
    unless you make changes via the WMI interface. 
     See

    ~~

    ~

    The parameters such as Startup Memory and Maximum Memory are available for Server 2008 R2: http://technet.microsoft.com/en-us/library/ff817651(v=WS.10).aspx

    ~

    The Minimum Memory parameter is only available via a powershell script: http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/09/15/scripting-dynamic-memory-part-5-changing-minimum-memory.aspx
    ~

    Thanks,
    Abhishek

    --

    ~

     Thanks both Michael Kelley and Abhishek Gupta for info

    and

    Thanks Benjamin Armstrong for the script 

    Script look like usefull even today for RHEL 6.4 on Win 2008 R2

    ~




    Thursday, December 12, 2013 7:03 AM
  •  Ok, we are return again to problem:

    https://bugzilla.redhat.com/show_bug.cgi?id=979257

    Bug 979257 - [Hyper-V][RHEL6.5][RFE]in-kernel online support for memory hot-add

    ==


    ~

    Michael Kelley say :

    RHEL 6.5 / CentOS 6.5 ; RHEL 6.4 / CentOS 6.4 include support for Hyper-V drivers

    The VM crash with 6.5 should be fixed by adding the udev rule.  The udev rule looks like: 

    SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory*[0-9]", TEST=="/sys$devpath/state", RUN+="/bin/sh -c 'echo online > /sys$devpath/state'"

    ~

    ~

    2013-12-13:

    {{

    Nikolay Pushkarev :

    Following rule works slightly faster for me (assuming that memory0 bank always in online state):

    SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory[1-9]*", RUN+="/bin/cp /sys$devpath/../memory0/state /sys$devpath/state"

    }}


        Copy-paste from russian part of this forum

    ( translate to English by V.V.Miasnikov )

     Nikolay Pushkarev write about another non-full good udev-script:

    {{

    Strange, that the native udev-script

    SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"

     triggered somehow through time ( VVM: very often not work as need ) ,
    it would be much easier ( VVM: if this udev-script work as need, then problem is solved :-(  )  .

    }}

    P.S.

      About pretty good solution see later



    Friday, December 13, 2013 10:22 AM
  • Copy-paste from russian part of this forum

    ( translate to English by V.V.Miasnikov )

    Nikolay Pushkarev :

    {{

    As a result, Nikolay Pushkarev write ​​a program putarg.c, that is even faster than /bin/cp method :

    ~

    #include <stdio.h>
    #include <fcntl.h>
    #include <sys/ioctl.h>
    #include <string.h>
    int main(int argc, char** argv) {
      int i, fd;
      if (argc < 2) return 0;
      if ((fd = open(argv[1], O_RDWR)) < 0) return 1;
      for (i = 2; i < argc; i++) {
        if (write(fd, argv[i], strlen(argv[i])) < 0) {
          close(fd);
          return i;
        }
      }
      close(fd);
      return 0;
    }

    The first argument - the name of the output file ,
    and argument number 2 (and all subsequent (if exist) ) - are text that are written in output file.

    Compile source code to executable file by run command: gcc -o putarg -s putarg.c



    The resulting binary put out an accessible location , such as /usr/bin, or wherever you want.


    Now udev rule using "putarg" can be written as :

    SUBSYSTEM=="memory", ACTION=="add", RUN+="/usr/bin/putarg /sys$devpath/state online"

    This complex solutions ( compiled file and udev-rule ) works exceptionally fast.

    }}

    Friday, December 13, 2013 12:02 PM
  •   K. Y.  Srinivasan  sent on July 25, 2013  message with Subject

    [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in
    context"

    ~

    ~

    ==

    ----- Original Message -----

    From: "K. Y. Srinivasan"
    Sent: Thursday, July 25, 2013 12:29 AM

    Subject: [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context"

      . . .

    ==

    ~

    ~

    ( and see some discussion in   https://bugzilla.redhat.com/show_bug.cgi?id=979257 )

    ~

    ~

    What new? Roadmap?

    Are You shue what Linux kernel team _discard_ this
    patch? May be _forget_ about this patch?



    Wednesday, February 26, 2014 6:59 AM
  • Q from Brian Wong:

    ----- Original Message -----
    From: "Brian Wong"
    To: <linux-kernel@vger.kernel.org>
    Sent: Thursday, March 06, 2014 9:24 AM
    Subject: Re: Linux does not use more than the startup RAM under Hyper-V with dynamic memory enabled


    On 3/6/2014 1:20 AM, Brian Wong wrote:

       . . .

    > The kernel is built with the full set of Hyper-V drivers, including the
    > key "Microsoft Hyper-V Balloon Driver" as well as memory hot-add and
    > hot-remove functionality. This is happening with both the Gentoo-patched
    > 3.10.32 kernel and the vanilla 3.12.5 kernel. The host machine has a
    > total of 24 GB of memory.
    >
    > For now, I am working around the issue by starting the VM with the
    > startup memory set to the maximum and letting Hyper-V take the usused
    > memory back when it is not in use. The VM will then get the extra memory
    > when it needs it.
    >
    > Have I encountered a bug in the Hyper-V balloon driver?
    >

    Just a correction: the vanilla kernel version is 3.13.5, not 3.12.5.
    Sorry for any confusion.

    --
    Brian Wong
    http://www.fierydragonlord.com
    --

     

    ----- Original Message -----
    From: "Brian Wong"
    To: <linux-kernel@vger.kernel.org>
    Sent: Thursday, March 06, 2014 9:20 AM
    Subject: Linux does not use more than the startup RAM under Hyper-V with dynamic memory enabled

    I'm new to LKML, so please don't be too hard on me :)

    I'm running Gentoo Linux under Microsoft Client Hyper-V on Windows 8.1
    Pro, and I've noticed some odd behavior with respect to dynamic memory
    (aka memory ballooning). The system will never use more than the startup
    memory defined in the virtual machine's settings.

    ( VVM: typewriting error viRtual fixed by me, for best search in future )

    For example, if I set the startup memory to 512 MB, and enable dynamic
    memory with a minimum of 512 MB and a maximum of 8192 MB, the system
    will never allocate than 512 MB of physical memory, despite Hyper-V
    assigning more memory to the VM and the added memory being visible in
    the output of "free" and "htop". Attempting to use more memory causes
    the system to start paging to swap, rather than actually allocating the
    memory above the startup memory assigned to the VM.

    The kernel is built with the full set of Hyper-V drivers, including the
    key "Microsoft Hyper-V Balloon Driver" as well as memory hot-add and
    hot-remove functionality. This is happening with both the Gentoo-patched
    3.10.32 kernel and the vanilla 3.12.5 kernel. The host machine has a
    total of 24 GB of memory.

    (
      Brian Wong wrote On 3/6/2014 1:20 AM:
     Just a correction: the vanilla kernel version is 3.13.5, not 3.12.5. )
    )

    For now, I am working around the issue by starting the VM with the
    startup memory set to the maximum and letting Hyper-V take the usused
    memory back when it is not in use. The VM will then get the extra memory
    when it needs it.

    Have I encountered a bug in the Hyper-V balloon driver?

    --
    Brian Wong
    http://www.fierydragonlord.com
    --

    Thursday, March 06, 2014 2:51 PM
  • ~

    ~

    Moved  P.P.P.S.

    Q from Brian Wong:

    ----- Original Message -----
    From: "Brian Wong"
    To: <linux-kernel@vger.kernel.org>
    Sent: Thursday, March 06, 2014 9:24 AM
    Subject: Re: Linux does not use more than the startup RAM under Hyper-V with dynamic memory enabled


    On 3/6/2014 1:20 AM, Brian Wong wrote:

       . . .

    > The kernel is built with the full set of Hyper-V drivers, including the
    > key "Microsoft Hyper-V Balloon Driver" as well as memory hot-add and
    > hot-remove functionality. This is happening with both the Gentoo-patched
    > 3.10.32 kernel and the vanilla 3.12.5 kernel. The host machine has a
    > total of 24 GB of memory.
    >
    > For now, I am working around the issue by starting the VM with the
    > startup memory set to the maximum and letting Hyper-V take the usused
    > memory back when it is not in use. The VM will then get the extra memory
    > when it needs it.
    >
    > Have I encountered a bug in the Hyper-V balloon driver?
    >

    Just a correction: the vanilla kernel version is 3.13.5, not 3.12.5.
    Sorry for any confusion.

    --
    Brian Wong
    http://www.fierydragonlord.com
    --

    ----- Original Message -----
    From: "Brian Wong"
    To: <linux-kernel@vger.kernel.org>
    Sent: Thursday, March 06, 2014 9:20 AM
    Subject: Linux does not use more than the startup RAM under Hyper-V with dynamic memory enabled

    I'm new to LKML, so please don't be too hard on me :)

    I'm running Gentoo Linux under Microsoft Client Hyper-V on Windows 8.1
    Pro, and I've noticed some odd behavior with respect to dynamic memory
    (aka memory ballooning). The system will never use more than the startup
    memory defined in the virtual machine's settings.

    ( VVM: typewriting error viRtual fixed by me, for best search in future )

    For example, if I set the startup memory to 512 MB, and enable dynamic
    memory with a minimum of 512 MB and a maximum of 8192 MB, the system
    will never allocate than 512 MB of physical memory, despite Hyper-V
    assigning more memory to the VM and the added memory being visible in
    the output of "free" and "htop". Attempting to use more memory causes
    the system to start paging to swap, rather than actually allocating the
    memory above the startup memory assigned to the VM.

    The kernel is built with the full set of Hyper-V drivers, including the
    key "Microsoft Hyper-V Balloon Driver" as well as memory hot-add and
    hot-remove functionality. This is happening with both the Gentoo-patched
    3.10.32 kernel and the vanilla 3.12.5 kernel. The host machine has a
    total of 24 GB of memory.

    (
      Brian Wong wrote On 3/6/2014 1:20 AM:
     Just a correction: the vanilla kernel version is 3.13.5, not 3.12.5. )
    )

    For now, I am working around the issue by starting the VM with the
    startup memory set to the maximum and letting Hyper-V take the usused
    memory back when it is not in use. The VM will then get the extra memory
    when it needs it.

    Have I encountered a bug in the Hyper-V balloon driver?

    --
    Brian Wong
    http://www.fierydragonlord.com
    --



    ----- Original Message -----
    From: "Victor Miasnikov"
    To:  linux-kernel@vger.kernel.org; "Brian Wong"
    Cc: "Abhishek Gupta (LIS)" ( zzzzzzzzzzzzzzz (at) microsoft.com>; "KY Srinivasan" zzzzzzzzzzz (at) microsoft.com
    Sent: Thursday, March 06, 2014 1:07 PM
    Subject: Re: Linux does not use more than the startup RAM under Hyper-V with dynamic memory enabled RE: [PATCH 2/2]
    Drivers: hv: balloon: Online the hot-added memory "in context" Re: [PATCH 1/1] Drivers: hv:

    Hi!

     Short:

     Question to Linux kernel team:

    may be patch

    >>> [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context"

    can solve problems with dynamic memory hot add in Hyper-V VMs with Linux OS ?

     Full:

    BW>> .., if I set the startup memory to 512 MB, and enable dynamic
    BW>> memory with a minimum of 512 MB and a maximum of 8192 MB,
    BW>>  the system will never allocate than 512 MB of physical memory

    BW>>
    BW>> Have I encountered a bug in the Hyper-V balloon driver?
    BW>>

     Unfortunately,  It's long story . . . :-(

    a)

     I already ( on January 09, 2014 2:18 PM )  write about problems with "Online the hot-added memory"  in "user space" see
    P.P.S.

    b)

      See

    Bug 979257 -[Hyper-V][RHEL6.5][RFE]in-kernel online support for memory hot-add
    https://bugzilla.redhat.com/show_bug.cgi?id=979257

     (  Info from this topic may be interessant not only for RedHat users )


    b2)

     Detail about pathes related problem "Online the hot-added memory"  in "user space" :

    >>> [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context"
    >>>
    >>>
    >>> === 0001-Drivers-base-memory-Export-functionality-for-in-kern.patch
    >>>  . . .
    >>> +/*
    >>> + * Given the start pfn of a memory block; bring the memory
    >>> + * block online. This API would be useful for drivers that may
    >>> + * want to bring "online" the memory that has been hot-added.
    >>> + */
    >>> +
    >>> +int online_memory_block(unsigned long start_pfn) {  struct mem_section
    >>> +*cur_section;  struct memory_block *cur_memory_block;
    >>>
    >>>  . . .
    >>> ===
    >>>
    >>>
    >>> ==
    >>>  . . .
    >>> == 0002-Drivers-hv-balloon-Online-the-hot-added-memory-in-co.patch
    >>>   . . .
    >>>    /*
    >>> -   * Wait for the memory block to be onlined.
    >>> -   * Since the hot add has succeeded, it is ok to
    >>> -   * proceed even if the pages in the hot added region
    >>> -   * have not been "onlined" within the allowed time.
    >>> +   * Before proceeding to hot add the next segment,
    >>> +   * online the segment that has been hot added.
    >>>     */
    >>> -  wait_for_completion_timeout(&dm_device.ol_waitevent, 5*HZ);
    >>> +  online_memory_block(start_pfn);
    >>>
    >>>   }


    c)

      Before apply patches ( see in P.S. about native udev-script) we are need use one of this methods:

     [ VVM:   URL of this topic skipped ]

    c1)

    "/bin/cp method" by Nikolay Pushkarev :

    Following udev rule works slightly faster for me (assuming that memory0 bank always in online state):

    SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory[1-9]*",
    RUN+="/bin/cp /sys$devpath/../memory0/state /sys$devpath/state"}}

    ( VVM : of course all need be place in one line, 2 line in this msg. -- only for good visual formating reasons )


    c2)

    udev rule using "putarg" by Nikolay Pushkarev :


     Even "/bin/cp method" udev rule work time to time not as need :-(


    As a result, Nikolay Pushkarev write a program putarg.c, that is even faster than "/bin/cp method" :

    ==
    #include <stdio.h>
    #include <fcntl.h>
    #include <sys/ioctl.h>
    #include <string.h>
    int main(int argc, char** argv) {
      int i, fd;
      if (argc < 2) return 0;
      if ((fd = open(argv[1], O_RDWR)) < 0) return 1;
      for (i = 2; i < argc; i++) {
        if (write(fd, argv[i], strlen(argv[i])) < 0) {
          close(fd);
          return i;
        }
      }
      close(fd);
      return 0;
    }
    ==

     The first argument - the name of the output file ,
    and argument number 2 ( and all subsequent (  if exist ) ) - are text that are wiil be written in output file.


     Compile source code to executable file by run command:
    gcc -o putarg -s putarg.c

    The resulting binary need be placed an accessible location , such as /usr/bin, or wherever you want.


    Now udev rule using "putarg" can be written as :

    SUBSYSTEM=="memory", ACTION=="add", RUN+="/usr/bin/putarg /sys$devpath/state online"


    This complex solutions ( compiled file and udev-rule ) works exceptionally fast.

    Best regards, Victor Miasnikov
    Blog:  http://vvm.blog.tut.by/

    P.S.

    Nikolay Pushkarev about standart udev-script :

    {{

    Strange, that the native udev-script

    SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"


    triggered somehow through time ( VVM: very often not work as need )


    }}

    P.P.S.

    ----- Original Message -----
    From: "Victor Miasnikov"
    To: "Dan Carpenter"; "K. Y. Srinivasan" ; <linux-kernel@vger.kernel.org>
    Cc: "Greg KH" ; <devel@linuxdriverproject.org>; <olaf (at) aepfle.de>; ""Andy Whitcroft"" <zzzzzzzzzzzz (at)
    canonical.com>;
    <jasowang (at) redhat.com>
    Sent: Thursday, January 09, 2014 2:18 PM
    Subject: RE: [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context" Re: [PATCH 1/1] Drivers: hv:
    Implement the file copy service


    Hi!

    > Is there no way we could implement file copying in user space?


      For "file copy service"  "user space"  may be pretty good

    But I ( and other Hyper-V sysadmin)  see non-Ok ( in "political correct" terminalogy) results with "hv: balloon: Online
    the hot-added memory" in "user space"


    ==
     [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context"
    ==


     What news?  Roadmap?


    Best regards, Victor Miasnikov
    Blog:  http://vvm.blog.tut.by/

    ~

    ~

    ~

    ~

    ~

    ~

    ~

    ~

    ~

    ----- Original Message -----
    From: "KY Srinivasan"
    To: "Victor Miasnikov"; linux-kernel@vger.kernel.org; "Brian Wong"
    Cc: "Abhishek Gupta (LIS)"
    Sent: Thursday, March 06, 2014 1:23 PM
    Subject: RE: Linux does not use more than the startup RAM under Hyper-V with dynamic memory enabled RE: [PATCH 2/2]
    Drivers: hv: balloon: Online the hot-added memory "in context" Re: [PATCH 1/1] Drivers: hv:

    > -----Original Message-----
    > From: Victor Miasnikov
    > Sent: Thursday, March 6, 2014 3:38 PM
    > To: linux-kernel@vger.kernel.org; Brian Wong
    > Cc: Abhishek Gupta (LIS); KY Srinivasan
    > Subject: Re: Linux does not use more than the startup RAM under Hyper-V
    > with dynamic memory enabled RE: [PATCH 2/2] Drivers: hv: balloon: Online
    > the hot-added memory "in context" Re: [PATCH 1/1] Drivers: hv:
    >


    Victor,

    I will try to get my in-context onlining patches accepted upstream.

    K. Y



    Thursday, March 06, 2014 2:52 PM