locked
Linux VM shows continous CPU load RRS feed

  • Question

  • Hello,

    I have a cluster of Windows 2012 R2 - Hyper-V enabled.

    I have installed a Centos 6.5 64bit VM on it.

    Now, the problem is whenever I enable dynamic memory for that VM the CPU load is showing 1.0 all the time, even when the VM is in idle state. As soon as I disable dynamic memory for the VM and check again then the load is normal around 0.10

    Please suggest a way so that the VM shows normal CPU load whenever dynamic memory is enabled.

    Please visit this link containing the screenshot for the dynamic memory enabled scenario.

    Thanks.

    -

    Hussain

    • Moved by BrianEhMVP Friday, February 7, 2014 4:04 PM
    Friday, February 7, 2014 11:39 AM

All replies

  • Please add the following udev rule:

    Create a file /etc/udev/rules.d/100-balloon.rules . You may use any other desired name for the file. Add the following content to the file:

                SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"

    Let me know if this solves your problem.

    Thanks,

    Abhishek

    Saturday, February 8, 2014 12:08 AM
  • Hello Abhishek,

    Many thanks for your reply.

    I created the file and added the content and finally rebooted the VM.

    Still the CPU load remains at 1.00

    Please suggest.

    Saturday, February 8, 2014 5:54 AM
  • What are the values for startup memory, minimum memory and maximum memory? Let me know. Our team is out for the weekend but we can get back to you on this by next week if I am unable to solve over the forum.

    Thanks,

    Abhishek

    Saturday, February 8, 2014 5:57 AM
  • Following are the RAM parameters of the VM.

    Startup RAM: 1024MB
    Minimum RAM: 1024MB
    Maximum RAM: 4096MB
    Memory Buffer: 20%

    Please let me know if you want some additional details.

    Thanks.

    Saturday, February 8, 2014 6:08 AM
  • Could you try setting the startup RAM to the maximum RAM. It could be that Hot-Add is having some issues and that way at least you can get ballooning to work. Let me know the results.

    Thanks,
    Abhishek


    Saturday, February 8, 2014 6:22 AM
  • Although I didn't understand as to how could setting startup RAM to maximum RAM help ballooning, I set it anyways as shown below.

    Startup RAM: 4096MB
    Minimum RAM: 1024MB
    Maximum RAM: 4096MB
    Memory Buffer: 20%

    But the VM still shows constant load of "1.00"

    Saturday, February 8, 2014 6:39 AM
  • Aargh! This must be frustrating. I guess I will have it triaged next week when our team returns. Please hold until then. One last thing - does setting the memory buffer to zero help?

    Setting Max RAM to Startup RAM ensures that all the memory is Hot-Added at boot time and therefore only ballooning is in operation for the rest of the time. I shall have this looked at next week. Sorry for the trouble.

    Thanks,

    Abhishek

    Saturday, February 8, 2014 6:44 AM
  • No not a problem at all. It feels good to troubleshoot such unique problems. ;)

    Now, I tried setting the RAM Buffer to 0%, but error pops up with "Out Of Bound" message and asks to set between 5% to 2000%.

    Thus, I set it to 5% and still the same issue.

    Although I can manually set it to 0% from the config file by stopping the service but I haven't tried this option.

    Also, no issues from my side to wait until next week. Perfectly OK. :)

    Thanks for your prompt responses.


    • Edited by Ethan52 Saturday, February 8, 2014 7:18 AM Typo
    Saturday, February 8, 2014 7:17 AM
  • ~~

     . . .  error pops up with "Out Of Bound" message and asks to set between 5% to 2000%.

    Thus, I set it to 5% and still the same issue.


    ~~

     To Ethan52: Please, try this:

    {{

    rule by Nikolay Pushkarev :

    RHEL 6.5 / CentOS 6.5 ; RHEL 6.4 / CentOS 6.4 include support for Hyper-V drivers

    ==

    Following rule works slightly faster for me (assuming that memory0 bank always in online state):

    SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory[1-9]*", RUN+="/bin/cp /sys$devpath/../memory0/state /sys$devpath/state"

    And one more thing, udev solution doesn't work in 32-bit kernel architecture, only for 64-bit. Is this by design or yet another bug?

    ==

    P.S.

    And see topic

    Dynamic Memory on Linux VM 

    }}}

     Rule work better what default?


    Tuesday, February 11, 2014 12:35 PM
  • ~

    ~

    . . .  I guess I will have it triaged next week when our team returns. Please hold until then.

    ~

    ~

    ==

    Leverage the newly exported functionality to bring memory online
    without involving user level code.

    Signed-off-by: K. Y. Srinivasan

    ---
     drivers/hv/hv_balloon.c |   20 +++-----------------
     1 files changed, 3 insertions(+), 17 deletions(-)

    diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
    index 2d094cf..c2eec17 100644
    --- a/drivers/hv/hv_balloon.c
    +++ b/drivers/hv/hv_balloon.c

     . . .

    @@ -606,12 +598,10 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size,
       }
     
       /*
    -   * Wait for the memory block to be onlined.
    -   * Since the hot add has succeeded, it is ok to
    -   * proceed even if the pages in the hot added region
    -   * have not been "onlined" within the allowed time.
    +   * Before proceeding to hot add the next segment,
    +   * online the segment that has been hot added.
        */
    -  wait_for_completion_timeout(&dm_device.ol_waitevent, 5*HZ);
    +  online_memory_block(start_pfn);
     
      }

     . . .

    ==

    +

    ==

    > On 07/24/2013 02:29 PM, K. Y. Srinivasan wrote:
    > >    /*
    > > -   * Wait for the memory block to be onlined.
    > > -   * Since the hot add has succeeded, it is ok to
    > > -   * proceed even if the pages in the hot added region
    > > -   * have not been "onlined" within the allowed time.
    > > +   * Before proceeding to hot add the next segment,
    > > +   * online the segment that has been hot added.
    > >     */
    > > -  wait_for_completion_timeout(&dm_device.ol_waitevent,
    > 5*HZ);
    > > +  online_memory_block(start_pfn);
    >
    > Ahhhhh....  You've got a timeout in the code in order to tell the
    > hypervisor that you were successfully able to add the memory?  The
    > userspace addition code probably wasn't running within this timeout
    > period.  right?

    As I have always said, the onlining would not occur within a specified amount
    of time (under some conditions). The timeout here is to ensure that we are able
    to online the memory before attempting to hot-add more memory. With the ability
    to online memory from within the kernel, we don't need this timeout and the code is
    much more predictable.

    Regards,

    K. Y

    ==

    ~

    ~

    ----- Original Message -----
    From: "Victor Miasnikov"
    To: "Dan Carpenter"; "K. Y. Srinivasan" ; <linux-kernel@vger.kernel.org>
    Cc: "Greg KH" ; <devel@linuxdriverproject.org>; <olaf (at) aepfle.de>; ""Andy Whitcroft"" <apw (at) canonical.com>;
    <jasowang (at) redhat.com>
    Sent: Thursday, January 09, 2014 2:18 PM
    Subject: RE: [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context" Re: [PATCH 1/1] Drivers: hv:
    Implement the file copy service


    Hi!

        . . .

    But I ( and other Hyper-V sysadmin)  see non-Ok ( in "political correct" terminalogy) results with "hv: balloon: Online
    the hot-added memory" in "user space"


    Best regards, Victor Miasnikov
    Blog:  http://vvm.blog.tut.by/

    P.S.

    ==
     [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context"
    ==

     What news?  Roadmap?


    Tuesday, February 11, 2014 12:50 PM
  • Hi folks, We are still triaging this issue but wanted to ask you something. In our tests it seems if we disable SELinux then you may not see the high CPU utilization. Could you guys test it for us on your side and confirm? Also please include either our udev rule or Olaf's rule. They should both work.

    Furthermore, Hot-Add will not work on 32 bit builds because MEMORY_HOTPLUG feature is turned off on 32 bit kernels. So Dynamic Memory will only work on 64-bit systems.

    Unfortunately, we need to continue using the udev rule because upstream has decided not to use the alternate patch. We are pursuing this with upstream though and will let you know when the situation resolves.

    Please let me know about the SELinux behavior.

    Thanks,

    Abhishek

    Wednesday, February 12, 2014 12:11 AM
  • Hello,

    I disabled the SELinux and still the same behavior.

    I also have tried the same on Centos 6.4 final and the same behavior.

    Thanks.

    Wednesday, February 12, 2014 5:09 AM
  • Hi Ethan, We did some more investigation it seems that this may be an artifact of our ballooning subsystem. Hyper-V requires a report of the memory pressure inside the guest and this report is generated once per second. It is likely that you are seeing the 1.00 because of this report. Do note that 1.00 does not mean 100% of CPU load. It merely implies 1% load of the CPU. Let me know if you have further questions.

    Thanks,

    Abhishek

    Thursday, February 13, 2014 1:47 AM
  • Hello Abhishek,

    I think load of 1.00 means a particular thread is fully utilized. And if the VM has just one vCPU then it is matter of concern.

    Now, as I can understand what you are trying to explain but that would be really difficult to explain it to customers who are novice at actual and have their VM running on Hyper-V.

    I would propose to have a Microsoft webpage mentioning this problem so that it would be good for us to let our customers know about it.

    Thanks for your support.

    Thursday, February 13, 2014 5:33 AM
  • To Ethan52:  Sorry I don't undestand / not shue:

    -- are You try this? Or not?

     

    {{{{{{{{{{{{{{{{

    To Ethan52: Please, try this:

    {{

    rule by Nikolay Pushkarev :

    RHEL 6.5 / CentOS 6.5 ; RHEL 6.4 / CentOS 6.4 include support for Hyper-V drivers

    ==

    Following rule works slightly faster for me (assuming that memory0 bank always in online state):

    SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory[1-9]*", RUN+="/bin/cp /sys$devpath/../memory0/state /sys$devpath/state"

    }}}}}}}}}}}}}}}}

    Thursday, February 13, 2014 9:41 AM
  • Hello Victor,

    I tried this one and still the load remains.

    Thursday, February 13, 2014 10:33 AM
  • ~

    ~

    I tried this one and still the load remains.

    ~

      :-(

    ~

     May be this variant can work better:

    ~

    Dynamic Memory on Linux VM

    {{{{

    Copy-paste from russian part of this forum

    ( translate to English by V.V.Miasnikov )

    Nikolay Pushkarev :

    {{

    As a result, Nikolay Pushkarev write ​​a program putarg.c, that is even faster than/bin/cp method :

    ~

    #include <stdio.h>
    #include <fcntl.h>
    #include <sys/ioctl.h>
    #include <string.h>
    int main(int argc, char** argv) {
      int i, fd;
      if (argc < 2) return 0;
      if ((fd = open(argv[1], O_RDWR)) < 0) return 1;
      for (i = 2; i < argc; i++) {
        if (write(fd, argv[i], strlen(argv[i])) < 0) {
          close(fd);
          return i;
        }
      }
      close(fd);
      return 0;
    }

    The first argument - the name of the output file ,
    and argument number 2 (and all subsequent (if exist) ) - are text that are written in output file.

    Compile source code to executable file by run command: gcc -o putarg -s putarg.c



    The resulting binary put out an accessible location , such as /usr/bin, or wherever you want.


    Now udev rule using "putarg" can be written as :

    SUBSYSTEM=="memory", ACTION=="add", RUN+="/usr/bin/putarg /sys$devpath/state online"

    This complex solutions ( compiled file and udev-rule ) works exceptionally fast.

    }}

    }}}}

    ~


    Thursday, February 13, 2014 12:12 PM
  • Hi Ethan, Let me do some further investigation and get back to you.

    Thanks,

    Abhishek

    Thursday, February 13, 2014 7:52 PM
  • Hi Ethan, This turned out to be a bug on our side. As I mentioned earlier, our ballooning mechanism does wake up every second to send a memory consumption report to the hypervisor. While it is not running, this thread sleeps in uninterruptible mode currently. Unfortunately, uninterruptible sleep is accounted in a way that leads to a higher near term load metric. Do note that this does not imply that actual CPU load is high. This is just a side effect of the uninterruptible sleep and processor load is not affected because the ballooning process itself has a short execution time.

    We will work on fixing this and I shall also publish a KB article for this issue. Thank you for bringing this to our notice.

    Regards,

    Abhishek



    Thursday, February 13, 2014 11:25 PM
  • Hello Abhishek,

    Many thanks for your time. Appreciate it.

    Will wait for your update related to KB and fix for the same.

    Friday, February 14, 2014 6:19 AM
  • Hi again,

    If you are able to then you can apply the following patch to your system:

    https://lkml.org/lkml/2014/2/13/562

    I have also filed a bug with Red Hat but it will take some time to percolate to RHEL 6.5/6.4.

    Thanks,

    Abhishek

    Friday, February 14, 2014 5:34 PM
  • ~

    ~

    . . . patch  . . . :

    https://lkml.org/lkml/2014/2/13/562

    ~

      Yes:

    ~

    ==

    ----- Original Message -----
    From: "K. Y. Srinivasan"
    To:   linux-kernel
    Sent: Thursday, July 25, 2013  14:29:59 -0700

    Subject: [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in
    context"

    The non-interruptible sleep of the memory pressure posting thread
    results in higher reported load average. Make this sleep interruptible.

    Signed-off-by: K. Y. Srinivasan

    --- a/drivers/hv/hv_balloon.c
    +++ b/drivers/hv/hv_balloon.c
    @@ -1171,7 +1171,8 @@ static int dm_thread_func(void *dm_dev)
      int t;
     
      while (!kthread_should_stop()) {
    -  t = wait_for_completion_timeout(&dm_device.config_event, 1*HZ);
    +  t = wait_for_completion_interruptible_timeout(
    +      &dm_device.config_event, 1*HZ);
       /*

    ==

     Thanks!

    P.S. And see in topic :

    Dynamic Memory on Linux VM

    ==

    ----- Original Message -----

    From: "K. Y. Srinivasan"
    Sent: Thursday, July 25, 2013 12:29 AM

    Subject: [PATCH 2/2] Drivers: hv: balloon: Online the hot-added memory "in context"

      . . .

    ==

    What new? Roadmap?

    Are You shue what Linux kernel team _discard_ this
    patch? May be _forget_ about this patch?



    Wednesday, February 26, 2014 6:58 AM