locked
Not enough memory to start _ Allow Virtual machines to span NUMA nodes is off RRS feed

  • Question

  • Occasionally I am having trouble starting a VM.

    Key is one VM I want to allocate 30Gb of ram to (better would be 32GB but I can't get this to work) even though this is the first VM to start.

    Server had two physical CPUs (each 12 cores). Server had 64Gb RAM

    The other virtual machines are set to dynamic memory and start later without issue.

    Any advice would be great.

    Note span Numa is deliberately off.


    Alistair


    • Edited by AlistairK Monday, September 26, 2016 3:16 PM Typo
    Monday, September 26, 2016 3:11 PM

Answers

  • This is the expected behavior for the configuration that you've chosen. Do you understand what a NUMA node is?


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Proposed as answer by Leo Han Monday, October 3, 2016 6:41 AM
    • Marked as answer by Leo Han Wednesday, October 5, 2016 6:03 AM
    Monday, September 26, 2016 3:46 PM
  • If the operating system had the ability to function that way, the overhead would exceed the microscopic gain that comes from disabling NUMA spanning. If you want an operating environment to exist in a completely distinct hardware silo without sharing resources with other operating environments, then the only solution is to not virtualize.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Proposed as answer by Leo Han Monday, October 3, 2016 6:41 AM
    • Marked as answer by Leo Han Wednesday, October 5, 2016 6:03 AM
    Monday, September 26, 2016 6:15 PM

All replies

  • This is the expected behavior for the configuration that you've chosen. Do you understand what a NUMA node is?


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Proposed as answer by Leo Han Monday, October 3, 2016 6:41 AM
    • Marked as answer by Leo Han Wednesday, October 5, 2016 6:03 AM
    Monday, September 26, 2016 3:46 PM
  • Thanks for the response.

    Yes I believe I understand the implication of disabling spanning.

    It still seems to me that while the host OS will obviously use some ram it is a shame we can't force the ram to come from one CPU and keep the other CPU (+ram) free for the VM. 


    Alistair

    Monday, September 26, 2016 3:54 PM
  • If the operating system had the ability to function that way, the overhead would exceed the microscopic gain that comes from disabling NUMA spanning. If you want an operating environment to exist in a completely distinct hardware silo without sharing resources with other operating environments, then the only solution is to not virtualize.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Proposed as answer by Leo Han Monday, October 3, 2016 6:41 AM
    • Marked as answer by Leo Han Wednesday, October 5, 2016 6:03 AM
    Monday, September 26, 2016 6:15 PM
  • Hi Alistair,

    Have you got aware of the mechanism?

    You could mark the reply as answer if it is helpful.

    Best Regards,

    Leo


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, October 3, 2016 6:41 AM
  • I am trying to build as closely as possible to the white paper https://www.microsoft.com/en-gb/download/details.aspx?id=41936 _ Planning a Lync Server 2013 Deployment on Virtual Servers

    In the white paper they describe Numa spanning off. They outline a server with 4 sockets and each CPU being allocated 32 gb of ram. They then describe a key guest OS being allocated 32 GB of ram.

    However if I understand Eric's point correctly this is not possible. The maximum amount of RAM that can be allocated to any single guest OS <32gb ram which is what I am seeing.

    I know VMware have brought the whitepaper into question http://blogs.vmware.com/apps/2015/01/virtualizing-microsoft-lync-server-lets-clear-confusion.html

    So any thoughts and comments welcome 


    Alistair

    Monday, October 3, 2016 7:35 AM
  • A quick read of the VMware document you reference makes me wonder if VMware understands what the Lync team is saying.  In real deployments, I have heard the Lync team say that they prefer that the application be deployed to physical machines due to the nature of how it operates.  In other words, they prefer that it not be run in a virtual environment, but they recognize that some people will try to virtualize it anyway.  Given that, the Lync team's document tries to offer configurations that will work better than default configurations.

    Lync is a very intensive application that has real-time execution considerations.  Real-time applications are dependent upon tasks completing in a specific period of time.  That is why the Lync team says to disable hyper-threading.  A hyper-thread will never execute at the same level as a physical core.  So if you leave hyper-threading enabled, you can significantly impact the way Lync runs, or make it inoperable.  VMware tries to discount disabling hyper-threading in their environment because ESX runs better with hyper-threading enabled.  Well, Hyper-V also recommends that hyper-threading be enabled for the same reasons as VMware, but if the application is impacted by hyper-threading, it doesn't make any difference what hypervisor is in use.  Not all applications are meant to be run as VMs, and when they are run as VMs, you might have to adjust the hypervisor to the characteristics of the application instead of the other way around.

    Similar with NUMA.  If you disable NUMA spanning to accommodate the needs of an application, you will need to live within the constraints of what that means.  If you have a NUMA machine with a total of 64 GB of physical memory, that means there is a maximum of 32 GB in each NUMA 'zone'.  The host operating system is not constrained by NUMA spanning, so some memory will be used in each NUMA 'zone' by the host.  Therefore, there will never be 32 GB of memory available for a VM that is defined to not use NUMA spanning.  So if you want to allocate 32 GB to a single VM with NUMA spanning disabled, you must have more that 32 GB in each NUMA 'zone'.


    . : | : . : | : . tim

    Monday, October 3, 2016 12:35 PM
  • Due to some restrictions, I am not allowed to speak entirely freely on the subject of virtualizing Lync. However, on a personal level based on personal experiences and conversations with other institutions doing the same thing that I am doing, I do not believe a single thing that the Lync team has to say about NUMA on Hyper-V. Like others, I have asked the Lync team for clarification and anything concrete that we can reference to make sense of their NUMA recommendations. Responses would probably be restricted by NDA. I can share all of their responses and supporting documentation and evidence without violating any NDAs, so here it is: . Did you get all that?

    I don't really have anything on the Hyperthreading bit, though. There are a lot of "ifs" and "maybes" and "buts" to go along with that, but no more supporting documentation or evidence than for NUMA. Hyperthreading doesn't add enough in even the best conditions for me to have felt it necessary to do any investigation.

    If you rely on Microsoft PSS for assistance, then one option that they fully support for virtualizating Lync is to disable NUMA in hardware. That makes your entire computer into a single NUMA node, so the spanning option doesn't even do anything. All available memory will be accessible to a VM. In the abstract sense, that should have a generally negative impact on performance, but without supporting documentation or evidence, there's no way to know for certain unless someone is willing to leave NUMA on and test it.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Monday, October 3, 2016 1:43 PM