locked
Hyper-V - Some questions on CPU / NUMA settings RRS feed

  • Question

  • Hi!

    I'm trying to unterstand how CPU / NUMA settings are working (Windows Server 2019). I know how NUMA basically is working, but there some additional I was not able to figure out. I know that each CPU has direct address access to the assigned RAM. But there are some question how this is exactly working. Until now I only had one CPU, so NUMA wasn't a topic until now.

    Following setting:

    • 2x CPU with 8 Cores per CPU (Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz)
    • HT enabled => 16 vCPU's (Threads) per physical CPU
    • 4x 16GB RAM, 2x 16GB each CPU
    • NUMA spanning enabled

    1. Case
      VM1 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB
      VM2 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      How is the physical CPU usage? Are running VM1 & VM2 on the same physical CPU (because there enough vCPUs and RAM to handle both on one physical CPU) or the vCPUs are used random over the 2 physical CPU's? If is random between the physical CPU => does this affect the performance because data in RAM's are splitted over the two nodes?

    2. Case
      VM1 = 20GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB
      VM2 = 20GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      How is now the usage? Is each VM now fixed to one physical CPU, because only each physical CPU can handle the amount of RAM without spanning and performance issue?

    3. Case
      VM1 = 40GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 40GB
      VM2 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      Is now spanning active because VM1 uses more RAM than 1 physical CPU has directly attached?

    4. Case
      VM1 = 10GB RAM (static), 16vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB
      VM2 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      How does the usage look like in this case? Is the usage between the CPU's random (assuming VM1 has only 10% CPU usage) or fixed?

    5. Case
      VM1 = 20GB RAM (static), 16vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB
      VM2 = 20GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      Now each VM fixed to one physical CPU?

    6. Case
      VM1 = 10GB RAM (static), 20vCPU, NUMA => max. CPU = 20vCPU's, max. RAM = 30GB
      VM2 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      Is this even possible => VM1 shared over two physical CPU's and VM2 fixed to one physical CPU? => are there performance issued to expect?

    7. Case
      VM1 = 40GB RAM (static), 20vCPU, NUMA => max. CPU = 20vCPU's, max. RAM = 40GB
      VM2 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      Possible performance issue because VM1 needs spanning between nodes?

    8. Case
      VM1 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB
      VM2 = 10GB RAM (static), 8vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB
      VM3 = 10GB RAM (static), 16vCPU, NUMA => max. CPU = 16vCPU's, max. RAM = 30GB

      Is now VM1 = CPU1, VM2 = CPU1 and VM3 = CPU 2?

    I was not able to figure out how this is exactly working. I don't want to configure something without unterstanding it and then I'm wondering why I'm having later performance issues.

    I have one additionally question which is more ore less related:

    Same physical setting as above, and this VM config:
    VM1 (SQL-Server) = 8vCPU
    VM2 (IIS-Webserver) = 8vCPU

    SQL-Server is licensed for 8 Cores (8 physical Cores / 8 vCPU's). Is Hyperthreading enabled in this case useful? Is the peformance better with hyperthreading enabled or not (or doesn't it matter)? So is Hyperthreading only necessary if I want to use more than two VM's (or one VM of this two needs more than 8vCPU's)? But if I'm disabling Hyperthreading I should reduce the Cores of VM2 because the HOST needs also cores to work.

    As I understand: If SQL-Server fully needs 8 physical Cores with 100% CPU usage, enabling Hyperthreading will not help because the CPU has no more capacity?

    Sorry for this many questions but reading articles on the web and books has not answered my questions.

    Friday, January 31, 2020 9:30 AM

Answers

  • NUMA can be hard to understand. Actually, it enables one CPU on multi-CPU systems to access the memory, which is controlled by this CPU. NUMA actually decrease RAM latency.

    Here are articles, which describes NUMA usage in virtualized environments:

    A closer look at NUMA Spanning and virtual NUMA settings

    NUMA and Cluster-on-die

    • Marked as answer by Domsik Tuesday, February 11, 2020 8:06 AM
    Friday, January 31, 2020 2:42 PM
  • In all cases, the scheduler will make the decision on where to place threads and memory. Placement is handled per thread, not per vCPU. Other VMs and environmental memory usage will also make an impact.

    Unless you have an application that is extremely sensitive to memory latency, this investigation is probably not a good use of your time. If you have your VMs set up with fixed memory and are letting Hyper-V manage the vNUMA topology, it will do a good job.

    Check with your license reseller for the rules on licensing SQL. I'm not certain that you can license it for a virtual core like that. But, technical forums aren't the place for license-related questions. Hyper-threading does not play into the licensing equation, though.

    Hyper-threading usually boosts SQL because queries use a lot of common instructions. Maybe a 10% boost over the same system with Hyper-threading disabled. I have not heard of any cases where disabling Hyper-threading helped SQL.

    vCPUs assigned to VMs are not deducted from a pool. Assigning a specific number of vCPUs to a VM only controls the maximum number of threads that it can simultaneously schedule on hardware. It's usually safe to oversubscribe to a degree.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Friday, January 31, 2020 3:01 PM

All replies

  • NUMA can be hard to understand. Actually, it enables one CPU on multi-CPU systems to access the memory, which is controlled by this CPU. NUMA actually decrease RAM latency.

    Here are articles, which describes NUMA usage in virtualized environments:

    A closer look at NUMA Spanning and virtual NUMA settings

    NUMA and Cluster-on-die

    • Marked as answer by Domsik Tuesday, February 11, 2020 8:06 AM
    Friday, January 31, 2020 2:42 PM
  • In all cases, the scheduler will make the decision on where to place threads and memory. Placement is handled per thread, not per vCPU. Other VMs and environmental memory usage will also make an impact.

    Unless you have an application that is extremely sensitive to memory latency, this investigation is probably not a good use of your time. If you have your VMs set up with fixed memory and are letting Hyper-V manage the vNUMA topology, it will do a good job.

    Check with your license reseller for the rules on licensing SQL. I'm not certain that you can license it for a virtual core like that. But, technical forums aren't the place for license-related questions. Hyper-threading does not play into the licensing equation, though.

    Hyper-threading usually boosts SQL because queries use a lot of common instructions. Maybe a 10% boost over the same system with Hyper-threading disabled. I have not heard of any cases where disabling Hyper-threading helped SQL.

    vCPUs assigned to VMs are not deducted from a pool. Assigning a specific number of vCPUs to a VM only controls the maximum number of threads that it can simultaneously schedule on hardware. It's usually safe to oversubscribe to a degree.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Friday, January 31, 2020 3:01 PM
  • Hi ,

    Just checking in to see if the information provided was helpful.

    Please let us know if you would like further assistance.

    Best Regards,

    Candy


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com   

    Monday, February 3, 2020 4:06 AM
  • Hi ,

    Did you have any updates on the question?

    Best Regards,

    Candy


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com   

    Thursday, February 6, 2020 7:13 AM