locked
Hyper-V Core Scheduler RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    We are currently running a 3 node Hyper-V Cluster on Windows Server 2016 Datacenter.  The VMs running on our cluster are using a total of 542 cores (Hyper Threaded) however there are only 352 cores available in total on the 3 nodes of the cluster.  It is possible to achieve successful over-subscription such as this using the Hyper-V classic scheduler, which has been the default for all version of Windows Hyper-V including Windows Server 2016.  Server 2016 now provides a new alternative called “the core scheduler”  Details about the different types of schedulers is explained here: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-scheduler-types To use the new core scheduler, we would need to have enough hyper threaded cores available on the nodes of the cluster to cover the 542 cores currently needed by our VMs plus enough to allow for growth and to allow for a node of the cluster to go offline.  We are considering buying more nodes and going to the core scheduler because it is recommended here https://support.microsoft.com/en-us/help/4457951/windows-server-guidance-to-protect-against-l1-terminal-fault to add an extra level to guard against speculative executions side-channel vulnerabilities.  Also, in the first link it says, “The core scheduler will be used by default starting in Windows Server 2019. On Windows Server 2016, the core scheduler is optional and must be explicitly enabled by the Hyper-V host administrator, and the classic scheduler is the default.”  We will want to upgrade our Hyper-V Cluster to 2019 at some point and the way it sounds we will be required to use core scheduler to migrate to 2019.  So my question is, is it possible to migrate to Server 2019 and still use classic scheduler by changing the setting in 2019 until we have enough nodes with enough cores to support core scheduler or is that not an option?

    Monday, February 4, 2019 7:40 PM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

    The language used in those and other articles on this topic is quite confusing. I see several places where it gives false impressions. Primarily: The core scheduler does not prevent CPU over-subscription.

    The core scheduler's behavior will cause a greater performance drag than the classic scheduler as you exceed 1 vCPU per pCore, but 1) "greater than" does not automatically mean "unacceptable" or even "noticeable", and 2) you can reduce the impact by tweaking the hardware thread count on your vCPUs.

    The documentation also makes some presumptions that you're not just over-subscribing but over-driving. It looks like you're well under a 3:1 ratio if you count only 2 nodes. If you're not pushing them hard today, I don't think the core scheduler will hurt you. I would want to see performance traces showing consistently high pCPU utilization or context switches before worrying about the core scheduler in your environment.

    I currently have a core scheduler build running at a 4:1 ratio. It works just fine, and even though I should go in and update the hardware thread count on the vCPUs like they recommend, I haven't felt any particular rush. Of course, my environment isn't yours.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Proposed as answer by Tim CerlingMVP Tuesday, February 5, 2019 1:56 PM
    • Marked as answer by WSUAL2 Thursday, February 7, 2019 8:14 PM
    Monday, February 4, 2019 9:16 PM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    The language used in those and other articles on this topic is quite confusing. I see several places where it gives false impressions. Primarily: The core scheduler does not prevent CPU over-subscription.

    The core scheduler's behavior will cause a greater performance drag than the classic scheduler as you exceed 1 vCPU per pCore, but 1) "greater than" does not automatically mean "unacceptable" or even "noticeable", and 2) you can reduce the impact by tweaking the hardware thread count on your vCPUs.

    The documentation also makes some presumptions that you're not just over-subscribing but over-driving. It looks like you're well under a 3:1 ratio if you count only 2 nodes. If you're not pushing them hard today, I don't think the core scheduler will hurt you. I would want to see performance traces showing consistently high pCPU utilization or context switches before worrying about the core scheduler in your environment.

    I currently have a core scheduler build running at a 4:1 ratio. It works just fine, and even though I should go in and update the hardware thread count on the vCPUs like they recommend, I haven't felt any particular rush. Of course, my environment isn't yours.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Proposed as answer by Tim CerlingMVP Tuesday, February 5, 2019 1:56 PM
    • Marked as answer by WSUAL2 Thursday, February 7, 2019 8:14 PM
    Monday, February 4, 2019 9:16 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Very interesting.  Apparently you have more knowledge and understanding than the engineer from the Microsoft Premier Support Case I opened to try and get some clarification on this.  After 6 hours of research and a $2000 bill, I was told that I needed to have at least a 1:1 ratio to be able to run VMs using core scheduler.  I asked for documentation saying that and why but they could not produce anything.  They spent hours researching this as no one had any experience using core scheduler so I am very happy that I found someone who does have some experience with it.  

    The way I was led to understand how core scheduler works is that it takes 1 physical core (2 Logical processors or vCPUs) from the Hyper-V node it is running on and assigns it to a VM that has been configured to use 2 vCPUs.  Core scheduler then would not allow any other VMs to use this specific physical core, hence eliminating any potential risk of a VM that was compromised because of a speculative side channel vulnerability reading data from another VM that was sharing this same core.  How is this understanding of core scheduler wrong?  How does core scheduler assign vCPUs to a VM while mitigating a compromised VM from reading data from another VM that is sharing the same core?

    Tuesday, February 5, 2019 4:33 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    In my experience, the phone-based engineers are better at break-fix than general knowledge. It goes with the job. And, you get wildly varying quality of engineer, mostly depending on what tier your support plan qualifies you to access. But, I did use their consulting service once a few years ago in a similar situation. I could hear the engineer Googling my questions. Fairly certain he read portions of my blog articles back to me. So, yeah. Don't get me wrong, there are a lot of insanely smart people at Microsoft. But, there are also a lot of people that should probably skip to "I don't know, let me ask/escalate" a lot faster. Like any other company of that size, I suppose. I am sorry for your experience.

    Anyway, you were told right, but from the wrong presumptions. The scheduler does indeed prevent other VMs from using an SMT core pair. But, we're talking about scheduling, not ownership. You can schedule exclusive access to a train car, a tractor-trailer, a move theater, a dining hall, a wallpaper steamer, etc. But, it's not yours. When you're done, they're going to clean up after you and let someone else use it. Maybe that someone else is you again.

    The bigger thing here, is that with the classic scheduler, different VMs can simultaneously, but separately, use two cores that belong to the same SMT pair. The core scheduler will not allow that. If a scheduled VM is not enabled for SMT, then the other core in the pair will sit idle during that VM's scheduled time.

    I do not possess sufficient knowledge of physical CPUs or any knowledge of the commands Microsoft uses that would be necessary to explain exactly what core scheduling entails. I could speculate from the documents, but given that we've found at least one place where it gives false impressions, it would stand to reason that any conclusions I draw from the rest of it would also be suspect.

    What I can tell you for certain, the core scheduler does not prevent oversubscription.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.


    Tuesday, February 5, 2019 6:03 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    The expertise of Microsoft engineers certainly varies from product group to product group.  The System Center DPM group is awesome, other areas such as Windows Server and SCCM, not so much in my experience.  Oh well, good thing for these forums to help one another out.

    Thank you for the very nice explanation.  

    Referring to the article above: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-scheduler-types -- in the section "Enabling SMT in guest virtual machines" Have you done this or is this what you said you haven't done yet?  

    Tuesday, February 5, 2019 6:43 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    That's the part that I have not yet done. 

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Tuesday, February 5, 2019 7:00 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    So I wonder if once the number of SMT threads per core the guest VM will see is set, will core scheduler stop allowing over-subscription?  I wish I had a test Hyper-V server to try that out on.  
    Tuesday, February 5, 2019 7:28 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Well, now that just made me have to update the configurations.

    No, setting the SMT thread count does not prevent oversubscription. I didn't think it would, but now I know for certain.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Wednesday, February 6, 2019 12:58 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Thanks for testing that.  Did you shut down and start the VMs back up as well?   This article About Hyper-V hypervisor scheduler type selection talks about enabling SMT on VMs some more and suggests that a full shutdown and restart is needed to fully complete the SMT enablement process or at least that's what I gathered from the article.
    Wednesday, February 6, 2019 7:38 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    The VM has to be off for Set-VMProcessor to work, if that's what you mean.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Wednesday, February 6, 2019 7:58 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Ok, so the VM must be powered off to run the "Set-VMProcessor" command?  I was looking for some documentation saying that but I had not come across that yet.  From what I have read, I understood it that you could run the "Set-VMProcessor" while the VM was running but you would have to power it off and then back on for the change to take effect.
    Wednesday, February 6, 2019 9:07 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    The cmdlet itself doesn't care if the VM is off or on. The various APIs that it calls have their own rules. Some work online, some don't.

    Changing the hw thread count says "Cannot change the processor functionality of a virtual machine now" if I try it while the VM is on.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Thursday, February 7, 2019 2:37 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Very good.  Thank you for all of your help and testing.  It is much appreciated.  
    Thursday, February 7, 2019 8:14 PM