none
Live Migrations No Longer Working After January 2019 Updates

    Question

  • We have a problem after the January updates (50 plus blades with AMD Bulldozer Family 15h processors) We have 400 plus VMs hosted on our hyper v clusters (Server 2012R2 and SCVMM 2012r2), and after January updates can no longer live migrate. We have put the KB4490512 update on one of the hosts but it requires a reboot, which in turn would require us to power down our VMs as we are unable to migrate them off now.  Is there/will there be a fix to enable us to not have to reboot the hosts?  or is there/ will there be a workaround to enable us to live migrate the vms ?

    Hopefully there will be as this will cause a lot of business disruption.

    Any help would be much appreciated

    Regards

    Andrew
    Wednesday, March 13, 2019 10:59 AM

All replies

  • Hi Andrew,

    Thanks for your question.

    Could you manually execute the VMs failover to another host?

    When shutting down or rebooting a node in a Failover Cluster, you first want to drain (move off) any roles running on that server (such as a virtual machine).  This ensures that the shutting down of a node is graceful to any applications running on that node.

    1. Open Failover Cluster Manager (CluAdmin.msc)
    2. Click on “Nodes”
    3. Right-click on the node name and under ‘Pause’ click on ‘Drain Roles’
    4. Under Status the node will appear as ‘Paused’.  At the bottom of the center pane click on the ‘Roles’ tab.  Once all roles have moved off this node, it is safe to shut down or reboot the node.

    Details:

    https://blogs.msdn.microsoft.com/clustering/2013/08/23/how-to-properly-shutdown-a-failover-cluster-or-a-node/

    Besides, Is there/will there be a fix to enable us to not have to reboot the hosts?>>>>

    We could configure highly available virtual machine priority when the host restarts. Please refer to the following blog,

    https://www.petri.com/clustered-hyper-v-virtual-machine-prioritization-2

    https://www.altaro.com/hyper-v/configure-start-order-priority-clustered-vms/

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    Hope above information can help you. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, March 14, 2019 7:34 AM
    Moderator
  • Duplicate post - https://social.technet.microsoft.com/Forums/en-US/c1828799-f639-4b92-8502-00307f796614/live-migrations-no-longer-working-after-january-2019-updates?forum=winserverhyperv

    tim

    Thursday, March 14, 2019 1:14 PM
  • Hi,

    Just want to confirm the current situations.

    Please feel free to let me know if you need further assistance.

    Best regards,

    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, March 18, 2019 10:01 AM
    Moderator
  • Hi,

    How are things going on? Was your issue resolved?

    Please feel free to let me know if you need further assistance.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Wednesday, March 20, 2019 2:51 PM
    Moderator
  • Hi,

    Was your issue resolved?

    If you resolved it using our solution, please "mark it as answer" to help other community members find the helpful reply quickly.

    If you resolve it using your own solution, please share your experience and solution here. It will be very beneficial for other community members who have similar questions.

    If no, please reply and tell us the current situation in order to provide further help.

    Best Regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, March 22, 2019 9:55 AM
    Moderator
  • Hi

    Sorry it has taken a while to reply to this, we have been unable to solve this problem as the patch (KB4490512 ) for the hosts requires a reboot.  We are unable to live migrate any VMs off the hosts unless the VMs are powered down and  as we have 400 hundred plus VMs on 30 plus hosts this would cause a lot of problems to the business.  I was hoping Microsoft would have supplied a non disruptive solution to this by now but as it is only a certain processor group maybe they don't believe it is a big enough problem which is a little disappointing.

    Anyway thanks for the time you have taken on this Michael it is much appreciated.

    Regards

    Andrew

    Tuesday, April 2, 2019 9:08 AM
  • "as it is only a certain processor group maybe they don't believe it is a big enough problem"

    First question to ask is if it is a supported processor for the operating system you are running?  http://www.windowsservercatalog.com  It could be that you are running on a platform for which the system vendor never certified the solution, meaning that a patch could cause issues.

    Second question to ask is have you contacted the system vendor to ask about anything they may have found with the latest patches.  It is up to the system vendor to perform the tests on all the different processors they support.  Microsoft does not do processor specific checking itself, but works with the vendors.  The vendors may have a patch their environment that corrects a problem exposed by a general patch from Microsoft.

    Third question to ask (if you have answered 'yes' to the above two questions) is have you logged a support call with Microsoft.  If this is truly specific to a certain processor group, Microsoft may have a specific hot fix for the issue, or they may be working with the vendor to develop one.  Microsoft is not testing on all processor and possible system configurations - they rely on their partnerships with the various vendors.  If this problem has not been reported to either the system vendor of a supported configuration or to Microsoft, it will not have an engineer assigned to work on it.


    tim

    Tuesday, April 2, 2019 2:05 PM
  • Hi

    Thanks for the time you've taken, we did a test applying Microsofts fix on a a 2 node cluster.  It worked but as I said to do this we have to power down the VMs on the hosts apply the patch reboot the hosts and  live migrations are working.  The unfortunate part of this in production we have over 450 VMs to do this to and as you may guess the bosses ain't over happy about it.

    But again thanks all for your time and effort on this.

    Andy

    Thursday, April 11, 2019 9:16 AM
  • According to another post you made (https://social.technet.microsoft.com/Forums/en-US/c1828799-f639-4b92-8502-00307f796614/live-migrations-no-longer-working-after-january-2019-updates?forum=winserverhyperv), it sounds like you might have a number of servers that are not supported for running Windows Server.  In that post you identify using Jaguar and Puma AMD chips.  Those chips were designed for use on tablets, not for use with Windows Server.  The only one that can address the use of those chips in Windows Server environments is AMD, and if they haven't certified them for use in Windows Server, you have unsupported environments.

    The chips may work just fine, but you need to make the business decision if you want to run 450 VMs on unsupported hardware.  Running on unsupported hardware may or may not be the root cause of your issue.  But without support, it means you have to perform your own troubleshooting with whatever support AMD may be willing to provide.

    "We have put the KB4490512 update on one of the hosts but it requires a reboot, which in turn would require us to power down our VMs as we are unable to migrate them off now.  Is there/will there be a fix to enable us to not have to reboot the hosts?"

    It is quite common to have to reboot a system after a patch has been applied.  There are many files that are not reloaded until a reboot has taken place.  A patch to any one of those files requires a reboot for the patch to take effect.  Reboot is fact of life if you want to apply all patches, particularly security patches.

    Not sure what you are saying about unable to migrate VMs.  In a clustered environment, it is a recommended practice to configure all VMs with processor compatibility.  This enables live migration between 'dissimilar' hosts.  Even if the two hosts are the same model, it is possible to have slight differences in them due to different patching levels.  Configuring VMs for processor compatibility generally ensures the ability to live migrate between different processors supported by Windows Server (as long as the processors are of the same manufacturer, e.g. AMD or Intel).


    tim

    Thursday, April 11, 2019 1:18 PM