locked
cluster failover question RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    Good day everyone.

    I have a slew of stand-alone hyper-v hosts running around 10-15 guest VMs each.  I'm looking to start clustering the hosts to have a higher degree of confidence in availability.

    I created a lab cluster with two 2016 hosts using CSV on an EMC SAN. I have 3 test VMs running on each host, and as a test i did a few live migrations to the other cluster member, no issues, 1 second to move.  So...today i wanted to replicate a real failure so with each host running (3) VMs each, i unplugged one of  the servers.  The fail-over was not immediate as i was expecting.  From cluster manager on the running host, it showed "unmonitored" for the 3 VMs on the dead host for approximately 3 or 4 minutes and then started running them natively on the up host.

    Any ideas why the failover wasn't immediate?

    Thanks,

    Danny

    Monday, December 9, 2019 7:54 PM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Danny,

    In Windows Server 2016 Failover Clustering there are were a few new features, for example:

    A VM continues to run on a node, even when it falls out of cluster membership. In this state, the node is considered to be in an “isolated” state and the VM is “unmonitored” – i.e., its health is not being actively monitored by the cluster service.

    By default, a cluster will wait 4 minutes before responding to a host failing to heartbeat. The 4 minutes is enough time for an operator to realize that they have pulled the wrong cable or for a top-of-rack switch to restart after a crash. During this time, a non-responding host has a status of Isolated in the cluster and failovers will not occur.

    If a host fails to return online after 4 minutes have passed, then the cluster will initiate a failover of every virtual machine. The virtual machines will behave differently depending on your storage system:

    • SMB 3.0: If the host is online and able to communicate with the storage, the virtual machines remain online.
    • CSV on Block Storage: The virtual machine is placed into a Paused-Critical state.

    If a host returns online before 4 minutes have expired, then it rejoins the cluster. What if the host goes offline again? Once again, the host has a status of Isolated and failovers will not take place. The default time is 2 hours. If the host becomes isolated for a third time in a 2 hour period, then the cluster will place that host into a Quarantined state. It will live migrate the virtual machines to more suitable hosts in the cluster.

    Note that the 4 minutes and 2 hours, are defaults and can be overridden. The 4-minute wait can be modified on a per-virtual machine basis. Compute Resiliency can be disabled on the cluster. This might make sense for clusters where transient issues are unlikely to isolate hosts or a completely self-contained cluster, such as a cluster-in-a-box.

    You'll find more information about these values over here:
    Virtual Machine Compute Resiliency in Windows Server 2016

    Best regards,
    Leon

    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, December 9, 2019 10:04 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    "Any ideas why the failover wasn't immediate?"

    In addition to what Leon provided, you also have to take into consideration the different types of failovers that can occur.  In the situation you created - unplugging one of the hosts - you created a situation where the virtual machines completely disappeared.  Since the host had a power failure, so did the VMs.  Once the cluster determines that the failed host is not coming back (Leon's information), the cluster has to start the VMs from scratch.  In other words, they have to perform a cold boot in order to get running.

    tim

    • Marked as answer by Danny Wacker Thursday, December 12, 2019 4:45 PM
    Tuesday, December 10, 2019 4:25 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    "I'm confused as tot why they would have to do this - with the storage, and memory .bin on the CSV, wouldn't the other cluster member know the state and be able to take over "on the fly"?"

    When a VM is running in the memory of a machine, the active memory pages are not being written out to disk.  The .bin file that may exist on disk is reserved space into which the memory will be written if you specify that the memory is to be saved when the machine is gracefully shut down.  By pulling the plug on the host that is running the VM, you have in essence pulled the plug on the VMs - there is no time for the hypervisor to write the physical memory in use by the VM to disk.  Therefore when the VM is moved to a surviving host there is nothing to read from disk to restore the VM's state.  It must be cold booted.

    tim

    • Marked as answer by Danny Wacker Thursday, December 12, 2019 4:45 PM
    Wednesday, December 11, 2019 2:05 PM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Danny,

    In Windows Server 2016 Failover Clustering there are were a few new features, for example:

    A VM continues to run on a node, even when it falls out of cluster membership. In this state, the node is considered to be in an “isolated” state and the VM is “unmonitored” – i.e., its health is not being actively monitored by the cluster service.

    By default, a cluster will wait 4 minutes before responding to a host failing to heartbeat. The 4 minutes is enough time for an operator to realize that they have pulled the wrong cable or for a top-of-rack switch to restart after a crash. During this time, a non-responding host has a status of Isolated in the cluster and failovers will not occur.

    If a host fails to return online after 4 minutes have passed, then the cluster will initiate a failover of every virtual machine. The virtual machines will behave differently depending on your storage system:

    • SMB 3.0: If the host is online and able to communicate with the storage, the virtual machines remain online.
    • CSV on Block Storage: The virtual machine is placed into a Paused-Critical state.

    If a host returns online before 4 minutes have expired, then it rejoins the cluster. What if the host goes offline again? Once again, the host has a status of Isolated and failovers will not take place. The default time is 2 hours. If the host becomes isolated for a third time in a 2 hour period, then the cluster will place that host into a Quarantined state. It will live migrate the virtual machines to more suitable hosts in the cluster.

    Note that the 4 minutes and 2 hours, are defaults and can be overridden. The 4-minute wait can be modified on a per-virtual machine basis. Compute Resiliency can be disabled on the cluster. This might make sense for clusters where transient issues are unlikely to isolate hosts or a completely self-contained cluster, such as a cluster-in-a-box.

    You'll find more information about these values over here:
    Virtual Machine Compute Resiliency in Windows Server 2016

    Best regards,
    Leon

    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, December 9, 2019 10:04 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    "Any ideas why the failover wasn't immediate?"

    In addition to what Leon provided, you also have to take into consideration the different types of failovers that can occur.  In the situation you created - unplugging one of the hosts - you created a situation where the virtual machines completely disappeared.  Since the host had a power failure, so did the VMs.  Once the cluster determines that the failed host is not coming back (Leon's information), the cluster has to start the VMs from scratch.  In other words, they have to perform a cold boot in order to get running.

    tim

    • Marked as answer by Danny Wacker Thursday, December 12, 2019 4:45 PM
    Tuesday, December 10, 2019 4:25 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Leon and Tim - Thank you very much for your answers. 

    A follow up on what Tim said about having to cold boot to come up on the "up" cluster member (which is exactly what happened in my test).  I'm confused as tot why they would have to do this - with the storage, and memory .bin on the CSV, wouldn't the other cluster member know the state and be able to take over "on the fly"?

    thanks again.

    Dan

    Tuesday, December 10, 2019 5:46 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi

    there are 2 elements to this that you need to understand which are better explained in the links below. So firstly, its how Cluster communicate over networks:

    https://techcommunity.microsoft.com/t5/Failover-Clustering/Tuning-Failover-Cluster-Network-Thresholds/ba-p/371834

    If the cluster nodes are in the same subnet, they try to ping each other for a heartbeat every second (SameSubnetDelay). If after 10 Heartbeats there is no response (SameSubnetThreshold), the node will be set to down.

    Then you have resiliency:

    https://techcommunity.microsoft.com/t5/Failover-Clustering/Virtual-Machine-Compute-Resiliency-in-Windows-Server-2016/ba-p/372027

    This is what Leon is referring to above. If there is a sudden unannounced failure such as a power loss, the node will go to Isolated and the VM's will go to Unmonitored for a period of 4 minutes or 240 seconds. 

    These values are all configurable - if you run "Get-Cluster -Name "YourClusterName" | fl *", this will give you a full listing of all values. As the article says, you can then run "(Get-Cluster).ResiliencyDefaultPeriod = <value>" if you want the failover to be less than 240 seconds.

    In a regular reboot scenario, the Cluster Service will shutdown and will announce to the other nodes that it is going into a down state. VMs will move immediately.

    Hope this helps

    Thanks

    Michael

    Tuesday, December 10, 2019 6:03 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    "I'm confused as tot why they would have to do this - with the storage, and memory .bin on the CSV, wouldn't the other cluster member know the state and be able to take over "on the fly"?"

    When a VM is running in the memory of a machine, the active memory pages are not being written out to disk.  The .bin file that may exist on disk is reserved space into which the memory will be written if you specify that the memory is to be saved when the machine is gracefully shut down.  By pulling the plug on the host that is running the VM, you have in essence pulled the plug on the VMs - there is no time for the hypervisor to write the physical memory in use by the VM to disk.  Therefore when the VM is moved to a surviving host there is nothing to read from disk to restore the VM's state.  It must be cold booted.

    tim

    • Marked as answer by Danny Wacker Thursday, December 12, 2019 4:45 PM
    Wednesday, December 11, 2019 2:05 PM