none
s2d quorum model RRS feed

  • Question

  • Hi,

    Does anybody know how s2d quorum works? Because I’m confused a bit.

    According to this article https://technet.microsoft.com/en-us/library/jj822937.aspx Storage Space requires “at least three physical disks” “for two-way mirroring on failover clusters” and “at least five physical disks” “for three-way mirroring on failover cluster”. It looks like s2d has the same requirements because I can’t create a 3-way mirror virtual disk unless I have 5 or more physical disks in my s2d pool. The only explanation I was able to find is here https://blogs.technet.microsoft.com/tip_of_the_day/2013/08/29/tip-of-the-day-3-way-mirrors/ and in more details here https://arstechnica.com/civis/viewtopic.php?t=1203237 (look at PreventRage’s post). They say it’s all about quorum. Fine! But clusters support different types of quorum now. We can have a cluster with a witness and dynamic quorum. However, it seems that s2d supports only majority disks quorum model. For example, I built a 3-node cluster with a cloud witness and created a 3-way mirror s2d volume. When I shut down two nodes I still have working cluster, but my s2d volume goes offline with “Majority Disks Unhealthy” detached reason. Very funny.

    But if it’s true and s2d uses majority disks quorum only, how does it work for a 2-node s2d cluster? Because when a node goes down in a 2-node s2d cluster we lose half of our disks.

    Could anyone help me to sort it out?

    Thank you,
    NS



    • Edited by SuchNS Tuesday, November 28, 2017 9:09 AM
    Tuesday, November 28, 2017 7:02 AM

All replies

  • Hi SuchNS,

    Could you please check the event viewer, anything wrong in the event viewer for cluster and storage space when you shutdown two nodes?

    Based on my knowledge, the Storage Space disk quorum are , and the requirement is that more than 50% of the disks must be present to make quorum and make the pool available. So that 2-way mirror could survive the loss of up to 1 physical disks and that 2-way mirror could survive the loss of up to 5 physical disks.

    But the cluster quorum is another concept. It's mainly for failover.  In my two-node S2D environment, shutdown one node, the S2D virtual disk still online.

    And in addition, I suggest you could refer to Elden's explanation for Cluster S2D.

    https://social.technet.microsoft.com/Forums/en-US/74dbd313-741e-424e-8acb-d7d68e22174c/two-node-s2d-cluster-disk-volume-fails-when-one-host-goes-down?forum=ws2016

    Best Regards,

    Mary


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.


    Tuesday, November 28, 2017 8:58 AM
    Moderator
  • Hi Mary,

    Thanks for the link! I’ve seen this post. I don’t think it’s relevant. We use a cloud witness and it works fine. As I mentioned before our cluster with just one node still stays online. There is no resyncing and our hardware supports SES.

    Regarding event viewer, I don’t need to check it, cause I can see volume status in FCM is detached and Get-VirtualDisk reports the reason as “Majority Disks Unhealthy”.

    You said that storage space disk quorum should have more than 50% of the disk to make, and I agree. But if you shutdown a node in your 2-node s2d cluster, you’ll have exactly 50% of the disks. It is not enough to make quorum.

    Thank you,
    NS



    • Edited by SuchNS Tuesday, November 28, 2017 11:50 AM
    Tuesday, November 28, 2017 11:49 AM
  • Hi SuchNS,

    Based on my understanding for cluster S2D. When we enable S2D,  the local disk are all used as shared disk.

    And according to the cluster quorum, if the cluster could still work, then in theory, the cluster S2D, could work.

    In my two node S2D  test lab, it seems still online. But I have to say that, the virtualdisk is in a warning status.

     

    Mary


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.


    Wednesday, November 29, 2017 2:40 AM
    Moderator
  • Hi Mary,

    Theory and guessing are great, but I’d like to know. Because in theory my 3-way mirror 3-node with a witness s2d cluster should survive 2 node failure, and the cluster does, but s2d doesn’t. And I want to know why.

    I couldn’t find the answer in so-called deep-dive articles. Maybe, they are not deep enough. Do you have a chance to ask your colleges at Microsoft?

    Thank you,
    NS

    Wednesday, November 29, 2017 6:25 AM
  • Hi NS,

    For now I haven't find more explanation from Microsoft, how the cluster quorum affect cluster S2D.

    And test a 3 node cluster S2D with a witness, when turn off 2 nodes. virtual disk online, S2D pool still health. But get-virtualdisk says warning.

    May I ask when you mean do you see the same results as mine?

    Best Regards,

    Mary


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, November 29, 2017 9:55 AM
    Moderator
  • Hi Mary,

    Is it a 3-way mirror virtual disk?

    My result is:

    Before

    

    After one node’s down.


    Two nodes are down.





    Thank you,

    NS

    Wednesday, November 29, 2017 11:03 AM
  • Hi,

    I am also trying to involve someone familiar with this topic to further look at this issue for more ideas. It may take some time to reply.

    Thanks for your support and understanding.


    Best Regards,

    Mary Dong


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, November 30, 2017 2:09 AM
    Moderator
  • Thank you, Mary, for trying to help!
    Thursday, November 30, 2017 5:35 AM
  • Hi SuchNs,

    Sorry for the late reply. After discussing, it may be related to disk/pool re-sync data. You could try the following test:

    shutdown one node first

    then waiting the virtual disk re-sync data, make sure that the status of virtual disk is complete and no waring, and there is no storage job

    after that, shutdown another node

    then verify whether the virtual disk and pool still get offline status

    At the same time, please install the latest Nov rollup.

    Best Regards,

    Mary


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, December 6, 2017 5:49 AM
    Moderator
  • Hi Mary,

    How could the virtual disk be synced with no warning after shuting down one node if we have 3-way mirror?

    Again, it's a 3-way mirror and as far as I understand we have 3 copies of data on 3 nodes. if I shutdown one node the virtual disk should go into warning state, because we lost one copy. But we still have 2 copies and 2 nodes.

    My question was why does the disk go offline when we lose 2 nodes, while we still have one copy of data and a cloud witness that keeps the cluster online?

    I didn't think it would be a hard question. Really nobody faced a similar issue?

    Thank you anyway, Mary.

    NS

    Wednesday, December 6, 2017 6:10 AM
  • Hi,

    For now, I still couldn't reproduce your issue. Appreciate your support and understanding.

    Please also patch your server to the latest in case some known issue lead to this behavior.

    And I'll also involve someone to  further look at this issue for suggestion. If there's more ideas, I will reply at once.

    Best Regards,

    Mary


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, December 6, 2017 9:58 AM
    Moderator
  • Mary,

    I installed the latest cumulative update but it changed nothing.

    I believe it's because s2d uses disk majority quorum model and when majority of disks go offline virtual disk goes offline too.

    Thank  you,

    NS

    Wednesday, December 6, 2017 2:10 PM
  • Hi NS,

    Could you please also test to  re-create virtual disk by powershell command and make sure the properties “ColumnIsolation” is  “StorageScaleUnit”. In your scenario now, the properties “ColumnIsolation” of virtual disk is “physical disk”.

    Best Regards,

    Mary


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, December 14, 2017 9:57 AM
    Moderator
  • Hi Mary,

    It's interesting suggestion. Unfortunately I can't check it right now, but thank you for it!

    Thank you,
    NS

    Thursday, December 14, 2017 11:25 AM
  • Hi SuchNs, I do have the same problem, did you ever found the cause/solution?

    Thank you,

    Alex

    Thursday, December 28, 2017 1:05 PM
  • Hi Alex,

    No, I didn't. Mary's suggestion looks promising, but I can't give it a try. My cluster is in production now. I'm afraid it's all about quorum model. s2d seems to use disk majority quorum with a witness when there is exactly half of the disks. It works fine for 2, 4 or more node clusters, but not for 3-node one.

    Thank you,
    NS

    Thursday, December 28, 2017 6:08 PM
  • Hi NS,

    I think it has something to do with Three Way Mirroring, it needs 50% of disks online, not 33%.

    As stated, Three Way Mirroring can be achieved with at least 5 disks and not only 3 and with 5 disks can suffer two disks breakdown.

    Someone calls the forth and fifth disk "quorum".

    I'm waiting for an official reply from Microsoft (I've opened a case).

    BTW, in the screenshots posted by @Mary Dong you can see a disk quorum, this means that are taken from a "normal" cluster and not from a S2D cluster, that's why she cannot reproduce our problem.

    I'll keep you informed, I've just opened a case and it will take sometime to reach the correct "level" and have the right answer (now they're talking about quorum but in our case is not the cluster that goes down but the CSV, taking with it all VMs (mine crashes when this happens)).

    Regards,

    Alex


    • Edited by Alessio.P Friday, December 29, 2017 7:38 AM
    Friday, December 29, 2017 7:36 AM
  • Hi Alex,

    This’s exactly what I’m talking about. 3-way mirroring uses (regardless of cluster) its own quorum model - disk majority with a witness (optionally). But it’s only a guess.

    There is also the ResiliencySettingName parameter which is blank on Mary’s screenshots. So, I’m not sure if she really used 3-way mirroring volume for the test.

    Anyway, it would be great to get an official answer.

    Thank you in advance for sharing it,

    NS

    Friday, December 29, 2017 8:53 AM
  • Hi NS,

    I've a first confirmation of our problem from Microsoft:

    "I've done some testing with my VMs and I'm facing the same behavior.

    For 2012 there was a workaround but is not applicable to 2016 (?).

    Today I've planned a meeting with the development team and if there is a solution I'll let you know.

    I'm sorry but, at the moment, I don't have a solid solution."

    Friday, December 29, 2017 9:46 AM
  • Another problem could be that, based on this document, in a three host scenario, we can sustain only a single disk failure (or more only if on the same host)!

    This is very worrying...

    Alex

    Friday, December 29, 2017 10:57 AM
  • Hi Alex,

    Cool! It looks like a chance to find out how it really works!

    It would be good to learn more about workaround for 2012. Just in case. Though 2012 could have only Storage Space (not s2d) on a single server. So, I don't understand how it's related to 3-node cluster.

    Thank you,

    NS

    Friday, December 29, 2017 10:59 AM
  • Alex,

    I've seen this document before, but I don't undestand how you came to this conclusion. It should sustain up to two disk failures on different nodes.

    Thank you,

    NS

    Friday, December 29, 2017 11:08 AM
  • My worries come from the fact that if have a disk failed on two nodes, you lose the 50% of healthy nodes, but this is just an extrapolation from what is explained in this image where you have a single disk failed on three nodes over four, always taken from this document

    Alex

    Friday, December 29, 2017 11:16 AM
  • Alex,

    Yes, you lose the 50% of healthy nodes, but not the 50% of healthy disks.

    As for this image, it's only correct for 3 failures, not 2 ones. Remember, we talk about 3-way mirror.

    Thank you,

    NS

    Friday, December 29, 2017 11:29 AM
  • NS,

    we are not talking only about three way mirroring, we are talking about D2S with three way mirroring!

    How many disks do you think you could lose with a four node D2S and three way mirroring? Only three?

    No, it depends on the number of hosts that survive, not on the number of disks!

    Have a look at picture number 5, the CSV stays online with four disks failed but goes offline with only three disks as seen in picture number 7, why? Because of the number of healthy hosts remaining, not on the number of disks!

    BTW, I've asked a confirmation (or a contradiction) of my worries to Microsoft, we'll see what they answer.

    Lately I may test my worries pulling two disks, one for host from my S2D and see what happens to my CSV.

    My configuration is 2 900Gb NVMe and 4 6Tb SAS per host.

    Alex

    Friday, December 29, 2017 11:41 AM
  • Alex,

    how many disks we can afford to lose depends on where these disks are located. our maximum is half of the disks, but 3-way mirror in s2d guarantees that our volume will stay online if we lose 2 disks on 2 different nodes in a 3 or more node cluster.

    At least that's how I see it.

    Thank you,

    NS

    Friday, December 29, 2017 1:42 PM
  • You are right, I've pulled two disks one from one host and one from another and everything stayed online, degraded but online, reinserting the disks everything went back to normal in a few minutes.

    The only problem left is the one with two hosts failed.

    Alex

    Friday, December 29, 2017 4:18 PM