none
Witness Server, Quorum and DAC Mode

    Question

  • Right, my Exchange 2010 knowledge quest continues and people have been very generous in helping me with my questions. However, the more I read the more I get confused! So here are today's two questons:

    1. I understand the wintess server's role in deciding if a DAG is quorate or not but I'm still a bit puzzled about what quorum, in an Exchange 2010 context, means. Does it just apply to implementations of DAGs which straddle two data centres? iIs it used to decide which data centre has the most available mailbox nodes and thus decides which is the primary site or does it have a use for DAGs on whihc all nodes are hosted n the same site? If so what is that purpose?  In short what is the definistion of a quorum in Exhcnage 2010?

    2. Why is DAC mode not enabled by defualt?

    Monday, May 14, 2012 2:21 PM

Answers

  • Hi Tyler,

    Quorum is a "majority vote" between mailbox servers.  If a mailbox server does not have quorum all mailbox databases automatically dismount.  Ok reading this probally makes no sense... let me try and explain.

    Lets take an example where I have two datacentres, datacenter A and datacenter B.  Datacenter A has 3 exchange mailbox servers and Datacenter B has 2 mailbox servers.  If datacenter B went offline, the servers in datacenter B would be offline howerver the servers in Datacenter A would still be accessible.  If Datacenter A went offline, all servers in Datacenter B will dismount their databases and the cluster would be down.  Why?  Because Datacenter A has majority vote, i.e. it has quorum.

    Still confused?

    Ok, well lets think about this for a second.  Why would Datacenter B dismount its databases if Datacenter A goes offline?  What happens if it wasnt Datacenter A that went down, what if it was the network link between datacenters?  Either way Datacenter B has no idea what the problem is, it might be the server failed in Datacenter A or the network link went down, either way it can't talk right?  If a server in Datacenter B mounts a mailbox database without being able to talk to Datacenter A this can result in serious issues.  If the same database is ever mounted in two locations at the same time, databases on both sides will be modified, emails, calendar appointments etc.  When connectivity is restored replication will no longer work.  You will need to delete one of the database instances and reseed it, you cannot keep both versions of the database. Either way dataloss, very bad!

    Still confused, every time I try and explain this I do find it difficult...

    Lets look at split brain in more detail... I found the below content from an email I sent to a client a few months back (saves me retyping

    A single Exchange mailbox database can never be active on more than one servers at the same time, if this happens different transaction logs will be written to each database, the databases would no longer contain the same content and replication cannot continue.  In this event one database needs to be deleted, and the other database needs to reseed as the authoritative copy.  This is known as split brain and will result in data loss.  Cluster’s must maintain “quorum” to avoid split brain.  The moment any cluster node loses communication with over 50% of the cluster nodes, it must deactivate.  Because it has lost communication with majority of the cluster nodes and it won’t know if the database is mounted on another node.

    In the event a DAG cluster has an even number of nodes such as 6, 8 or 10… a file witness share (FWS) is required.  The FSW acts as a deciding voter to ensure there is always an odd number of nodes in the cluster.  For example take two datacentres, each datacentre contains 4 DAG nodes in a single DAG cluster.  Both datacentres have exactly 50% of the total cluster nodes.  The FWS acts as the deciding vote.  If DC1 contains the FWS and the link between the datacentres fails, all servers in DC1 will maintain cluster quorum.  All cluster nodes in DC2 will the immediately dismount any active databases and failover to DC1 will occur.

     <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f">
     <v:stroke joinstyle="miter">
    <v:formulas>What happens if the datacentre containing the FWS failed?  Well both datacentres will lose quorum and there will be an outage.</v:formulas></v:stroke></v:shapetype>

    <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f"><v:stroke joinstyle="miter"><v:formulas></v:formulas></v:stroke></v:shapetype>

    For this scenario Microsoft developed an Alternate Witness Share (AWS) which must be manually invoked by an Administrator.  So DC1 failed, the environment is down, an Administrator executes the PowerShell command to invoke the AWS, this brings the environment back online from DC2.

    <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f"><v:stroke joinstyle="miter"><v:formulas><v:f eqn="if lineDrawn pixelLineWidth 0">
    So how can a network failure result in Split Brain?  <v:f eqn="sum @0 1 0">Let’s say both DC1 and the link between datacentres went down.  Administrator activates the AWS, DC2 is up and running, all is good.  <v:f eqn="sum 0 0 @1">
      <v:f eqn="prod @2 1 2">
      However what happens if the administrator restored DC1 BEFORE the link was back online?  DC1 does not know the AWS has been activated, it will mount its databases.<v:f eqn="prod @3 21600 pixelWidth">
      </v:f></v:f></v:f></v:f></v:f></v:formulas></v:stroke></v:shapetype>

    This is the only time split brain can occur in Exchange 2010 DAG environments.

    Microsoft invented a technology called Datacenter Activation Coordination (DAC) to deal with this exact scenario.   I do not recommend implementing DAC mode initially… for more information about DAC mode and how it works please refer to http://technet.microsoft.com/en-us/library/dd979790.aspx

    <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f"> </v:shapetype>


    Clint Boessen MVP - Exchange Server, MCSE, MCITPx4, Dip Network Engineering
    Perth, Western Australia

    Blog: http://clintboessen.blogspot.com/

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Monday, May 14, 2012 3:04 PM
  • Quorum simply means majority. It knows nothing of Data Centers.

    See:

    Exchange 2010 High Availability Misconceptions Addressed

    For the second question, its probably not enabled by default because it requires some understanding and positive action by the Exchange Admin when its implemented and there should be no suprises when you are doing a datacenter switchover.

    Monday, May 14, 2012 2:48 PM
  • Correct, for the cluster to say up, a majority (quorum) of nodes have to be able to communicate with each other

    Exchange will automatically dismount the stores if quorum is lost. You can manually "force" Exchange to mount the stores if there is only 1 node left however. But that is a manual process.

    Yes. this is where DAC Mode comes handy. In a DR scenario, you switchover to the backup Data Center. Its a manual process outlined here:

    Datacenter Switchovers

    Monday, May 14, 2012 5:58 PM

All replies

  • Quorum simply means majority. It knows nothing of Data Centers.

    See:

    Exchange 2010 High Availability Misconceptions Addressed

    For the second question, its probably not enabled by default because it requires some understanding and positive action by the Exchange Admin when its implemented and there should be no suprises when you are doing a datacenter switchover.

    Monday, May 14, 2012 2:48 PM
  • Hi Tyler,

    Quorum is a "majority vote" between mailbox servers.  If a mailbox server does not have quorum all mailbox databases automatically dismount.  Ok reading this probally makes no sense... let me try and explain.

    Lets take an example where I have two datacentres, datacenter A and datacenter B.  Datacenter A has 3 exchange mailbox servers and Datacenter B has 2 mailbox servers.  If datacenter B went offline, the servers in datacenter B would be offline howerver the servers in Datacenter A would still be accessible.  If Datacenter A went offline, all servers in Datacenter B will dismount their databases and the cluster would be down.  Why?  Because Datacenter A has majority vote, i.e. it has quorum.

    Still confused?

    Ok, well lets think about this for a second.  Why would Datacenter B dismount its databases if Datacenter A goes offline?  What happens if it wasnt Datacenter A that went down, what if it was the network link between datacenters?  Either way Datacenter B has no idea what the problem is, it might be the server failed in Datacenter A or the network link went down, either way it can't talk right?  If a server in Datacenter B mounts a mailbox database without being able to talk to Datacenter A this can result in serious issues.  If the same database is ever mounted in two locations at the same time, databases on both sides will be modified, emails, calendar appointments etc.  When connectivity is restored replication will no longer work.  You will need to delete one of the database instances and reseed it, you cannot keep both versions of the database. Either way dataloss, very bad!

    Still confused, every time I try and explain this I do find it difficult...

    Lets look at split brain in more detail... I found the below content from an email I sent to a client a few months back (saves me retyping

    A single Exchange mailbox database can never be active on more than one servers at the same time, if this happens different transaction logs will be written to each database, the databases would no longer contain the same content and replication cannot continue.  In this event one database needs to be deleted, and the other database needs to reseed as the authoritative copy.  This is known as split brain and will result in data loss.  Cluster’s must maintain “quorum” to avoid split brain.  The moment any cluster node loses communication with over 50% of the cluster nodes, it must deactivate.  Because it has lost communication with majority of the cluster nodes and it won’t know if the database is mounted on another node.

    In the event a DAG cluster has an even number of nodes such as 6, 8 or 10… a file witness share (FWS) is required.  The FSW acts as a deciding voter to ensure there is always an odd number of nodes in the cluster.  For example take two datacentres, each datacentre contains 4 DAG nodes in a single DAG cluster.  Both datacentres have exactly 50% of the total cluster nodes.  The FWS acts as the deciding vote.  If DC1 contains the FWS and the link between the datacentres fails, all servers in DC1 will maintain cluster quorum.  All cluster nodes in DC2 will the immediately dismount any active databases and failover to DC1 will occur.

     <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f">
     <v:stroke joinstyle="miter">
    <v:formulas>What happens if the datacentre containing the FWS failed?  Well both datacentres will lose quorum and there will be an outage.</v:formulas></v:stroke></v:shapetype>

    <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f"><v:stroke joinstyle="miter"><v:formulas></v:formulas></v:stroke></v:shapetype>

    For this scenario Microsoft developed an Alternate Witness Share (AWS) which must be manually invoked by an Administrator.  So DC1 failed, the environment is down, an Administrator executes the PowerShell command to invoke the AWS, this brings the environment back online from DC2.

    <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f"><v:stroke joinstyle="miter"><v:formulas><v:f eqn="if lineDrawn pixelLineWidth 0">
    So how can a network failure result in Split Brain?  <v:f eqn="sum @0 1 0">Let’s say both DC1 and the link between datacentres went down.  Administrator activates the AWS, DC2 is up and running, all is good.  <v:f eqn="sum 0 0 @1">
      <v:f eqn="prod @2 1 2">
      However what happens if the administrator restored DC1 BEFORE the link was back online?  DC1 does not know the AWS has been activated, it will mount its databases.<v:f eqn="prod @3 21600 pixelWidth">
      </v:f></v:f></v:f></v:f></v:f></v:formulas></v:stroke></v:shapetype>

    This is the only time split brain can occur in Exchange 2010 DAG environments.

    Microsoft invented a technology called Datacenter Activation Coordination (DAC) to deal with this exact scenario.   I do not recommend implementing DAC mode initially… for more information about DAC mode and how it works please refer to http://technet.microsoft.com/en-us/library/dd979790.aspx

    <v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f"> </v:shapetype>


    Clint Boessen MVP - Exchange Server, MCSE, MCITPx4, Dip Network Engineering
    Perth, Western Australia

    Blog: http://clintboessen.blogspot.com/

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Monday, May 14, 2012 3:04 PM
  • Thanks for the replies, things are a bit clearer but the replies  from Clint and A_D go some way to explaining why I'm confused! Clint, your answer is , which I understand but couched in terms of two data centres but this contrasts with A_D who says "It knows nothing of Data Centers" (which I take to mean sites as well as in most cases they are analogous). I guess, what I need to know is does the FWS have any purpose in a single site\data centre implementation? I know the FWS is created automatically so assume it must be but I've never seen anything that tells me what it does in a single site?
    Monday, May 14, 2012 4:42 PM
  • Thanks for the replies, things are a bit clearer but the replies  from Clint and A_D go some way to explaining why I'm confused! Clint, your answer is , which I understand but couched in terms of two data centres but this contrasts with A_D who says "It knows nothing of Data Centers" (which I take to mean sites as well as in most cases they are analogous). I guess, what I need to know is does the FWS have any purpose in a single site\data centre implementation? I know the FWS is created automatically so assume it must be but I've never seen anything that tells me what it does in a single site?

    The FSW is used only when there is an even number of DAG members and is used by the cluster as a tie-breaker if the there is a danger of loosing quorum.

    The FSW doesnt know about sites. So it can be used in a single site or multi-site DAG.

    DAC Mode was designed with Data Center Swithovers in mind, but that doesnt mean it wont work if there is only 1 Datacenter. DAC Mode was created because clusters in of themselves are not data center aware. Hope that makes sense.

    Monday, May 14, 2012 4:55 PM
  • I guess I'm not explaining myself properly but what does 'losing quorum' mean? In a single site what are the two entities the tie breaker is between?

    Monday, May 14, 2012 5:12 PM
  • I guess I'm not explaining myself properly but what does 'losing quorum' mean? In a single site what are the two entities the tie breaker is between?

    Less than the majority. For purposes of discussing the quorum, you need to divorce the concept of sites from the discussion.

    Database Availability Group Quorum Models

    Quorum requires a majority of voters to be able to communicate with each other. Consider a DAG that has four members. Because this DAG has an even number of members, an external witness server is used to provide one of the cluster members with a fifth, tie-breaking vote. To maintain a majority of voters (and therefore quorum), at least three voters must be able to communicate with each other. At any time, a maximum of two voters can be offline without disrupting service and data access. If three or more voters are offline, the DAG loses quorum, and service and data access will be disrupted until you resolve the problem

    For the second question, please read the article I linked in my first post - specifically:

    Quorum requires a majority of voters to achieve a consensus.Thus, when you have an even number of members in a DAG, you need an external component to provide a weighted vote for one of the actual quorum voters to prevent ties from occurring. ◦In a Windows Failover Cluster, only members of the cluster are quorum voters. When the cluster is one vote away from losing quorum and the Witness Server is needed to maintain quorum, one of the DAG members that can communicate with the Witness Server places a Server Message Block (SMB) lock on a file called witness.log that is located in the Witness Directory. The DAG member that places the SMB lock on this file is referred to as the locking node. Once an SMB lock is placed on the file, no other DAG member can lock the file. ◦The locking node then acquires a weighted vote; that is, instead of its vote counting for 1, it counts for 2 (itself and the Witness Server). ◦If the number of members that can communicate with the locking node constitutes a majority, then the members in communication with the locking node will maintain quorum and continuing servicing clients. DAG members that cannot communicate with the locking node are in the minority, and they lose quorum and terminate cluster and DAG operations
    Monday, May 14, 2012 5:18 PM
  • I think I might have just had a eureka moment!

     I was given under the impression that if a node had sufficient resources it could host all the databases in a cluster if neccessary. I assumed that if a 4 node cluster hosted 4 databases in total if three nodes went down the 4 databases could hosted on the remaining node. Now, from what I've just read Exchange won't 'allow' this and that at least half (or half plus the FSW!) nodes in cluster need to be available for the cluster to continue running. It is this 'rule' that the majority of servers in a cluster must be up for a cluster to continue operating that I missed.

    Is that correct? Please say yes! If so then this leads me to another question (sorry!)

    If you spread your cluster across two sites, e.g. 3 at the prmiary and one at the DR site if you lose the primary you will never be able to achieve bring a aquorum at the DR site even if you move the FSW. Is it possible to over ride this in a DR situation so you can provide a aprtial service in the DR site when the primary is down?

    Thanks for you patience, it is much appreciated.

    Monday, May 14, 2012 5:52 PM
  • Correct, for the cluster to say up, a majority (quorum) of nodes have to be able to communicate with each other

    Exchange will automatically dismount the stores if quorum is lost. You can manually "force" Exchange to mount the stores if there is only 1 node left however. But that is a manual process.

    Yes. this is where DAC Mode comes handy. In a DR scenario, you switchover to the backup Data Center. Its a manual process outlined here:

    Datacenter Switchovers

    Monday, May 14, 2012 5:58 PM
  • I really dont know why my post formatting messed up..

    Clint Boessen MVP - Exchange Server, MCSE, MCITPx4, Dip Network Engineering
    Perth, Western Australia

    Blog: http://clintboessen.blogspot.com/

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Thursday, May 17, 2012 6:37 AM