none
Storage for 2-node cluster

    Question

  • Hey,

    We currently have a simple 2-node cluster (Server 2012 R2 DC with Node + Disk for quorum) which uses an iSCSI NetApp FAS2020 with a few different LUNs.

    We are looking at switching to SAS JBODs and doing the disk setup (pools etc.) within Hyper-V, so I am researching that at the moment.

    One thing I want to achieve over and above what we have with the NetApp is entire disk enclosure/tray failure.  I've read about 'Enclosure Awareness' and SES requirements for this, but is there anything else I need to think about?

    All the articles I seem to find are about humungous data centres and loads of big 60-bay JBODs.  We don't and will never need that!  The NetApp is 12 bays with 450GB disks and we've had it for 7 years, so we're very much SMB scale.

    Thanks.

    Tuesday, April 18, 2017 2:00 PM

All replies

  • Hi Sir,

    Generally , 'JBOD' is a solution replaced of SAN (due to Cost Budget).

    As you mentioned , you don't need so many disks , you may consider using "storage space direct" to achieve that.

    This is a new feature brought with server 2016 (if upgrading 2012R2 to 2016 is possible):

    https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/storage-spaces-direct-overview?f=255&MSPPError=-2147217396

      

    Best Regards,

    Elton


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, April 19, 2017 6:18 AM
    Moderator
  • Second Elton. S2D is a nice choice since it is using DAS which is faster and has smaller latency. Unfortunately, S2D requires datacenter licensing which might be too expensive for an SMB scale so you might consider using StarWind VSAN https://www.starwindsoftware.com/starwind-virtual-san or HPE StoreVirtual https://www.hpe.com/us/en/storage/storevirtual.html as they do basically the same job. 

    Best regards,

    Taras Shved

    StarWind Software

    Blog:   Twitter:   LinkedIN:  

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Wednesday, April 19, 2017 8:56 AM
  • Thanks both.

    We have Datacenter with Software Assurance on the Hyper-V nodes so that's not a problem.

    Having a quick read on S2D, the data exists locally on the node but is shared/replicated between any nodes in the cluster, so Quick/Live Migration still works, failover to another node when a node fails still works?  In both directions?

    My only concern with that is that we have HP DL380 G8 servers with 8 slots for disks - 4 are currently used for the OS so it doesn't leave much room for additional disks.

    I've also been reading up on enclosure failure and I've seen you need at least 3?  Why is that - if you had 2 enclosures with 12 disks and each disk in enclosure 1 was mirrored with it's counterpart disk in enclosure 2 (so disk 1 & 1, disk 2 & 2 etc.) surely the other half of each mirror is still up if an entire enclosure is lost?

    Obviously a 3-way mirror would need 3 enclosures.

    I've also seen most articles mention SOFS - is that needed in-between the nodes and back-end storage?  Can't each node just have access to each disk in each enclosure?


    Wednesday, April 19, 2017 12:15 PM
  • "so Quick/Live Migration still works, failover to another node when a node fails still works?  In both directions?"

    Yes.  S2D clustering is just presenting a different type of storage.  The cluster still operates the same as it would using other forums of storage.

    "we have HP DL380 G8 servers with 8 slots for disks - 4 are currently used for the OS "

    Why so many disks for OS?  It is common to set up hardware mirroring for the OS disk, but rarely do you need more than about 50 GB for the OS, particularly in a cluster.

    "you need at least 3? "

    Depends upon the level of redundancy you want.  Obviously three provides a higher level of redundancy than does two.  Besides, with S2D, you are not talking about storage enclosures because the storage is local.  It does not matter if the storage is really local or if you have a storage shelf locally (i.e connected just to a single server).  It is treated the same.

    "I've also seen most articles mention SOFS - is that needed in-between the nodes and back-end storage?  "

    SoFS is something additional to just S2D.  It is not needed.  You can set up a S2D cluster with just two nodes.  Two nodes limits the amount of failure you can have in the cluster.  Three nodes provides a higher level of resistance to failure.  And four nodes or more is considered optimal.

    "Can't each node just have access to each disk in each enclosure?

    Each host in a S2D cluster has access to its own local disks.

    BTW, if you are considering a S2D cluster, you need to know that RAID controllers are NOT supported for data disks.  The data disks to be presented as storage to the cluster must reside on HBAs.  You should talk with your system vendor about the configuration they would recommend.  Though it may be possible to roll your own, S2D puts a different type of stress on the components of the system than does a on-S2D cluster.  System vendors have been testing specific solutions and working the bugs out of these specific solutions.  Better to rely on the  work they have been doing for the last several months instead of you undertaking all that same work on your own.


    tim

    Wednesday, April 19, 2017 2:23 PM
  • [ ... ]

    BTW, if you are considering a S2D cluster, you need to know that RAID controllers are NOT supported for data disks.  The data disks to be presented as storage to the cluster must reside on HBAs.  You should talk with your system vendor about the configuration they would recommend.  Though it may be possible to roll your own, S2D puts a different type of stress on the components of the system than does a on-S2D cluster.  System vendors have been testing specific solutions and working the bugs out of these specific solutions.  Better to rely on the  work they have been doing for the last several months instead of you undertaking all that same work on your own.


    Right. RAID controllers are out. And you can't use disks used for anything else (boot etc) as S2D "claims" your disks to become storage pool members once and forever. Not very flexible...

    Cheers,

    Anton Kolomyeytsev [MVP]

    StarWind Software

    Profile:   Blog:   Twitter:   LinkedIn:   

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Sunday, April 23, 2017 6:27 PM
  • Thanks Tim.

    To clarify, when I say "you need at least 3" I was referring to disk enclosures to sustain entire enclosure failure without outage.

    I have just now found this.  The first post is the configuration I had in mind, although without the (SAS) switch in between.  Each Hyper-V node would have direct connectivity to each JBOD.

    https://social.technet.microsoft.com/Forums/office/en-US/69676515-1c4f-49ed-94e0-3290d23a5aad/use-storage-spaces-hosted-by-hyperv-nodes-with-failover-clustering?forum=winserverhyperv

    I think you and Anton have answered my questions in that thread.  So 3 enclosures minimum for 2-way mirrors.  2 of the enclosures would contain the "data" disks and the 3rd would have the witness disk.



    Tuesday, April 25, 2017 7:44 AM
  • A number of options come to mind here, below is some examples to begin looking at:

    Some of these are free and some are not.

    FreeNAS/TrueNAS: http://www.freenas.org/blog/freenas-vs-truenas/

    Windows Storage Spaces: https://technet.microsoft.com/en-us/library/hh831739(v=ws.11).aspx

    Starwind VSAN: https://www.starwindsoftware.com/starwind-virtual-san

    FreeNAS and TrueNAS are not Windows based, they’re based on the FreeBSD ZFS filesystem which is a different

    system altogether but is rock stable and free (excluding TrueNAS).

    I’ve never used Starwind before so can’t talk about that much but it seems quite popular.

    If you want to stick with Windows then I’d recommend the Storage Spaces.

    Tuesday, April 25, 2017 12:34 PM
  • You appear to be mixing implementations of Storage Spaces.  Storage Spaces Direct would not be using shared SAS storage shelves unless the disks in the storage shelves were configured so that individual disks are owned by individual hosts.  S2D depends upon local, in-shared storage.  Having multiple storage shelves for redundancy is no different than having multiple local disks for redundancy.  You do not define your RAID - S2D does that for you depending upon what storage and number of nodes are presented.

    If you start talking about shared SAS, then you are talking about using clustered Storage Spaces and implementing your own definitions of RAID and storage groups.

    So your first order of business is to determine what type of Storage Spaces you want to implement.  Think of Storage Spaces as a name Microsoft has assigned to its new methods (plural) of presenting storage.


    tim

    Tuesday, April 25, 2017 1:00 PM
  • Thanks again, Tim.

    I realised I never answered about why 4 disks for OS before in the the HPV nodes; we use 2 for OS and 2 for page file (2 separate mirrors).  We also have 1 disk as a global hot-spare, so 5 slots are used of the 8.

    I have just checked the servers though.  They have 3 bays at the front for disk 'shelves', with only 1 currently populated, so we could add another 2 to get 16 extra slots.  That would be enough I reckon.

    With S2D, as thet storage is housed in the server, scalability is a bit more limited, right? in that if we need more than 16 disks going forwards, we'd be stuck.  When we decommision the server, the storage is really going with them so we have to re-buy?  So the cheaper long-term option is external shelves.  We'd need additional HBA's either way.  Does S2D rely on the same 3 disk minimum for 2-way mirrors?

    Just thinking out loud there.

    PS:  Isn't S2D 2016 only?

    Wednesday, April 26, 2017 8:19 AM
  • "realised I never answered about why 4 disks for OS before in the the HPV nodes; we use 2 for OS and 2 for page file (2 separate mirrors).  We also have 1 disk as a global hot-spare, so 5 slots are used of the 8."

    This is just my opinion, but if that were my configuration, I would say I am wasting two disks for page file.  A system configured with the proper amount of memory (which will most likely cost less than your two disk drives), needs about 1-2 GB for a page file.  Starting with Windows Server 2008 I started using a page file of 1-4 GB for all my systems, even those with 256 GB of physical memory.  There was no need for a monster page file.   And the existing page file is used minimally so there is no noticeable performance benefit to placing the page file on a separate disk.  Then the hot spare is a belt and suspenders approach.  You already have redundancy on the OS by using a mirrored volume.  That means you can survive the loss of a single drive failure.  By having a hot spare, you can now have two failures.  It is your business decision, but getting three more storage slots in the chassis might make more fiscal sense.

    "With S2D, as thet storage is housed in the server, scalability is a bit more limited, right? in that if we need more than 16 disks going forwards, we'd be stuck.  When we decommision the server, the storage is really going with them so we have to re-buy?  So the cheaper long-term option is external shelves. "

    External shelves can be presented to the host as local storage, so having a limited number of internal bays does not limit the amount of storage that can be presented.

    "Does S2D rely on the same 3 disk minimum for 2-way mirrors?"

    With S2D, mirroring occurs among the nodes.  With a two node S2D cluster, you have a two-way mirror.  With a three node S2D cluster, you have a three-way mirror.  With more than three nodes, Microsoft uses a different form of RAID.  S2D automatically configures the storage to the optimal configuration.


    tim

    Wednesday, April 26, 2017 12:48 PM
  • I have a feeling I'm going to be thanking you in every post!

    Re the cross-node disk mirroring, does the Hyper-V cluster still need a witness disk - we do Node + Disk Majority by having a small LUN provisioned on the NetApp SAN at the moment.

    I suppose we could move to 3 nodes and drop the witness disk to still have majority in case of single node failure, and we would have 3-way mirroring.

    Re putting more backplanes in the servers and using S2D, the controller in them is a HP SmartArray P440ar (http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04496202), that supports upto 26 disks using an expander.  If we used an expander, does the expander operate in the same mode as the controller its connected to, in our case RAID.  Or can the expander present the disks connected to it as S2D requires, or is it better to get a completely separate HBA supporting the number of disks we need (16).

    Oh, how does SS/S2D actually affect CSV and live migration.  Does the storage a VM has still need to be on a CSV?

    Thanks!



    Thursday, April 27, 2017 8:46 AM
  • My recommendation is to always have a witness disk.  Clustering has 'dynamic quorum', which means it brings in the witness disk (or file share) as a quorum member when needed and removes when not needed.  This means that it is actually possible to continue operations in a cluster down to a single node through multiple failures when a witness is in place.  If you do not have a witness, you can lose a single node in a three node cluster, but any fault after that may bring down the cluster.  Not a big thing, but just an added level of availability.  My personal preference.  A disk witness is 'cheap' protection.

    You need to talk to HP about which controllers they support in in an S2D environment.  However, in my quick read of your link, I seriously doubt it does.  the P440ar is described as an array controller.  Microsoft does not support RAID controllers for S2D.  Some people have gotten them to work by performing some hacking, but Microsoft does not support it.

    "how does SS/S2D actually affect CSV and live migration. "

    Fully supports it.  Storage for a VM never needs to be on CSV.  It is just that CSV gives you a lot more options and flexibility than does using a non-CSV cluster disk.  There are very, very few instances where it makes any sense to not use CSV in an Hyper-V or SQL cluster.


    tim

    Thursday, April 27, 2017 2:03 PM
  • the P440ar is described as an array controller.


    tim

    Yeah I know - I was just wondering if an expander can be set to HBA mode independently to the controller its connected to.

    If not, rather than chucking in an expander, we would use an actual HBA instead.

    Wednesday, May 03, 2017 9:40 AM
  • "was just wondering if an expander can be set to HBA mode independently to the controller its connected to."

    The expansion shelf does not care whether it is controlled by an HBA or a RAID controller.  It's job is to present the disks.  However, it generally requires that the HBA or RAID controller own all disks in the shelf.

    Or am I missing what you mean by 'expander'?


    tim

    Wednesday, May 03, 2017 12:35 PM