locked
REFS for a Hyper-V CSV Volume RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    I am currently running a 4 node Server 2016 Hyper-V cluster storage is provided by a DELL MD3200 SAN normal concepts for CSV volume on NTFS are in place as previously nodes where 2012R2 and 2008R2 prior. My main question is I am going to be adding another SAN DELL MD3420 and I have been wondering whether to use REFS for the file system for VM's situated on this SAN.

    I came across a Veeam article advising moving to this scenario however I would like to get the opinion of you all.

    Regards

    Chris

    MCP, MCTS, MCITP, MCITPVA, MCSA 2012, MCSE Private Cloud, MCSE Server Infrastructure

    Thursday, February 23, 2017 7:15 PM

Answers

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    in short, stay with NTFS in the SAN Config, have a look on the supported deployments

    https://technet.microsoft.com/en-us/windows-server-docs/storage/refs/refs-overview

    cheers

    Udo

    Thursday, February 23, 2017 11:16 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    I agree with Udo.

    I suppose using NTFS is a better choice. 

    Best Regards,

    Leo

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, February 24, 2017 2:03 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Maybe, this could also be helpful around Cluster size:

    https://blogs.technet.microsoft.com/filecab/2017/01/13/cluster-size-recommendations-for-refs-and-ntfs/

    https://www.windowspro.de/marcel-kueppers

    I write here only in private interest

    Disclaimer: This posting is provided AS IS with no warranties or guarantees, and confers no rights.

    • Proposed as answer by Leo Han Thursday, March 9, 2017 12:12 PM
    Friday, February 24, 2017 9:43 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    You could mark the reply as answer if it is helpful. 

    Best Regards,

    Leo

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, March 9, 2017 12:12 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    in short, stay with NTFS in the SAN Config, have a look on the supported deployments

    https://technet.microsoft.com/en-us/windows-server-docs/storage/refs/refs-overview

    cheers

    Udo

    Hi,

    jumping on this bandwagon as well. I've read that article you mention. The date on top of the page says 2016-1-3, but maybe it's been updated in the past, I'm not sure about that. But under 'Fcuntionality' it clearly stated ReFS is supported in cluster environments. While it might be working with redirected IO (I'm gonna test that in a test environment), you seem to say it's not supported. But as far as that document goes it is.

    By the way, the reason I'm looking into this is that although we have 16Gbps SAN which sequentially almost reaches that (around 1400MB/sec sequential write) we see huge, huge IO when Hyper-V zeroes-out a vdisk when expanding or creating one. We deliberately use fixed vhdx. Creating or expanding a vdisk practically stalls all VM's on that same array. I'm not sure what Hyper-V does there, as writing for example 100GB of random data to the same array writes with somewhere between 1200-1400MB/sec and all VM's keep running just fine.

    Anyway with ReFS we shouldn't have that issue anymore as that handles creating files completely different.

    I'll set something up in my test environment and let you know.

    Tuesday, June 20, 2017 9:06 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    jumping on this bandwagon as well. I've read that article you mention. The date on top of the page says 2016-1-3, but maybe it's been updated in the past, I'm not sure about that. But under 'Fcuntionality' it clearly stated ReFS is supported in cluster environments. While it might be working with redirected IO (I'm gonna test that in a test environment), you seem to say it's not supported. But as far as that document goes it is.

    Hi,

    ReFS has a lot of benefits and in my opinon best practices only to use it with CSV in 2016 (On top of Storage Spaces / Direct). But your test regarding redirected IO is correct.

    Regards,
    Marcel

    https://www.windowspro.de/marcel-kueppers

    I write here only in private interest

    Disclaimer: This posting is provided AS IS with no warranties or guarantees, and confers no rights.



    Tuesday, June 20, 2017 10:01 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Adding some great links to help you with the choice ReFS/NTFS. 

    https://blogs.technet.microsoft.com/askpfeplat/2013/01/01/windows-server-2012-does-refs-replace-ntfs-when-should-i-use-it/

    https://www.starwindsoftware.com/blog/refs-performance

    https://mangolassi.it/topic/11362/windows-administration-ntfs-and-refs-filesystems

    We are still running NTFS though.

    Best regards,

    Taras Shved

    StarWind Software

    Blog:   Twitter:   LinkedIN:  

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Tuesday, July 18, 2017 8:06 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hello,

    Staying with NTFS for now would be the best option for now. NTFS disk are recoverable in case of failure. REFS can not be recovered as Microsoft state that REFS is self recoverable. In case of severe failure backup is the only option when it comes to recover from a REFS data drive loss.

    Regards,

    Nick.

    Wednesday, July 19, 2017 7:15 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    I've not dived THAT deep into it yet, but I think it's recoverability is based partly on the metadata. However, for VHDX files that part is disabled in the first place, making it a bad choice based on the recoverability I guess as it can't detect failures in there anyway.

    But then, the metadata is not like a copy of the data, just some form of parity in this sense. When it detects a failure, it can copy the datablock from another copy, if you have build a storage spaces direct setup. We don't use that though, and I don't see us using that in the near future. We utilize proper SANs that do data scrubs constantly to somewhat prevent the same issue. Given all severe bugs we've had so far in Server 2016 as well as the atrocious cumulative patching system, meaning if a bug is in a specific fix in a cumulative you have the choice to either update everything or nothing.

    Take KB3200970 which was released in November 2016. It breaks the functionality of Hyper-V to take snapshots of virtual failover-clusters, breaking all hypervisor-level backup tools like Veeam. It's now July 2017 and still no fix, which was promised us in the June update already. In the past we'd be able to uninstall the specific patch that screwed things up and keep all other patches installed to still be secure for the rest. With 2016 that's not possible anymore. And we've had so many more bugs, in AppV for example, rendering our 2016 AppV unusable for 4 months. In our 2008R2 all we just uninstalled the specific update and we were fine again.

    I can see the advantages of cumulative updates of course, making for a lot less diversity in the field. Yet, I can't imagine the trouble when there's a bug in REFS / Storage Spaces in a specific cumulative. With the speed of MS releasing patches (for good reasons) it could bring you down for months.

    Having said all that, we've decided to stay with NTFS as we don't trust MS in any way with our data at this point. I hope that will change in the future, but for now, if you don't have storage spaces, and no 'proper' storage solution, go for NTFS. With Hyper-V, as there's no metadata on VHDX files, stay with NTFS period until they fix that.

    Thursday, July 20, 2017 6:44 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Sigdarkside!

    Refs is not supported with SAN Storage Systems, for this scenario use only NTFS, you can see updated documentation with that information:

    https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview

    Monday, September 4, 2017 12:59 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    This note is very new and on some sites (e.g. german) not available.

    https://www.windowspro.de/marcel-kueppers

    I write here only in private interest

    Disclaimer: This posting is provided AS IS with no warranties or guarantees, and confers no rights.

    Tuesday, September 19, 2017 6:55 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Sigdarkside!

    Refs is not supported with SAN Storage Systems, for this scenario use only NTFS, you can see updated documentation with that information:

    https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview

    Wow. That's a bold statement there on that site. ReFS is not supported on SAN storage. That would render ReFS unusable in, well, let's guess, 60% of all somewhat bigger environments? At least the ones I know. My own backup solution runs Veeam on ReFS as it has a somewhat nice integration with it, and it has it's own dedicated SAN for that. That statement renders my backup solution unsupported? I can't believe that's the way MS wants ReFS to go. I know MS wants to push their thinking to as much of it's customers as possible, but we don't all want to or can use S2D.

    I'd have so much more trust in a good hardware SAN than a MS software solution for storage. I'm really curious where this will go.

    Tuesday, September 19, 2017 7:47 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    The documentation has changed. But, the problem seems that if you are using ReFS for CSVs on a SAN, then all IO is redirected through the coordinator. Check https://www.vembu.com/blog/windows-server-2016-hyper-v-refs-vs-ntfs/

    MarianoC.


    • Edited by MarianoC Wednesday, August 8, 2018 1:32 PM
    Wednesday, August 8, 2018 1:31 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi does it also aply to Windows server 2019, is there some changes in REFS to avoid IO redirection on CSV volumes. I'm building Hyper-V cluster with HPE 3par and 10 GB iSCSI network.

    Thanks OSO



    • Edited by OOSOO Wednesday, August 7, 2019 1:44 PM
    Wednesday, August 7, 2019 12:58 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    https://docs.microsoft.com/en-us/windows-server/failover-clustering/failover-cluster-csvs

    documention doesnt mention it, though doesnt mention it for 2016 as well. my assumption would be it didn't change, but if its a fresh cluster you plan to implement, its propably quickest to try it out. getting new servers in october, can test it then if you couldnt before

    Wednesday, August 7, 2019 9:31 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi FZB what should i have to test, what methodology do you sugest, which tools etc.

    Thanks

     
    Thursday, August 8, 2019 7:19 AM
  • Question
    Sign in to vote
    1
    Sign in to vote

    Hi so I'can confirm that in Windows 2019 REFS has same behavior as on Windows 2016

    BlockRedirectedIOReason      : NotBlockRedirected
    FileSystemRedirectedIOReason : FileSystemReFs
    Name                         : Cluster Disk 1
    Node                         : xxxx
    StateInfo                    : FileSystemRedirected
    VolumeFriendlyName           : Volume2
    VolumeName                   : \\?\Volume{b6ec8000-6768-462e-b05b-9c66a6f06a8f}\

    Friday, August 9, 2019 7:44 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Given the immaturity of ReFS, I'd not trust my VM's on that, even if we have a very sturdy backup plan. Maybe on S2D. But certainly not on a standalone ReFS volume. There is basicallyno technical documents on it, and when you run into issues (and we've ran into them multiple times now) you're basically f...ed. There is refsutil which up to now is quite useless. We've had multiple occurances on our Veeam ReFS storages, which corrupted files for still unknown reason. However, when that happens a file gets flagged unaccessible and then there is no way to delete it, repair it or reclaim the space. That's just one of the quirks we've seen with ReFS. We will not use it again on standalone volumes.

    • Proposed as answer by MysticFoxDE Wednesday, August 5, 2020 6:20 AM
    • Unproposed as answer by MysticFoxDE Wednesday, August 5, 2020 6:20 AM
    • Proposed as answer by K. Wester-Ebbinghaus Wednesday, August 5, 2020 9:28 AM
    Friday, August 9, 2019 7:58 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Hi Robert,

    I can confirm this behavior with Veeam and ReFS from several installations. 😡
    On the other hand, Veeam always complains when I use an NTFS volume as a backup destination and recommends that I use ReFS instead. ?!?

    But I still believe that ReFS is inherently a good filesystem and that for these problems that we all experience, "intermediate" components are more responsible.
    I may have found out yesterday why a ReFS formatted SAN LUN cannot currently be used in "Direct Mode" with CSV and also believe that I may can fix the problem.

    As yet, there is not a single documentation from Microsoft that actively contradicts this scenario. !!!

    We still have an Hyper-V cluster with a ReFS formatted LUN in operation at a customer. 
    And this weekend I'm trying to teach this "Direct Mode" with ReFS. 🤪

    Best Regards from Germany

    Alex
    Wednesday, August 5, 2020 6:42 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    We still have an Hyper-V cluster with a ReFS formatted LUN in operation at a customer. 
    And this weekend I'm trying to teach this "Direct Mode" with ReFS. 🤪

    Best Regards from Germany

    Alex

    Wow, I hope that is a test environment. I wouldn't try something like that out at a customer site :-) Nor would I run ReFS at my clister but thats another story. It's spme years past but I'm still utterly NOT convinced of ReFS with S2D vs a proper SAN.

    Anyway I'm very curious what you found and whether or not it works out. Good luck and let us know!

    Thursday, August 6, 2020 3:55 PM