locked
ReFS corruption RRS feed

  • Question

  • A few weeks ago, I created a striped virtual disk using storage spaces on Win2K12R2 formatted in ReFS.  I copied a large amount of data to the disk with apparently no issues.  I am now finding problems and corrupted files in the new disk.  Files that apparently cannot be accessed or deleted from the ReFS disk.  "Error 0x80070143: A data integrity checksum error occurred.  Data in the file  stream is corrupt".  What gives?  This seems to be a pretty significant flaw.  I know that ReFS is supposed to have built in error detection and recovery hence the lack of chkdsk. Is there no way to scan the file system for errors and correct?  I can't even delete the bad file so obviously there is something wrong with the file system.  Any advice is appreciated.

    Joe

    Sunday, November 17, 2013 8:22 PM

Answers

  • A few weeks ago, I created a striped virtual disk using storage spaces on Win2K12R2 formatted in ReFS.  I copied a large amount of data to the disk with apparently no issues.  I am now finding problems and corrupted files in the new disk.  Files that apparently cannot be accessed or deleted from the ReFS disk.  "Error 0x80070143: A data integrity checksum error occurred.  Data in the file  stream is corrupt".  What gives?  This seems to be a pretty significant flaw.  I know that ReFS is supposed to have built in error detection and recovery hence the lack of chkdsk. Is there no way to scan the file system for errors and correct?  I can't even delete the bad file so obviously there is something wrong with the file system.  Any advice is appreciated.

    Joe

    ReFS will detect (that's what you see, based on a strong metadata and optional data checksums) and try to fix (with underlying supported mechanism - spaces with mirror or parity, NOT simple or striped) data errors. In your case you've allocated wrong container so ReFS can only report "data has issues". It's not a flaw by design, it's a flaw by implementation... Advice - recover from backup and don't use unprotected spaces. See:

    http://www.refs-data-recovery.com/new-features-in-refs.aspx

    Repair of damaged data

    If ReFS encounters a read error on a mirrored storage space, it can read the data from the "healthy" copy and correct the damaged one. If a write error occurs, the ReFS driver allocates another space to write the data automatically.

    http://msdn.microsoft.com/en-us/library/windows/desktop/hh848060(v=vs.85).aspx

    • Integrity: ReFS stores data so that it is protected from many of the common errors that can cause data loss. File system metadata is always protected. Optionally, user data can be protected on a per-volume, per-directory, or per-file basis. If corruption occurs, ReFS can detect and, when configured with Storage Spaces, automatically correct the corruption. In the event of a system error, ReFS is designed to recover from that error rapidly, with no loss of user data.

    Hope this helped :)


    StarWind iSCSI SAN & NAS

    • Marked as answer by Mandy Ye Thursday, December 5, 2013 8:23 AM
    Sunday, November 17, 2013 11:02 PM
  • Hi,

    As VR38DETT said, for your current situation we do not have any method to try when encountering a file corruption issue. It is still recommended to perform regular backup so that at least we could restore a corrupted file.

    For more information about ReFS, please refer to the article below:

    Windows Server 2012: Does ReFS replace NTFS? When should I use it?
    http://blogs.technet.com/b/askpfeplat/archive/2013/01/02/windows-server-2012-does-refs-replace-ntfs-when-should-i-use-it.aspx

    Regards,

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    • Marked as answer by Mandy Ye Thursday, December 5, 2013 8:23 AM
    Wednesday, November 27, 2013 2:53 AM
  • Thanks for the response.  Let me ensure that I am explaining my storage implementation clearly. 

    I have have a single thin provisioned basic volume formatted ReFS. The volume is contained in a virtual disk with parity layout.  This is part of a pool of 3 disks.  Windows is reporting health status OK for both the volume and individual disks.   Is this a non supported configuration?

    Joe

    I have evaluated ReFS on a VHDX disk and its hosted in a single high capacity disk. So far as a data store it is working OK.

    I suggest abandoning the storage pool approach as that is not how it is supposed to be used. Pools of servers not disks.


    Corsair Carbide 300R with window
    Corsair TX850V2 12V1 70A
    Asus M5A99FX PRO R2.0 CFX/SLI
    AMD Phenom II 965 C3 Black Edition @ 4.0 GHz
    G.SKILL RipjawsX DDR3-2133 8 GB
    EVGA GTX 660 Ti FTW Signature
    Asus PA238QR IPS LED HDMI DP 1080p
    ST2000DM001 & Windows 8.1 x64 Professional
    LG WH14NS40 14X Blu-Ray Writer BDXL 128GB
    Microsoft Wireless Desktop 2000
    Wacom Bamboo CHT470M

    • Marked as answer by Mandy Ye Thursday, December 5, 2013 8:23 AM
    Wednesday, November 27, 2013 3:24 AM

All replies

  • A few weeks ago, I created a striped virtual disk using storage spaces on Win2K12R2 formatted in ReFS.  I copied a large amount of data to the disk with apparently no issues.  I am now finding problems and corrupted files in the new disk.  Files that apparently cannot be accessed or deleted from the ReFS disk.  "Error 0x80070143: A data integrity checksum error occurred.  Data in the file  stream is corrupt".  What gives?  This seems to be a pretty significant flaw.  I know that ReFS is supposed to have built in error detection and recovery hence the lack of chkdsk. Is there no way to scan the file system for errors and correct?  I can't even delete the bad file so obviously there is something wrong with the file system.  Any advice is appreciated.

    Joe

    ReFS will detect (that's what you see, based on a strong metadata and optional data checksums) and try to fix (with underlying supported mechanism - spaces with mirror or parity, NOT simple or striped) data errors. In your case you've allocated wrong container so ReFS can only report "data has issues". It's not a flaw by design, it's a flaw by implementation... Advice - recover from backup and don't use unprotected spaces. See:

    http://www.refs-data-recovery.com/new-features-in-refs.aspx

    Repair of damaged data

    If ReFS encounters a read error on a mirrored storage space, it can read the data from the "healthy" copy and correct the damaged one. If a write error occurs, the ReFS driver allocates another space to write the data automatically.

    http://msdn.microsoft.com/en-us/library/windows/desktop/hh848060(v=vs.85).aspx

    • Integrity: ReFS stores data so that it is protected from many of the common errors that can cause data loss. File system metadata is always protected. Optionally, user data can be protected on a per-volume, per-directory, or per-file basis. If corruption occurs, ReFS can detect and, when configured with Storage Spaces, automatically correct the corruption. In the event of a system error, ReFS is designed to recover from that error rapidly, with no loss of user data.

    Hope this helped :)


    StarWind iSCSI SAN & NAS

    • Marked as answer by Mandy Ye Thursday, December 5, 2013 8:23 AM
    Sunday, November 17, 2013 11:02 PM
  • Thanks for the response.  Let me ensure that I am explaining my storage implementation clearly. 

    I have have a single thin provisioned basic volume formatted ReFS. The volume is contained in a virtual disk with parity layout.  This is part of a pool of 3 disks.  Windows is reporting health status OK for both the volume and individual disks.   Is this a non supported configuration?

    Joe

    Sunday, November 17, 2013 11:39 PM
  • Hi,

    As VR38DETT said, for your current situation we do not have any method to try when encountering a file corruption issue. It is still recommended to perform regular backup so that at least we could restore a corrupted file.

    For more information about ReFS, please refer to the article below:

    Windows Server 2012: Does ReFS replace NTFS? When should I use it?
    http://blogs.technet.com/b/askpfeplat/archive/2013/01/02/windows-server-2012-does-refs-replace-ntfs-when-should-i-use-it.aspx

    Regards,

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    • Marked as answer by Mandy Ye Thursday, December 5, 2013 8:23 AM
    Wednesday, November 27, 2013 2:53 AM
  • REFS does NOT support VHD or VHDX with file integrity(the corruption fixing and detection) routines enabled.  If you create a virtual disk on a refs volume the file integrity system is automatically turned off.  If you want the integrity system enabled remove the virtual disks...tear down the volume and rebuild it as a regular storage space volume with ReFs.  That's what i have done on my server.  I have two ssd's for the boot volume using windows dynamic disks in a mirror.  For the twin 3.2 TB drives I created a ReFs formatted storage space in a mirror.  Doing this enabled the file integrity system and would have stopped this error in it's tracks.
    Sunday, November 29, 2015 10:12 PM
  • So I encountered the "volume repair is not successful" error on some of the files I have within a directory on a ReFS partition.  I was able to recover them from a backup so I didn't lose any data which is good.  My question here is that I am unable to then delete the files that show that error.  I would like to remove them.  Any ideas? Or do I need to redo the partition?
    Wednesday, December 23, 2015 4:06 AM
  • REFS does NOT support VHD or VHDX with file integrity(the corruption fixing and detection) routines enabled.  If you create a virtual disk on a refs volume the file integrity system is automatically turned off.  If you want the integrity system enabled remove the virtual disks...tear down the volume and rebuild it as a regular storage space volume with ReFs.  That's what i have done on my server.  I have two ssd's for the boot volume using windows dynamic disks in a mirror.  For the twin 3.2 TB drives I created a ReFs formatted storage space in a mirror.  Doing this enabled the file integrity system and would have stopped this error in it's tracks.

    It does. You cannot run VMs from these VHDX but storing VHDX with integrity on ReFS is OK.

    Either way Windows Server 2016 is going to fix what you're complaining about completely: integrity can be enabled everywhere (performance impact from doing that is another story).


    Cheers,

    Anton Kolomyeytsev [MVP]

    StarWind Software Chief Architect

    Profile:   Blog:   Twitter:   LinkedIn:  

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Wednesday, December 23, 2015 9:59 AM
  • I hope anyone reading this thread actually tests ReFS/Storage Spaces with some test corruption.  I tested with the new Workstations SKU of W10 Pro.  Microsoft specifies it should detect, log and repair corruption.  It fails miserably.  Even the scheduled scrubber fails.  Please try it and report back.

    The problem is getting too little attention because:

    1) The functionality is so fundamental to the product everybody figures it must work.

    2) ReFS does catch a few errors and reports them, so it looks like it works even though it misses most of the time.

    3) People have difficulty creating test corruption.  Most editors will only make changes that are seen as legitimate to the OS.

    Check out the tests described in these threads:

    ReFS test with corrupt data. Does it work?

    Why use ReFS  (near the end of this thread another party verifies my results)

    How to create file corruption

     

    Perhaps things work fine on the Server product, but as of tests done at the end of August 2018, ReFS/Storage spaces is a complete failure on the "Windows 10 Pro for Workstations" version of the OS.

    • Edited by Mark_391 Wednesday, September 12, 2018 11:59 PM
    Sunday, September 9, 2018 4:44 AM