locked
Storage Spaces pool does not reclaim free space from deleted files RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    1
    Sign in to vote

    I have 8x6TB disks in a storage pool with several thin-provisioned dual-parity volumes on it. One of them is currently occupying 30+ TB on the pool, but only about 10TB of data are actually on the volume:

    PS C:\WINDOWS\system32> get-storagepool -FriendlyName "Storage pool" | Get-Volume | select-object size,sizeremaining | fl
size          : 17592050753536
sizeremaining : 11069258166272

size          : 17592118935552
sizeremaining : 15741980147712

size          : 38373183979520
sizeremaining : 26477607419904
    Accounting for resiliency, the total occupied space across all volumes should be 24.5TB. However, the pool reports itself as being totally full:

    PS C:\WINDOWS\system32> get-storagepool -FriendlyName "Storage pool" | select-object AllocatedSize,Size | fl
AllocatedSize : 47977469050880
Size          : 48004966907904

    I'm missing 19TB of space!

    I tried the below command multiple times, to no avail:

    PS C:\WINDOWS\system32> optimize-volume r -retrim -SlabConsolidate -verbose

Post Defragmentation Report:
VERBOSE:
 Volume Information:
VERBOSE:   Volume size                 = 34.89 TB
VERBOSE:   Cluster size                = 4 KB
VERBOSE:   Used space                  = 10.81 TB
VERBOSE:   Free space                  = 24.08 TB
VERBOSE:
 Allocation Units:
VERBOSE:   Slab count                  = 35736
VERBOSE:   Slab size                   = 1 GB
VERBOSE:   Slab alignment              = 895.00 MB
VERBOSE:   In-use slabs                = 12486
VERBOSE:
 Slab Consolidation:
VERBOSE:   Space efficiency            = 88%
VERBOSE:   Potential purgable slabs    = 1
VERBOSE:   Slabs pinned unmovable      = 0
VERBOSE:   Successfully purged slabs   = 0
VERBOSE:   Recovered space             = 0 bytes
VERBOSE:
 Retrim:
VERBOSE:   Backed allocations          = 21514
VERBOSE:   Allocations trimmed         = 9854
VERBOSE:   Total space trimmed         = 9.62 TB
    I've had a nightmarish two weeks following the 1903 update, with all sorts of storage and filesystem issues that almost cost me all my data (not out of the woods yet, but getting there I hope.) If I can't sort this out, I'll have no choice but pass on Storage Spaces and go back to my previous storage solution. Please help, I've scoured the Internet and I'm pretty sure the answer isn't a click away. Thank you so much!
    Friday, June 14, 2019 2:27 PM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote
    Hi,
    Thanks for your question.
    Would you try the windows feature Data Deduplication on the volumes of storage space?
    We can try the following which demonstrated configuring Data Deduplication on the volume and optimize it. Hope this helps.
    Data deduplication -- often called intelligent compression or single-instance storage -- is a process that eliminates redundant copies of data and reduces storage overhead.
    Reference link:
    Data Deduplication Overview
    Furthermore, please note, we need to backup the important data or the volume before this critical proceeding in case of any unexpected risky.

    Hope above information can help you. If you have any question or concern, please feel free to let me know.
    Best regards,
    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, June 17, 2019 7:23 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    I would love to try dedup but alas this is Windows 10 Pro, not a server SKU. I have a Windows 2019 1809 evaluation copy installed on a spare SSD with 170 days to go, but it doesn't even recognize the pool because the metadata was updated by Windows 10 1903. If I could get a Windows Server 2019 1903 evaluation ISO, I might try dedup, but I don't have an MSDN sub and I see no other way to obtain a copy.

    What I tried over the weekend was optimize-volume -defrag (5 passes, including free space consolidation,) followed by optimize-volume -retrim -slabconsolidate. Alas, that didn't do anything either. The volume still has a 30+ TB footprint even though there are only 10TB worth of files on it.

    It's almost as if there's a disconnect between the ReFS and Storage Spaces layers, whereas ReFS "thinks" it has passed the trim instructions on certain blocks, but the pool never received them. One would think retrim would take care of that, but no. Is there an equivalent to optimize-volume -slabconsolidate but at the pool level? It looks lie optimize-storagepool only does rebalancing, but with 100% slab usage there's not much to rebalance so it quits almost immediately.

    Monday, June 17, 2019 8:22 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    Thanks for your detailed update.

    Have you tried Defragment Drives Tools for your implementation?

    Please try the following threads which discussed how to optimize the drives in windows 10 to see if it helps.

    https://social.technet.microsoft.com/wiki/contents/articles/51825.windows-10-how-to-optimize-hard-disk.aspx

    https://www.isumsoft.com/windows-10/how-to-optimize-and-defragment-drives-in-windows-10.html

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    Hope this helps. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, June 18, 2019 1:07 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    Just checking in to see if the information provided was helpful. Please let us know if you would like further assistance.

    Best Regards,

    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, June 20, 2019 10:45 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    As I wrote earlier, I did defrag, retrim and consolidates slabs on the volume. It didn't help, although it claims it should have (that's how I read "Total space trimmed = 9.62 TB" anyway.) I honestly don't see what else I can do at this point other than delete and recreate the volume, or tear down the storage space altogether and use the drives elsewhere. It doesn't seem like Storage Spaces is production ready, at least not on the consumer SKUs.

    Thursday, June 20, 2019 6:16 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Thin provisioned disk expand in size as they get used. The only point is they never automaticaly shrink again since the empty blocks still stay inside the virtual disk.

    According to storage space documentation (https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-standalone-storage-spaces):

    "With thin provisioning, space is allocated on an as-needed basis. This optimizes the usage of available storage. However, because this enables you to over-allocate storage, you must carefully monitor how much disk space is available."

    You can use the command "Optimize-VHD" to reclaim unused space inside a thin provisioned disk, however the disk cannot be in use during this process.
    https://docs.microsoft.com/en-us/powershell/module/hyper-v/optimize-vhd?view=win10-ps

    More on thin provisioning:
    https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/thin-provisioning



    Thursday, June 20, 2019 6:31 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Malcolm,

    You're referring both Storage Spaces and Hyper-V documentation. I know Hyper-V VHDs don't automatically shrink and you can't do it while online, but are you sure this applies to Storage Spaces virtual disks as well? All the Storage Spaces documentation points to the opposite, including registry tweaks on how to disable delayed TRIM notifications.

    The last link you provide is very interesting and something I hadn't seen before. Specifically, it refers to SCSI and ATA commands to retrieve deleted slabs. You may be on to something there. These disks are connected to a SAS HBA with SAS-to-SATA breakout cables. So, to the host, they are SCSI/SAS disks, but they are, in fact, SATA3. Even more interestingly, when the storage spaces virtual disk is made available to a guest VM as a passthrough device, it is identified as a thinly-provisioned LUN. It doesn't seem impossible that somewhere in all of those layers, something was lost in translation and the FS believes it's TRIM'ed a bunch of slabs, but that message never got to the controller itself, or at least not in the right way. I don't think I ever ran optimize-volume -retrim on the client VM. Perhaps that could have helped? Some more information about the APIs would be useful, or a PowerShell module that implements them, including the IOCTL calls.

    Anyway, I ended up deleting and recreating the volume from scratch, and am in the process of restoring from backup media. It's been almost a month since this whole ordeal started with the 1903 update, and at this stage I'd rather get back to my life and get my weekends back. Thanks for all the help. If nothing else, it was a valuable lesson in trusting unproven technology with critical data.

    Friday, June 21, 2019 12:11 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    Just want to check your current situation. 

    Please feel free to let us know if you need further assistance.

    Best regards,

    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, June 27, 2019 6:52 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    How are things going on? Was your issue resolved?

    Please feel free to let me know if you need further assistance.

    Best regards,

    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Friday, June 28, 2019 9:14 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    As I wrote in literally the post above yours, I had to give up on this whole concept and restore my data from backup. I don't know what broke, or how, or how I could have prevented it, or fixed it, if it was even possible, and I wasn't going to shell out $499 per incident to debug a feature that used to work prior to the upgrade. So no, it wasn't resolved, I just gave up so I no longer care about it. I guess you could say it's technically no longer an issue :)
    Thursday, July 4, 2019 8:48 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

    I'll stand by with you, if you have any question or concern, please feel free to let me know.

    Thanks for your sharing and support.

    Best regards,

    Michael

    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, July 5, 2019 9:35 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi everyone, I am seeing this still in the Windows Insider of Win 10 Pro 20H1. I understood this thread is reporting for Windows 10 Pro 1903 or later.

    It only affects Storage Spaces that have ReFS volumes. NTFS volumes are fine.

    Here is my insert on twitter and feedback hub. 

    This is really critical and may only affect Server OS not only Win 10 Pro. The thread here really sums it up perfectly. Technically nothing to add.

    tldr: In Windows 10 Pro, ReFS does unclaim freed space on volume level but not in Storage Spaces Spool nor the actual storage.

    Feedback Hub: https://aka.ms/AA6z402
    Twitter discussion: sorry it got a bit cluttered due involving many other people / MVP and MS Insider team.

    original post: https://twitter.com/tweet_alqamar/status/1216101761381031939

    asking around: https://twitter.com/etguenni/status/1216492148000002049

    MCP Windows Server, MCSA Windows 2008 / 2012, MCITP Hyper-V SCCM


    Sunday, January 12, 2020 11:46 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Malcolm also running defrag will do this since Windows 8.0 / 2012, however it does not solve this problem. 

    Neither Optimizing the container nor the pool does solve it for ReFS. 

    MCP Windows Server, MCSA Windows 2008 / 2012, MCITP Hyper-V SCCM

    Monday, January 13, 2020 12:17 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    some more insights.

    Once the the thin provisioned pool in storage is reportingly full you cannot reclaim any space on the ReFS as Powershell quits with even if you free up space on NTFS it does not affect the "full" ReFS drive.

    Optimize-Volume -DriveLetter h -Analyze

Optimize-Volume -DriveLetter h -SlabConsolidate

Optimize-Volume : Volume optimization was unable to complete because the disk backing the volume is full.               Activity ID: {64e2db38-859b-4e00-811d-e662aac5df85}                                                                     In Zeile:1 Zeichen:1                                                                                                    + Optimize-Volume -DriveLetter h -SlabConsolidate                                                                       + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                           + CategoryInfo          : NotSpecified: (StorageWMI:ROOT/Microsoft/...age/MSFT_Volume) [Optimize-Volume], CimExcep     tion                                                                                                                     + FullyQualifiedErrorId : StorageWMI 43023,Optimize-Volume                                                                                                                                                                                  


Optimize-Volume -DriveLetter h -ReTrim
Optimize-Volume : Volume optimization was unable to complete because the disk backing the volume is full.
Activity ID: {8a9d3ef6-9520-437f-b0f4-fd064cbaa195}
In Zeile:1 Zeichen:1
+ Optimize-Volume -DriveLetter h -ReTrim
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (StorageWMI:ROOT/Microsoft/...age/MSFT_Volume) [Optimize-Volume], CimExcep
   tion
    + FullyQualifiedErrorId : StorageWMI 43023,Optimize-Volume

Optimize-Volume -DriveLetter h -TierOptimize
Optimize-Volume : The volume optimization operation requested is not supported by the hardware backing the volume.
Activity ID: {4028bd55-9fed-43ca-9ad1-c6ef5a574545}
In Zeile:1 Zeichen:1
+ Optimize-Volume -DriveLetter h -TierOptimize
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (StorageWMI:ROOT/Microsoft/...age/MSFT_Volume) [Optimize-Volume], CimExcep
   tion
    + FullyQualifiedErrorId : StorageWMI 43022,Optimize-Volume

    MCP Windows Server, MCSA Windows 2008 / 2012, MCITP Hyper-V SCCM


    Monday, January 13, 2020 12:40 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I confirm the problem. Is there any workaround other than abandoning the use of refs?
    Sunday, April 19, 2020 4:47 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Hi there, the problem is at least improved in the latest Windows 10 Insider Fast. I have picked this up as any release 1903 and after is affected and passed this on to the Insider team, it was investigated by the storage team. I cannot tell if it was fixed for 1903 or 1909 or the soon 2004.

    MCP Windows Server, MCSA Windows 2008 / 2012, MCITP Hyper-V SCCM

    • Proposed as answer by largelos Saturday, April 25, 2020 4:16 AM
    Friday, April 24, 2020 3:59 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I also have this problem on 2004 update, Win10 Pro Workstation. I can confirm it isn't fixed. Oddly, prior to 2004 update, Storage Spaces showed thin priovisioning working fine and at approx 60% full - post 2004 upgrade, I was being prompted to upgrade my Storage Spaces pool and noticed, prior to the upgrade, that the pool was now showing as 99% full. Post upgrade of the pool, it's still at 99% full. So the upgrade to 2004 broke my thin provisioning capability. I use ReFS specifically with mirrored pools.
    Wednesday, June 10, 2020 10:11 AM
  • Question
    Sign in to vote
    1
    Sign in to vote

    Hi Balders,

    at the moment has been fixed in Insider Fast only. No backport to 2004, 1909, 1903 as of today.

    Also these possibly related Defrag issues have been fixed in Insider fast only. 
    - running defrag on Storage pool with thin prov. volumes will virtually allocate space that gets slab"bed" (zeroed) in the Pool but the space allocation will not be freed up, even after hours on ReFS mirrored volumes.

    - Defrag does "dice" the optimization level on thin provisioned ReFS Storage Space volumes (90%,78%,94%) everytime you run it.

    There is no backport for this yet for 1903, 1909, 2004

    MCP Windows Server, MCSA Windows 2008 / 2012, MCITP Hyper-V SCCM


    Thursday, June 11, 2020 8:50 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Is there anything that consumer users can specifically track about this or should do in the interim?

    I found this thread while researching a sidebar notification about my 2-drive mirrored ReFS Storage Space having "no capacity" (Win 10 Pro 2004).  The actual usage is well below capacity.

    Is there a risk of data loss or other malfunction in the interim or is this just a misreporting issue?

    Thursday, July 9, 2020 12:14 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Is there a risk of data loss or other malfunction in the interim or is this just a misreporting issue?

    In fact there is a high risk for malfunction. no data loss (except you count that you have to backup the whole storage space and rebuild is acceptable.

    it still does affect 1903-2004 (including Server SAC) as of Patching state 07-2020. 

    Windows Update team is deaf ears. I cannot submit a ticket for this, they want to charge me for that ticket. No way I will bother with this. It is NOT listed in the known bug list. 

    Malfunction happens for following scenarios:

    Mirrored storage spaces with ReFS 3.1 or later, Storage Spaces Level Server2019 or ServerVNext

    if running an affected version: Do not run "optmization" via defrag, it will cause the Storage Space to be filled with zeroes (SLAB optimization) which does not get deleted so the Storage Space, Physical Volumes are even "full" only for this optimization process.

    MCP Windows Server, MCSA Windows 2008 / 2012, MCITP Hyper-V SCCM

    Thursday, July 9, 2020 9:45 PM