none
Storage spaces Direct File deleting problem RRS feed

  • Question

  • Hello All,

    SETUP:

    In our organisation we installed 2 node,  2-way mirror, all NVme S2D. On top of S2D we have SOFS role, and we have second "Compute" cluster witch saves all VHD's on that share.  Vhd are stored on Volume01, general share is on Volume02, Backup Volume03 (all with dedicated SOFS role).

    Problem that we face was not noticed until we migrated all of our VM from old cluster to new one. So now our problem is even worse than before.

    PROBLEM:

    When we copy files on any of our shares "some" files "cannot" be deleted. We delete files and they lose all Security (no owner or any other security)  but files are still visible on storage but we cannot do anything with them. For first we thought it is security rights issue but then we tried to "Take offline" one of Cluster volumes and bring it online. And problematic files disappear.

    Problem still persist and we cannot take our volume offline and online just for deleting files !!  ...files that wont be deleted are in most cases .iso, .exe, but there was some .pdf and so on. What is Interesting, we were doing some test for user folder redirection (documents, desktop, pictures), and if user puts same file (that won t be deleted) inside redirected folder user can delete it and there is no problem.  But if you access same file outside user machine a try to delete it ... \\sofs01\volume02\share\folder_redirection\user\desktop\file.zip    (and i give myself ownership or without same problem) after i click delete file disappears for second or two and appears again, but file is unusable. Can t do nothing with it.  User can't see it in it's shared folder cannot copy new file with same name or extansion i that folder either, while i can still see file on \\sofs01\volume02\share\folder_redirection\user\desktop\file.zip path outside users pc.  So we turn of and on volume and file is no more. 

    Some details of S2D:

     Get-ClusterS2D | FL

    CacheMetadataReserveBytes : 34359738368
    CacheModeHDD              : ReadWrite
    CacheModeSSD              : WriteOnly
    CachePageSizeKBytes       : 16
    CacheState                : Disabled
    Name                      : HCLSS2D01
    ScmUse                    : Cache
    State                     : Enabled

    ------------------------------

     Get-VirtualDisk

    FriendlyName              ResiliencySettingName FaultDomainRedundancy OperationalStatus HealthStatus  Size FootprintOnPool StorageEfficiency
    ------------              --------------------- --------------------- ----------------- ------------  ---- --------------- -----------------
    Volume02                  Mirror                1                     OK                Healthy       4 TB            8 TB            49,99%
    Volume03                  Mirror                1                     OK                Healthy       2 TB            4 TB            49,98%
    Volume01                  Mirror                1                     OK                Healthy      10 TB           20 TB            50,00%
    ClusterPerformanceHistory Mirror                1                     OK                Healthy      16 GB           34 GB            47,06%

    --------------------------------------

    Get-StoragePool -FriendlyName "S2D on HCLSS2D01" | FL


    ObjectId                          : {1}\\HCLSS2D01\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StoragePool.ObjectId="{2c461cff-2d46-4283-8baf-f08f9ebe9d51}:S
                                        P:{e2e70bb7-158f-4273-88a6-528eef21e90e}"
    PassThroughClass                  :
    PassThroughIds                    :
    PassThroughNamespace              :
    PassThroughServer                 :
    UniqueId                          : {e2e70bb7-158f-4273-88a6-528eef21e90e}
    AllocatedSize                     : 35235911696384
    ClearOnDeallocate                 : False
    EnclosureAwareDefault             : False
    FaultDomainAwarenessDefault       : StorageScaleUnit
    FriendlyName                      : S2D on HCLSS2D01
    HealthStatus                      : Healthy
    IsClustered                       : True
    IsPowerProtected                  : True
    IsPrimordial                      : False
    IsReadOnly                        : False
    LogicalSectorSize                 : 4096
    MediaTypeDefault                  : Unspecified
    Name                              :
    OperationalStatus                 : OK
    OtherOperationalStatusDescription :
    OtherUsageDescription             : Reserved for S2D
    PhysicalSectorSize                : 4096
    ProvisioningTypeDefault           : Fixed
    ReadOnlyReason                    : None
    RepairPolicy                      : Parallel
    ResiliencySettingNameDefault      : Mirror
    RetireMissingPhysicalDisks        : Never
    Size                              : 102385846321152
    SupportedProvisioningTypes        : Fixed
    SupportsDeduplication             : True
    ThinProvisioningAlertThresholds   : {70}
    Usage                             : Other
    Version                           : Windows Server 2019
    WriteCacheSizeDefault             : Auto
    WriteCacheSizeMax                 : 18446744073709551614
    WriteCacheSizeMin                 : 0
    PSComputerName                    :

    ----------------

    All disks healthy and operational status OK.

    Please Help, we are out of ideas and this problem only happends on brand new high availability S2D. 

    Thanks all helpers  in advance :)


    Monday, July 29, 2019 9:36 AM

All replies

  • Hi,

    From description, it seems disk not flush metadata after deleting file.

    For now, please check if there are disk/ntfs error for the disk which store problematic files. Checking logs on the owner node of disk.

    -------------------------------

    1. Open event viewer.
    1. Click system:

    -------------------


    1. Click “filter current logs”.

    -----------------------


    1. Check disk and ntfs boxes.

    ------------------------------

    1. Check if there are disk related errors.

    Best Regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, July 30, 2019 9:22 AM
    Moderator

    • Edited by Pero Corko Tuesday, July 30, 2019 12:46 PM So there are warnings. Disk 17 has been surprise removed. First 5 warnings are all for Disk 17. And there was no surprise removing :/ ...20, 18, 19,17 also have same warnings but one or two repeats of event.
    • Proposed as answer by Hamid Sadeghpour Saleh Tuesday, August 6, 2019 7:02 AM
    Tuesday, July 30, 2019 11:49 AM

    • Edited by Pero Corko Tuesday, July 30, 2019 12:45 PM This warning goes for all disk.
    Tuesday, July 30, 2019 11:54 AM
  • Tuesday, July 30, 2019 11:54 AM
  • But yes, metadata is my thought but how can I manipulate metadata on disks. Cluster shows no errors. Volume is healthy ??
    Tuesday, July 30, 2019 12:02 PM
  • Hi Michael,

    I noticed that there are event id 157 for disks. And event id 157 indicate that something disrupts the system’s communication with a disk. Like a SAN fabric error or a SCSI bus problem. For now, I would like to suggest you try to engage storage vendor to check storage.

    For more information about Event ID 157, please check:

    ---------------------------

    https://blogs.msdn.microsoft.com/ntdebugging/2013/12/27/event-id-157-disk-has-been-surprise-removed/

    Best Regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, August 2, 2019 1:57 AM
    Moderator
  • Hi,

    Just checking in to see if the information provided was helpful. Please let us know if you would like further assistance.

    Best Regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, August 6, 2019 6:21 AM
    Moderator
  • Hello, Sorry for late response.

    First of all thank You for help. 

    We found a partial solution. It is so simple but nobody tried it.  

    If these "Ghost files" are inside some folder.  You can rename folder and files disappear.  But this is no good. When this folders will contain TB of data, a lot of files will be in use and this solution will not work because we will not be able to rename folders. 

    I never seen something like this before.  Yes we have "event id 157" but i don't think it has something to do with this particular problem...

    Thursday, August 22, 2019 1:16 PM
  • Hi, this happens because of the "zone information in file attachments" feature - a.k.a. the unblock tick when you right click a file you have downloaded from external location.

    You can check the zone.identifier with dir /r (woshub.com/how-windows-determines-that-the-file-has-been-downloaded-from-the-internet)

    This seems to happen only with CSV Refs on Windows Server 2019. On 2016/2012R2 the zone.identifier does not seem to interfere.

    I have yet to find a solution to this (rather than not copying .iso, .exe and etc. files) to the CS volume, so if anyone has any suggestions they will be much appreciated. 

    Tuesday, October 8, 2019 12:54 PM
  • In the end...I removed SOFS for general share and installed "normal general" file share role on cluster. It works normally.    Microsoft...very very bad :(
    Wednesday, October 9, 2019 7:31 AM
  • Hi,

    Thanks for your reply. 

    I'm sorry for your issue bothered you all this time. We'll continue to monitor this thread and if you want to recreate file server by using SOFS, we continue to give the advise. 

    Thanks for your sharing and support.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Wednesday, October 9, 2019 9:36 AM
    Moderator
  • Thank you,

    It wolud be nice to resolve this problem but for now we will stay on "General FS".

    Best regards,

    Pero

    Friday, October 11, 2019 5:37 AM