none
Storage Spaces Disk High Activity No Throughput from Devices RRS feed

  • Question

  • Hello, TechNet! 

    I've been fighting this issue for a few weeks now, and I cannot for the life of me figure out what's causing it. I use Microsoft Storage Spaces on a SAN, providing iSCSI targets to a pair of Hyper-V hypervisors in a Failover Cluster. The SAN has:

    • 3 X 480GB SSD
    • 4 X 4TB HDD

    The original plan was to use FreeNAS and create a two-tier, RAID5 array, which failed miserably (<1MB/s read/write). I moved to Windows and decided mirroring was more important than the space saved by parity, and created a storage pool containing all 7 disks, with two tiers and an ReFS volume on it; this volume holds virtual disks which are shared via iSCSI (both NTFS and ReFS virtual disks, for files and VMs respectively). I do have a 4th SSD ready, just waiting on another part to install it.

    So what's the problem? I initially copied about 5 TB of data from an old file server onto the new SAN, all user files. Speeds were amazing, I was sustaining 150+MB/s and during testing could read it back at 250MB/s. I started migrating a few VMs onto it (on a separate VHDX, formatted as ReFS) and still, speed was available in spades. At some point after the 3rd VM, the SAN locked up. Investigating on the SAN, I found reading/writing at 65/65MB/s. The data was virtually inaccessible via the hypervisors, and by bringing the volumes offline and copying the VHDX files off the SAN directly I cannot exceed 5MB/s, recovering it onto my backup media (which can sustain 115MB/s write) is going to take around 2 weeks. This is the second time this happened, I thought the first was a configuration mistake and rebuilt the SAN. Why is this happening, and can I stop it to get my data back?

    Wednesday, November 13, 2019 12:56 PM

All replies

  • Hi ,

    >>At some point after the 3rd VM, the SAN locked up. 

    >>by bringing the volumes offline and copying the VHDX files off the SAN directly I cannot exceed 5MB/s

    Based on your description, It seems the SAM performance is not very good.

    As far as I know ,the storage pool is based on device hardware , if physical level's performance is not very well. Storage pool cannot improve performance too much.

    So I would suggest contact the vendor of SAN for further help.

    Best Regards,

    Candy


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com   

    Thursday, November 14, 2019 2:57 AM

  • Based on your description, It seems the SAM performance is not very good.

    As far as I know ,the storage pool is based on device hardware , if physical level's performance is not very well. Storage pool cannot improve performance too much.

    So I would suggest contact the vendor of SAN for further help.

    Hey Candy, thanks for the reply!

    I'm sorry, I should have clarified, this is not a 3rd party SAN, this is a Microsoft Windows Server 2019 box with directly attached (SATA) storage operating as an iSCSI Target Storage Provider, with the following specs:

    CPU: Intel i5 750

    RAM: 8GB 1333MHz

    Storage via PERC H310 (in JBOD mode):

    • 4 X 4TB HDD

    Storage via Motherboard:

    • 3 X 480GB SSD

    Initially, connecting to this "SAN" benchmarked at 250MB/s read, 100MB/s write, from a pair of Hypervisors (also Server 2019 machines) via iSCSI. This held true for some time, I was able to push files onto the SAN at these speeds. The connection was through 4 MPIO-enabled gigabit ethernet links. However, at one point, sometime after migrating the 3rd VM, the Windows box providing the storage essentially stopped functioning. My VMs froze, and I was unable to pull files off the "SAN" using the failover file server on the Hypervisors that I had used to load files onto the machine. I have since disabled all iSCSI access to the machine, RDP'd into it, and started manually copying my most important data off the server, and have not been able to exceed 8MB/s. It's estimated to take about 2 weeks. There is no other program accessing the disk, it's this file recovery attempt.

    I have a screenshot, but am unable to post it, of Task manager displaying 65MB/s read and 65MB/s write on the "Microsoft Storage Space Device". I've since rebooted the server, and this has disappeared, and now I only see the files being pulled off the drive.

    What I want to know now is, one, how do I recover my data in a more reasonable time, and how do I prevent Microsoft Storage Spaces from accepting all my data, and then refusing to give it back at a reasonable speed?

    Thursday, November 14, 2019 6:01 PM
  • Hi ,

    >> how do I recover my data in a more reasonable time, and how do I prevent Microsoft Storage Spaces from accepting all my data, and then refusing to give it back at a reasonable speed?

    Please understand, recover data is beyond our forum support. If your issue is urgent ,I would suggest you open a case with Microsoft, more in-depth investigation can be done so that you would get a more satisfying explanation and solution to this issue. 

    Here is the link:

    https://support.microsoft.com/en-us/help/4051701/global-customer-service-phone-numbers

    Best Regards,

    Candy


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com   

    Friday, November 15, 2019 1:55 AM