none
PCIe SSD + Storage Spaces = HUGE read performance drop? RRS feed

  • Question

  • So I'm seeing some strange, slow behavior with a simple Storage Spaces pool with only a pair of mirrored drives.  I've seen lots of folks having problems with SS, but most are issues with parity spaces.

    I've got a new Dell R730xd with 4 of their 400GB nvme ExpresFlash SSDs.  The machine also has a PERC card with a bunch of 15K rpm SAS drives and will be used for SQL Server. (and a pair of E5-2643v3 xeons and 256GB ram)

    This is running Windows Server 2012R2 with all the latest patches.  I've used both Storage Spaces / pools and traditional windows disk management to create a mirrored pair of these drives.  I also tested a single drive by itself.  The SS volume actually showed huge promise for a moment where it was obviously reading simultaneously from both drives.  These are some of the fastest SSDs available now, so these results for just using a pair of them are pretty awesome.  Using the old-school windows disk mirroring does show a performance penalty vs. a bare drive, but it's not drastic, so it's probably what I'll be forced to use unless I can fix the problem.

    *note I later rebuilt this pool using a 4k physical & logical sector size hoping that was the culprit, but it didn't help.  It sorta looks like I'm running into some PCIe bus contention, possibly overloading the backplane. *shrug*

    You can see the PS syntax used to create this here:
    photobucket image

    And here's what a single drive looks like by itself:

    Saturday, April 25, 2015 4:41 AM

Answers

  • Thanks adminxyz, the new firmware/drivers for the SSDs changed the UniqueID reported by the drives so that they're actually unique and, therefore, can be managed by the Storage Spaces GUI. 

    So the read problems all appear to be flaws with ATTO.  I've been doing a variety of tests with SQLIO instead and it doesn't seem to have any bottlenecks.  In fact, I put all 4 drives in a single storage pool & created a mirrored volume with the following results... which are borderline insane :)

    PS C:\Program Files (x86)\SQLIO> .\sqlio -kR -t48 -s120 -dK -o8 -fsequential -b64 -BH -LS Testfile.dat

    sqlio v1.5.SG

    using system counter for latency timings, 3320312 counts per second

    48 threads reading for 120 secs from file K:Testfile.dat

            using 64KB sequential IOs

            enabling multiple I/Os per thread with 8 outstanding

            buffering set to use hardware disk cache (but not file cache)

    using current size: 24576 MB for file: K:Testfile.dat

    initialization done

    CUMULATIVE DATA:

    throughput metrics:

    IOs/sec: 164569.01

    MBs/sec: 10285.56

    latency metrics:

    Min_Latency(ms): 0

    Avg_Latency(ms): 1

    Max_Latency(ms): 19

    histogram:

    ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

    %:  7 40 31 14  5  2  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

    PS C:\Program Files (x86)\SQLIO> .\sqlio -kR -t16 -s120 -o16 -frandom -b8 -BH -LS K:\TestFile.dat

    sqlio v1.5.SG

    using system counter for latency timings, 3320312 counts per second

    16 threads reading for 120 secs from file K:\TestFile.dat

            using 8KB random IOs

            enabling multiple I/Os per thread with 16 outstanding

            buffering set to use hardware disk cache (but not file cache)

    using current size: 24576 MB for file: K:\TestFile.dat

    initialization done

    CUMULATIVE DATA:

    throughput metrics:

    IOs/sec: 524070.82

    MBs/sec:  4094.30

    latency metrics:

    Min_Latency(ms): 0

    Avg_Latency(ms): 0

    Max_Latency(ms): 5

    histogram:

    ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

    %: 99  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0


    • Edited by xyvyx Thursday, May 14, 2015 8:35 PM needed bold
    • Marked as answer by xyvyx Thursday, May 14, 2015 8:35 PM
    Thursday, May 14, 2015 8:33 PM

All replies

  • Hi,

    Sorry for the delay.

    I'm currently researching on this case and will update when I got enough information. 


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Tuesday, April 28, 2015 5:04 AM
    Moderator
  • Thanks.  Here's some info about the virtual disk and the underlying drives..

    Info reported by powershell Get-VirtualDisk command

    The "disk drives" reported in Device Manager are shown as NVMe Dell Express Fla SCSI Disk Device.
    Driver - Provider: Microsoft, Date: 6/21/2006, Driver Version: 6.2.9600.16384

    The "Storage controller" reported for these drives, however, uses the more recent driver from Samsung.
    The device is listed as Dell Express Flash NVMe XS1715 400GB Controller.  Driver version: 1.4.6.22 dated 5/14/2014, the latest version I found on Dell's support website.

    Within OpenManage, all 4 of the SSDs report this info (again, shows the latest firmware & drivers):

    Physical devices
    ID 0:1:20
    Status OK
    Name Physical Device 0:1:20
    State Ready
    Device Name Not Available
    Bus Protocol PCIe
    Device Protocol NVMe 1.0
    Media SSD
    Remaining Rated Write Endurance 100%
    Firmware Revision IPM0FD3Q
    Driver Version 1.4.6.22
    Model Number Dell Express Flash NVMe 400GB
    Capacity 372.61GB
    Vendor ID Samsung
    Product ID Samsung5197
    Serial No. S1J0NYAF901012
    PCIe Negotiated Link Speed 8.0 GT/s
    PCIe Maximum Link Speed 8.0 GT/s
    PCIe Negotiated Link Width x4
    PCIe Maximum Link Width x4
    Form Factor 2.5 inches
    Sub Vendor Dell

    And they're all plugged into backplane which is in a 16x PCIe Gen3 slot.  
    Maybe Dell has some insight...   

    Wednesday, April 29, 2015 7:58 PM
  • Hi,

    Sorry for the long delay. It takes some time to discuss this issue.

    At these high transfer rates, PCI bandwidth becomes a limitation. You will need to simultaneously benchmark both SSDs, configured as individual volumes to see if you have bottleneck in the system.

    You can also try putting a single PCIe SSD in a Simple VD, so that it uses the Spaces stack. You should test both drives, each one at a time, and each in a Simple Space. There may be an issue with one of the drives or the system.


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Wednesday, May 6, 2015 2:10 AM
    Moderator
  • no problem Shaon, thanks for looking into this. 

    I attempted to run the same benchmark using a variety of disk configurations.  This has lead me to believe the problem lies within Windows and is common to both Storage Spaces & conventional software raid.  

    When I used a software-striped pair of the drives, it exhibited similar read behavior to that of the mirrored SS pool.  For gits & shiggles, I even tried using all 4 disks in a software and SS raid-0 and "2 column, 2 copy" volumes... same behavior.  Just a few minutes ago, I tried running the simultaneous tests as you suggested.  I used a batch file + START command to initiate the tests(at first, I just tried clicking on both start buttons as quickly as possible)... and they both ran perfectly fine w/o any indicate of bus bandwidth limitation.  They both hit ~ 2,700 MB/sec read transfer rates.  I wouldn't have expected us to hit PCIe bus bandwidth limitations just yet given the capabilities of the pcie 3.0 bus and the number of lanes in use, and I think this confirmed it:

    The older software mirror logic doesn't (or so it seems) read from both disks like the SS pool does, so the read transfer rates never approach the limit that causes the bandwidth issues.  

    I also found a very recent review of some of Samsung's new M.2 PCIe drives where they hit the same sort of limitation while running a RAID0 test:
    http://www.legitreviews.com/samsung-sm951-m2-pcie-ssds-raid0-performance_161753/3

    Granted, this was with Windows 8.1 and an X99-based test system, but neither of these systems should be running into these limitations. 

    I found some instructions on how to use XPerf to monitor for filter drivers causing extra latency, but I didn't find anything... no AV software on here yet to slow things down.

    Thursday, May 7, 2015 9:28 PM
  • Hi,

    We think that the NVME devices may explain the 4GB/s throughput. I wonder whether the storage pool includes other kinds of devices as well. If so, I think I can explain the performance drop at 2MB transfer size. Could you please share the output of the following command?

    Get-VirtualDisk VirtualSSD1 | Get-PhysicalDisk | Format-List *


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Monday, May 11, 2015 7:44 AM
    Moderator
  • It's certainly possible the NVMe devices have some outstanding issues... of course part of the reason for their existence is that they remove much of the overhead associated with previous storage protocols like AHCI.

    I know for certain there are only SSDs in this storage pool since I specified the individual disks to use. 
    That said, here is the output from that command:



    PS C:\> Get-VirtualDisk VirtualSSD1 | Get-PhysicalDisk | Format-List *
    
    
    Usage                            : Auto-Select
    OperationalStatus                : OK
    HealthStatus                     : Healthy
    BusType                          : NVMe
    CannotPoolReason                 : In a Pool
    SupportedUsages                  : {Auto-Select, Manual-Select, Hot Spare, Retired...}
    MediaType                        : SSD
    ObjectId                         : {1}\\PROD-DB04-NEW\root/Microsoft/Windows/Storage/Providers_v2\SPACES_PhysicalDisk.ObjectId="{cc689b5e-e87c-11e4-80b4-806e6f6e6963}:PD:{b8f0221b-f440-11e4-80c2-806e6f6e6963}"
    PassThroughClass                 :
    PassThroughIds                   :
    PassThroughNamespace             :
    PassThroughServer                :
    UniqueId                         : NVMe    Dell Express Fla00000000
    AllocatedSize                    : 398895087616
    CanPool                          : False
    Description                      :
    DeviceId                         : 8
    EnclosureNumber                  :
    FirmwareVersion                  : IPM0
    FriendlyName                     : PhysicalDisk8
    IsIndicationEnabled              :
    IsPartial                        : False
    LogicalSectorSize                : 512
    Manufacturer                     : NVMe
    Model                            : Dell Express Fla
    OtherCannotPoolReasonDescription :
    PartNumber                       :
    PhysicalLocation                 :
    PhysicalSectorSize               : 512
    SerialNumber                     : 0000_0000_0000_0000.
    Size                             : 399163523072
    SlotNumber                       :
    SoftwareVersion                  :
    SpindleSpeed                     : 0
    PSComputerName                   :
    CimClass                         : ROOT/Microsoft/Windows/Storage:MSFT_PhysicalDisk
    CimInstanceProperties            : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
    CimSystemProperties              : Microsoft.Management.Infrastructure.CimSystemProperties
    
    Usage                            : Auto-Select
    OperationalStatus                : OK
    HealthStatus                     : Healthy
    BusType                          : NVMe
    CannotPoolReason                 : In a Pool
    SupportedUsages                  : {Auto-Select, Manual-Select, Hot Spare, Retired...}
    MediaType                        : SSD
    ObjectId                         : {1}\\PROD-DB04-NEW\root/Microsoft/Windows/Storage/Providers_v2\SPACES_PhysicalDisk.ObjectId="{cc689b5e-e87c-11e4-80b4-806e6f6e6963}:PD:{b8f0221c-f440-11e4-80c2-806e6f6e6963}"
    PassThroughClass                 :
    PassThroughIds                   :
    PassThroughNamespace             :
    PassThroughServer                :
    UniqueId                         : NVMe    Dell Express Fla00000000
    AllocatedSize                    : 398895087616
    CanPool                          : False
    Description                      :
    DeviceId                         : 9
    EnclosureNumber                  :
    FirmwareVersion                  : IPM0
    FriendlyName                     : PhysicalDisk9
    IsIndicationEnabled              :
    IsPartial                        : False
    LogicalSectorSize                : 512
    Manufacturer                     : NVMe
    Model                            : Dell Express Fla
    OtherCannotPoolReasonDescription :
    PartNumber                       :
    PhysicalLocation                 :
    PhysicalSectorSize               : 512
    SerialNumber                     : 0000_0000_0000_0000.
    Size                             : 399163523072
    SlotNumber                       :
    SoftwareVersion                  :
    SpindleSpeed                     : 0
    PSComputerName                   :
    CimClass                         : ROOT/Microsoft/Windows/Storage:MSFT_PhysicalDisk
    CimInstanceProperties            : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
    CimSystemProperties              : Microsoft.Management.Infrastructure.CimSystemProperties
    
    
    
    PS C:\>
    
    
    Monday, May 11, 2015 3:33 PM
  • On the Homepage from Dell I found an new driver.

    - Added support for additional Dell PowerEdge Express Flash NVMe PCIe SSDs.
    - Added support for Windows Storage Spaces.

    Version

    Version 1.4.6.24, A04

    http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=2CGP0




    • Edited by adminxyz Wednesday, May 13, 2015 3:37 PM
    Tuesday, May 12, 2015 1:57 PM
  • Thanks adminxyz, the new firmware/drivers for the SSDs changed the UniqueID reported by the drives so that they're actually unique and, therefore, can be managed by the Storage Spaces GUI. 

    So the read problems all appear to be flaws with ATTO.  I've been doing a variety of tests with SQLIO instead and it doesn't seem to have any bottlenecks.  In fact, I put all 4 drives in a single storage pool & created a mirrored volume with the following results... which are borderline insane :)

    PS C:\Program Files (x86)\SQLIO> .\sqlio -kR -t48 -s120 -dK -o8 -fsequential -b64 -BH -LS Testfile.dat

    sqlio v1.5.SG

    using system counter for latency timings, 3320312 counts per second

    48 threads reading for 120 secs from file K:Testfile.dat

            using 64KB sequential IOs

            enabling multiple I/Os per thread with 8 outstanding

            buffering set to use hardware disk cache (but not file cache)

    using current size: 24576 MB for file: K:Testfile.dat

    initialization done

    CUMULATIVE DATA:

    throughput metrics:

    IOs/sec: 164569.01

    MBs/sec: 10285.56

    latency metrics:

    Min_Latency(ms): 0

    Avg_Latency(ms): 1

    Max_Latency(ms): 19

    histogram:

    ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

    %:  7 40 31 14  5  2  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

    PS C:\Program Files (x86)\SQLIO> .\sqlio -kR -t16 -s120 -o16 -frandom -b8 -BH -LS K:\TestFile.dat

    sqlio v1.5.SG

    using system counter for latency timings, 3320312 counts per second

    16 threads reading for 120 secs from file K:\TestFile.dat

            using 8KB random IOs

            enabling multiple I/Os per thread with 16 outstanding

            buffering set to use hardware disk cache (but not file cache)

    using current size: 24576 MB for file: K:\TestFile.dat

    initialization done

    CUMULATIVE DATA:

    throughput metrics:

    IOs/sec: 524070.82

    MBs/sec:  4094.30

    latency metrics:

    Min_Latency(ms): 0

    Avg_Latency(ms): 0

    Max_Latency(ms): 5

    histogram:

    ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+

    %: 99  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0


    • Edited by xyvyx Thursday, May 14, 2015 8:35 PM needed bold
    • Marked as answer by xyvyx Thursday, May 14, 2015 8:35 PM
    Thursday, May 14, 2015 8:33 PM