Storage Spaces Direct (S2D) - Poor write performance with 5 nodes with 24 Intel P3520 NVME SSDs each over 40Gb IB network


  • Need a little help with my S2D cluster which is not performing as I had expected.


    5 x Supermicro SSG-2028R-NR48N servers with 2 x Xeon E5-2643v4 CPUs and 96GB RAM

    Each node has 24 x Intel P3520 1.2TB NVME SSDs

    The servers are connected over an Infiniband 40Gb network, RDMA is enabled and working.

    All 120 SSDs are added to S2D storage pool as data disks (no cache disks). There are two 30TB CSVs configured with hybrid tiering (3TB 3-way mirror, 27TB Parity)

    I know these are read intensive SSDs and that parity write performance is generally pretty bad but I was expecting slightly better numbers then I'm getting:

    Tested using CrystalDiskMark and diskspd.exe

    Multithreaded Read speeds: < 4GBps (seq) / 150k IOPs (4k rand)

    Singlethreaded Read speeds: < 600MBps  (seq) 

    Multithreaded Write speeds: < 400MBps  (seq) 

    Singlethreaded Write speeds: < 200MBps (seq) / 5k IOPS (4k rand)

    I did manage to up these numbers by configuring a 4GB CSV cache on the CSVs and forcing write through on the CSVs:

    Max Reads: 23GBps/500K IOPs 4K IOPS, Max Writes:2GBps/150K 4KIOPS

    That high read performance is due to the CSV cache which uses memory. Write performance is still pretty bad though. In fact it's only slight better than the performance I would get for a single one of these NVME drives. I was expecting much better performance from 120 of them!

    I suspect that the issue here is that Storage Spaces is not recognising that these disks have PLP protection which you can see here:

    Get-storagepool "*S2D*" | Get-physicaldisk |Get-StorageAdvancedProperty
    FriendlyName          SerialNumber       IsPowerProtected IsDeviceCacheEnabled
    ------------          ------------       ---------------- --------------------                   
    NVMe INTEL SSDPE2MX01 CVPF7165003Y1P2NGN            False                     
    WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
    NVMe INTEL SSDPE2MX01 CVPF717000JR1P2NGN            False                     
    WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.
    NVMe INTEL SSDPE2MX01 CVPF7254009B1P2NGN            False                     
    WARNING: Retrieving IsDeviceCacheEnabled failed with ErrorCode 1.

    Any help with this issue would be appreciated.


    joi, 12 iulie 2018 14:50

Toate mesajele

  • Hi,
    Based on the complexity and the specific situation, we need do more researches. If we have any updates or any thoughts about this issue, we will keep you posted as soon as possible. Your kind understanding is appreciated. If you have further information during this period, you could post it on the forum, which help us understand and analyze this issue comprehensively.
    Sorry for the inconvenience and thank you for your understanding and patience.
    Best Regards,


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact

    vineri, 13 iulie 2018 07:59
  • Did  you ever resolve this?  We have struggled with poor S2D performance with SuperMicro servers, and I've just realized that all the Intel DC SSDs which have PLP are not recognized as such - return to query is the same as yu are seeing.


    luni, 27 august 2018 19:38
  • Also checking in to see if this has been resolved yet, as I too am running into this issue with S2D not recognizing that my Intel DC S3610 have PLP. This is killing my lab deployment, which in turn is also killing my (would-be) prod deployment.

    sâmbătă, 27 octombrie 2018 06:09
  • I also have the same symthoms. My Virtual Disk is not using Cache(Jurnal) devices and advanced properties returns the same error.
    marți, 20 noiembrie 2018 10:01