none
Strange blazing-fast IO performance inside Hyper-V VM

    Question

  • Hello!

    We have just set up couple of clusters:

    - One cluster for VMs: 7 nodes 6x1Gbps NetAdapters (team), 160 shared disks in JBODs, connected through LSI SAS Switch (disks are in mirrored storage spaces)

    - One cluster for SoFS: 2 nodes, 2x1Gbps (team) + 2x10Gbps NetAdapters, 60 shared disks in JBOD (disks are in mirrored storage spaces too).

    OS on hosts and guests is Win 2012 R2. Guests are Generation1 VMs.

    Before using them in production and decide, which variant is better/cheaper/easier to deploy/etc, we want to test these scenarios:

    - IO performance inside VM when virtual disk located on CSV, Direct Mode

    - IO performance inside VM when virtual disk located on CSV, Redirected Mode

    - IO performance inside VM when virtual disk located on SoFS

    We expected moderate or, maybe, worse performance (especially in last two scenarios), bus unexpectedly it was blazing-fast and just theoretically unreachable. We have spent almost a week, searching for any explanations with no success :(

    So, we're testing virtual SCSI disk (500 Gb, dynamic) with SQLIO.

    First - disk on CSV, Direct mode (VM host and owner of CSV is the same node)

    random write 8Kb IO:

    IOs/sec: 239856.70
    MBs/sec:  1873.88

    Avg_Latency(ms): 0

    This CSV is two-way mirror created on pool of 24x2.5 600Gb SAS disks. I just can't believe in 250K random 8kb write IOPS on this array. Other hdd performance measurement tools report same counters.

    Then, when I move CSV to another node, leaving VM itself where it was (as I understand - this forces use of redirected mode for operations on that CVS).

    Almost nothing changes in IOPS/throughput counters, but I see network activity on VM host and CSV owner. However, it is just 60-80 Mbps!

    Finally, the SoFS.

    random write 8Kb IO:

    IOs/sec: 125737.93
    MBs/sec:   982.32

    Avg_Latency(ms): 0

    SQLIO shows almost 1Gigabytes/s throughput, but Network usage on SoFS node is just 160-180 Mbit/s.

    Using sqio with 256KB sequential writing IO shows 5829.57 MB/s, but just 1Gbps of network activity (it's a limit, because 10Gbps adapters and SMB multichannel were not set up).

    I've tried to test SoFS share and CSV directly from host and got more realistic results:

    random write 8Kb IO on \\sofs1\share1\test\testfile.dat:

    IOs/sec:  4931.03
    MBs/sec:    38.52

    Avg_Latency(ms): 18

    random write 8Kb IO on C:\ClusterStorage\pool1vd1\test\testfile.dat

    IOs/sec:  2206.29
    MBs/sec:    17.23

    Avg_Latency(ms): 42

    So, can anybody explain these strange results, when performing IO inside VM?

    I can run any other test or post any specific performance counters/screenshots. I would like to understand, what’s happening before moving to production.

    Thanks in advance,

    Best regards, Dmitry.

    Tuesday, February 18, 2014 10:26 PM

Answers

All replies

  • "Then, when I move CSV to another node, leaving VM itself where it was (as I understand - this forces use of redirected mode for operations on that CVS)."

    Not true.  VMs have direct access to the CSV as long as the host has a working physical connection to the CSV.  Moving its ownership from one node of a cluster to another node of a cluster should not impact the performance of the VM in any way if you are using fixed size disks.  There will be a small, unnoticeable without measurement tools, difference in performance if using dynamic disks.  Redirected means that the host has lost physical access to the disk and must perform its IO across the network.  Simply moving the ownership like you did does not cause a failure of the original host's connection.

    What sort of cacheing is in place on your disks?


    .:|:.:|:. tim

    Wednesday, February 19, 2014 1:32 AM
  • Hi, Tim!

    As I've mentioned, we're using mirrored storage spaces. This is asymmetric storage and so, all IO should be performed through coordinator node (that is CSV owner, I assume).

    PS C:\> Get-ClusterSharedVolumeState -node HVC1N5 | ?{$_.volumefriendlyname -eq "pool1vd1"}


    Name                         : Cluster Virtual Disk (test1)
    VolumeName                   : \\?\Volume{ecb328a3-3fea-461f-8d1e-4744af50e4ac}\
    Node                         : HVC1N5
    StateInfo                    : BlockRedirected
    VolumeFriendlyName           : pool1vd1
    FileSystemRedirectedIOReason : NotFileSystemRedirected
    BlockRedirectedIOReason      : StorageSpaceNotAttached

    HVC1N5 is VM Host. "Pool1vd1" - name of virtual disk, where CSV is located.

    Owner of the CSV is node HVC1N2:

    PS C:\> Get-ClusterSharedVolumeState -node HVC1N2 | ?{$_.volumefriendlyname -eq "pool1vd1"}


    Name                         : Cluster Virtual Disk (test1)
    VolumeName                   : \\?\Volume{ecb328a3-3fea-461f-8d1e-4744af50e4ac}\
    Node                         : HVC1N2
    StateInfo                    : Direct
    VolumeFriendlyName           : pool1vd1
    FileSystemRedirectedIOReason : NotFileSystemRedirected
    BlockRedirectedIOReason      : NotBlockRedirected

    Physical disks are just simple SAS disks (Toshiba AL13SEB600, they have 64Mb buffer) in JBODs. No caches/BBUs on HBAs. No SSDs.

    There might be a CSV cache, but memory for CSV Cache is not configured (so caching is disabled):

    PS C:\> Get-ClusterSharedVolume -name "Cluster Virtual Disk (test1)" | Get-ClusterParameter -name EnableBlockCache | fl

    Name          : EnableBlockCache
    IsReadOnly    : False
    ParameterType : UInt32
    Value         : 1

    PS C:\> get-cluster| select BlockCacheSize |fl

    BlockCacheSize : 0

    But still, cache does not explain throughput of 5829.57 MB/s over 1Gbps network and getting realistic results while running test directly on hosts, but not inside VMs. Looks like Hyper-V's storage subsystem uses some kind of IO batching and data compression while reading/writing to storage, but I haven't heard about such technologies.

    Wednesday, February 19, 2014 6:28 AM
  • Hi Demonixed ,

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
    Thank you for your understanding and support.

    Best regards
    Elton Ji

     

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.

    Monday, February 24, 2014 9:09 AM
    Moderator
  • Hi, Elton!

    I've just noticed, this happens on dynamic VHD's only.

    If I use fixed VHD (VHDX) - IO performance is predictable (8kb random writes, 12 threads 8 outstanding i/o requests):

    throughput metrics:
    IOs/sec:  2657.87
    MBs/sec:    20.76
    latency metrics:
    Min_Latency(ms): 1
    Avg_Latency(ms): 35
    Max_Latency(ms): 517

    If I use dynamic VHD - IO performance is strange (8kb random writes, 12 threads 8 outstanding i/o requests):

    throughput metrics:
    IOs/sec: 276689.26
    MBs/sec:  2161.63
    latency metrics:
    Min_Latency(ms): 0
    Avg_Latency(ms): 0
    Max_Latency(ms): 24

    Both disks are on the same file share and were created with same maximum size (40 Gb). Btw, size of SQLIO's test file is 35 gigabytes, but size of corresponding dynamic vhd file is just 100 megabytes. So, vhd file wasn't expanded. I think, that's because SQLIO uses file filled with zeroes, which not causes dynamic vhd file to expand. Looks like fast throughput is due to some kind of IO compression :)

    But I still can't find any official explanation of these things.


    • Edited by Demonixed Monday, March 03, 2014 3:50 PM
    Monday, March 03, 2014 3:50 PM
  • Hi,

    Please refer to the following link:

    http://technet.microsoft.com/en-us/library/dd723635(v=office.12).aspx

    Best Regards,

    Vincent Wu


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Wednesday, March 05, 2014 2:12 AM
  • Hi, Vincent!

    Thanks for reply, but I know how to monitor disk performance via perfmon (it shows same high values for IOPS and throughput). why I'm seeing these huge numbers?".

    Underlying storage is 30 * 15k disks. So there can't be 280k *real* IOPS.

    My initial problem was - to check performance of HyperV over SMB3.0 solution with teamed 1Gbps network adapters. I observed strange behavior and asked question here.

    Frankly, I've never tested virtual machine's disk with sqlio before. I've tried to do this last week in VM, hosted on Win 2008 R2 and I've seen similar results (Numbers were smaller, but still unreal).

    Tuesday, March 11, 2014 10:15 AM
  • If you are using Disk Benchmarking tools such as SQLIO \ DISKSPD\ DISKBASH. These tools write zeros to a file to simulate IO which in case of Dynamic Disk are perceived as Zeros and are optimized by simply updating the metadata for the VHD and hence not committing any IO to the underlying disk. You may want to run these tests by copying a file such as a Video to the Dynamic Disk which shall show real numbers.
    Thursday, June 04, 2015 12:44 PM