none
Drive performance with hyper-v cluster

    Question

  • I have a Dell VRTX which contains 4 server blades each running Server 2016 and internal shared storage which is made available via cluster shared volumes.

    The server has a mix of drives as its been upgraded over time:-

    4 x 600GB 10k SAS in RAID 10

    5 x 600GB 10k SAS in RAID 5

    9 x 900GB 10k SAS in RAID 5

    4 x 1TB 7.2k SAS in RAID 5

    I have run some disk performance tests using ATTO Disk Benchmark from within various VM's (Some Server 2012 R2 and some 2016). Some VM's are on different drive arrays (CSV's)

    I am getting pretty consistent results from different VM's on the same arrays, so I'm comfortable the results are correct.

    I still have some older Server 2012 R2 VM's which give averages of 900MB/s write and 1.7GB/s read on the RAID 10 array. These would have been Gen 1 original VM's.

    I have one Server 2016 VM which was in-place upgraded from 2012 R2, still original IDE controller as first boot drive. This shows speeds of 1.1GB/s write and 2.46GB/s read.

    I then created a brand new Server 2016 VM in the cluster, storing it's VHDX also on the RAID 10 array. This had SCSI as the boot drive type and gave averages of 3.2GB/s write and 3.33GB/s read.

    I found these results interesting as it implied that server 2016 was faster on the same hardware, but also that the upgraded server was not as quick as a new one. Could this be as a result of fragmentation, deduplication, or the upgrade process?

    So I then created some new small (only 20-30GB in size) VHDX files for both these VM's, creating one on each of the drive arrays to test with.

    With the upgraded 2016 server I was now seeing speeds of 5.75GB/s write and the same for read. Similar speeds were encountered with the brand new Server 2016 VHDX's.

    I also tried a new VM running Server 2019 and saw similar speeds (So some consistency here)

    So it appears that my old Server 2012 R2 VM's and the upgraded 2016 VM are experiencing much slower disk throughput than either a new Server 2016 VM or a new VHDX assigned to them.

    Any thoughts as to the reasons? Disk fragmentation over time, dedplication or something else?

    My plan is to run the in place upgrade on the remaining Server 2012 R2 servers to make them 2016, however I would like to fix this performance problem as part of this, to ensure they are all running as fast as they can be.

    Wednesday, January 9, 2019 1:53 PM

All replies

  • Here's some more information to backup my issue:-

    This is a file server VM running Server 2016. It was originally 2012R2 but I did an in place upgrade on it.

    C: drive is obviously IDE as this was created in 2012 R2 on a 2012 R2 cluster.

    D: drive is on a different storage spaces drive, different physical set of RAID disks.

    H: drive is a new drive I have just created, on same physical RAID array and CSV as D:, both are SCSI in the VM settings.

    I have included a screenshot of the drive types below, all are formatted as NTFS.

    The only difference, apart from the fact the server and drives were upgraded from 2012, is that there is deduplication running on the D: drive VHDX.

    I do not understand why there is such a dramatic speed difference between these drives.

    Thursday, January 10, 2019 9:10 AM
  • And for comparison, here are the details of the brand new VM I created, 2016 from scratch.

    These are using the same drives, CSV's etc as the VM above, but with quite dramatic differences in speed.

    I have run the speed tests on a number of my other file server VM's and am finding comparable results and issues. Some of those are still running Server 2012 R2.

    For some reason new VHDX files are running much, much faster than the old ones.

    Thursday, January 10, 2019 9:14 AM
  • Hi,

    Thanks for your question.

    Please check the following articles which talked about how to optimize Hyper-V VM performance to see if it helps.

    23 Best Practices to improve Hyper-V and VM Performance

    95 Best Practices for Optimizing Hyper-V Performance

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    Highly appreciate your effort and time. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, January 10, 2019 9:28 AM
    Moderator
  • Thanks for the links Michael

    I have read through them and made some notes to check. However, nothing I have read explains why an existing VHDX performs much worse than a new one I have just added to the server.

    This is the core thing I wish to resolve.


    • Edited by PMThomas Thursday, January 10, 2019 1:28 PM
    Thursday, January 10, 2019 12:26 PM
  • "4 server blades each running Server 2016 and internal shared storage which is made available via cluster shared volumes."

    How is the internal storage shared?  Microsoft does not support such a configuration unless it is in an Storage Spaces Direct environment.  Since you state that the storage is configured in on RAID arrays, and S2D does not support RAID controllers, that means you must be using something else to be able to present local storage as shared storage.


    tim

    Thursday, January 10, 2019 2:45 PM
  • Hi Tim

    Its a quite unique hardware solution from Dell, there are 2 shared PERC controllers that manage the internal drive arrays. Each blade server has drivers installed for the PERC controller and from memory MPIO is used.

    VRTX Guide for Hyper-V Cluster

    This is a supported installation scenario.

    It did take me a while to get my head around how it all went together, but we have been running successfully for 3 years now. Originally on 2012 R2, but then migrated to 2016.

    The VRTX is a good one-box solution for a Hyper-V Cluster, although I think it was originally envisaged for branch office scenarios.



    • Edited by PMThomas Thursday, January 10, 2019 3:04 PM
    Thursday, January 10, 2019 2:59 PM
  • Hi,

    Thanks for your detailed information.

    For now, I’m wondering of more information about this issue so that we can find more clue.

    Are these existing VHDX all stored in the Drive arrays on the PERC controllers?

    What data are stored in these existing VHDX? Or I mean what are these VHDX used for in the VMs?

    What data are stored in the Storage? All are VHDX?

    Highly appreciate your effort and time. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, January 11, 2019 4:15 AM
    Moderator
  • Hi Michael

    Yes, all VHDX files are stored on the CSV's on the RAID, through the PERC.

    The coloured bars at the top of each disk test explains which RAID array they are on. Same colour = same array/CSV.

    New disks(VHDX) added to the exact same array/PERC/CSV are faster than existing.

    The VHDX's are used for user files as part of a Fileserver VM.

    All files on the storage are either VM's and their associated files or VHDX's for the VM's.

    I am doing a series of further tests, ensuring latest MS updates installed on host and VM. The restarting both, checking speed tests again etc.

    If you can think of anything else for me to test, please let me know.

    Friday, January 11, 2019 11:38 AM
  • Hi,

    This is a quick note to let you know that I am currently performing research on this issue and will get back to you as soon as possible. I appreciate your patience.

    If you have any updates during this process, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Wednesday, January 16, 2019 10:05 AM
    Moderator
  • Morning Michael, thank you for your time on this topic :)

    I am doing extensive testing at the moment to try and improve performance.

    The primary server I am testing with is simply a VM File Server for sharing software we use within IT.

    I have taken a copy of it's D: drive VHDX which contains all of the shared files.

    Mounted this onto a new Server 2016 VM and had similar performance results.

    I have gone through the lengthy procedure of removing deduplication within the VM and then un-deduplicating the data. Then done a defrag on this. The file wouldn't shrink back down. It's a dynamicaly expanding VHDX, set to 1TB max. It was sitting at just under 1TB in size with only 414GB of data in it (no deduplication).

    After a lot of faffing about I managed to compact the file and bring it down to 451GB, as shown on the Host CSV.

    Running further tests gave me improved performance:-

    I am now going through the process of compacting the original VHDX, but leaving the data deduplicated as there is a 41% saving on this due to the nature of all the files and software stored on it.




    Wednesday, January 16, 2019 10:18 AM
  • Ok, just finished compacting the original VHDX (Obviously had to shut down VM):-

    Mount-VHD xxxx -ReadOnly

    Optimize-VHD xxxx -mode Full

    Dismount-VHX xxxx

    Here are the test results, as you can see much, much better than originally:-

    I have a number of other VHDX files that are performing similarly slow, however, these are live VM's and I cannot just take them down to test with.

    I have some new storage coming in the next few weeks, so will be able to make copies of the Live VM's to test and play with.

    So optimizing the VHDX files does seem to massively improve performance, but I don't understand why?

    Making a copy of the VHDX to another CSV (completely different drive and physical array) showed the same performance problems. This is all Dell hardware RAID. The only common denominator is the VHDX itself.

    It appears the act of Compacting (Optimizing) a VHDX improves the performance, regardless of being deduplicated or not

    Are there any PowerShell commands we can run on live running VM's to analyse the VHDX structure, capacity, etc and then compare results?


    • Edited by PMThomas Wednesday, January 16, 2019 10:36 AM
    Wednesday, January 16, 2019 10:35 AM
  • Hi,

    Thanks for your detailed update.

    Here're articles talked about getting VHD info using Powershell command, hope it can helpful to you. 

    https://www.altaro.com/hyper-v/gathering-vhd-info-get-vhd-powershell/

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information. 

    https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/hyper-v-server/storage-io-performance

    Highly appreciate your effort and time. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, January 17, 2019 7:46 AM
    Moderator
  • I have captured the output from those commands, to be honest it doesn't look very detailed, but here it is anyway.

    IT Software server is the one I have sped up by compacting the D: drive, its a 1TB dynamic drive with 754GB free, deduplication enabled within the VM and saves 41% (175GB) of space.

    The other two VM's are both slow, these are all a few years old now. 

    Groups server has a 2TB dynamic D: drive with 297GB free, deduplication is enabled within the VM and saves 43% (1.3TB) of space.

    Archive server has a 500gb D: dynamic drive with 411GB free, deduplication enabled within the VM and saves 41% (63.5GB) of space.

     

    The fragmentation doesn't look huge to me, you may think differently.

    Are there any other parameters I can capture and compare?

    Thursday, January 17, 2019 1:23 PM
  • Hi,

    Thanks for your detailed information.

    For now, I couldn't find any clue about this issue. I'm afraid you might need to contact  Microsoft Customer Support Services (CSS) so that a dedicated Support Professional can help you on this issue.

    To obtain the phone numbers for specific technology request, please refer to the website listed below:

    https://www.microsoft.com/en-us/worldwide.aspx

    Appreciate your support and understanding.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Wednesday, January 23, 2019 5:58 AM
    Moderator