Same disk copy in VM very slow compared to network copy (HyperV 2012) RRS feed

  • Question

  • I have a weird behavior on some of my VMs:

    Host OS Windows Server 2012 with Hyper-V

    Guest OS both 2008R2 and 2012

    Storage Equallogic SAN, iSCSI

    Gigabit ethernet

    The host is connected to a 1 GB Volume where the VHD of the VM are stored

    If I copy a large file (say a template .VHDX)  from the HD of the host, via network, to the drive (a. VHDXfile ) of the guest, transfer speeds are acceptable (start well over 300 MB/s then stabilizes to 80 MB/s), and compatible with the virtual gigabit network card on the VM

    If I copy the same file on the VM (a simple ctrl-c ctrl-v on the same disk, but the same results happen copying from one drive (VHD) to another) the copy speed crawl to a 17 MB/s average and it's generally unstable.

    I've seen many posts about the opposite behavior, slow network speeds and fast internal copy speeds and I can say that I have VMQ disabled on both my host and guest machines.

    What could be the cause of this issue?

    Thanks in advance for any help that you may give.

    Wednesday, July 17, 2013 3:34 PM


All replies

  • Consider your physical layer.

    In which one of these copy operations are you reading from and writing to the same spindles?

    As long as the set of spindles is different, you should see good throughput - if you don't then you have a network issue.

    As long as the set of spindles is the same (same LUN, same array) - you have a disk IO issue.

    The other possibility is to exercise the same storage path in different ways to see if you have a controller / driver issue / array configuration issue.

    Brian Ehlert
    Learn. Apply. Repeat.
    Disclaimer: Attempting change is of your own free will.

    Wednesday, July 17, 2013 3:57 PM
  • Is your HIT Kit installation in the host current?

    You may want to disable ODX and try again. Instructions and more information are in this document (disabling is under step 3): http://technet.microsoft.com/en-us/library/jj200627.aspx

    Eric Siron

    Wednesday, July 17, 2013 7:18 PM
  • Hi Brian,

    In all the copy operations I'm reading and writing to/from the same spindles.

    The VM VHDs and the test file (another VHD, a template one, 11 GB in size) reside on a LUN mapped via iSCSI to the HyperV host.

    So physically I'm reading and writing from the same Volume on the same spindles.

    In the fastest case I'm reading from that volume (same spindles) mounted on the HyperV host accessed via network, in the slowest case I'm reading from that same volume (set of spindles) but from the VHD mounted inside the guest OS.

    I did some other tests, with a dedicated VHD mounted on the guest on the virtual SCSI, same results, fast copy thru the network, slow copy to the dedicated VHD, and also creating a volume (on the same set of spindles) on the SAN and used it as PTD to the guest VM, same results. Even opyng a file from the VHD to the PTD gave the same results.

    It seems that there is an issue in the way the guest OS handles read/writes on its disks, an issue that does not happen when reading from network.,.

    Thursday, July 18, 2013 8:18 AM
  • Hi Eric,

    I believe it's 4.5, I downloaded it when I setup the Equallogic SAN in April.

    Besides I have no trouble accessing the SAN volumes from the host machine.

    I followed your suggestion to disable ODX but I could not see changes in the copy performance.

    Any further info you may need that can help me shed a light on this?



    Thursday, July 18, 2013 12:58 PM
  • I haven't had an Equallogic device of my own since November 12 so I don't know what "current" is. I saw a document from them that seemed to indicate that ODX is, or was, a work-in-progress. To the limits of my ability to parse what you're saying, it really sounds like your issue is related to ODX. This does not sound like a connectivity or accessibility issue to me. With host ODX disabled, do you see any difference between 08 and 12 guests? What happens if you run a straight-up I/O benchmarking tool like SQLIO or IOMeter?

    Eric Siron

    Thursday, July 18, 2013 1:30 PM
  • From the dell/equallogic website 4.5 is current and there's a development 4.6 available, that I don't feel safe to install in a production environment.

    I disabled ODX both in the W2012 host and in the 2012 guest, with no effect (should I reboot the servers for the settings to be effective?)

    I see no difference in performance with ODX disabled on the 2012 guest or the 2008R2 guest

    I'll try IOMETER and let you know the results.

    Thursday, July 18, 2013 1:50 PM
  • If it were me, I'd bounce the systems after making the ODX change. We had to disable ODX for a different issue and the Microsoft engineer assigned to our case instructed us to reboot them. I don't know if that's just a matter of being thorough or not.

    Eric Siron

    Thursday, July 18, 2013 1:59 PM
  • OK, Ill schedule a downtime for tonight, and redo the tests.

    By the way, IOMETER does not recognize the 2 volumes attached to the guest via the virtual SCSI interface, so I'll try with SQLIO for the performance tests.

    Thursday, July 18, 2013 2:34 PM
  • The release version of IOMeter cannot see them, but 1.1.0 RC1 can.

    edit: removed incorrect link, see Brian's comment below

    Eric Siron

    Thursday, July 18, 2013 3:36 PM
  • This link might work better: http://sourceforge.net/projects/iometer/

    Brian Ehlert
    Learn. Apply. Repeat.
    Disclaimer: Attempting change is of your own free will.

    Thursday, July 18, 2013 4:35 PM
  • How are things going? Please let us know if there is any progress.

    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

    Sunday, July 21, 2013 12:32 PM
  • Eric,

    I finally managed to restart the systems during the weekend, after disabling ODX and that seems to have fixed the issues.

    Now network speeds and same disk copy speeds are comparable (around 80 MB/s both in 2008R2 and 2012 OSes)

    Thanks a lot for helping me solve this problem!

    • Edited by Sir Action Monday, July 22, 2013 1:01 PM more typos
    Monday, July 22, 2013 12:51 PM