none
VM virtual disks and read caching RRS feed

  • Question

  • I have a server with abundant memory running Windows Server 2008 R2 and Hyper-V. Through my testing, it seems that for some reason the host OS isn't doing any kind of read caching of .vhd files. If a VM reads from disk, it will always hit the physical disk. The test was performed by creating a Windows 7 guest OS, and running IOMeter from inside it. IOMeter was configured to read from a file 1GB in size - the guest OS only had 1GB of memory but the host had a very large amount of free memory.

    Because the guest OS is so constrained on memory, I wouldn't expect it to do much read caching. But the host OS has plenty of memory. Why doesn't the host OS cache reads to .vhd files? It knows reads are being made to the .vhd, and that the reads are being made to the same place over and over again without any intervening writes. It seems insane not to cache those reads especially when there is a superabundance of free memory available.

    Obviously the host OS can't cache writes, because that would be dangerous. But caching reads never violates consistency. What's the rationale for this? And is there a way to force the host OS to cache reads to .vhd files to improve performance to VMs?

    Saturday, June 9, 2012 5:17 AM

All replies

  • I have a server with abundant memory running Windows Server 2008 R2 and Hyper-V. Through my testing, it seems that for some reason the host OS isn't doing any kind of read caching of .vhd files. If a VM reads from disk, it will always hit the physical disk. The test was performed by creating a Windows 7 guest OS, and running IOMeter from inside it. IOMeter was configured to read from a file 1GB in size - the guest OS only had 1GB of memory but the host had a very large amount of free memory.

    Because the guest OS is so constrained on memory, I wouldn't expect it to do much read caching. But the host OS has plenty of memory. Why doesn't the host OS cache reads to .vhd files? It knows reads are being made to the .vhd, and that the reads are being made to the same place over and over again without any intervening writes. It seems insane not to cache those reads especially when there is a superabundance of free memory available.

    Obviously the host OS can't cache writes, because that would be dangerous. But caching reads never violates consistency. What's the rationale for this? And is there a way to force the host OS to cache reads to .vhd files to improve performance to VMs?

    That's because you need ARC and not LRU caching mechanism implemented :) VM reads are always random so any upcoming read will effectively remove cached block from memory. Think about this situation as about "I/O blender" or "anti-cache" :) So it's NTFS issue say ZFS is very different here.

    You can cache writes just fine. You need however distribute cache between multiple hypervisor nodes (and keep cache coherent of course) and confirm write when you have all caches synchronized. Proper SAN virtualization software works this way :)

    -nismo

    Sunday, June 10, 2012 5:17 PM
  • I have a server with abundant memory running Windows Server 2008 R2 and Hyper-V. Through my testing, it seems that for some reason the host OS isn't doing any kind of read caching of .vhd files. If a VM reads from disk, it will always hit the physical disk. The test was performed by creating a Windows 7 guest OS, and running IOMeter from inside it. IOMeter was configured to read from a file 1GB in size - the guest OS only had 1GB of memory but the host had a very large amount of free memory.

    Because the guest OS is so constrained on memory, I wouldn't expect it to do much read caching. But the host OS has plenty of memory. Why doesn't the host OS cache reads to .vhd files? It knows reads are being made to the .vhd, and that the reads are being made to the same place over and over again without any intervening writes. It seems insane not to cache those reads especially when there is a superabundance of free memory available.

    Obviously the host OS can't cache writes, because that would be dangerous. But caching reads never violates consistency. What's the rationale for this? And is there a way to force the host OS to cache reads to .vhd files to improve performance to VMs?

    OK, have some good news for you :) Upcoming version of Hyper-V is going to have CSV cache. Finally we'll have pinned down logical block level cache! It's still LRU or MRU (@ least I don't see any ARC claims enywhere) and it's write-thru for a reason (no BBU like in RAIDs to store pending transations, no "magic" memory to store log like on NetApp boxes with WAFL and no distributed cache like on properly implemented SANs), but it should and does cache reads. Exactly what you wanted! Please take a look at the following link:

    http://blogs.msdn.com/b/clustering/archive/2012/03/22/10286676.aspx

    and here are some preliminary results (OK, ATTO is not I/O Meter but whetever):

    http://www.thomasmaurer.ch/2012/04/windows-server-8-csv-cache-benchmark/

    Hope this helped :)

    -nismo

    • Proposed as answer by VR38DETTMVP Friday, June 15, 2012 10:52 AM
    Friday, June 15, 2012 10:52 AM
  • But I'm not using clustering, or CSVs. Is this caching available without using clustering?
    Friday, June 15, 2012 4:29 PM
  • But I'm not using clustering, or CSVs. Is this caching available without using clustering?

    From what I understand you need to use CSV to get these benefits...

    There's a workaround for you: you may use iSCSI coming with caching and run in a loopback config. See this thread for more information:

    http://social.technet.microsoft.com/Forums/en-US/winserverfiles/thread/8a48d7e5-ccf4-4af8-bd17-3d9c9d0ca1b3

    -nismo

    Friday, June 15, 2012 8:36 PM
  • That seems like an awful lot of work and third-party software for something so simple that really should be built into the OS...
    Sunday, June 17, 2012 8:31 PM
  • That seems like an awful lot of work and third-party software for something so simple that really should be built into the OS...

    1) There's very little to no work to apply (no pain - no gain)

    2) Referenced software is free in functionality you need

    3) Stuff you're looking for is part of Windows Server 2012

    4) There are A LOT of issues with caching files being actually a containers for file systems (exactly the case for VHDs). Re-read my first post.

    -nismo

    Sunday, June 17, 2012 8:37 PM