none
Error: "hv_vmbus: Buffer too small..." - how to increase buffer in linux IS/driver? RRS feed

  • Question

  • Hi everybody!

    Since I am getting the error message above if I put heavy load on my "virtual network" of the hyper-v server/VMs, I wonder whether there is a way to increase the vmbus buffer in the relevant linux hyper-v drivers?

    Here are more details why I believe increasing the buffer may solve my errors:

    The Server:

    Key hardware: Xeon E3-1225 v3, 32GB ECC Ram, Intel i350-T4 NIC, Dell Powerraid Perc H200

    Host-OS: Hyper-V Server Technical Preview (Windows 10 base?), version 6.4.9841

    The Server runs mainly 2 VMs: 1x Datacenter Server Technical Preview (also version 6.4.9841) ("Windows-VM") and 1x Linux Ubuntu Server ("Ubuntu-VM", which is giving the above error).

    The "problematic" Ubuntu-VM:

    Gen2, 4 virtual processors, 1x VHDX BootDisc, 4x4TB HDD (connected to the H200, forwarded to the VM as "physical drives"), 1x NIC@i350-T4 (external switch, not shared, 192.x.x.7), 1x NIC@PrivateNetwork (10.x.x.5)

    Dynamic RAM: Startup 8192MB, Min 8192MB, Max 16384MB

    Guest-OS: Ubuntu 14.04.2 LTS 3.16.0-34-generic x86_64 

    Note: ZFS on Linux installed, ZFS-filesystem (raidz) on the 4x4TB HDD (mainly managed through napp-it, has limitations though)

    I set up network filesharing on the ubuntu-server for both networks, i.e. I can connect to the share both through the external network (192.x...) as well as via the hyper-v private switch (10.x...). For testing purposes I mapped the share in the Windows-VM as drive N:

    The "problem":

    If I put heavy load on the "10Gbit virtual NIC" of the Ubuntu-VM (e.g. by running CrystalDiskMark in the Windows-VM on the mapped ZFS drive N:), the screen of the Ubuntu-VM spits pages of errors like this: 

    [87202.240500] hv_vmbus: Buffer too small - needed 3056 bytes but got space for only 2048 bytes

    The numbers in the [] constantly increase. the "needed" bytes also vary from time to time, available space is always 2048 bytes.

    However, CrystalDiskMark completes the std-test without complaints, speeds are nothing to complain about (in my view, whatever that is worth):

    Seq: 734.2read 349.5write

    512K: 606.2read 468.9write

    4k: 28.11read 15.06write

    4kQd32: 28.41read 15.44

    If I run the tool on a 2nd physical PC (Win7) connected to the share through the "real network", I do not get any errors in the Ubuntu-VM (speeds are way slower though). 

    My conclusions:

    It seems that the std. buffer size was not meant to deal with heavy load on a 10Gbit network, at least not a virtual one internally connecting only VMs running all on the same physical server, and at least not while being accessed through the SMB/NFS protocol (dunno whether that makes any difference though).

    I found a rather old technet posting (http://blogs.technet.com/b/winserverperformance/archive/2010/02/02/increase-vmbus-buffer-sizes-to-increase-network-throughput-to-guest-vms.aspx), which seemed to somewhat relate to my problem. Unfortunately, I couldn't find any ressources how to increase the buffer size as described in the blog for a Linux system...

    On a side note: I am happy and quite proud of myself that I managed to get ZFS running a) on linux and b) under Hyper-V! I am not an IT-pro and nothing described above is meant for or used as production system. But still I hope somebody takes an interest and could point me in the right direction. Yes, I am willing to recompile stuff and do a lot (more) trial-and-error exercises, whatever it takes...

    Monday, April 20, 2015 11:18 PM

Answers

  • The good news is that the message "[87202.240500] hv_vmbus: Buffer too small - needed 3056 bytes but got space for only 2048 bytes" is informational and not an indication of a serious problem.

    The kernel will automatically increase the ring buffer size if needed to receive the incoming vmbus message. You can safely ignore this and we will submit a patch request to the Ubuntu folks to increase the default buffer size and decrease the frequency of seeing messages like this in the future.

    The number in brackets is a timecode. You do not need to be concerned that it is increasing.

    You do not need to increase any kernel parameters and make a custom kernel to work around this, it should be working.

    • Marked as answer by Besterino Thursday, April 30, 2015 6:36 AM
    Wednesday, April 29, 2015 6:00 PM

All replies

  • The good news is that the message "[87202.240500] hv_vmbus: Buffer too small - needed 3056 bytes but got space for only 2048 bytes" is informational and not an indication of a serious problem.

    The kernel will automatically increase the ring buffer size if needed to receive the incoming vmbus message. You can safely ignore this and we will submit a patch request to the Ubuntu folks to increase the default buffer size and decrease the frequency of seeing messages like this in the future.

    The number in brackets is a timecode. You do not need to be concerned that it is increasing.

    You do not need to increase any kernel parameters and make a custom kernel to work around this, it should be working.

    • Marked as answer by Besterino Thursday, April 30, 2015 6:36 AM
    Wednesday, April 29, 2015 6:00 PM
  • Thank you, this is a relief. The error only appears if SENDING data to the VM, read is ok. Is there a mailing list or website or other where I could follow the development of the linux hyper-v drivers more closely?
    Thursday, April 30, 2015 6:36 AM
  • We plan to be more active posting announcements to this forum as well as the Virtualization Blog http://blogs.technet.com/b/virtualization/ in the future, as well as continually updating the "Linux and FreeBSD Virtual Machines on Hyper-V" https://technet.microsoft.com/en-us/library/dn531030.aspx pages.


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Friday, May 1, 2015 4:18 PM