none
VMBus Network Adapter Ping bug?

    Question

  • Hello,

     

    I've noticed pinging within a guest OS with the VMBus network adapter seems to be broken:

     

     

    Reply from 1.2.3.4: bytes=32 time=449ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time=449ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time=449ms TTL=128

    Reply from 1.2.3.4: bytes=32 time=449ms TTL=128

    Reply from 1.2.3.4: bytes=32 time=449ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

    Reply from 1.2.3.4: bytes=32 time<1ms TTL=128

     

    These are all pings to the host machine itself.  Of note are the 449ms ping times.  These times actually display instantaneously, and they also seem to incriment the longer the server has been running (I've seen them get up to 3.5s).  This behavior is exhibited regardless of what I'm pinging, be it an internal server, website, or the host itself.  Also, the host machine pings our internal servers without this issue.  Every guest OS on this system has this problem.  Integration components are installed, and TCP/IP offloading is disabled on both physical and virtual network adapters.  This problem was also in RC0. 

     

    Specs:

     

    Dell R900 w/32gigs ram

    VHDs live on a Dell MD3000i iSCSI SAN

    Server 2008 Enterprise

    Hyper-V RC1

    Guest OS is Server 2003 Enterprise SP2

     

    Ideas?  Bug?

     

    -Russ

    Friday, May 23, 2008 10:53 PM

Answers

  • This issue is currently being investigated.
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, May 30, 2008 10:52 PM
  • Hi All,
    We had a similar issue on our HP AMD X2 systems. After puzzling with the same trouble's as with Hyper-V on our HP Blade Servers and Server 2003 I tried to the set affinity option in Task Manager on the AMD X2. It appeared that one of the CPU's was working ok but the other wasn't. And google helped us to a solution for both the problems on the Guest OS Virtual 2003 in Hyper-V and on our clients pc's :).

    The solution is posted here:  http://wiki.ljackson.us/Negative_Ping_Time

    Adding this parameter /usepmtimer to the BOOT.ini on our 2003 servers seems to be working.

    A collegae of mine has asked Microsoft if this is the solution to the problem at hand.

    Best regards Jacco

    • Edited by JvdTol Monday, June 02, 2008 8:16 AM Update
    • Proposed as answer by Pavel Penicka Tuesday, June 03, 2008 8:30 PM
    • Marked as answer by Chang Yin Thursday, June 05, 2008 9:42 AM
    Monday, June 02, 2008 8:14 AM

All replies

  • Russ,

     

    Which version of the VMBus NIC Drivers are being loaded for the Guest NIC's?

     

    (I ask this as I am also having 'some issues' with the guest NIC's in RC1)

     

    In 'Device Manager' are they listed as:- "Microsoft Virtual Machine Bus Network Adaptor" or "Microsoft VMBus Network Adaptor".?

     

    The "Microsoft Virtual Machine Bus Network Adaptor" C:\Windows\System32\Drivers\NETVSC50.sys v6.0.6001.18010 is the correct RC1 Hyper-V Guest NIC Driver whereas the previous "Microsoft VMBus Network Adaptor" C:\Windows\System32\Drivers\NETVSC50.sys v6.0.6001.18004

    Tuesday, May 27, 2008 3:34 AM
  • Hi,

     

    I noticed the same behaviour:

    C:\Documents and Settings\Administrator>ping 192.168.0.1 -t

    Pinging 192.168.0.1 with 32 bytes of data:

    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time<1ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time<1ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time<1ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128

     

    This is a ping from my Windows 2003 SP2 X86 VM to the host OS. It looks like this only occurs on VM's with multiple virtual processors. When rebooting the response time is back to 1 or 2 ms but after some time it increases again. But, my Windows 2008 X64 VM's with multiple virtual processors don't seem to behave this way.

     

    I use a Dell PowerEdge 2950 with 32 GB memory and dual Broadcomm and dual Intel NICs. Furthermore I was talking to somebody over the weekend who saw similar behaviour on his HP servers...

     

    Time for more research I guess.

     

    Cheers,

    Jaap

     

    Tuesday, May 27, 2008 2:51 PM
  •  Stephen Edgar wrote:

    Russ,

     

    Which version of the VMBus NIC Drivers are being loaded for the Guest NIC's?

     

    (I ask this as I am also having 'some issues' with the guest NIC's in RC1)

     

    In 'Device Manager' are they listed as:- "Microsoft Virtual Machine Bus Network Adaptor" or "Microsoft VMBus Network Adaptor".?

     

    The "Microsoft Virtual Machine Bus Network Adaptor" C:\Windows\System32\Drivers\NETVSC50.sys v6.0.6001.18010 is the correct RC1 Hyper-V Guest NIC Driver whereas the previous "Microsoft VMBus Network Adaptor" C:\Windows\System32\Drivers\NETVSC50.sys v6.0.6001.18004

     

    The guest OS NIC is "Microsoft Virtual Machine Bus Network Adapter" which is driver version "6.0.6001.18010".

     

    -Russ

    Tuesday, May 27, 2008 2:56 PM
  •  Jaap Wesselius wrote:

    Hi,

     

    I noticed the same behaviour:

    C:\Documents and Settings\Administrator>ping 192.168.0.1 -t

    Pinging 192.168.0.1 with 32 bytes of data:

    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time<1ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time<1ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=537ms TTL=128
    Reply from 192.168.0.1: bytes=32 time<1ms TTL=128
    Reply from 192.168.0.1: bytes=32 time=538ms TTL=128

     

    This is a ping from my Windows 2003 SP2 X86 VM to the host OS. It looks like this only occurs on VM's with multiple virtual processors. When rebooting the response time is back to 1 or 2 ms but after some time it increases again. But, my Windows 2008 X64 VM's with multiple virtual processors don't seem to behave this way.

     

    I use a Dell PowerEdge 2950 with 32 GB memory and dual Broadcomm and dual Intel NICs. Furthermore I was talking to somebody over the weekend who saw similar behaviour on his HP servers...

     

    Time for more research I guess.

     

    Cheers,

    Jaap

     

     

    It's interesing you're having the same issue.  The issue also "resets" for me after a reboot, and then steadily increases.  Is it possible there is some sort of crazy bug with the virtual core syncing with multiple cores?  Both VMs on this server have multple cores on 2003 x86 SP2.

     

    -Russ

    Tuesday, May 27, 2008 3:01 PM
  • I did a little poking in the task manager with ping.exe.  Here's what I found, at least for a dual virtual core VM:

     

    Setting affinity to CPU 0 for ping.exe causes all of the erroneous pings to be -256ms (the time is arbitrary, these pings all show up instantaneously, like the packets aren't even being sent).  Setting affinity to CPU 1 yields a positive 256ms for the erroneous pings.  These don't show up instantaneously, but rather at a regular interval comperable to the correct <1ms pings.  So it's like the packet is going out and returning, but the wrong results are being displayed. 

     

    Giving affinity to both cores yields a mix of all these results.  I pinged localhost for these tests.

     

    I then decided to try pinging a local server on the network to see if the results were the same (I expected them to be).  They're not.  CPU 1 yields 100% incorrect pings, CPU 0 yields 100% correct pings.  Setting both is, again, a mixed bag.

     

    Not sure if this helps, but I figured I'd post my findings in case it does.

     

    -Russ

    Tuesday, May 27, 2008 3:15 PM
  • This issue is currently being investigated.
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, May 30, 2008 10:52 PM
  • Hi All,
    We had a similar issue on our HP AMD X2 systems. After puzzling with the same trouble's as with Hyper-V on our HP Blade Servers and Server 2003 I tried to the set affinity option in Task Manager on the AMD X2. It appeared that one of the CPU's was working ok but the other wasn't. And google helped us to a solution for both the problems on the Guest OS Virtual 2003 in Hyper-V and on our clients pc's :).

    The solution is posted here:  http://wiki.ljackson.us/Negative_Ping_Time

    Adding this parameter /usepmtimer to the BOOT.ini on our 2003 servers seems to be working.

    A collegae of mine has asked Microsoft if this is the solution to the problem at hand.

    Best regards Jacco

    • Edited by JvdTol Monday, June 02, 2008 8:16 AM Update
    • Proposed as answer by Pavel Penicka Tuesday, June 03, 2008 8:30 PM
    • Marked as answer by Chang Yin Thursday, June 05, 2008 9:42 AM
    Monday, June 02, 2008 8:14 AM
  • Hi All,

    I have tested this on a Windows 2003 R2 SP2 32-Bits and a Windows 2003 R2 SP2 64-Bits.
    The two servers are running on a Hyper-V 2008 server.
    The performance is stable and looks like they working fine now. It seems the solution for the problem.



    Best regards Gerben.

    Monday, June 02, 2008 9:13 AM
  • I am also seeing this issue with 2003 SP2 VM's and NO issues with any 2008 VM's.

    After rebooting the 2003 VM and doing a ping test again the times were in the range from <1ms - 9ms whereas before the reboot the 'long' ping times were ~800ms and that server had an uptime of ~15 days.

    I am also not using an AMD processor and am using an Intel E6600.


    Cheers, Stephen Edgar
    Tuesday, June 10, 2008 10:39 PM
  • Hi,

    I have run in to same issue with Server 08/64 as a host and WinXP SP3 as a guest OS. Pinging host from guest with multiple virtual CPUs results in:

    Reply from 10.101.25.79: bytes=32 time=2ms TTL=127
    Reply from 10.101.25.79: bytes=32 time=48766ms TTL=127
    Reply from 10.101.25.79: bytes=32 time=1ms TTL=127
    Reply from 10.101.25.79: bytes=32 time=48768ms TTL=127

    Hardware is Dell Precision 490 with single quad core Intel Xeon CPU.
    Please advise if any progress have been made in identifying and solving this issue.
    I would also like to remark tha the overall performance of VM (even as light as XPSP3) leves much to be desired. On two virtual CPUs of 2.66 GHz xeon and 2GB RAM assigned to VM I would expect beter performance.

    Thank you,

    Vladimir
    Vladimir Yakovlev
    Wednesday, June 25, 2008 1:17 PM
  • We are seeing this issue with Hyper-V RTM as well, on a new Dell Poweredge 2950 with a single Quad Core Xeon processor. 

    Interstingly, we have two virutal machines running on the server...one running Windows 2003 SP2 utilizing 1 processor, and one running Windows 2003 R2 utilizing 2 processors.  The one running R2 exhibits the ping timing issue...the one running 2003 SP2 does not.

    I will try the /usepmtimer switch and report back here if it seems to fix the problem on the R2 virutal machine.
    Thursday, July 17, 2008 6:03 PM
  •  I found this blog entry, and can confirm that /usepmtimer fixed the problem in our environment.

    http://blogs.msdn.com/tvoellm/archive/2008/06/05/negative-ping-times-in-windows-vm-s-whats-up.aspx
    Friday, July 18, 2008 11:36 AM