Server 2012 NIC speed
-
Dienstag, 8. Januar 2013 07:37
Hi everybody!
In process of migrating to Hyper-V from ESXi I encountered really strange problem that cannot resolve it by myself...
I have "Whitebox" server for testing only, on ESXi 5.1 all worked great.
I installed Server 2012 standard and 2 2008 R2 guests on it.
Underlying storage is RAID 10 with read up to 250 MB/s and write speed up to 130 MB/s.
I have 2 NICs one Intel GT 1000 PRO and one onboard Atheros, both 1 Gbps.
On host I disabled Atheros, with Intel I created one External vSwitch for VMs and server management (like I had on ESXi)....The OK part:
1. Test ISO file, 5 GB in size
2. If I copy this file between VMs I get speed 116 MB/s
3. If I copy this file from VM to HOST I get speed 211 MB/s
4. If I copy this file from HOST to VM I get speed 117 MB/sThe NOT OK part:
5. If I copy this file from HOST to my WORKSTATION I get speed ~50 MB/s
6. If I copy this file from WORKSTATION to HOST I get speed ~50MB/s
7. If I copy this file from VM to my WORKSTATION I get speed ~50 MB/s
8. If I copy this file from WORKSTATION to VM I get speed ~50 MB/sThis same server under ESXi 5.1 control copied files over network with 110-120 MB/s !!!
So my question is, how come Server 2012 and its guests perform 1/2 slower over the network than ESX?
Switch is also gigabit and proven working (transfers between 2 WKS are 109 MB/s)Any idea?
Thanks!
- Bearbeitet Grega_SLO Dienstag, 8. Januar 2013 07:38
Alle Antworten
-
Dienstag, 8. Januar 2013 08:13
Hi everybody!
In process of migrating to Hyper-V from ESXi I encountered really strange problem that cannot resolve it by myself...
I have "Whitebox" server for testing only, on ESXi 5.1 all worked great.
I installed Server 2012 standard and 2 2008 R2 guests on it.
Underlying storage is RAID 10 with read up to 250 MB/s and write speed up to 130 MB/s.
I have 2 NICs one Intel GT 1000 PRO and one onboard Atheros, both 1 Gbps.
On host I disabled Atheros, with Intel I created one External vSwitch for VMs and server management (like I had on ESXi)....The OK part:
1. Test ISO file, 5 GB in size
2. If I copy this file between VMs I get speed 116 MB/s
3. If I copy this file from VM to HOST I get speed 211 MB/s
4. If I copy this file from HOST to VM I get speed 117 MB/sThe NOT OK part:
5. If I copy this file from HOST to my WORKSTATION I get speed ~50 MB/s
6. If I copy this file from WORKSTATION to HOST I get speed ~50MB/s
7. If I copy this file from VM to my WORKSTATION I get speed ~50 MB/s
8. If I copy this file from WORKSTATION to VM I get speed ~50 MB/sThis same server under ESXi 5.1 control copied files over network with 110-120 MB/s !!!
So my question is, how come Server 2012 and its guests perform 1/2 slower over the network than ESX?
Switch is also gigabit and proven working (transfers between 2 WKS are 109 MB/s)Any idea?
Thanks!
You're tuning device with multiple handles @ the same which is always considered to be a bad engineering practice. With your tests you test not only network stack but also storage stack (and particularly cache implementation which is a BIG question) and SMB redirector. If you're interested in raw network performance (I beleive you really need TCP) then stick with NTtcp and IPerf tools to figure out how your NETWORK is performing. Here are some download links for Windows and also a guide how to use the tools (for *nix but CLI semantics are the same):
http://lcp.nrl.navy.mil/nuttcp/
http://timelordz.com/wiki/Network_Speed_Test
after you'll make sure it's really network being slow it makes sense to continue troubleshooting. I still beleive it's something different (say write-thru
access for VHDs with Hyper-V versus block-level cached acccess with ESXi).
- Als Antwort vorgeschlagen VR38DETTMicrosoft Community Contributor Donnerstag, 10. Januar 2013 16:11
-
Dienstag, 8. Januar 2013 08:25
Thanks for this but read again please.
I already 100% confirmed network problem. Disks can perform way more than 50MB/s... Also copy from VM to VM is normal.
From host to my workstation file copy has NOTHING to do with VHD.
-
Dienstag, 8. Januar 2013 14:20
hello
sorry didn undeerstand.. but is your workstation a vm ? or an outside client.. ?
if it is a client what hardware ?
------------------------------------------------------- I understand a little computers.
-
Dienstag, 8. Januar 2013 14:27
can you try to copy with xcopy..
and use /j parameter with it..
i suggest you tu use a 10 gb file for a better test.. use a timer and then calculate the transfer..
regards
------------------------------------------------------- I understand a little computers.
-
Dienstag, 8. Januar 2013 16:11
Hi!
Tried already.
Its the same thing...
My workstation is not VM it is PC on hardware...
-
Dienstag, 8. Januar 2013 16:18
is your workstation nic RSS capable ? can you try to optimize thread usage during the copy ?
is the nic on the hyper host rss capable ?and optimize it there too ?
can you you try a robocopy with \mt parameter.
Regards
------------------------------------------------------- I understand a little computers.
- Bearbeitet Ales75 Dienstag, 8. Januar 2013 16:19 correction
-
Dienstag, 8. Januar 2013 17:05
Ha ha found where problem is...
I booted in safe mode with networking and speeds are as tehy should be...
I suspected hyper-v and vswitch. Booted up in normal mode, deleted vswitch and speeds from host to my hardware workstation are again as they should be...
This is interesting one, drivers are OK, hardware is OK, disks are OK, seems that Hyper-v vswitch does some limit or whatever...
Any idea?
-
Dienstag, 8. Januar 2013 17:07Check the rss thing..
------------------------------------------------------- I understand a little computers.
-
Dienstag, 8. Januar 2013 17:18
RSS is enabled on all Ethernet cards, both server side as client side.
RSS is not the issue read post above, vSwitch makes things fuzzy...
-
Dienstag, 8. Januar 2013 17:23
It not correct, your nic could be rss capable but for any reason using only 1 thread and that makes difference.
This is why i suggest to check this in powershell.
------------------------------------------------------- I understand a little computers.
-
Dienstag, 8. Januar 2013 17:24
I used: Get-NetAdapterRss
In output was:
Enabled: True
-
Dienstag, 8. Januar 2013 17:32
Are you multichanneling while copying ?
Get-smbmultichannelconnection
------------------------------------------------------- I understand a little computers.
-
Dienstag, 8. Januar 2013 17:49
I don`t know where this is going but it does not sound right for me :)
I have issue ONLY when Hyper-V vSwitch is created.
Before that speeds are normal.
-
Dienstag, 8. Januar 2013 19:28Do you have the latest 2012 drivers for the NIC? What about other NIC settings besides RSS? For example, do you have any VMQ settings enabled/disabled? Have you tried changing some of those settings? I haven't seen it specifically with the Intel card, but I saw an incident with a Broadcom card that had a 'Virtual Machine Queues' setting enabled by default. The experience sounded very similar to yours. They disabled that setting and performance became equal across the board.
tim
-
Dienstag, 8. Januar 2013 19:57yes correct, virtual machine queues must be disabled on nics performing rss operations.
------------------------------------------------------- I understand a little computers.
-
Dienstag, 8. Januar 2013 20:50
Hehe we clearly do not understand each other...
PHYSICAL server 2012 with Hyper-V installed which has created vSwitch and 3 VM POWERED off (or in any state actually)
PHYSICAL workstation Windows 8
FILE COPY between them = 50 MB/s
IF I remove vSwitch on 2012 server FILE COPY between them = 82-87 MB/s
VMQ has absolutley NOTHING to do with scenario above.
-
Dienstag, 8. Januar 2013 23:43
is the switch shared with the os ?
------------------------------------------------------- I understand a little computers.
-
Mittwoch, 9. Januar 2013 06:03
Yes it is.
EDIT:
Since I have 2 NICs I created a team and then vSwitch.
Now incoming (weird) and outgoing speeds are better (82 MB/s).Huh now I really am stumbeled :)
- Bearbeitet Grega_SLO Mittwoch, 9. Januar 2013 07:20
-
Mittwoch, 9. Januar 2013 14:58
You state "PHYSICAL server 2012 with Hyper-V installed which has createdvSwitch ".
You state "VMQ has absolutley NOTHING to do with scenario above"
In answer to the question if the switch is shared with the OS, you state "Yes it is".
As soon as you create a virtual switch on a physical NIC and share that virtual switch with the host, you have altered the environment. The physical host now no longer has a dedicated physical NIC, but it has a virtual NIC connected to the virtual switch. As a result, any virtual function, such as VMQ, could very well be having some sort of impact. I'm curious to know how you determined that, unequivocally, VMQ is not involved here.
It makes sense that teaming would increase speed. After all, that is one aspect of teaming.
tim
-
Mittwoch, 9. Januar 2013 18:09
Hi!
I checked in settings, there was no VMQ option at all...
This is why I did not tought it has something to do with speed...
But it is interesting, VM-VM copy on same Vswitch and same disk storage happens to be 113 MB/s while VM-PHYSICAL WORKSTATION is 80 MB/s...
-
Donnerstag, 10. Januar 2013 00:22
it is strange.. what hardware is ? nic ?
But it is interesting, VM-VM copy on same Vswitch and same disk storage happens to be 113 MB/s while VM-PHYSICAL WORKSTATION is 80 MB/s...
i would think for example that storage on the workstation is not capable as the server sotrage..
post here server controller and hard drives and workstation controller and hard drives
by this i can think that the worksation is not write capable as the server, so your limit is the storage and not the network.
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 07:11
Hi!
Both are RAID 10 on intel ICH "fake" raid controller...
But interesting is also that copy file from Workstation to server is 80 MB/s. Like it is capped at that speed...
4 stata disks seagate constallation on workstation in RAID 10 config are surley capable of read speed larger than 80 MB/s right?
This is how it looks from server/vm to wks or vice-versa.
- Bearbeitet Grega_SLO Donnerstag, 10. Januar 2013 07:29
-
Donnerstag, 10. Januar 2013 07:41
Hello
Well.. In a raid 1 with a sw raid that setup would perform similar to that.. But with a 10 you can expect some more..
I suggest to start separating the problem..in two areas.. Network and then storage.
Since i suppose network is involved and not storage, use nttcp tool, the result of that will give your pure speed link..
I would not be surprised if you get 1 gb .. Thats what your setupnis capable.
Regards
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 09:33
There we go:
C:\Tools\nuttcp-5.5.5.win32>nuttcp-5.5.5-pre1.exe -i1 -w1m 10.10.0.50 58.7852 MB / 1.01 sec = 486.3178 Mbps 71.5568 MB / 1.00 sec = 601.4648 Mbps 70.1415 MB / 1.00 sec = 588.9788 Mbps 83.3140 MB / 1.00 sec = 700.2890 Mbps 83.2910 MB / 1.00 sec = 700.0961 Mbps 83.2674 MB / 1.00 sec = 699.1969 Mbps 83.4138 MB / 1.00 sec = 701.1282 Mbps 83.3215 MB / 1.00 sec = 699.6508 Mbps 83.2323 MB / 1.00 sec = 699.6026 Mbps 65.9192 MB / 1.01 sec = 545.3356 Mbps 767.1875 MB / 10.36 sec = 621.3203 Mbps 13 %TX 35 %RX C:\Tools\nuttcp-5.5.5.win32>nuttcp-5.5.5-pre1.exe -i1 -w5m 10.10.0.50 83.1424 MB / 1.01 sec = 687.8193 Mbps 83.1415 MB / 1.00 sec = 698.8387 Mbps 83.0783 MB / 1.00 sec = 697.6092 Mbps 83.1125 MB / 1.00 sec = 698.5951 Mbps 83.1808 MB / 1.00 sec = 699.1692 Mbps 83.1514 MB / 1.00 sec = 698.2224 Mbps 83.1561 MB / 1.00 sec = 698.9621 Mbps 83.2010 MB / 1.00 sec = 698.6389 Mbps 83.1934 MB / 1.00 sec = 699.2757 Mbps 83.9254 MB / 1.01 sec = 694.2967 Mbps 837.0625 MB / 10.36 sec = 677.9097 Mbps 12 %TX 37 %RX C:\Tools\nuttcp-5.5.5.win32>nuttcp-5.5.5-pre1.exe -i1 -w8m 10.10.0.50 57.1017 MB / 1.01 sec = 472.3906 Mbps 45.2693 MB / 1.00 sec = 380.5077 Mbps 61.6591 MB / 1.00 sec = 517.7521 Mbps 85.3210 MB / 1.00 sec = 717.1591 Mbps 83.7889 MB / 1.00 sec = 704.2805 Mbps 64.2774 MB / 1.00 sec = 539.7380 Mbps 83.5903 MB / 1.00 sec = 702.6111 Mbps 83.7449 MB / 1.00 sec = 703.2064 Mbps 83.8886 MB / 1.00 sec = 705.1186 Mbps 84.9676 MB / 1.01 sec = 702.9194 Mbps 741.4375 MB / 10.12 sec = 614.3450 Mbps 10 %TX 35 %RX C:\Tools\nuttcp-5.5.5.win32>nuttcp-5.5.5-pre1.exe -i1 -w10m 10.10.0.50 82.2609 MB / 1.01 sec = 680.5273 Mbps 82.3913 MB / 1.00 sec = 692.5336 Mbps 83.0749 MB / 1.00 sec = 697.5807 Mbps 57.2131 MB / 1.00 sec = 480.9003 Mbps 91.4971 MB / 1.00 sec = 769.0716 Mbps 83.2833 MB / 1.00 sec = 699.3307 Mbps 66.1273 MB / 1.00 sec = 555.8274 Mbps 83.2941 MB / 1.00 sec = 699.4208 Mbps 83.3209 MB / 1.00 sec = 700.3467 Mbps 60.8578 MB / 1.01 sec = 503.4634 Mbps 783.3125 MB / 10.14 sec = 648.0179 Mbps 11 %TX 37 %RX C:\Tools\nuttcp-5.5.5.win32>nuttcp-5.5.5-pre1.exe -i1 -w10m 10.10.0.50 56.6604 MB / 1.01 sec = 468.7393 Mbps 82.9675 MB / 1.00 sec = 697.3763 Mbps 82.5735 MB / 1.00 sec = 693.3700 Mbps 82.9487 MB / 1.00 sec = 697.2187 Mbps 83.0481 MB / 1.00 sec = 698.0539 Mbps 82.9143 MB / 1.00 sec = 696.2314 Mbps 82.5291 MB / 1.00 sec = 693.6918 Mbps 83.0442 MB / 1.00 sec = 697.3226 Mbps 82.9674 MB / 1.00 sec = 697.3756 Mbps 83.8872 MB / 1.01 sec = 693.9811 Mbps 814.2500 MB / 10.14 sec = 673.6118 Mbps 10 %TX 38 %RX
Both gigabit, so I can assume something is clogging my network?
-
Donnerstag, 10. Januar 2013 09:50
Well well well...
C:\Tools\nuttcp-5.5.5.win32>nuttcp-5.5.5-pre1.exe -i1 -w10m 10.10.0.55 113.0014 MB / 1.01 sec = 934.8367 Mbps 112.9681 MB / 1.00 sec = 949.5445 Mbps 112.9653 MB / 1.00 sec = 948.5706 Mbps 112.9708 MB / 1.00 sec = 949.5666 Mbps 112.9612 MB / 1.00 sec = 949.4859 Mbps 112.9680 MB / 1.00 sec = 948.5927 Mbps 112.9640 MB / 1.00 sec = 949.5094 Mbps 112.9708 MB / 1.00 sec = 948.6160 Mbps 112.9681 MB / 1.00 sec = 949.5445 Mbps 114.7305 MB / 1.01 sec = 949.1413 Mbps 1141.1875 MB / 10.12 sec = 945.5724 Mbps 16 %TX 28 %RX
Other workstation has correct speed...
So I can say that NIC on my primary physical workstation is to blame, correct?
-
Donnerstag, 10. Januar 2013 14:30
you are correct :)
try to move the nic on another pci slot.. for example and repeat the test, plus to a clean reinstall of its drivers
Regards
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 14:31do it if you are really sure that any parameter is mispelled !
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 14:44I`m not at home right now, but I will replace NIC with other one, this one is gold old Dlink DGE-528T from 2006 hehe :)
-
Donnerstag, 10. Januar 2013 14:46
this one is gold old Dlink DGE-528T from 2006
you gave yourself the answer.. without any further investigation..
grrrr.. :)
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 15:00
Previously I had ESXi 5.1 and Win7 client
Now I have Hyper-V and Windows 8 client (upgraded on same PC)...
So Win8 probably borked something up...
-
Donnerstag, 10. Januar 2013 15:27
no.. i dont think.. but the lack of features on that nic.. i suppose that may be would not certanly created the proper scenario with features avaible nowadays..
it's just a thought
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 16:00
Confirmed.
It was bad NIC.
I replaced nic and speeds are now from 111-115MB/s constant!
Geez...
-
Donnerstag, 10. Januar 2013 16:01
glad it worked... !
Regards
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 16:02
ok now you can pay the bill :)
propose it as an answer..
Regards
------------------------------------------------------- I understand a little computers.
-
Donnerstag, 10. Januar 2013 16:07
Awsome, now I can go on.
BTW it is Windows 8 that borked drivers somehow, I tested same nic with Win 7 and I get 105-109MB/s....
Huh got to buy some Intel CT desktop adapters :)
P.S.
Huge thanks to all involved in helping me out!
- Bearbeitet Grega_SLO Donnerstag, 10. Januar 2013 16:08
.png)
.png)
