none
NIC Teaming is reducing throughput. Why? RRS feed

  • Question

  • Hello everyone!

    I have been trying to improve our network here. First we acquired two very nice HP 5900 10 Gbps switches (48 port). Got a bunch of Intel 10Gb NICs. The plan is to have redundant network connectivity and maximize network throughput. I am having a problem... 

    Short version:

    Using iPerf I can saturate 10Gb links easily (meaning I get nearly 10Gbps) however when I team the same ports, I get only 3-4 Gbps. Anything I could do to improve the network throughput over a teamed link?

    Long version:

    Here is how two physical test machines are connected, network wise:

    • The two network switches are configured to behave as a single chassis (using HP IRF).
    • Each machine has a dual port 10GB NIC. Top port goes to switch 1. Bottom port goes to switch 2.
    • All connections are via CAT 7 patch cords (really nice ones).
    • IP of team on server 1 : 192.168.3.30
    • IP of team on server 2: 192.168.3.13

    Test results:

    I am using iPerf for running the throughput tests. This tool eliminates any disk I/O type bottlenecks on the machines.

    > Iperf is in server mode on server 2

    > Iperf is in client mode on server 1

    > TCP window size has been maximized to 1.5g

    > Jumbo Frames have been setup properly and tested to be working fine on all ports.

    Notice that I get only around 3.5 Gbps:

    Teaming settings:

    Here is what I have set it up like (for now). I can setup LACP later.

    I do want to double the throughput "eventually" but for now I expect at least the same performance as a single link.

    Any tips?


    -Rajeev rajdude.com

    Tuesday, March 24, 2015 3:45 PM

Answers

  • Hi Rajeev,

    If we choose the Swith Independent mode, the inbound traffic will only received by single NIC.

    Dynamic mode may cause out-of-order delivery of packets.

    Please try to change the Load balancing mode to Address hash distribution.

    Besides, teaming doesn't increase bandwidth to a single user.  Teaming increases the aggregate throughput, which means that more people can use the speed of the individual NICs simultaneously.

    For detailed information about NIC teaming, please click the link below:

    https://www.microsoft.com/en-us/download/details.aspx?id=40319

    Best Regards


    Steven Lee Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Wednesday, March 25, 2015 3:32 AM
    Moderator
  • Hi Steven,

    Thanks, Yes I do (now) understand that NIC teaming won't increase bandwidth to a single server/user. That is why I am telling iPerf to use more than one stream. However I suspect that the OS still treats it as going to a single user/server. That is why I get capped at around 12 Gbps even with  4 iPerf streams. Notice that it really is over 10Gbps...so I "AM" getting slightly higher throughput than one single 10Gbps link.

    I think what I will have to do is to setup two machines (server 2 and server3) as iPerf servers and run iPerf client from server 1, sending two streams to each server. Maybe that way I will see nearly 20Gbps....but again, that may be only in theory...I do remember that ! I'll work on this type of test next week.

    LACP Update:

    I configured LACP on the switch and both machines. After a bit of troubleshooting the correct settings on my switches, I am successfully able to connect using LACP.

    However, I find that the throughput results are pretty much the same as seen without LACP.

    So my conclusion is:

    It may not be really worth spending time and effort in setting up LACP just for NIC teaming. The switch independent mode with address hash achieves pretty much the same throughput.


    -Rajeev rajdude.com

    Thursday, March 26, 2015 6:23 PM

All replies

  • Hi Rajeev,

    If we choose the Swith Independent mode, the inbound traffic will only received by single NIC.

    Dynamic mode may cause out-of-order delivery of packets.

    Please try to change the Load balancing mode to Address hash distribution.

    Besides, teaming doesn't increase bandwidth to a single user.  Teaming increases the aggregate throughput, which means that more people can use the speed of the individual NICs simultaneously.

    For detailed information about NIC teaming, please click the link below:

    https://www.microsoft.com/en-us/download/details.aspx?id=40319

    Best Regards


    Steven Lee Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Wednesday, March 25, 2015 3:32 AM
    Moderator
  • Oh wow! the moment I changed both servers to Switch Independent / Address Hash / all adapters active, I am (almost) able to saturate the 10Gb connection:

    So thanks a ton! :-)

    Now lets see if I can increase the streams in iPerf and see higher than 10Gbps....

    Will LACP give me higher throughput over a single stream?


    -Rajeev rajdude.com

    Wednesday, March 25, 2015 5:29 PM
  • I tried iPerf with 2 and 4 simultaneous streams over the teamed NICs. (see screenshots below)

    Two streams topped off around 8-9 Gbps total

    Four streams varied around 8 to 18 Gbps.

    So, overall the throughput is rather flaky in multistream mode. It is not 20Gbps, and that is probably just wishful thinking. Well, as long as it is at or above a single link's speed -10Gbps, I am "almost" happy :-)

    Will try LACP next.....I guess this will take longer since switch needs to be configured.


    -Rajeev rajdude.com

    Wednesday, March 25, 2015 6:13 PM
  • Hi Rajeev,

    Glade to hear that your issue is "almost" resovled and thanks for the sharing.

    Actually, from my point of view, network speed depends many factors. Normally, we won't and can't let it reach the limit. The 20Gbps is a theoretical value. Teaming will cause additional cost.

    >>Will LACP give me higher throughput over a single stream?

    As I have mentioned above, theoretically, it can let single stream distributed on multiple NICs.(According to the loadbalance setting of the switch) But it will cause out-of-order delivery of packets. It will cause additional cost to reorder the packets.

    Teaming doesn't increase bandwidth to a single user.  Teaming increases the aggregate throughput, which means that more people can use the speed of the individual NICs simultaneously

    Best Regards.


    Steven Lee Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.


    Thursday, March 26, 2015 3:25 AM
    Moderator
  • Hi Steven,

    Thanks, Yes I do (now) understand that NIC teaming won't increase bandwidth to a single server/user. That is why I am telling iPerf to use more than one stream. However I suspect that the OS still treats it as going to a single user/server. That is why I get capped at around 12 Gbps even with  4 iPerf streams. Notice that it really is over 10Gbps...so I "AM" getting slightly higher throughput than one single 10Gbps link.

    I think what I will have to do is to setup two machines (server 2 and server3) as iPerf servers and run iPerf client from server 1, sending two streams to each server. Maybe that way I will see nearly 20Gbps....but again, that may be only in theory...I do remember that ! I'll work on this type of test next week.

    LACP Update:

    I configured LACP on the switch and both machines. After a bit of troubleshooting the correct settings on my switches, I am successfully able to connect using LACP.

    However, I find that the throughput results are pretty much the same as seen without LACP.

    So my conclusion is:

    It may not be really worth spending time and effort in setting up LACP just for NIC teaming. The switch independent mode with address hash achieves pretty much the same throughput.


    -Rajeev rajdude.com

    Thursday, March 26, 2015 6:23 PM
  • This is an excellent post glad to see someone doing real world test instead of always talking about theory.
    Sunday, May 31, 2015 3:55 PM