none
TCP download speed over high latency connections is poor (compared to Linux)

    Question

  • I've been transferring some large (1gb+) files over HTTP (TCP) between two Windows servers, and noticed that the download speed never exceeds 10Mb/s. Even though the available bandwidth is 1000Mb/s.
    The servers are located in Roubaix, France and New York, USA. The RTT/ping between the servers is 72ms.

    What I've noticed is that over the same network, with a Linux guest in Hyper-V, the download speed exceeded 400Mb/s!

    When downloading from the Linux machine the performance was similar to the Windows > Windows transfer.
    After doing some tests it became clear that the TCP receive window doesn't grow large enough on the Windows machines, when the network latency is "high" (above 50ms).

    I've tried fiddling around with the TCP auto tuning levels and even disabling TCP auto tuning, but this didn't cause a increase in download speeds. (disabling auto tuning even reduced the download speed)

    My question is: What settings should I change in Windows to increase the download speed, to get similar results to Linux?

    I think this can be achieved by making TCP auto tuning in Windows Server 2016 more aggressive on high latency connections.

    • Edited by Gijs007 Friday, June 8, 2018 3:22 PM
    Friday, June 8, 2018 2:39 PM

Answers

  • In the end it wasn't the download speeds which were poor. It turned out that sending data from Linux to Windows also had good download speeds.

    It's simply the Windows TCP stack algorithms when uploading data ,which perform poor on high latency connections. Unfortunately I couldn't find a way to improve it's performance.

    I've even upgraded these system to Windows Server 2019. As it would enable TCP Cubic, which should perform similarly to Cubic on Linux. Microsoft even calls it "well suited for high bandwidth, high latency links" on their blog: https://techcommunity.microsoft.com/t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-8-A-Faster/ba-p/339749

    I've just retested this with Filezilla and the IIS webserver on Windows Server 2019.
    Serving large files over a high latency connection is still very slow when the file is transmitted from a Windows system..


    • Marked as answer by Gijs007 Sunday, June 9, 2019 10:33 AM
    • Edited by Gijs007 Sunday, June 9, 2019 10:46 AM
    Tuesday, June 4, 2019 5:55 AM

All replies

  • Hi,

    Thanks for your question.

    Please try the following suggestions to see if it could be of help.

     

    Maximum achievable throughput for a single TCP connection is determined by different factors.

    We can consider from these aspects:

    • Window size

    In all recent Microsoft Windows implementations, windows scaling is enabled by default.

    You can change registry values by editing the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters\DefaultSendWindow

    • Packet loss

    When packet loss occurs in the network, an additional limit is imposed on the connection.

    The TCP selective acknowledgment option (SACK, RFC 2018) allows a TCP receiver to precisely inform the TCP sender about which segments have been lost. This increases performance on high-RTT links, when multiple losses per window are possible.

    The SackOpts value in the following registry key can be edited to control the use of selective acknowledgements:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    • Windows scaling

    For more efficient use of high bandwidth networks, a larger TCP window size may be used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte.

    The Tcp1323Opts value in the following registry key can be added to control scaling windows and timestamp:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters

    • TCP retransmission behavior and fast retransmit

    By default, Windows resends a segment if it receives three ACKs for the same sequence number, (one ACK and 2 duplicates) and that sequence number lags the current one. This is controllable with the TcpMaxDupAcks registry parameter.

    The TcpMaxDupAcks value in the following registry key can be edited to control the number of ACKs necessary to start a fast retransmits:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

     

    Specific operation please refer to the following link:

    Description of Windows 2000 and Windows Server 2003 TCP Features

    https://support.microsoft.com/en-us/help/224829/description-of-windows-2000-and-windows-server-2003-tcp-features

     

    Hope you have a nice day!

    Best regards,

    Travis 



    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, June 11, 2018 10:11 AM
    Moderator
  • Hi,

    Was your issue resolved?

    If you resolved it using our solution, please "mark it as answer" to help other community members find the helpful reply quickly.

    If you resolve it using your own solution, please share your experience and solution here. It will be very beneficial for other community members who have similar questions.

    If no, please reply and tell us the current situation in order to provide further help.

    Best Regards,

    Travis


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, June 15, 2018 6:08 AM
    Moderator
  • Hi Travis,

    Thank you for your reply.

    Unfortenetly my download speed is still at 3.52MB/s.

    I've set in: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    SackOpts=1
    Tcp1323Opts=3
    TcpMaxDupAcks=2

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters

    DefaultReceiveWindow=64512
    DefaultSendWindow=64512


    • Edited by Gijs007 Friday, June 15, 2018 1:35 PM
    Friday, June 15, 2018 1:34 PM
  • Hi @Gijs007,

    I have been trawling the net trying to find an answer to my high-latency Internet speed issue and I found this post which seems to be exactly the same thing I am experiencing.

    Did you eventually work out what the cause of the problem was?

    I would appreciate any info you have on this!

    Thanks,

    Sam.

    Tuesday, June 4, 2019 2:52 AM
  • In the end it wasn't the download speeds which were poor. It turned out that sending data from Linux to Windows also had good download speeds.

    It's simply the Windows TCP stack algorithms when uploading data ,which perform poor on high latency connections. Unfortunately I couldn't find a way to improve it's performance.

    I've even upgraded these system to Windows Server 2019. As it would enable TCP Cubic, which should perform similarly to Cubic on Linux. Microsoft even calls it "well suited for high bandwidth, high latency links" on their blog: https://techcommunity.microsoft.com/t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-8-A-Faster/ba-p/339749

    I've just retested this with Filezilla and the IIS webserver on Windows Server 2019.
    Serving large files over a high latency connection is still very slow when the file is transmitted from a Windows system..


    • Marked as answer by Gijs007 Sunday, June 9, 2019 10:33 AM
    • Edited by Gijs007 Sunday, June 9, 2019 10:46 AM
    Tuesday, June 4, 2019 5:55 AM
  • Hey, thanks for the info!  You've saved me a lot of work!

    Did you ever try to capture some packets to see whether the TCP window was expanding to a large enough size to accommodate the conditions?

    Tuesday, June 4, 2019 6:30 AM
  • I just wanted to add a note here that I have just discovered that currently, iPerf3 v3.1.3 for Windows comes bundled with a version of Cygwin1.dll (2.5.1) which has a hard-coded TCP window size setting of 278,775.

    This causes results to be limited to 12.6Mb/s per thread.

    This is why iPerf speeds in Linux are much better that in Windows.

    Updating the Cygwin dll to the latest version resolves this issue.

    I have informed the iPerf devs of this issue.

    Thanks to the Argon Systems article which directed me to this line of inquiry!
    https://argonsys.com/microsoft-cloud/library/windows-network-performance-suffering-from-bad-buffering

    Sunday, June 9, 2019 8:41 AM
  • I just wanted to add a note here that I have just discovered that currently, iPerf3 v3.1.3 for Windows comes bundled with a version of Cygwin1.dll (2.5.1) which has a hard-coded TCP window size setting of 278,775.

    This causes results to be limited to 12.6Mb/s per thread.

    This is why iPerf speeds in Linux are much better that in Windows.

    Updating the Cygwin dll to the latest version resolves this issue.

    I have informed the iPerf devs of this issue.

    Thanks to the Argon Systems article which directed me to this line of inquiry!
    https://argonsys.com/microsoft-cloud/library/windows-network-performance-suffering-from-bad-buffering


    Interesting, are you able to get near line speed throughput when using the updated cygwin.dll?

    I'm a bit surprised, as Filezilla and the build in IIS webserver both perform poorly (on Windows server 2019). If the article from argonsys is correct, this could indicate even IIS uses statically allocate TCP buffers!
    Sunday, June 9, 2019 10:52 AM
  • Yes, my iPerf speeds went from 100 to 500Mb/s (line speed) due only to the change in Cygwin.

    As for IIS; I'd be pretty surprised if it statically allocated TCP buffers, but I guess you could always check with wireshark or similar.

    Sunday, June 9, 2019 11:53 AM