none
Slow download speed from Azure Blob Storage

    Question

  • While testing our storage virtualization solution for a client we found interesting situation:

    1. When object is downloaded directly from Azure Blob Storage to outside of Azure - the speed is around 0.2 to 0.5 MB/s

    ubuntu@flexify-engine:~$ wget -O /dev/null https://dvideo.blob.core.windows.net/                                                 videos/Angra%20-%20Final%20Light.avi
    
    /dev/null                        100%[=======================================================>]  54.24M   299KB/s    in 3m 12s

    2. When the same object is downloaded to a VM in Azure - the speed is around 60 MB/s

    flexify@CWAYDOCKER:~$ wget -O /dev/null https://dvideo.blob.core.windows.net/videos/Angra%20-%20Final%20Light.avi
    
    /dev/null                         100%[=============================================================>]  54.24M  62.3MB/s    in 0.9s

    It may seem like a network issue, but....

    3. When the same object is downloaded via our proxy that also runs in the same Azure region and actually downloads the same object from Azure Blob Storage without any caching (and converts to S3 which is irrelevant here) - the speed is 30 MB/s and is basically limited by client's network. 

    ubuntu@flexify-engine:~$  wget -O /dev/null https://kandaurov.s3.test.flexify.io/Angra%20-%20Final%20Light.avi --no-check-certificate
    
    /dev/null                        100%[=======================================================>]  54.24M  31.0MB/s    in 1.8s

    Is Azure somehow throttles traffic coming from Blob storage to outside of Azure?




    Thursday, October 26, 2017 5:17 AM

All replies

  • There is no throttling on the Azure side below the published 60 MB/s scalability target for a single blob. If the network and client machine can handle the traffic then we will send it.

    Here's a guide for troubleshooting high E2E latency:

    https://docs.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting#metrics-show-high-AverageE2ELatency-and-low-AverageServerLatency

    Thursday, October 26, 2017 2:58 PM
  • Thanks, I will try to troubleshoot. 

    But definitely there is something wrong here because network/machine can accept much more than 500 KB/s as it does when the traffic is redirected through the proxy on a VM in the same Azure region. 

    Can I ask you a favour to check the speed in your network by downloading https://dvideo.blob.core.windows.net/videos/Angra%20-%20Final%20Light.avi?

    • Proposed as answer by JHSS Thursday, October 26, 2017 6:57 PM
    • Unproposed as answer by JHSS Thursday, October 26, 2017 6:57 PM
    Thursday, October 26, 2017 3:19 PM
  • I have the same problem. VM in the East US.

    Thursday, October 26, 2017 6:58 PM
  • @JHSS

    Could you describe your set-up and issue in more detail?

    -----------------------------------------------------------------------------------------------------

    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

    Monday, October 30, 2017 8:04 AM
    Moderator
  • @ Sergey Kandaurov

    The link you’ve shared doesn’t seem to be a valid one.

    -----------------------------------------------------------------------------------------------------

    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

    Wednesday, November 01, 2017 10:02 AM
    Moderator
  • Sorry for delay with the reply - we were really busy with this project. Managed to get sustained 4-5 gbps download speed by splitting between multiple threads and multiple machines.

    However the initial problem still exists. I renamed another blob to make the link above work again:

    wget -O /dev/null https://dvideo.blob.core.windows.net/videos/Angra%20-%20Final%20Light.avi --no-check-certificate
    wget -O /dev/null https://kandaurov.s3.test.flexify.io/Angra%20-%20Final%20Light.avi --no-check-certificate

    The first link is direct link to Azure Blob Storage in North Europe region.

    The second link point to Flexify.IO engine that runs as a VM (actually a VM scale set with load balancer) in North Europe region and proxies data from the first link without any caching. 

    Interesting that I see this behaviour only when testing from outside of Azure. If I ssh to an Azure VM - both link give me about the same speed.

    Friday, November 10, 2017 7:56 PM
  • @Sergey Kandaurov

    Have you had a chance to go over the high E2E latency troubleshooting doc? That should certainly give more pointers in the right direction to diagnose this issue.

    -----------------------------------------------------------------------------------------------------

    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

    Sunday, November 12, 2017 9:13 AM
    Moderator
  • Yes, I looked though the doc. Unfortunately it's not helpful. 

    The metrics show super high E2E latency (in minutes) and server latency about 1s.

    The doc basically says "fix your app code" and "use message analyzer". But we have already checked the code in different conditions (on Azure) and it works as expected. Also we can reproduce the problem with any code, including wget. 

    Increasing number of parallel connections helps - we see around 3mbps in a single connection, and 256 simultaneous connections gave us around 700 mbps. However it's not practical to increase the number of connection further as the service starts to close connections.

    Also I don't see any throttling errors.

    The basic problem here is that the results are very different if you download from outside of Azure vs. to when you download from within Azure. And it looks like a problem on Azure side. 

    Sunday, November 12, 2017 1:29 PM
  • Did you try Message Analyzer or Wireshark? The fact that it works fine from within Azure lets us rule out server-side issues. It would be interesting to see what is going on at the networking layer when you are seeing the slowness.

    I'm also interested in knowing how the connections are being closed. You should be able to maintain 512 connections easily.

    Monday, November 13, 2017 11:04 PM