locked
PUT Blobs of size greater than 5.5MB fail with HTTPS but not HTTP RRS feed

  • Question

  • I have written a Cygwin app that uploads (using the REST API PUT operation) Block Blobs to my Azure storage account, and it works well for different size blobs when using HTTP. However, use of SSL (i.e. PUT using HTTPS) fails for Blobs greater than 5.5MB. Blobs less than 5.5MB upload correctly. Anything greater and I find that the TCP session (as seen by Wireshark) reports a dwindling window size that goes to 0 once the aforementioned number of bytes have been transferred. The failure is very repeatable and consistent. As a point of reference,  PUT operations against my Google/AWS/HP accounts work fine when using HTTPS for various object sizes, which suggests my problem is not in my client but specific to the HTTPS implementation on the MSAZURE storage servers. 

    If I upload the 5.5MB blob as two separate uploads of 4MB and 1.5MB followed by a PUT Block List, the operation succeeds as long as the two uploads used separate HTTPS sessions. Notice the emphasis on separate. That same operation fails if I attempt to maintain an HTTPS session across both uploads. This is another data point that seems to suggest that the Storage server has a problem 

    Any ideas on why I might be seeing this odd behavior that appears very specific to MS Azure HTTPS, but is not seen when used against AWS/Google/HP cloud storage servers?

    Saturday, March 15, 2014 3:22 AM

All replies

  • Hi,

    Which method did your use Put Block or put Blob ? Did you set the Content-Length or x-ms-blob-content-length in request header? For block blob, it will be up to 4MB size. Any update info, please let me know

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, March 17, 2014 6:15 AM
  • I don't think Put Blob vs. Put Block matters.  What does matter is use of SSL/TLS. Also, I do specify the Content-Length attribute in all PUT requests.

    A few examples of how Put Block and Put Blob perform when using HTTP vs. HTTPS:

    1. Size 10MB over HTTPS: Single Put Blob operation fails after transfer of 5.5MB (socket not writable as TCP window size provided by server is 0). This should technically work up to 64MB if I am reading your documentation correctly.

    2. Size 7MB over HTTPS: Two Put Blocks operations, one of 4MB and another of 3MB. First one works. Second one (3MB) fails after 1.5MB transfer if it reuses the same (i.e., does not close and open a new) HTTPS session that was used for the 4MB session. 

    3. Size 10MB over HTTP : Single Put Blob works. Note: no SSL/TLS used.

    4. Size 7MB over HTTP : Dual Put Block works. Note: no SSL/TLS used.


    • Edited by Prrplexed Monday, March 17, 2014 2:57 PM
    Monday, March 17, 2014 2:05 PM
  • Not sure why. Can you just use different SSL sessions for different requests? At least this serves as a workaround.
    Wednesday, March 26, 2014 7:27 AM
  • Different SSL sessions, one for each PUT Block is an extremely inefficient workaround due to the overhead associated with SSL session establishment. So I'm forced to use two PUT Block sessions instead of just one PUT Blob even for a small file like 7MB.

    The fundamental issue here is: Why do SSL sessions fail after 5.5MB? I would be ok if the limitation were to occur at the max PUT Blob size of 64MB, but 5.5MB is just too small a threshold for an SSL session to fail.

    Wednesday, March 26, 2014 11:54 AM
  • Hi,


    From a support perspective this is really beyond what we can do here in the forums. If you cannot determine your answer here or on your own, consider opening a support case with us. Visit this link to see the various support options that are available to better meet your needs:  http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Jun Zh - MSFT Microsoft Online Community Support

    Tuesday, April 1, 2014 7:25 AM
  • Thank you for reporting this and we apologize for the inconvenience. We have managed to recreate the issue and have filed a bug. Unfortunately we cannot share a timeline for the fix at this time, but we will respond to this forum when the fix has been deployed. In the meantime, a plausible workaround (and a recommended best practice) is to break large uploads into smaller chunks (using the Put Block and Put Block List APIs), thus enabling the client to parallelize the upload.
    Tuesday, April 8, 2014 9:40 PM
  • RE:  a plausible workaround (and a recommended best practice) is to break large uploads into smaller chunks (using the Put Block and Put Block List APIs)

    This does not appear to be possible with the PHP sdk. By default BlobRestProxy.php appears to do this for files 32MB+ in size at 4MB chunks however I called setSingleBlobUploadThresholdInBytes(1048576) (1MB) to force it to chunk files less than 5MB in size and it still fails at the upload of the 6th chunk, around the 5+MB mark. 

    If we cannot upload 5MB+ files to the azure blob service via HTTPS then the service is basically useless to us and we will have to consider taking our business elsewhere. I hope I am missing something and am open to suggestions. 



    Wednesday, April 30, 2014 3:41 PM
  • Can you please look at the unit test sample that we have here to see the usage: https://github.com/Azure/azure-sdk-for-php/blob/841a8f3d1e160034eeee4a60ed5da9436e017230/tests/unit/WindowsAzure/Blob/BlobRestProxyTest.php#L1595

    If it still didn't work please file an issue here with the details and we'll work actively on getting it done quickly: https://github.com/Azure/azure-sdk-for-php/issues

    Btw, are you using PEAR or Composer for pulling the SDK?

    Saturday, May 3, 2014 12:24 AM
  • Yes, lines 1598 thru 1612 matches my usage. I assume the rest of the code in that test method are not needed in my case. 

    $restProxy->setSingleBlobUploadThresholdInBytes(1048576); //1MB
    $restProxy->createBlockBlob($container, $blobName, $blob, $props);

    In stepping through the code it pushes 5 chunks quickly and then on the 6th call of
      $this->createBlobBlock($container, $blob, $block->getBlockId(), $body);
    it hangs for about 3 minutes and I get a php warning: Warning 2 fwrite(): SSL: Broken pipe app/Lib/PEAR/HTTP/Request2/SocketWrapper.php line 202

    I am using PEAR. I will submit an issue via github. Thanks. 

    Sunday, May 4, 2014 9:48 PM
  • Hi,

    I'm getting this problem also when trying to upload blobs > 5.5mb using the Azure PHP SDK with HTTPS.

    There is no way I can find to get a blob > 5.5mb to upload, unless you use http, rather than https, which is not a good solution.

    I've written my own scripts to use the HTTP_Request2 library, to send the request as a test, and it fails with that also when using the 'socket' method.

    However, if I write a script using the PHP Curl extension directly, then it works fine, and blobs > 5.5mb get uploaded.

    It seems to be irrelevant which method is used, uploading in 1 go, or using smaller chunks, the PHP SDK seems broken.

    Also, I think I've found another bug in the SDK, when you do the smaller chunks, the assigning of the BlockID is not correct.

    In: WindowsAzure/Blob/BlobRestProxy.php

    Line: $block->setBlockId(base64_encode(str_pad($counter++, '0', 6)));

    That is incorrect usage of the str_pad function, and if you upload a huge blob that needs splitting, then the blockIDs will after a while become a different length and therefore fail.

    It should be: str_pad($counter++, 6, '0',STR_PAD_LEFT);

    I also think there is 1 too many base64_encodes() in there, as I think its being done twice, once in that line, and then again within the createBlobBlock() just before the send() for a 2nd time.

    Can someone please advice, when this/these bug(s) will be fixed in the PHP SDK, as at the moment its useless to me as I cant upload things securely.

    Friday, December 5, 2014 9:23 AM