none
Unrecoverable error occurred during Flush operation

    Question

  • We have been backing up our on premise sql instances (3 different servers) to various azure storage accounts for the past several months without issue. 

    Starting on July 14th, we have been increasingly unable to complete backups with the following message recorded in the logs:

    A nonrecoverable I/O error occurred on file "https://<removed>.blob.core.windows.net/sql-full/<removed>.bak:" Backup to URL received an exception from the remote endpoint. Exception Message: Unrecoverable error occurred during Flush operation.  Msg 3013, Level 16, State 1, Server <removed>, Line 1  BACKUP DATABASE is terminating abnormally.  Outcome: Failed  Duration: 00:03:05  Date and time: 2016-07-17 03:33:05   

    The error is occurring on full, differential and log backups.

    We are using SQL 2014 standard (sp 2) running on fully patched Windows 2012 R2 servers. At the point the errors started happening we had only installed sp 1, but have since installed sp 2 to see if it would help. 

    We generally use Ola Hallengren's backup scripts executed via sql jobs, but we have also tried manually backing up directly to azure via SSMS and that has failed as well.

    We've searched around but have been unable to find any solution. 

    Any suggestions on how to resolve this issue would be greatly appreciated.

    Sunday, July 17, 2016 11:31 AM

All replies

  • Hi dynamismike,

    Based on your description, I’m sensing that the issue is likely with your Azure storage. To verify that, I would suggest you try backup to local device first. In this case, I would suggest you turn on trace flag 3051 for verbose logging.

    Also, according to this article, this kind of error message may related to proxy errors. I would suggest you also check your network/proxy settings see if it helps.

    If you have any other questions, please let me know.

    Regards,
    Lin
    Tuesday, July 19, 2016 3:14 AM
    Moderator
  • Hi Lin,

    Thanks for the response.

    We do not use a proxy server.

    I have turned on trace flag 3051, the results are below:

    Error: 18210, Severity: 16, State: 1.

    BackupVirtualDeviceFile::RequestDurableMedia: Flush failure on backup device "". Operating system error Backup to URL received an exception from the remote endpoint. Exception Message: Unrecoverable error occurred during Flush operation.

    Error: 3041, Severity: 16, State: 1.

    BACKUP failed to complete the command BACKUP LOG. Check the backup application log for detailed messages.

    Error: 18210, Severity: 16, State: 1.

    BackupVirtualDeviceFile::CloseMedia: Completion failure on backup device "". Operating system error Error could not be gathered from Remote Endpoint.

    In addition, i examined the azure logs, and found the following:

    It looks like the timeout is being hit - however i don't know why that would be the case, and i'm not sure if/where the timeout could be increased.

    Tuesday, July 19, 2016 12:23 PM