locked
Web.Contents 403 error handling RRS feed

  • Question

  • I'm trying to use the Edgar API. Somewhere around 1% of API calls will return a 403 error randomly. Retrying works, unless the retry also randomly gets a 403 error. I'm trying to write code to retry with a back-off. Will this code do what I hope it will do? Or, is there a better way to do it?

    let
        APICall = "http://edgaronline.api.mashery.com/v2/corefinancials/ann?appkey=" & appkey & "&entityids=" & Number.ToText(entityids),
        APIData1 = Web.Contents(APICall, [ManualStatusHandling={403}]),
        APIData2 = Function.InvokeAfter( () => Web.Contents(APICall, [ManualStatusHandling={403}]), #duration(0,0,0,0.5)),
        APIData3 = Function.InvokeAfter( () => Web.Contents(APICall), #duration(0,0,0,1)),
    
        APIData = if Value.Metadata(APIData1)[Response.Status] = 200 then APIData1
            else if Value.Metadata(APIData2)[Response.Status] = 200 then APIData2
            else APIData3
    in
        APIData

    What I'm hoping this will do is through lazy execution, the APIData2 and APIData3 calls will only happen if the ones before them fail.

    I'm still getting runtime errors, saying auth failed. I'm trying to confirm that the code is working. I've enabled tracing but I'm struggling to read it, multiple trace files get created. I think these are the relevant log entries from one of the files. I would expect to see the same entityid multiple times in a row before an error. But the last 2 entityids logged, 89104, and 703428, don't show up anywhere else in the log file. (I added bolding to make it a little easier to see) (Why are the last 3 lines like from 2 minutes earlier?)

    DataMashup.Trace Information: 24579 : {"Start":"2017-12-20T16:36:21.5040150Z","Action":"Library/Common/RetryPolicy/TryExecuteAction","HostProcessId":"19596","Attempt":"1","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:00:00.4772822"}
    DataMashup.Trace Information: 24579 : {"Start":"2017-12-20T16:36:21.5039998Z","Action":"Library/Common/RetryPolicy/ExecuteAction","HostProcessId":"19596","TryCount":"3","RetryAlgorithm":"ExponentialBackoff","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:00:00.4773206"}
    DataMashup.Trace Information: 24579 : {"Start":"2017-12-20T16:36:21.5039757Z","Action":"Engine/IO/Web/Request/GetResponse","HostProcessId":"19596","RequestMethod":"GET","RequestUri":"http://edgaronline.api.mashery.com/v2/corefinancials/ann?appkey=appkey&entityids=89104","RequestHasContent":"False","RequestHasHeaders":"False","RequestHasTimeout":"True","UseCache":"True","ResponseStatusCode":"200","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:00:00.4773527"}
    DataMashup.Trace Warning: 24579 : {"Start":"2017-12-20T16:36:22.4821279Z","Action":"Library/Common/RetryPolicy/TryExecuteAction","HostProcessId":"19596","Attempt":"1","Exception":"Exception:\r\nExceptionType: System.Net.WebException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\r\nMessage: The remote server returned an error: (403) Forbidden.\r\nStackTrace:\n   at System.Net.HttpWebRequest.GetResponse()\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetWebResponse(WebRequest webRequest)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.CreateResponse(ResourceCredentialCollection credentials)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetResponseCore(ResourceCredentialCollection credentials)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.<>c__DisplayClass42.<GetResponse>b__40()\r\n   at Microsoft.Mashup.Engine1.Library.Common.RetryPolicy.Execute[TResult](IEngineHost host, Func`1 func)\r\n\r\n\r\n","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:00:00.0429423"}
    DataMashup.Trace Error: 24579 : {"Start":"2017-12-20T16:36:22.4820971Z","Action":"Library/Common/RetryPolicy/ExecuteAction","HostProcessId":"19596","TryCount":"3","RetryAlgorithm":"ExponentialBackoff","Exception":"Exception:\r\nExceptionType: System.Net.WebException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\r\nMessage: The remote server returned an error: (403) Forbidden.\r\nStackTrace:\n   at System.Net.HttpWebRequest.GetResponse()\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetWebResponse(WebRequest webRequest)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.CreateResponse(ResourceCredentialCollection credentials)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetResponseCore(ResourceCredentialCollection credentials)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.<>c__DisplayClass42.<GetResponse>b__40()\r\n   at Microsoft.Mashup.Engine1.Library.Common.RetryPolicy.Execute[TResult](IEngineHost host, Func`1 func)\r\n\r\n\r\n","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:00:00.0433333"}
    DataMashup.Trace Error: 24579 : {"Start":"2017-12-20T16:36:22.4820766Z","Action":"Engine/IO/Web/Request/GetResponse","HostProcessId":"19596","RequestMethod":"GET","RequestUri":"http://edgaronline.api.mashery.com/v2/corefinancials/ann?appkey=appkey&entityids=703428","RequestHasContent":"False","RequestHasHeaders":"False","RequestHasTimeout":"True","UseCache":"True","Exception":"Exception:\r\nExceptionType: System.Net.WebException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\r\nMessage: The remote server returned an error: (403) Forbidden.\r\nStackTrace:\n   at System.Net.HttpWebRequest.GetResponse()\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetWebResponse(WebRequest webRequest)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.CreateResponse(ResourceCredentialCollection credentials)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetResponseCore(ResourceCredentialCollection credentials)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.<>c__DisplayClass42.<GetResponse>b__40()\r\n   at Microsoft.Mashup.Engine1.Library.Common.RetryPolicy.Execute[TResult](IEngineHost host, Func`1 func)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetResponse(ResourceCredentialCollection credentials, RetryPolicy retryPolicy, AuthorizationThrowFunction authorizationFunction)\r\n\r\n\r\n","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:00:00.0444368"}
    DataMashup.Trace Information: 24579 : {"Start":"2017-12-20T16:34:23.7104723Z","Action":"SimpleDocumentEvaluator/GetResult<IDataReaderSource>","HostProcessId":"19596","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:01:58.8013295"}
    DataMashup.Trace Warning: 24579 : {"Start":"2017-12-20T16:34:25.3045127Z","Action":"RemotePageReader/RunStub","HostProcessId":"19596","Exception":"Exception:\r\nExceptionType: Microsoft.Mashup.Engine.Interface.ResourceAccessForbiddenException, Microsoft.MashupEngine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35\r\nMessage: Exception of type 'Microsoft.Mashup.Engine.Interface.ResourceAccessForbiddenException' was thrown.\r\nStackTrace:\n   at Microsoft.Mashup.Engine1.Library.Http.HttpServices.ThrowIfAuthorizationError(WebException exception, IResource resource)\r\n   at Microsoft.Mashup.Engine1.Library.Web.HttpRequest.ThrowIfAuthorizationError(WebException exception)\r\n   at Microsoft.Mashup.Engine1.Library.Web.Request.GetResponse(ResourceCredentialCollection credentials, RetryPolicy retryPolicy, AuthorizationThrowFunction authorizationFunction)\r\n   at Microsoft.Mashup.Engine1.Library.Web.WebContentsBinaryValue.GetResponse(Request request)\r\n   at Microsoft.Mashup.Engine1.Library.Web.WebContentsBinaryValue.Open()\r\n   at Microsoft.Mashup.Engine1.Runtime.BinaryValue.MetaTypeBinaryValue.Open()\r\n   at Microsoft.Mashup.Engine1.Runtime.BinaryValue.OpenText(Encoding encoding)\r\n   at Microsoft.Mashup.Engine1.Runtime.BinaryValue.OpenText(Value encoding)\r\n   at Microsoft.Mashup.Engine1.Library.Json.JsonModule.Json.DocumentFunctionValue.TypedInvoke(Value jsonText, Value encoding)\r\n   at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValue2`3.Invoke(Value arg0, Value arg1)\r\n   at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValue2.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Runtime.FunctionValue.MetaTypeFunctionValue.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Runtime.SerializationApplyingModule.SerializableFunctionValue.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Language.ExpressionInvocationExpression1.Execute(MembersFrame0& frame)\r\n   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue0.Invoke()\r\n   at Microsoft.Mashup.Engine1.Runtime.Library.Function.InvokeAfterFunctionValue.TypedInvoke(FunctionValue function, DurationValue delay)\r\n   at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValue2`3.Invoke(Value arg0, Value arg1)\r\n   at Microsoft.Mashup.Engine1.Runtime.FunctionValue.MetaTypeFunctionValue.Invoke(Value arg0, Value arg1)\r\n   at Microsoft.Mashup.Engine1.Runtime.SerializationApplyingModule.SerializableFunctionValue.Invoke(Value arg0, Value arg1)\r\n   at Microsoft.Mashup.Engine1.Language.ExpressionInvocationExpression2.Execute(MembersFrame1& frame)\r\n   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRecordValue.Force(Int32 index)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRecordValue.get_Item(Int32 index)\r\n   at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)\r\n   at Microsoft.Mashup.Engine1.Language.FieldAccessExpression.Execute(Value frame)\r\n   at Microsoft.Mashup.Engine1.Language.FieldAccessExpression.Execute(Value frame)\r\n   at Microsoft.Mashup.Engine1.Language.RuntimeFunctionValue1.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRecordValue.Force(Int32 index)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRecordValue.get_Item(Int32 index)\r\n   at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)\r\n   at Microsoft.Mashup.Engine1.Language.FieldAccessExpression.Execute(Value frame)\r\n   at Microsoft.Mashup.Engine1.Language.FieldAccessExpression.Execute(Value frame)\r\n   at Microsoft.Mashup.Engine1.Language.RuntimeFunctionValue1.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRecordValue.Force(Int32 index)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRecordValue.get_Item(Int32 index)\r\n   at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)\r\n   at Microsoft.Mashup.Engine1.Language.FieldAccessExpression.Execute(Value frame)\r\n   at Microsoft.Mashup.Engine1.Language.FunctionInvocationExpression2.Execute(Value frame)\r\n   at Microsoft.Mashup.Engine1.Language.RuntimeFunctionValue1.Invoke(Value arg0)\r\n   at Microsoft.Mashup.Engine1.Language.RecordExpression.RuntimeRe","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:01:57.2351187"}
    DataMashup.Trace Information: 24579 : {"Start":"2017-12-20T16:34:23.6973864Z","Action":"RemoteDocumentEvaluator/Service/OnBeginGetResult","HostProcessId":"19596","evaluationID":"1099","ProductVersion":"2.46.4732.721 (PQ-CY17SU05)","ActivityId":"78c2fd50-f70e-4518-914a-e54a6b511cfe","Process":"Microsoft.Mashup.Container.NetFX40","Pid":16308,"Tid":1,"Duration":"00:01:58.8424146"}

    Wednesday, December 20, 2017 5:15 PM

Answers

  • Hi Ed. Your code should do what you're wanting, since M is lazy (i.e. values only get calculated when they're asked for). However, you may need to adjust the backoff times.

    Were you able to get this to work?

    Ehren

    Wednesday, January 10, 2018 6:35 PM