none
Error: ServiceBusyException - ProxyAuthenticationRequired RRS feed

  • Question

  • Hello Everyone!

    I have configured a Cluster in 5 Virtual Machines: WN11, WN12, CN11, CN12, SQL11.

    It worked well but then something changed and it stopped working. I say "something" because I didn't change anything in the Server (neither a configuration file, nor a server config) but I'm not sure if any change was made by the Infraestructure Department. (They say no)

    What is happening now is that WN11 throws 'Microsoft.RServer.Domain.Exceptions.ServiceBusyException' when I try to execute remote code trought this Web Node. WN02 Works just fine.

    Some tests I've made:

    • Using  mrsdeploy package and login into WN11 Works if I keep session = FALSE parameter. I could also list the 96 webServices of the Cluster but I cant predict through the API (503). If I use session = TRUE parameter it throws 503 - ServiceBusyException.
    • Configuration Test and Get Raw Server Status fails and they print a BIG traceback in the screen.
    • Running diagnostics tests from WN12 works well
    • I tried installing and configuring a third computer as a web node and it Works
    • Re-Configuring WN11 with other Compute Nodes from other Cluster and trying to execute code from WN11 to those Compute Nodes also fails.

    Log Files:

    • stdout.txt is full of {"CorrelationId":"3f088e82-7ae2-488d-8dc3-401da8cffaab","Subject":{"uri":"http://192.0.3.156:12805","details":-1}}
    • A different error is being logged sometimes when trying to execute diagnostic test
    {"CorrelationId":"95afac8e-8939-4f59-a67c-78900be434dc","Subject":{"Message":"Exception of type 'Microsoft.RServer.Domain.Exceptions.ServiceBusyException' was thrown.","StackTrace":"   at Microsoft.RServer.Domain.Extensions.LoadBalancingStrategyExtensions.<InvokeOperationAsync>d__1`1.MoveNext() in C:\\swarm\\workspace\\deployr-9.0.2\\product\\server\\src\\Microsoft.RServer.Domain\\Extensions\\LoadBalancingStrategyExtensions.cs:line 45\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.RServer.Domain.Engine.ShellProxyFactory.<CreateShellAsync>d__3.MoveNext() in C:\\swarm\\workspace\\deployr-9.0.2\\product\\server\\src\\Microsoft.RServer.Domain\\Engine\\ShellProxyFactory.cs:line 28\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.RServer.Domain.Sessions.DefaultShellHandlingStrategy.<CreateShellAsync>d__3.MoveNext() in C:\\swarm\\workspace\\deployr-9.0.2\\product\\server\\src\\Microsoft.RServer.Domain\\Sessions\\DefaultShellHandlingStrategy.cs:line 35\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.RServer.Domain.Sessions.SessionService.<CreateSessionAsync>d__4.MoveNext() in C:\\swarm\\workspace\\deployr-9.0.2\\product\\server\\src\\Microsoft.RServer.Domain\\Sessions\\SessionService.cs:line 49\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.RServer.WebNode.Controllers.SessionsController.<CreateSessionAsync>d__5.MoveNext() in C:\\swarm\\workspace\\deployr-9.0.2\\product\\server\\src\\Microsoft.RServer.WebNode\\Controllers\\SessionsController.cs:line 59\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ObjectMethodExecutor.<CastToObject>d__40`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__27.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextExceptionFilterAsync>d__24.MoveNext()"}}
    • stderr.txt shows nothing.

    I can't reproduce it but once stderr.txt thrown a weird log with some HTML inside:

    {"CorrelationId":"b579ea31-8e96-4922-8b51-1e7cef76d6f7","Subject":"{\"StatusCode\":-1,\"Details\":{\"logPath\":\"C:\\\\Program Files\\\\Microsoft\\\\R Server\\\\R_SERVER\\\\o16n\\\\Microsoft.RServer.WebNode\\\\logs\"},\"ComponentStatus\":{\"authentication\":{\"StatusCode\":0,\"Details\":{},\"ComponentStatus\":{\"ldap\":{\"StatusCode\":0,\"Details\":{\"Host\":\"192.0.1.70\",\"Port\":3268,\"Version\":3},\"ComponentStatus\":null}}},\"backends\":{\"StatusCode\":-1,\"Details\":{},\"ComponentStatus\":{\"http://192.0.3.156:12805/\":{\"StatusCode\":-1,\"Details\":{\"exception\":{\"Message\":\"Generic BackEnd communication error: StatusCode=ProxyAuthenticationRequired; RequestUri=http://192.0.3.156:12805/status Content=<HTML><HEAD>\\n<TITLE>Access Denied</TITLE>\\n</HEAD>\\n<BODY>\\n<FONT face=\\\"Helvetica\\\">\\n<big><strong></strong></big><BR>\\n</FONT>\\n<blockquote>\\n<TABLE border=0 cellPadding=1 width=\\\"80%\\\">\\n<TR><TD>\\n<FONT face=\\\"Helvetica\\\">\\n<big>Access Denied (authentication_failed)</big>\\n<BR>\\n<BR>\\n</FONT>\\n</TD></TR>\\n<TR><TD>\\n<FONT face=\\\"Helvetica\\\">\\nYour credentials could not be authenticated: \\\"Credentials are missing.\\\". You will not be permitted access until your credentials can be verified.\\n</FONT>\\n</TD></TR>\\n<TR><TD>\\n<FONT face=\\\"Helvetica\\\">\\nThis is typically caused by an incorrect username and/or password, but could also be caused by network problems.\\n</FONT>\\n</TD></TR>\\n<TR><TD>\\n<FONT face=\\\"Helvetica\\\" SIZE=2>\\n<BR>\\nFor assistance, contact your network support team.\\n</FONT>\\n</TD></TR>\\n</TABLE>\\n</blockquote>\\n</FONT>\\n</BODY></HTML>\\n\",\"Data\":{},\"InnerException\":null,\"TargetSite\":{\"Name\":\"MoveNext\",\"DeclaringType\":\"Microsoft.RServer.Domain.Engine.RestClient+<HandleErrorResponseAsync>d__10, Microsoft.RServer.Domain, Version=1.0.0.0, Culture=neutral,

    The important piece of the log is: Generic BackEnd communication error: StatusCode=ProxyAuthenticationRequired after that it says: Your credentials could not be authenticated: "Credentials are missing. ". You will not be permitted access until your credentials can be verified. This is typically caused by an incorrect username and/or password, but could also be caused by network problems. For assistance, contact your network support.

    I think this may be the problem. The credentials are ok because I'm running same diagnostic from WN12 with same credentials and they work. (assuming this credentials are the same as configured in R Server with LDAP)

    I tried the utils::setInternet2(TRUE) solution but it didn't work, also because the other nodes don't have this problema and all the Machines are under the same network.

    Any ideas or thoughts about this behaviour? i've contacted Network Support (infraestructure team) and the say they weren't changes.

    Regards,

     

    <u5:p></u5:p>


    Thursday, August 31, 2017 3:59 PM