none
Retrieving R object exception: R session object (variableName) could not be fetched, rse=eval failed, request status: error code: 127 RRS feed

  • General discussion

  • I am currently facing the following issue with DeployR Open 8.0.0 in our production environment:

    Retrieving R object exception: R session object (PosteriorProbabilityDS1) could not be fetched, rse=eval failed, request status: error code: 127

    We're using a shared connection by our application, to the R-Server, to execute calculations against R, passing inputs and retrieving outputs dynamically.  After each execute, we're recycling the session to prevent any data overflow between executions as well.  However, sporadically, we get an error where it cannot pull the outputs back out of the workspace.  Here is an excerpt from the catalina.out file when debug tracing is enabled and our execution is running:

    [ INFO 22:43:36:46] ServerFilters DEPLOYR-EVENT[API-POST][192.168.33.92][F0D8D0D3D029CBA39A4AC0C695F857B5][admin][/deployr/r/project/execute/code]
    [ INFO 22:43:36:57] ServerFilters DEPLOYR-EVENT[API-PRE][192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][admin][/deployr/r/project/execute/code][nan:, consoleoff:False, tag:, format:json, code:##########################################################
    # Name: Post Prob By DS
    ########################


    PPDS1 <- IEDS1/ (IEDS1+ IEDS2+ IEDS3);
    PPDS2<- IEDS2/ (IEDS1+ IEDS2+ IEDS3);
    PPDS3<- IEDS3/ (IEDS1+ IEDS2+ IEDS3);

    print (PPDS1);
    print (PPDS2);
    print (PPDS3);, infinity:, enableConsoleEvents:False, graphicsheight:0, project:PROJECT-4f1cfccd-73cf-42e4-a498-ca120ea40a42, inputs:{
      "IEDS1": {
        "rclass": "numeric",
        "type": "primitive",
        "value": 0.4
      },
      "IEDS2": {
        "rclass": "numeric",
        "type": "primitive",
        "value": 0.045
      },
      "IEDS3": {
        "rclass": "numeric",
        "type": "primitive",
        "value": 0.010000000000000002
      }
    }, preloadbydirectory:, graphicswidth:0, graphics:, robjects:PosteriorProbabilityDS1,PosteriorProbabilityDS2,PosteriorProbabilityDS3, echooff:False, encodeDataFramePrimitiveAsVector:False]
    [ INFO 22:43:36:59] User getAuthorities: admin roles=[ADMINISTRATOR]
    [ INFO 22:43:36:66] ServerFilters DEPLOYR-EVENT[API-PRE][192.168.33.92][F0D8D0D3D029CBA39A4AC0C695F857B5][admin][/deployr/r/project/recycle][project:PROJECT-d9ea4286-0ab9-4a38-ba31-0eda84634ee0, format:json]
    [ INFO 22:43:36:68] ResponseBuilder DEPLOYR-EVENT[RESPONSE][F0D8D0D3D029CBA39A4AC0C695F857B5][deployr:[response:[success:true, call:/r/project/recycle, httpcookie:F0D8D0D3D029CBA39A4AC0C695F857B5, project:[project:PROJECT-d9ea4286-0ab9-4a38-ba31-0eda84634ee0, name:null, descr:null, longdescr:null, live:true, shared:false, cookie:null, origin:Project original., author:admin, authors:[admin], lastmodified:1474292616012]]]]
    [ INFO 22:43:36:69] ServerFilters DEPLOYR-EVENT[API-POST][192.168.33.92][F0D8D0D3D029CBA39A4AC0C695F857B5][admin][/deployr/r/project/recycle]
    [ INFO 22:43:36:149] ResponseBuilder DEPLOYR-EVENT[RESPONSE][BA36DBE5C5168CB0D4123B757D4854E3][deployr:[response:[success:false, call:/r/project/execute/code, httpcookie:BA36DBE5C5168CB0D4123B757D4854E3, interrupted:false, workspace:[objects:[]], project:[project:PROJECT-4f1cfccd-73cf-42e4-a498-ca120ea40a42, name:null, descr:null, longdescr:null, live:true, shared:false, cookie:null, origin:Project original., author:admin, authors:[admin], lastmodified:1474292616113], repository:[files:[]], execution:[console:, code:##########################################################
    # Name: Post Prob By DS
    ########################

    PPDS1 <- IEDS1/ (IEDS1+ IEDS2+ IEDS3);
    PPDS2<- IEDS2/ (IEDS1+ IEDS2+ IEDS3);
    PPDS3<- IEDS3/ (IEDS1+ IEDS2+ IEDS3);

    print (PosteriorProbabilityDS1);
    print (PosteriorProbabilityDS2);
    print (PosteriorProbabilityDS3);, interrupted:false, results:[], artifacts:[], warnings:[In strptime(paste(.leap.seconds, "23:59:60"), "%Y-%m-%d %H:%M:%S") :,  unable to identify current timezone 'C':, set environment variable 'TZ', In strptime(paste(.leap.seconds, "23:59:60"), "%Y-%m-%d %H:%M:%S") :,  unknown timezone 'localtime', Retrieving R object exception: R session object (PPDS1) could not be fetched, rse=eval failed, request status: error code: 127, Retrieving R object exception: R session object (PPDS2) could not be fetched, rse=eval failed, request status: error code: 127, Retrieving R object exception: R session object (PPDS3) could not be fetched, rse=eval failed, request status: error code: 127], info:[], help:null, execution:EXEC-c0c7dea0-2676-4bc7-be95-14f1aa61962e, timeStart:1474292616058, timeCode:15, timeTotal:55, tag:null, actor:admin, resultsGenerated:0, resultsAvailable:0], error:cannot open the connection, errorCode:900]]]
    [ INFO 22:43:36:150] ServerFilters DEPLOYR-EVENT[API-POST][192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][admin][/deployr/r/project/execute/code]
    [ INFO 22:43:36:215] ServerFilters DEPLOYR-EVENT[API-PRE][192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][admin][/deployr/r/project/close][project:PROJECT-4f1cfccd-73cf-42e4-a498-ca120ea40a42, format:json]
    [ INFO 22:43:36:254] LiveEngineService DEPLOYR-EVENT[GRID-SESSION]: RELEASED LiveToken[C]: [192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][PROJECT-4f1cfccd-73cf-42e4-a498-ca120ea40a42][admin] [Temporary Project false projectClose NODE-e04412e2-48d6-4206-82c7-d0e84fd4c678 DeployR Default Node Authenticated ]
    [ INFO 22:43:36:254] ResponseBuilder DEPLOYR-EVENT[RESPONSE][BA36DBE5C5168CB0D4123B757D4854E3][deployr:[response:[success:true, call:/r/project/close, httpcookie:BA36DBE5C5168CB0D4123B757D4854E3]]]
    [ INFO 22:43:36:255] ServerFilters DEPLOYR-EVENT[API-POST][192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][admin][/deployr/r/project/close]
    [ INFO 22:43:36:267] ServerFilters DEPLOYR-EVENT[API-PRE][192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][admin][/deployr/r/user/logout][format:json]
    [ INFO 22:43:36:268] ResponseBuilder DEPLOYR-EVENT[RESPONSE][BA36DBE5C5168CB0D4123B757D4854E3][deployr:[response:[success:true, call:/r/user/logout, httpcookie:BA36DBE5C5168CB0D4123B757D4854E3]]]
    [ INFO 22:43:36:269] ServerFilters DEPLOYR-EVENT[API-POST][192.168.33.92][BA36DBE5C5168CB0D4123B757D4854E3][null][/deployr/r/user/logout]

    Has anyone else seen behavior like this?  We are executing a high volume of calculations through our shared connection to the R Server, thus the recycling of the session after each calculation.  This error appears very random, as it doesn't always happen on the same script or at the same point in a job.  The only thing is reproducible is when we have 2 users executing analyses concurrently, each with their own connection to R server to execute calculations with.  One user, everything is fine, as soon as 2 login, then we sporadically get this issue.

    Items to Note:

    1. We're using DeployR Open 8.0.0, and the 8.0.0 Client Libraries (not RBroker).

    2. We're executing all calculations over an authenticated user with an authenticated, temporary project.  We are using the same credentials to the R Server for all application users, as you would with a database connection.  

    3. Have ran this against a developer edition of DeployR Server (similar to Enterprise Edition) and sporadically getting the issue there as well.

    4. checked the MongoDB, Catalina.out and all logs that DeployR contains to find information.  The server we're running DeployR Server on is within the specs identified by Revolution and Microsoft.

    Execution Stats:

    User 1:

    - Executing 26 analyses, with 154 calculations per analysis, in R

    User 2

    - Executing 21 analyses, with 154 calculations per analysis, in R

    When only one user runs their analyses, everything is fine.  When both run them, we see issues (sporadically) with the 

    Any help would be GREATLY appreciated as this is a current production issue.

    Sincerely,

    SEHUNLEY

    • Changed type sehunley Monday, September 19, 2016 9:05 PM Analyzing our design approach, need more feedback.
    Monday, September 19, 2016 3:56 PM

All replies

  • Hello,

    How are you calling the code execution? Are you passing any project parameters? As I go through your logs the executed project and the recycle project are different. Are you using dropWorkspace option as TRUE while recycling the project.

    I encountered similar issue with my application. My application is similar to yours where multiple users are using the same R scripts for their calculations. For each users I create 2 projects for their analysis. I keep the track of projects per users so when they submit the script it is on that particular project ID.

    Is it possible for you to explain how are you calling the these R codes?

    Regards,

    Anish


    Sunday, September 25, 2016 9:51 PM
  • Hi Anish, thanks for reviewing my post.  Each user in my application gets a project, and executes 1 to many r-scripts within the project to complete their workflow.  Between each R-Script execution, I execute a project.recycle().  I don't believe I have set a drop workspace option to true.  Is it a default?  

    My application creates a project to be shared, per user, for their workflow (calculations to be executed).  For each R-Script in their calculation workflow, I gather inputs from our application and create inputs and outputs to DeployR, using the .NET Client libraries.  I then execute the calculations, passing the raw R code and options with rinputs and routputs, then pull the variables, plots, etc from the results.  Once this is done, I recycle the project so I am ready for the next execution.  

    My intent, since sometimes users execute the sames scripts multiple times within the same project, was to clear the workspace after each execution, to give the user a "clean slate" to work with.  I haven't seen issues until the latest client, has put more load on the application than previous ones.  And they are performing these large scale calculation workflows across multiple users.  This hasn't been a problem, until I started getting errors such as "cannot open the connection" and issues reading output variables and sometimes the input variables to scripts.  

    It's almost as though the workspace is being cleared by one user's project, but affecting another user's workspace while scripts are being executed, but that shouldn't happen unless threads are crossing.  Each user has a unique connection/session or project to work in.  I see that each has a unique guid.  If I place waits in the code, to slow it down, the issue goes away, but if the application runs at full speed, these issues are happening sporadically.  At any one time, there are only 2 projects on the grid, so I know it's not a resource issue on the grid.

    I am really perplexed as I see this issue on the open and Enterprise version of the DeployR server.  I hope I have explained how I am using the DeployR server and API, but if I haven't I can send or Post a code sample of what we doing.  I'll look into the drop workspace option, though.

    Thanks again for responding to my post.

    -Scott



    • Edited by sehunley Monday, September 26, 2016 7:04 PM
    Monday, September 26, 2016 2:16 AM
  • Hello Scott,

    Thank you for explaining the issue in detailed manner. What I understand from your explanation I come down to 2 points.

    How are you encoding your inputs and decoding the outputs?

    When you execute the code, are you sending additional parameters like Project Execute Options?

    It will be very helpful if you can post a snippet of your code as it will be easy to understand the flow and i can replicate the same in my environment.

    Thanks,

    -Anish

    Tuesday, September 27, 2016 6:12 AM
  • Anish,

      My application passes in "parameters" (inputs and outputs) that have been defined by the user and then, through our UI, linked to a workflow that will pass variables in from queries, outcomes from other R-Scripts, etc...  I take these parameters and convert them to ProjectExecutionOptions as follows:

    private RData MapRParameterToRVariable(DeployRCommandParameter parameter)
    {
        switch (parameter.DataType)
        {
            case RParameterDataType.Character:
                return RDataFactory.createString(parameter.Name, parameter.Value as string);
            case RParameterDataType.CharacterVector:
                return RDataFactory.createStringVector(parameter.Name, parameter.Value as List<string>);
            case RParameterDataType.CharacterMatrix:
                return RDataFactory.createStringMatrix(parameter.Name, parameter.Value as List<List<string>>);
            case RParameterDataType.Numeric:
                return RDataFactory.createNumeric(parameter.Name, (parameter.Value as double?).Value);
            case RParameterDataType.NumericVector:
                return RDataFactory.createNumericVector(parameter.Name, parameter.Value as List<double?>);
            case RParameterDataType.NumericMatrix:
                return RDataFactory.createNumericMatrix(parameter.Name, parameter.Value as List<List<double?>>);
            case RParameterDataType.Date:
            case RParameterDataType.DateTime:
                return CreateRIsoDateString(parameter.Name, parameter.Value, DateTime.MinValue);
            case RParameterDataType.DateVector:
                return RDataFactory.createDateVector(parameter.Name, CreateRIsoDateStrings(parameter.Value as IList<DateTime?>, DateTime.MinValue), "yyyy-MM-dd HH:mm:ss");
            case RParameterDataType.Logical:
                return RDataFactory.createBoolean(parameter.Name, (parameter.Value as bool?).Value);
            case RParameterDataType.LogicalVector:
                return RDataFactory.createBooleanVector(parameter.Name, parameter.Value as List<bool?>);
            case RParameterDataType.LogicalMatrix:
                return RDataFactory.createBooleanMatrix(parameter.Name, parameter.Value as List<List<bool?>>);
            case RParameterDataType.Binary:
                return null;
            case RParameterDataType.DataFrame:
                return RDataFactory.createDataFrame(parameter.Name, parameter.Value as List<RData>);
            default:
                throw new Exception("Incorrect Data Type defined for Parameter: " + parameter.Name);
        }
    }

    Each DeployRParameter just contains 

    public class DeployRCommandParameter : IRCommandParameter
    {
        public string Name { get; set; }
     
        public object Value { get; set; }
     
        public RParameterDataType DataType { get; set; }
     
        public RParameterDirection Direction { get; set; }
     
        public DeployRCommandParameter(string ParameterName, RParameterDataType ParameterDataType, RParameterDirection ParameterDirection, object ParameterValue)
        {
            Name = ParameterName;
            Value = ParameterValue;
            DataType = ParameterDataType;
            Direction = ParameterDirection;
        }
    }

    This  I also take the parameters that are outputs and create a list of there names to populate the routputs: 

    // Map the Output Parameters to the Project
    projectExecutionOptions.routputs = Parameters.Where(x => x.Direction == RParameterDirection.Output).Select(x => x.Name).ToList();

    So, that is how I populate the inputs and outputs of the R-Script.  We allow our users to define their own R-Scripts, or to modify/use predefined R-Scripts delivered with our application.  These scripts are persisted to our data store in a JSON format and pulled back prior to execution.  The raw code to be executed, plus the metadata for parameter definition is retrieved and used to prepare and execute the R-Script.  Once we have the code and our parameters, we execute the R-Script as follows:

    // Execute the R-Code via the Authenticated Project
    projectExec = RSession.executeCode(CommandText, projectExecutionOptions);
     
    // Workspace objects are where we'll find our output variables
    var workspaceObjects = projectExec.about().workspaceObjects;

    Once this is done, I pull out the workspace objects so we can retrieve our values, plots, files, etc..  What ever the user has defined that needs to be returned.

    // look up the output variables supplied and get the value if they wrote the script correctly it should be there, 
    // else we give them null...
    var outParameters = Parameters.Where(x => x.Direction == RParameterDirection.Output).ToList();
    for (int i = 0; i < outParameters.Count; i++)
    {
        var workspaceObject = workspaceObjects.Where(w => w != null && w.Name.Trim() == outParameters[i].Name.Trim()).FirstOrDefault();
        if (workspaceObject != null)
        {
            /*
            In code below, if standard parsing does not work, first 19 characters of date string are considered.
            This is because RScript attaches timezone information to returned date string. Standard date value never goes beyond 19 chars.
            Eg: when date string is "2016-03-03 08:08:08 EST"
            */
            RParameterDataType fieldDataType = Parameters[Parameters.IndexOf(outParameters[i])].DataType;
            if (fieldDataType == RParameterDataType.DateTime || fieldDataType == RParameterDataType.Date)
            {
                DateTime outputDateValue;
                string date = workspaceObject.Value.ToString();
                if (DateTime.TryParse(date, out outputDateValue) || 
                    DateTime.TryParse(date.Substring(0, 19), out outputDateValue))
                {
                    outputDateValue = DateTime.SpecifyKind(outputDateValue, DateTimeKind.Utc);
                    Parameters[Parameters.IndexOf(outParameters[i])].Value = outputDateValue;
                }
            }
            else if(fieldDataType == RParameterDataType.DateVector)
            {
                if(workspaceObject.Value != null)
                {
                    if((workspaceObject.Value as List<DateTime?>) != null)
                    {
                        Parameters[Parameters.IndexOf(outParameters[i])].Value = workspaceObject.Value as List<DateTime?>;
                    }
                    else if((workspaceObject.Value as List<string>) != null)
                    {
                        List<DateTime?> dateList = new List<DateTime?>();
                        foreach(string date in (workspaceObject.Value as List<string>))
                        {
                            
                            DateTime outputDateValue;
                            if (DateTime.TryParse(date, out outputDateValue) ||
                                DateTime.TryParse(date.Substring(0, 19), out outputDateValue))
                            {
                                dateList.Add(DateTime.SpecifyKind(outputDateValue, DateTimeKind.Utc));
                            }
                            else
                            {
                                dateList.Add(null);
                            }
                        }
     
                        Parameters[Parameters.IndexOf(outParameters[i])].Value = dateList;
                    }
                }
            }
            else
            {
                Parameters[Parameters.IndexOf(outParameters[i])].Value = workspaceObject.Value;
            }
        }
    }

    This is where I consider a calculation complete and I then recycle:

    RSession.recycle();
    

    RSession, is just a parameter that is an instance of RProject, created and shared across multiple calculations, for performance reasons.

    Created as follows: 

    RSession = rUser.createProject();
    

    In the current scenario that is causing errors our client has 2 users operating concurrently with workflows, that when combined, can total some 7300 R-Script executions.  It is under this load that I am seeing issues with the "cannot open the connection" or it fails to find an input or output variable in the workspace.  Occaisionally there is a memory issue, but that was determined to be a resource issue on their server and has been resolved.  I can send you a copy of the actually implementation that I have if you want, to allow for testing on your side.  

    But the gist is we login (connect, authenticate and create the temporary project), then for each script:

    1. we map, create and encode inputs

    2. create outputs ( a list)

    3. pass the code and the parameters (as ProjectExecutionOptions) to the DeployR API for execution.

    4. Pull back the outputs from the workspace, populate our output parameters and pass back to the application.

    5. recycle the workspace on the session (RProject) to prepare for the next execution.

    This seemed pretty simple, and I may change my design to not match the familiar interface to calling database executions, but still, I can't seem to track down where the errors are coming from.  Let me know your thoughts on this and if you want me to zip up a sample project with my implementation and a few sample scripts to test with. 

    Thanks in advance,

    Scott

    Tuesday, September 27, 2016 1:13 PM
  • Hi Scott and Anish 

    I am having the same problem, 

    When I try to fetch objects (strings) that are larger than 32775 characters. I get the same error 

    I am running DeployR versio 8 on SQL server 2016 and using .net API 

    I have tried to dig through the problem and I am getting hints that it might have something to do with the TomCat server settings. 

    Have you been able to resolve the issue ? 

    command (C#) : 

    var X = exec.about().workspaceObjects;

    Exception error: 

    Retrieving R object exception: R session object (Stats) could not be fetched, rse=eval failed

    Thanks 

    Sam

    Monday, November 14, 2016 7:29 PM