none
The Orchestrator Runbook Service is no longer running. Verify the status of the Runbook Server and try to restart Orchestrator Runbook Service. If the problem persists, contact Technical Support. RRS feed

  • Question

  • Hi All,

    The Orchestrator Runbook Service  is no longer running. Verify the status of the Runbook Server and try to restart Orchestrator Runbook Service. If the problem persists, contact Technical Support.Please let me know the remedy for the same.

    The orchestrator Runbook service  is stopping for every 5 days .we are trying to start ,later its getting Stopped , All other runbooks are getting stopped instantly due to this.

    Regards,

    Kiran.


    Phanikiran Pisipati

    Wednesday, November 18, 2015 11:42 AM

All replies

  • Did you check the windows event logs when it the service got stopped any warning or error. 

    Do you get any message when you try to start the service after being stopped. 


    Priyabrata

    Wednesday, November 18, 2015 12:36 PM

  • Phanikiran Pisipati

    Wednesday, November 18, 2015 12:47 PM
  • Can you please check on the system logs. The one which you have shown is the application event logs.  Can you see the error related to your service account .

    Priyabrata

    Wednesday, November 18, 2015 1:02 PM
  • "C:\ProgramData\Microsoft System Center 2012\Orchestrator\RunbookService.exe\Logs".

    pls chk the logs here too..


    Priyabrata

    Wednesday, November 18, 2015 1:20 PM
  • Hi 

    I have Checked the logs here ,but there are no logs  generated for the warning messages time shown in error message.

    Regards,

    Kiran


    Phanikiran Pisipati

    Wednesday, November 18, 2015 4:06 PM
  • Check the log location in the registry from the Runbook Server(s): HKLM\SOFTWARE\Wow6432Node\Microsoft\SystemCenter2012\Orchestrator\TraceLogger\RunbookService.exe

    www.sc-orchestrator.eu , Blog sc-orchestrator.eu

    Wednesday, November 18, 2015 4:10 PM
  • Hi Proyabrata,

    here is the logfile for RunbookService.exe

    Process ID: 2320
    Version   : 7.2.84.0
    Computer  : CO1STSCORCHRS02
    User      : REDMOND\stpatcha

    2015-11-16 22:33:00 [4104] 1 Exception caught in void __thiscall PublishedPoliciesControl::start(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,const class _variant_t &,const __int64 &,const class boost::optional<class Credential> &)
    PublishedPoliciesControl.cpp(74):
    <Exception>
    <Type>Opalis::Exception</Type>
    <Location>
    class boost::shared_ptr<class WorkflowControl> __thiscall ComWorkflowConnector::createWorkflow(const struct WorkflowHandle &,const __int64 &,const struct PolicyId &,const class boost::optional<class Credential> &)
    ComWorkflowConnector.cpp(126)
    </Location>
    <MsgCode>Cannot create workflow</MsgCode>
    <Params>
    <Param>1027</Param>
    <Param>166310</Param>
    </Params>
    <Prev><Exception>
    <Type>Opalis::Exception</Type>
    <Location>
    __thiscall WorkflowControlComProxy::WorkflowControlComProxy(const struct _GUID &,const struct WorkflowHandle &,const class boost::optional<class Credential> &,const class std::tr1::function<void __cdecl(class OpalisEvent const &)> &)
    WorkflowControlComProxy.cpp(44)
    </Location>
    <MsgCode>Cannot create WorkflowControl object</MsgCode>
    <Prev><Exception>
    <Type>Opalis::Exception</Type>
    <Location>
    __thiscall WorkflowControlComProxy::WorkflowControlComProxy(const struct _GUID &,const struct WorkflowHandle &,const class boost::optional<class Credential> &,const class std::tr1::function<void __cdecl(class OpalisEvent const &)> &)
    WorkflowControlComProxy.cpp(38)
    </Location>
    <MsgCode>_com_error</MsgCode>
    <Params>
    <Param>Server execution failed</Param>
    <Param></Param>
    <Param>-2146959355</Param>
    </Params>
    </Exception></Prev>
    </Exception></Prev>
    </Exception>

    Creating new trace log file


    Phanikiran Pisipati

    Thursday, November 19, 2015 6:02 AM
  • here is the logfile for RunbookService.exe

    Process ID: 2320
    Version   : 7.2.84.0
    Computer  : CO1STSCORCHRS02
    User      : REDMOND\stpatcha

    2015-11-16 22:33:00 [4104] 1 Exception caught in void __thiscall PublishedPoliciesControl::start(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,const class _variant_t &,const __int64 &,const class boost::optional<class Credential> &)
    PublishedPoliciesControl.cpp(74):
    < Exception>
    < Type>Opalis::Exception</Type>
    < Location>
    class boost::shared_ptr<class WorkflowControl> __thiscall ComWorkflowConnector::createWorkflow(const struct WorkflowHandle &,const __int64 &,const struct PolicyId &,const class boost::optional<class Credential> &)
    ComWorkflowConnector.cpp(126)
    < /Location>
    < MsgCode>Cannot create workflow</MsgCode>
    < Params>
    < Param>1027</Param>
    < Param>166310</Param>
    < /Params>
    < Prev><Exception>
    < Type>Opalis::Exception</Type>
    < Location>
    __thiscall WorkflowControlComProxy::WorkflowControlComProxy(const struct _GUID &,const struct WorkflowHandle &,const class boost::optional<class Credential> &,const class std::tr1::function<void __cdecl(class OpalisEvent const &)> &)
    WorkflowControlComProxy.cpp(44)
    < /Location>
    < MsgCode>Cannot create WorkflowControl object</MsgCode>
    < Prev><Exception>
    < Type>Opalis::Exception</Type>
    < Location>
    __thiscall WorkflowControlComProxy::WorkflowControlComProxy(const struct _GUID &,const struct WorkflowHandle &,const class boost::optional<class Credential> &,const class std::tr1::function<void __cdecl(class OpalisEvent const &)> &)
    WorkflowControlComProxy.cpp(38)
    < /Location>
    < MsgCode>_com_error</MsgCode>
    < Params>
    < Param>Server execution failed</Param>
    < Param></Param>
    < Param>-2146959355</Param>
    < /Params>
    < /Exception></Prev>
    < /Exception></Prev>
    < /Exception>


    Phanikiran Pisipati

    Thursday, November 19, 2015 6:07 AM
  • Policy Module exe

    Process ID: 10708
    Version   : 7.2.84.0
    Computer  : CO1STSCORCHRS02
    User      : REDMOND\stpatcha

    2015-11-16 19:14:05 [16976] 1 Name="Start" ID={AFF3CC78-C86B-4AA7-93AF-58C09627A490} :  Start executing QIK Object

    2015-11-16 19:16:15 [16976] 1 Name="Start" ID={AFF3CC78-C86B-4AA7-93AF-58C09627A490} :  Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

    Exception: SqlException
    Target site: SqlInternalConnection.OnError

    Stack trace:
       at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
       at System.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean encrypt, Boolean trustServerCert, Boolean& marsCapable)
       at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, I

    Creating new trace log file


    Phanikiran Pisipati

    Thursday, November 19, 2015 6:09 AM
  • Policy Module exe file:

    Process ID: 10708
    Version   : 7.2.84.0
    Computer  : CO1STSCORCHRS02
    User      : REDMOND\stpatcha

    2015-11-16 19:14:05 [16976] 1 Name="Start" ID={AFF3CC78-C86B-4AA7-93AF-58C09627A490} :  Start executing QIK Object

    2015-11-16 19:16:15 [16976] 1 Name="Start" ID={AFF3CC78-C86B-4AA7-93AF-58C09627A490} :  Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

    Exception: SqlException
    Target site: SqlInternalConnection.OnError

    Stack trace:
       at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
       at System.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean encrypt, Boolean trustServerCert, Boolean& marsCapable)
       at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, I

    Creating new trace log file


    Phanikiran Pisipati

    Thursday, November 19, 2015 6:09 AM
  • Hi All

    As per the analysis,

    We noticed as per the attached RunbookService.exe, Below error it showing up

    2015-11-16 22:33:00 [4104] 1 Exception caught in void __thiscall PublishedPoliciesControl::start(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,const class _variant_t &,const __int64 &,const class boost::optional<class Credential> &)

    PublishedPoliciesControl.cpp(74):

    In the PolicyModule.exe attached, It showing SQL Timeout Expired.

    2015-11-16 19:16:15 [16976] 1 Name="Start" ID={AFF3CC78-C86B-4AA7-93AF-58C09627A490} :  Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.


    Phanikiran Pisipati

    Thursday, November 19, 2015 6:14 AM
  • Alright, open a blank notepad file save as "test.udl"

    Double click on the udl file. Can you now enter the server name , uid and pwd of the service account used to connect the orchestrator db , from the drop down select the DB name. And click on test connection. 

    Run it from the Management server and runbook server both. Let me know the results.



    Priyabrata

    Thursday, November 19, 2015 8:32 AM
  • Hi

    The Test-Connection is Suceeded  for both the scenarios i.e  When I Run it from the Management server and runbook server both,

    and for your Information

    The orchestrator Runbook service ( state is set to automatic ) and is stopping intermittently and later starting by itself, All other runbooks are getting stopped instantly due to this.

    Regards,

    Kiran.


    Phanikiran Pisipati

    Thursday, November 19, 2015 9:05 AM
  • Do you have only one run book server? If not , are the other servers having same issues. 

    Can you pls verify the udl connection when the issue persists. 

    If you have scom in place, pls place a custom ping monitoring from the orchestrator server to DB server. This will let you know if there is any network latency or any other network related issue. 

    Since it is intermittent I should suggest run network trace with 300MB file log rotation so that you can capture any network related errors during the issue time. 

    If you need help for the network logging with rotation let me know. 


    Priyabrata

    Thursday, November 19, 2015 11:30 AM
  • Hello,

    On the same context we found some other pointers which might help us to track the issue. Please have a look.

    Scenario:

    >>18th Nov 2015(6:50PM) the Runbook Service at the Runbook Server crashed , and when we digged the error message, we got the below error recorded at RunBookService.exe Log.(No Error recorded at eventvwr)

    2015-11-18 18:50:08 [4104] 1 Exception caught in void __thiscall PublishedPoliciesControl::start(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,const class _variant_t &,const __int64 &,const class boost::optional<class Credential> &)
    PublishedPoliciesControl.cpp(74):
    <Exception>
    <Type>Opalis::Exception</Type>
    <Location>
    class boost::shared_ptr<class WorkflowControl> __thiscall ComWorkflowConnector::createWorkflow(const struct WorkflowHandle &,const __int64 &,const struct PolicyId &,const class boost::optional<class Credential> &)
    ComWorkflowConnector.cpp(126)
    </Location>
    <MsgCode>Cannot create workflow</MsgCode>
    <Params>
    <Param>1192</Param>
    <Param>167295</Param>
    </Params>
    <Prev><Exception>
    <Type>Opalis::Exception</Type>
    <Location>
    __thiscall WorkflowControlComProxy::WorkflowControlComProxy(const struct _GUID &,const struct WorkflowHandle &,const class boost::optional<class Credential> &,const class std::tr1::function<void __cdecl(class OpalisEvent const &)> &)
    WorkflowControlComProxy.cpp(44)
    </Location>
    <MsgCode>Cannot create WorkflowControl object</MsgCode>
    <Prev><Exception>
    <Type>Opalis::Exception</Type>
    <Location>
    __thiscall WorkflowControlComProxy::WorkflowControlComProxy(const struct _GUID &,const struct WorkflowHandle &,const class boost::optional<class Credential> &,const class std::tr1::function<void __cdecl(class OpalisEvent const &)> &)
    WorkflowControlComProxy.cpp(38)
    </Location>
    <MsgCode>_com_error</MsgCode>
    <Params>
    <Param>Server execution failed</Param>
    <Param></Param>
    <Param>-2146959355</Param>
    </Params>
    </Exception></Prev>
    </Exception></Prev>
    </Exception>

    >>18th Nov 2015 (6:56 PM) - The same was reported in the Management Server, with a very generic error Message.

    The Orchestrator Runbook Service on XXX Server is no longer running. Verify the status of the Runbook Server and try to restart Orchestrator Runbook Service. If the problem persists, contact Technical Support.

    NOTE:

    This issue is very intermittent, and as the Service is set to Automatic, we may not be able to repro the issue manually.


    Phanikiran Pisipati

    Thursday, November 19, 2015 12:17 PM
  • Well you may have to look into this blog.

    https://support.microsoft.com/en-us/kb/3069115

    Hope this rectifies your issue. Please access your environment as per the blog.


    Priyabrata

    Thursday, November 19, 2015 12:51 PM
  • Hi

    This seems to be similar issue but not the same issue ,as we compared both the logs. So roll up 7 installation may not help here .

    Regards,

    Kiran.


    Phanikiran Pisipati

    Thursday, November 19, 2015 1:02 PM