locked
Skype for Business 2015 Front End Server not Starting (Error: 0xe8, state: 4) RRS feed

  • Question

  • Hi technet,

    I've had a Skype for Business instance up and running for around a year and all was working fine. We had a move around of some of the servers so had to shut down Skype for Business, when I turned it back on the Front End Server wasn't coming up, making the instance unusable. I've spent the better part of a week trying to find a solution for this but haven't had any luck.

    The details of the issue are below.

    From Event Viewer, the first error I see is:

    Server local connection provider has stopped listening on [ \\.\pipe\SQLLocal\RTCLOCAL ] due to a failure. Error: 0xe8, state: 4. The server will automatically attempt to re-establish listening.

    This is following around 5 seconds later with:

    Server named pipe provider has stopped listening on [ \\.\pipe\MSSQL$RTCLOCAL\sql\query ] due to a failure. Error: 0xe8, state: 4. The server will automatically attempt to reestablish listening.

    Next with .NET runtime error:

    Application: RtcHost.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.ComponentModel.Win32Exception

    Exception Info: System.Data.SqlClient.SqlException
       at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)
       at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
       at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
       at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
       at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
       at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
       at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
       at System.Data.SqlClient.SqlConnection.Open()
       at Microsoft.Rtc.Common.Data.DBCore.PerformSprocContextExecution(SprocContext sprocContext)

    Exception Info: Microsoft.Rtc.Common.Data.SqlConnectionException
       at Microsoft.Rtc.Management.Store.Sql.XdsSqlConnection.ReadDocItems(ICollection`1 key)
       at Microsoft.Rtc.Management.ScopeFramework.AnchoredXmlReader.Read(ICollection`1 key)
       at Microsoft.Rtc.Management.ServiceConsumer.CachedAnchoredXmlReader.Read(ICollection`1 key)
       at Microsoft.Rtc.Management.ServiceConsumer.TypedXmlReader.RegisterChangeNotification(SchemaId schemaId, IList`1 scopeContextList, EventHandler`1 changeEventHandler)
       at Microsoft.Rtc.Management.ServiceConsumer.TopologyWatcher.RegisterForCertificateChanges()
       at Microsoft.Rtc.Management.ServiceConsumer.TopologyWatcher..ctor(TypedXmlReader typedXmlReader)
       at Microsoft.Rtc.Management.ServiceConsumer.TopologyConsumer..ctor()
       at Microsoft.Rtc.Management.ServiceConsumer.ServiceConsumer..ctor(RoleName roleName, Boolean ignoreMyServiceNotFound)
       at Microsoft.Rtc.OutboundRouting.Settings.Start(ManualResetEvent applicationEnabledEvent, RoleName roleName)
       at Microsoft.Rtc.OutboundRouting.OutboundRouting.Start(ApplicationManifest& am, ManualResetEvent applicationEnabledEvent, RoleName roleName)
       at Microsoft.Rtc.ApplicationService.ApplicationService.Start(EventWaitHandle shutdownEvent, RoleName roleName)
       at Microsoft.Rtc.ApplicationService.ApplicationService.Main(String[] args)
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssemblyByName(String assemblyName, String[] args)
       at System.AppDomain.ExecuteAssemblyByName(System.String, System.String[])
       at Microsoft.Rtc.AppDomainHost.Launcher+AppDomainLauncher.RunThread()
       at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
       at System.Threading.ThreadHelper.ThreadStart()

    And finally with Application error:

    Faulting application name: RtcHost.exe, version: 6.0.9319.259, time stamp: 0x57606ec7
    Faulting module name: KERNELBASE.dll, version: 6.3.9600.18292, time stamp: 0x56fcb50a
    Exception code: 0xe0434352
    Fault offset: 0x0000000000008a5c
    Faulting process id: 0x14a4
    Faulting application start time: 0x01d2fa3a2ac80e04
    Faulting application path: C:\Program Files\Skype for Business Server 2015\Server\Core\RtcHost.exe
    Faulting module path: C:\Windows\system32\KERNELBASE.dll
    Report Id: b3447eba-662d-11e7-80db-000c29443757
    Faulting package full name: 
    Faulting package-relative application ID: 

    My setup is as follows:

    • Windows 2012 Skype for business 2015 (192.168.5.25, sfb2015.lab.local)
    • Windows 2012 SQL server 2014 (192.168.5.26, sfb2015sql.lab.local)

    The database name is sfb2015db.

    Things I've done:

    • Ensure named pipes are enabled, as well as TCP/IP (on both SfB server and SQL 2014 server)
    • Restarted both the SQL server and the SfB 2015 server.
    • Setting up a new database to try connect to it (as a test)

    Unsure what to do next really, any tips would be greatly appreciated. All online resources I've found so far say the same thing - To check the names pipes are enabled.

    Thank you.

    Best,

    Gary

    Tuesday, July 11, 2017 12:59 PM

All replies

  • Hi gashergill,

    From your description, I understand that in your Skype for Business environment your FE server services cannot start successfully.

    I would like to confirm some situations with your help:

    1.Please help to simply explain your topology: how many FE servers, Back End servers?

    2.As you said that you have had a move around of some of the servers, please help to explain what did you do?

    3.How many Skype for Business services cannot start in FE servers?

    4.Is there any other error about Skype for Business server in Event Log?

     

    Based on our research, we firstly give you following suggestion:

    1.Please help to check the services “SQL Server (RTCLOCAL)” and “SQL Server(LYNCLOCAL)” are running.

    2.Try to restart Skype for Business services by the following command “Stop-CsWindowsServices -graceful” and “Start-CsWindowsServices”.

    If anything I can help you, please feel free to let me know.

    Thank you for your understanding and patience!

     

    Best Regards,

    Shaw Lu

    Wednesday, July 12, 2017 10:14 AM
  • Hi Shaw Lu,

    Thank you for the reply.

    1. We have a simple setup - A single Skype for Business server and no back end servers. We also have the SQL server.

    2. This was a physical move in the rack down one u. EDIT: Essentially, the server was turned off and back on.

    3. It's just the Front End service which is unable to run.

    4. Those are the only errors when starting the Front End service.

    With respect to your suggests:

    1. Both these services are running and can be restarted with no issue.

    2. So I actually ran through a number of these commands on the management shell this morning to try and figure out what's going on, and the end all of it is simple - RTCSRV fails to start in all cases. Please see errors below along with the command ran:

    command: sc queryex RTCSRV
    result: 1 STOPPED

    command: Start-CsWindowsService
    result: "Service RTCSRV failed to start. Check to make sure the service is not disabled. Failure to start could also be because this front end was previously failed over (using computer fail over), in which case please invoke fail back (using computer failback). Consult logs for further details.
    CategoryInfo: Operation Timeout: (:Sourcecollection) [Start-CsWindowsService], InvalidOperationException
    FullyQualifiedErrorId: ServiceStartFailed,Microsoft.Rtc.Management.Deployment.StartServiceCmdlet

    command: Invoke-CsComputerFailBack -ComputerName sfb2015.lab.local
    result: waiting for rtcsrv to be in routinggroupplacement state

    command: CsPoolRegistrarState -ResetType QuorumLossRecovery 
    result: WARNING: Waiting for RtcSrv service at sfb2015pool.lab.local to be Running

    command: Reset-CsPoolRegistrarState
    result: no issue, but didn't fix the problem

    So from the log file on Cs-StartWindowsService:

    └ ▼ RTCSRV

    Service start requested "Skype for Business Server Front-End (RTCSRV)".

    Service start requested "Skype for Business Server Front-End (RTCSRV)".

    ServiceStartFailed: Failed to start service.

    Error: Service RTCSRV entered into an unexpected state while waiting. Expected State: Running. Actual State:Stopped.
    ▼ Details
    └ Type: InvalidOperationException
    └ ▼ Stack Trace
        └   at Microsoft.Rtc.Management.Deployment.Core.NTService.WaitForStatus(ServiceControllerStatus expectedStatus, ServiceControllerStatus unexpectedStatus, TimeSpan TotalWaitTime) 
    at Microsoft.Rtc.Management.Deployment.Core.NTService.Start(Boolean wait)

    I've spent a while looking for ways to get the RTCSRV back up and running but I'm really at a loss now.

    Note that I re-ran the step 2 of the Sykpe for Business 2015 install, which is the components part, but to no avail.

    Thank you.

    Kind Regards,

    Gary Shergill


    • Edited by gashergill Wednesday, July 12, 2017 1:39 PM
    Wednesday, July 12, 2017 11:08 AM
  • Yes,This SQL is not the backend server.This is from the FE server SQL.Verify the SQL services are running fine in FE servers?does it listening?did you tried re-run the deployment wizard?

    Jayakumar K

    Wednesday, July 12, 2017 12:56 PM
  • Hi Jayakumar,

    The FE SQL services -  SQL Server (LYNCLOCAL, RTC, RTCLOCAL and SFB2015DB) - are all running fine.

    I've re-run the deployment wizard step 2:

    Install Or Update Skype for Business Server System > Step 2: Setup or Remove Skype for Business Components

    Kind Regards,

    Gary Shergill

    Wednesday, July 12, 2017 1:29 PM
  • Hi gashergill,

    Thanks for your reply.

    I checked the protocols for “RTCLOCAL” in my lab, “Named Pipes” disabled and “Shared Memory” and “TCP/IP” enabled. I think you could disable “Named Pipes”.

    You may check “Logon User” for the “RTCLOCAL”, it is “Network Service” by default; and also the ports in Firewall.

    Another point, have you installed any updates for Skype for Business? If not, you may try to install the latest update:https://support.microsoft.com/en-us/help/3061064/updates-for-skype-for-business-server-2015

    Thank you for your understanding and patience!

    Best Regards,

    Shaw Lu

    Thursday, July 13, 2017 7:10 AM
  • Hi Shaw Lu,

    The Logon User for SQL Server (RTCLOCAL), as well as the other SQL Server services, is "Network Service".

    The Firewall is disabled on both the SfB 2015 and SQL servers.

    I ran though the full suite of updates yesterday using Cumulative Server Update Installer, but unfortunately no change.

    Kind Regards,

    Gary Shergill

    Thursday, July 13, 2017 10:42 AM
  • Hi gashergill,

    Thanks for your reply.

    I checked lots of documents and similar cases, but didn’t find other new related resolutions.

    So could you re-run all the deployment wizard steps (step1~3) to see if it can be luck. 

    Hope any one in our forum can share the experience when encountering this kind of issue.

    Best Regards,
    Shaw Lu

    Monday, July 17, 2017 10:50 AM
  • Hi Shaw Lu,

    Thank you again for your time. I've re-run those steps but unfortunately no joy, still failing to start up on the same issue.

    I'm on holiday next week, when I come back I may do a complete reinstall on a new box as a solution may not be out there.

    Kind Regards,

    Gary Shergill

    Monday, July 17, 2017 3:07 PM
  • Hi,

    Just thought I'd update here, I'm still seeing the same problem. I had this working for a short while following the steps in the final post here:
    https://social.technet.microsoft.com/Forums/ie/en-US/5a486300-b66e-4067-a5d9-1b975bab8411/the-skype-for-business-server-frontend-service-terminated-with-the-following-servicespecific?forum=lyncdeploy

    However, after the last restart of the Skype for Business server I'm still running into the same issue.

    I really don't understand what's going on here, I'm just constantly seeing this same RTCSRV failure every time I try to start Skype for Business front-end server.

    Thanks!

    Best,

    Gary

    Monday, January 8, 2018 4:48 PM
  • Hello,

    had a similiar problem where Skype Frond End Services would not start with the same error.

    Solved it by:

    - Running the Deployment Wizard as Admin, choose Install or Update Skype for Business Server System and reran Step2: Setup or Remove Skype for Business Server Component

    - Run Skype for Business Management Shell as Admin, type Enable-CsComputer, Start-CsWindowsService

    After that it was working again.

    Hope it helps and Best Regards

    Timo

    Wednesday, August 8, 2018 9:07 AM