locked
Skype for Business Front End Service Will not Start RRS feed

  • Question

  • Hey guys. I have a brand new Skype for business Enterprise pool set up with a dedicated, brand new SQL server back end. I published the topology with no issue and installed Skype for Business with no issues on all three of my enterprise front end pool servers. But when I went to start the services I was unable to.

    After some digging i determined that the Skype for Business Server Front-End server service would not start. The event log reveals an error stating: "Enterprise Edition Server successfully registered with the back-end, But a stored procedure version mismatch was detected. the service will not start until this problem is resolved." I began this installation with the RTM installation media. So far I have tried:

    • Googling for hours (haha)
    • Reboot all servers in environment multiple times
    • Reset-CsPoolRegistrarState -ResetType Fullreset
    • Install-CsDatabase –ConfiguartionDatabases –SqlServerFqdn MySQLServer -Verbose
    • Install-CsDatabase –CentralManagementDatabase –SqlServerFqdn MySQLServer -Verbose
    • Republished topology
    • Ran June CU installer on all servers
    • Rebooted entire environment
    • Re-ran above steps after updating
    • Ran Windows update on my SQL servers just in case
    • Install-CsDatabase -Update -DatabaseType Registrar
    • Reboot again

    After completing all of this I still have made zero progress and I cannot, for the life of me, figure out how to get past this annoying issue. Has anybody here ever seen/successfully resolved this?

    Thanks for any and all help.

    Thursday, July 14, 2016 7:45 PM

Answers

  • Just wanted to follow-up with status,

    Does the newly installed Pool bring the FE services to "Running" ? 


    Regards, Rajukb | MCSE (Communication ), MCSA (o365) ,Certified "Lync server 2013 depth support engineer"| This posting is providedwith no warranties and confers no rights. If my reply answers your question please mark as answer/helpful if its helpful.

    • Marked as answer by Brolide Monday, July 18, 2016 5:58 PM
    Saturday, July 16, 2016 5:21 PM
  • Have you tried running start-cspool -poolfqdn pool1.domain.com -verbose on one FE? to attempt to start all servers services? 

    Also the fact that you are getting this problem even with a rebuild, it is unlikely that windows updates is your answer. 

    I am not familiar with the blog you mention, so there may be a few assumptions with the blog that your environment may not match. Perhaps have a read of this ebook written by one of the more famous MVPs

    https://gallery.technet.microsoft.com/lync/Skype-for-Business-Server-f4dac647

    With regard to the "missing" intermediate certs. you can check this by looking at the issued certificates to the server and look at the certification path. all certs between the issued cert and root cert are intermediates. If there are any listed all of them should be listed in intermediate stores.

    Another note, disable windows firewall on server to see if that is preventing the starting of services, and also put the servers into a sterile OU not affected by any group policies that could be preventing things too.

    thanks


    Note: Please remember to `Mark as Answered` a post that answers your question and/or `Vote as Helpful` posts that have helped you. This will help others find answers to similar problems. For more Skype for Business help visit: http://www.skype4b.uk Please note that answers are based on my experience and opinion only and do not necessarily represent the views of my employer.

    Monday, July 18, 2016 8:57 AM

All replies

  • Is everything with your certificates in order? Did they install ok etc?

    If you look at event viewer under Application and Services Logs>Lync Server did you have anything else that might be a clue?

    Thursday, July 14, 2016 8:07 PM
  • Hello. There are some other errors but they seem very likely to be caused by my inability to start the pool and get the servers communicating with one-another.

    For example, I see Error 32178 indicating that the server has failed to sync data for routing group {ID}. Then I get a lot of 56011 which states that "One or more Health Agent probes encountered an unexpected error. The component/Services intended to be monitored by the probe may be functioning correctly." But nothing about certificates and I did get my certs from an internal CA that we have setup in our domain and they seem happy. 

    Thursday, July 14, 2016 8:19 PM
  • Brolide,

    I see that you already ran windows update. Run windows update on your front ends in addition to your backend if that has not been completed.

    Update your backend database.

    Follow this KB article go to step 2. https://support.microsoft.com/en-us/kb/2809243

    After all of this run.

    Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn "SQL NAME.litwareinc.com"

    Verify that the versions the most recent. https://blogs.technet.microsoft.com/dodeitte/2013/07/02/how-to-verify-if-lync-server-2013-database-updates-completed-successfully/

    Let us know if this works,

    Don

    Thursday, July 14, 2016 8:38 PM
  • Hi

    Are the front end servers running Server 2012 R2?

    It may seem odd, but can you check the local server trusted root certificates store on each front end. What you are looking for are NON Self Signed Certificates. e.g. issued to cybertrust intermediate CA signed by cybertrust root CA for example. If you find these move them to the correct stores i.e. intermediate or personal if they are device certificates.

    Then reboot all front end servers.

    I have seen it in the past and there is a bug that prevents the RTCSRV service from starting if certificates are stored in the incorrect stores.

    thanks


    Note: Please remember to `Mark as Answered` a post that answers your question and/or `Vote as Helpful` posts that have helped you. This will help others find answers to similar problems. For more Skype for Business help visit: http://www.skype4b.uk Please note that answers are based on my experience and opinion only and do not necessarily represent the views of my employer.

    Thursday, July 14, 2016 8:43 PM
  • Have you tried this ? check if the winfabric is online or offline

    http://techbridle.blogspot.com/2016/06/front-service-in-lync-server-2013.html

    however, as Mark said you can try to run this command to find the certificates which are misplaced in Trusted root certificate authority. then move them to right location and reboot server.

    Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Format-List * | Out-File "c:\computer_filtered.txt"


    Regards, Rajukb | MCSE (Communication ), MCSA (o365) ,Certified "Lync server 2013 depth support engineer"| This posting is providedwith no warranties and confers no rights. If my reply answers your question please mark as answer/helpful if its helpful.


    Thursday, July 14, 2016 8:47 PM
  • Hi

    Are the front end servers running Server 2012 R2?

    It may seem odd, but can you check the local server trusted root certificates store on each front end. What you are looking for are NON Self Signed Certificates. e.g. issued to cybertrust intermediate CA signed by cybertrust root CA for example. If you find these move them to the correct stores i.e. intermediate or personal if they are device certificates.

    Then reboot all front end servers.

    I have seen it in the past and there is a bug that prevents the RTCSRV service from starting if certificates are stored in the incorrect stores.

    thanks


    Note: Please remember to `Mark as Answered` a post that answers your question and/or `Vote as Helpful` posts that have helped you. This will help others find answers to similar problems. For more Skype for Business help visit: http://www.skype4b.uk Please note that answers are based on my experience and opinion only and do not necessarily represent the views of my employer.

    I have run into this as well, it ended up being one intermediate cert was missing from the store on a non working FE, comparing manually against a working FE I saw the extra cert in there and exported/imported to fix.

    Twas a nightmare!

    Thursday, July 14, 2016 8:54 PM
  • Hello all, thanks for the replies. For the certificates, I actually tried that powershell script before and did not get any results. I also went into Trusted Root Certification Authorities and checked the certificated in there. All had matching Issued to/from. I would, however, like some elaboration on a MISSING certificate from intermediate authorities. Seeing as how all of my servers are not coming up, and not just the one, how would I check for this?

    As far as windows updates and checking the databases, I will check on that first thing tomorrow afternoon when I get into the office and get back to you guys ASAP as I truly do appreciate all of the support you guys are giving and I know there were a couple of updates those boxes were eligible for. I will say, however, that I did try step 2 in https://support.microsoft.com/en-us/kb/2809243 already but I will give that another shot. I have not yet tried looking at

    https://blogs.technet.microsoft.com/dodeitte/2013/07/02/how-to-verify-if-lync-server-2013-database-updates-completed-successfully/

    As suggested above.

    In the meantime would anyone else have any further ideas? I am starting to get to this point where I am thinking about just scrapping this and rebuilding the whole thing from scratch as this is a test lab but I'm afraid I'm just going to run into the issue again when the time comes. Thanks again SO much you guys.

    Friday, July 15, 2016 1:43 AM
  • Did you get a chance to verify the winfabric status ? when you see everything seems working and no idea where the issue is ? then check the winfabric status

    get the Lync server events and winfabric events post here to provide more help.

    i have covered all the details in this article which is related to Lync 2013 but there is much difference in Lync 2013 and SFB Server.

     http://techbridle.blogspot.com/2016/06/front-service-in-lync-server-2013.htm


    Regards, Rajukb | MCSE (Communication ), MCSA (o365) ,Certified "Lync server 2013 depth support engineer"| This posting is providedwith no warranties and confers no rights. If my reply answers your question please mark as answer/helpful if its helpful.


    Friday, July 15, 2016 6:38 AM
  • Hi Brolide,

    Please make sure that you have applied the back-end database update before or right after applying the front-end updates.

    Then after update BE update, make sure all FE Server update to the latest version as well.

    For more information, please refer to

    http://networksteve.com/windows/topic.php/Skype_for_Business_Server_Front-End_will_not_start/?TopicId=74464&Posts=2

    Please check the steps that you deployed for  SFB enterprise edition

    https://blogs.technet.microsoft.com/canitpro/2015/05/05/step-by-step-installation-of-skype-for-business-server-2015/

    Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.

    Best regards,

    Alice Wang


    Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Friday, July 15, 2016 7:06 AM
  • Hello all. Small update, I have done the following:

    • Checked windows fabric per the link (Was set to 0 instead of a 1 but following the guide did not resolve)
    • Made sure my back end databases were updated
    • I already followed the above guide when deploying my installation
    • Test-csdatabase passes

    The only thing i am still waiting on is a Windows update which is technically a skype update rollup but it has been sitting on preparing to install for about a half hour now so I'm not even sure if this is going to go through.

    I think I am going to give this about another half hour and if i don't come up with anything else I am going to completely blow away this test lab and start over from scratch as I believe something is wrong, I'm just not sure what. Please let me know if you have any further ideas in the mean time. Thanks!

    Friday, July 15, 2016 7:52 PM
  • Really surprising what went wrong in your environment. after deleting winfabric logs and bootstrapping and reinstalling will bring the Winfabric status to active. have you done this in all servers ? or single server ?

    As you said RU updates are already installing wait for that and hope this should fix.

    scratching the servers you can do anytime, but learning what went wrong will help you to face similar situations if encounter in production in future. may be these are your environmental specific issues like network or anything ?

    run the ocs logger for XDS all components and ALL flags and AD All components and ALL flags connect for a while and read through it, it will help you whats happening in the network.

    read through events and share with us to help you.


    Regards, Rajukb | MCSE (Communication ), MCSA (o365) ,Certified "Lync server 2013 depth support engineer"| This posting is providedwith no warranties and confers no rights. If my reply answers your question please mark as answer/helpful if its helpful.



    Friday, July 15, 2016 8:40 PM
  • Hello. I ended up scrapping the servers as I am under a timeline and I had already sunk way too much time into it. I did the fabric reinstall on a single server. Unfortunately I can no longer run diagnostics on these servers but i am running through the configs on a fresh set of servers now so I will update you shortly on the progress of the new servers. Thank you!
    Friday, July 15, 2016 8:55 PM
  • Thank you for the update - wish new installation should fix all issues to bring the services to online.

    Regards, Rajukb | MCSE (Communication ), MCSA (o365) ,Certified "Lync server 2013 depth support engineer"| This posting is providedwith no warranties and confers no rights. If my reply answers your question please mark as answer/helpful if its helpful.

    Friday, July 15, 2016 9:05 PM
  • Just wanted to follow-up with status,

    Does the newly installed Pool bring the FE services to "Running" ? 


    Regards, Rajukb | MCSE (Communication ), MCSA (o365) ,Certified "Lync server 2013 depth support engineer"| This posting is providedwith no warranties and confers no rights. If my reply answers your question please mark as answer/helpful if its helpful.

    • Marked as answer by Brolide Monday, July 18, 2016 5:58 PM
    Saturday, July 16, 2016 5:21 PM
  • Hello and thanks for following up. The services still appear to be unable to start, attempting to start the pool causes the shell to loop through a message every few seconds saying that my servers failed to get status data.

    I followed the above deployment guide to the letter:

    https://blogs.technet.microsoft.com/canitpro/2015/05/05/step-by-step-installation-of-skype-for-business-server-2015/

    In the error log there is a repeating message indicating faulting application RtcHost.exe.

    Once again there are some Windows updates, I am going to begin them, give them a few hours to run, and check it again. If the Windows updates do not resolve then I am thoroughly confused as I do not see why this would not work on all of three separate, completely new servers along with a completely new SQL server. If anybody has any more suggestions, please feel free to throw them my way as I am absolutely tearing my hair out over this. Thanks all.

    Sunday, July 17, 2016 8:59 PM
  • Have you tried running start-cspool -poolfqdn pool1.domain.com -verbose on one FE? to attempt to start all servers services? 

    Also the fact that you are getting this problem even with a rebuild, it is unlikely that windows updates is your answer. 

    I am not familiar with the blog you mention, so there may be a few assumptions with the blog that your environment may not match. Perhaps have a read of this ebook written by one of the more famous MVPs

    https://gallery.technet.microsoft.com/lync/Skype-for-Business-Server-f4dac647

    With regard to the "missing" intermediate certs. you can check this by looking at the issued certificates to the server and look at the certification path. all certs between the issued cert and root cert are intermediates. If there are any listed all of them should be listed in intermediate stores.

    Another note, disable windows firewall on server to see if that is preventing the starting of services, and also put the servers into a sterile OU not affected by any group policies that could be preventing things too.

    thanks


    Note: Please remember to `Mark as Answered` a post that answers your question and/or `Vote as Helpful` posts that have helped you. This will help others find answers to similar problems. For more Skype for Business help visit: http://www.skype4b.uk Please note that answers are based on my experience and opinion only and do not necessarily represent the views of my employer.

    Monday, July 18, 2016 8:57 AM
  • Hello, so after full Windows updates and a good bit of patience, everything has finally started properly in my test environment. Not sure why it would not start in my old lab, but all seems to be resolved now. Thank you so much for all of the help!
    Monday, July 18, 2016 3:28 PM
  • Hi, I have had this a few times and it surprises me each time. Turn windows firewall to on and re-deploy the server using deployment wizard. Apparently the firewall must be set to on for a front end server to start for the first time. I'm just posting this as it is a known issue and not well documented.
    Thursday, June 20, 2019 12:18 PM