none
Failed to connect, Failed to initialize session with document engine: Unable to resolve Contentdistributor RRS feed

  • Question

  • I have FAST Search for SharePoint installed and going in my environment. While its been working smooth until a few months back when I ran the last crawl, now when I started a crawl from SharePoint it is having a hiccup. The crawl is stuck in Crawling state for a very long time and isn't proceeding.

    SharePoint reports the following in its event log:
    • Failed to connect to SERVER.com:XXXX Failed to initialize session with document engine: Unable to resolve Contentdistributor

    Upon looking at the Event Log for FAST Search, I see the following messages:

    • WARNING: Module (SPRel) at SERVER.com:XXXX is not responding
    • INFO: AdminLibrary.dll:AvailableListCheckerTimerWorkerInternal - Verifying sam worker node "net.tcp://SERVER.com:XXXX/" is still available.
    • WARNING: Process sprel was not running, restarting it

    These messages are continuing to be repeated over and over in the event log. Considering it mentions SPRel, I looked at the SPRel logs at <DIR>\FASTSearch\var\log\sprel and did not notice anything too interesting. Here are the most recent messages which are also repeated over and over:

    • [2012-06-05 23:43:56.201] VERBOSE    systemmsg ConfigServer is up again
    • [2012-06-05 23:43:56.201] VERBOSE    systemmsg Setting up XML-RPC
    • [2012-06-05 23:43:56.201] VERBOSE    systemmsg Connecting to ConfigServer at SERVER.com:XXXX..

    Any ideas on how to fix this issue and have a successful crawl completed? Please let me know if you need any more info.

    Note - I ended up stopping the crawl to troubleshoot the issue, btw it took an extremely long time to stop.



    Wednesday, June 6, 2012 7:00 AM

Answers

  • Ok. I found the root cause of this issue. Please find the symptoms and solution below:

    Symptoms:

    FAST and SharePoint 2010 have been working perfectly on this particular development server since installation.  Recently the content sources had stopped working and the crawl is stuck, yet nothing significant had changed on the server.

    • Content sources start crawling and never stop.
    • Manually stopping the content source crawl hangs on “Stopping” for a very long time.
    • No documents are ever actually crawled
    • Numerous errors on the SharePoint server under windows application log: ‘Failed to connect to SERVER.com:XXXX Failed to initialize session with document engine: Unable to resolve Contentdistributor‘.

    Potential Cause:

    During installation on development servers a self-signed certificate can be created for communication between FAST and SharePoint.  It turns out that the self-signed certificate is only valid for one year and when it expires the above problems will occur.  Unfortunately there is no mechanism making it obvious to the user that the certificate has expired, hence the potential for confusion.

    Steps to validate this as a cause:

    • On the FAST Server, open the Microsoft Management Console (MMC). On Run, type mmc or mmc.exe and hit <enter>.
    • Add the Snap-in for Certificates and choose local computer. When the Certificate store opens, check the FASTSearchCert certificate under Certificates (Local Computer) -> Personal -> Certificates.
    • If the FASTSearchCert's Expiration date is before todays date, then the certificate has expired.

    You have just validated the root cause for your problem. Please try the solution below,

    Solution:

    The fix is to generate and deploy new self-signed certificate and this can be achieved with the following steps:

    1. On the FAST Server, make sure the FAST Search for SharePoint & FAST Search for SharePoint Monitoring windows services are stopped.
    2. Open the Microsoft FAST Search Server for 2010 for SharePoint PowerShell on the FAST server as an Administrator.
    3. Navigate to the FAST directory (i.e. <FASTSearchFolder>\installer\scripts)
    4. Run the following command: .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true
    5. You may be asked to enter a password. Type a password and hit <enter>. Note - Keep this password handy, as it will be required in a later step.
    6. This will generate a new certificate, valid for one year in the following folder: <FASTSearchFolder>\data\data_security\cert\FASTSearchCert.pfx
    7. This certificate will need to be copied to the SharePoint server (if SharePoint is not hosted on the same server as FAST).
    8. Similarly copy the <FASTSearchFolder>\installer\scripts\SecureFASTSearchConnector.ps1 script from the FAST server to the Sharepoint Server (if SharePoint is not hosted on the same server as FAST).
    9. Start the FAST Search for SharePoint & FAST Search for SharePoint Monitoring  windows services.
    10. Now the certificate needs to be loaded on the SharePoint server.
    11. On the SharePoint Server, Start –> All Programs –> Microsoft SharePoint 2010 Products –> right click on SharePoint 2010 Management Shell and Run as administrator.
    12. Navigate to the location where the SecureFASTSearchConnector.ps1 script (this script may need to be copied from the FAST server as mentioned in step 7) exist.
    13. Run the following command (userName should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) windows service): .\SecureFASTSearchConnector.ps1 –certPath “path of the certificate\FASTSearchCert.pfx” –ssaName “name of your content SSA” –username “domain\username”
    14. You may be asked to enter a password. Please enter the password from step 5 and hit <enter>. Your powershell command prompt should look like this.

    Assuming there were no errors when running the PowerShell scripts the SharePoint server certificate has been deployed and will be valid for another year. The content sources should begin working normally again and crawls should complete succeesfully.

    Thanks to the following reference article on SharePoint Malarkey.

    • Marked as answer by DesiHero-007 Friday, June 8, 2012 5:19 AM
    Thursday, June 7, 2012 8:11 AM

All replies

  • Ok. I found the root cause of this issue. Please find the symptoms and solution below:

    Symptoms:

    FAST and SharePoint 2010 have been working perfectly on this particular development server since installation.  Recently the content sources had stopped working and the crawl is stuck, yet nothing significant had changed on the server.

    • Content sources start crawling and never stop.
    • Manually stopping the content source crawl hangs on “Stopping” for a very long time.
    • No documents are ever actually crawled
    • Numerous errors on the SharePoint server under windows application log: ‘Failed to connect to SERVER.com:XXXX Failed to initialize session with document engine: Unable to resolve Contentdistributor‘.

    Potential Cause:

    During installation on development servers a self-signed certificate can be created for communication between FAST and SharePoint.  It turns out that the self-signed certificate is only valid for one year and when it expires the above problems will occur.  Unfortunately there is no mechanism making it obvious to the user that the certificate has expired, hence the potential for confusion.

    Steps to validate this as a cause:

    • On the FAST Server, open the Microsoft Management Console (MMC). On Run, type mmc or mmc.exe and hit <enter>.
    • Add the Snap-in for Certificates and choose local computer. When the Certificate store opens, check the FASTSearchCert certificate under Certificates (Local Computer) -> Personal -> Certificates.
    • If the FASTSearchCert's Expiration date is before todays date, then the certificate has expired.

    You have just validated the root cause for your problem. Please try the solution below,

    Solution:

    The fix is to generate and deploy new self-signed certificate and this can be achieved with the following steps:

    1. On the FAST Server, make sure the FAST Search for SharePoint & FAST Search for SharePoint Monitoring windows services are stopped.
    2. Open the Microsoft FAST Search Server for 2010 for SharePoint PowerShell on the FAST server as an Administrator.
    3. Navigate to the FAST directory (i.e. <FASTSearchFolder>\installer\scripts)
    4. Run the following command: .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true
    5. You may be asked to enter a password. Type a password and hit <enter>. Note - Keep this password handy, as it will be required in a later step.
    6. This will generate a new certificate, valid for one year in the following folder: <FASTSearchFolder>\data\data_security\cert\FASTSearchCert.pfx
    7. This certificate will need to be copied to the SharePoint server (if SharePoint is not hosted on the same server as FAST).
    8. Similarly copy the <FASTSearchFolder>\installer\scripts\SecureFASTSearchConnector.ps1 script from the FAST server to the Sharepoint Server (if SharePoint is not hosted on the same server as FAST).
    9. Start the FAST Search for SharePoint & FAST Search for SharePoint Monitoring  windows services.
    10. Now the certificate needs to be loaded on the SharePoint server.
    11. On the SharePoint Server, Start –> All Programs –> Microsoft SharePoint 2010 Products –> right click on SharePoint 2010 Management Shell and Run as administrator.
    12. Navigate to the location where the SecureFASTSearchConnector.ps1 script (this script may need to be copied from the FAST server as mentioned in step 7) exist.
    13. Run the following command (userName should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) windows service): .\SecureFASTSearchConnector.ps1 –certPath “path of the certificate\FASTSearchCert.pfx” –ssaName “name of your content SSA” –username “domain\username”
    14. You may be asked to enter a password. Please enter the password from step 5 and hit <enter>. Your powershell command prompt should look like this.

    Assuming there were no errors when running the PowerShell scripts the SharePoint server certificate has been deployed and will be valid for another year. The content sources should begin working normally again and crawls should complete succeesfully.

    Thanks to the following reference article on SharePoint Malarkey.

    • Marked as answer by DesiHero-007 Friday, June 8, 2012 5:19 AM
    Thursday, June 7, 2012 8:11 AM
  • Thanks for the detailed directions.  

    I have 3 servers SharePoint, SharePoint Indexing Server, and SharePoint Fast, and then the SQL database server.  On my indexing server I am getting a ton of logs all with the same error message you posted.  I followed your directions to a T and it turns out my certificate was expired so I used a new one and it went through perfectly.  However I am still getting the error messages.  I dont know what else to do at this point I hate this damn mechanism.   


    • Edited by pslager Friday, October 12, 2012 4:03 PM
    Friday, October 12, 2012 3:36 PM
  • Hi Pslager,

    I have faced the same problem with you. My environment contains of 2 SharePoint Application Servers, 2 FS4SP Servers (which is started contentdistributor process on both servers, FT) and 1 DB Server. I have 2 contentdistributors so i decided to stop one of them and start crawling again.

    And it works like a charm!

    Hope this help.

    Bo

    Monday, October 15, 2012 3:50 AM
  • Thanks the config file was using port 13390 instead of 13391 i updated the text file and rebooted everything and it fixed it. 
    Monday, October 15, 2012 8:01 PM
  • pslarger

    I have the same problem, which machine was the port change (SP or fastsearch) and what was the file modified ?

    Thanks!!

    Friday, November 30, 2012 5:12 AM
  • Ok. I found the root cause of this issue. Please find the symptoms and solution below:

    Symptoms:

    FAST and SharePoint 2010 have been working perfectly on this particular development server since installation.  Recently the content sources had stopped working and the crawl is stuck, yet nothing significant had changed on the server.

    • Content sources start crawling and never stop.
    • Manually stopping the content source crawl hangs on “Stopping” for a very long time.
    • No documents are ever actually crawled
    • Numerous errors on the SharePoint server under windows application log: ‘Failed to connect to SERVER.com:XXXX Failed to initialize session with document engine: Unable to resolve Contentdistributor‘.

    Potential Cause:

    During installation on development servers a self-signed certificate can be created for communication between FAST and SharePoint.  It turns out that the self-signed certificate is only valid for one year and when it expires the above problems will occur.  Unfortunately there is no mechanism making it obvious to the user that the certificate has expired, hence the potential for confusion.

    Steps to validate this as a cause:

    • On the FAST Server, open the Microsoft Management Console (MMC). On Run, type mmc or mmc.exe and hit <enter>.
    • Add the Snap-in for Certificates and choose local computer. When the Certificate store opens, check the FASTSearchCert certificate under Certificates (Local Computer) -> Personal -> Certificates.
    • If the FASTSearchCert's Expiration date is before todays date, then the certificate has expired.

    You have just validated the root cause for your problem. Please try the solution below,

    Solution:

    The fix is to generate and deploy new self-signed certificate and this can be achieved with the following steps:

    1. On the FAST Server, make sure the FAST Search for SharePoint & FAST Search for SharePoint Monitoring windows services are stopped.
    2. Open the Microsoft FAST Search Server for 2010 for SharePoint PowerShell on the FAST server as an Administrator.
    3. Navigate to the FAST directory (i.e. <FASTSearchFolder>\installer\scripts)
    4. Run the following command: .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true
    5. You may be asked to enter a password. Type a password and hit <enter>. Note - Keep this password handy, as it will be required in a later step.
    6. This will generate a new certificate, valid for one year in the following folder: <FASTSearchFolder>\data\data_security\cert\FASTSearchCert.pfx
    7. This certificate will need to be copied to the SharePoint server (if SharePoint is not hosted on the same server as FAST).
    8. Similarly copy the <FASTSearchFolder>\installer\scripts\SecureFASTSearchConnector.ps1 script from the FAST server to the Sharepoint Server (if SharePoint is not hosted on the same server as FAST).
    9. Start the FAST Search for SharePoint & FAST Search for SharePoint Monitoring  windows services.
    10. Now the certificate needs to be loaded on the SharePoint server.
    11. On the SharePoint Server, Start –> All Programs –> Microsoft SharePoint 2010 Products –> right click on SharePoint 2010 Management Shell and Run as administrator.
    12. Navigate to the location where the SecureFASTSearchConnector.ps1 script (this script may need to be copied from the FAST server as mentioned in step 7) exist.
    13. Run the following command (userName should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) windows service): .\SecureFASTSearchConnector.ps1 –certPath “path of the certificate\FASTSearchCert.pfx” –ssaName “name of your content SSA” –username “domain\username”
    14. You may be asked to enter a password. Please enter the password from step 5 and hit <enter>. Your powershell command prompt should look like this.

    Assuming there were no errors when running the PowerShell scripts the SharePoint server certificate has been deployed and will be valid for another year. The content sources should begin working normally again and crawls should complete succeesfully.

    Thanks to the following reference article on SharePoint Malarkey.

    This is an addendum to your steps, but I've included them all together to avoid confusion.

    Symptoms:

    FAST and SharePoint 2010 have been working perfectly on this particular development server since installation.  Recently the content sources had stopped working and the crawl is stuck, yet nothing significant had changed on the server.

    • Content sources start crawling and never stop.
    • Manually stopping the content source crawl hangs on “Stopping” for a very long time.
    • No documents are ever actually crawled

    Numerous errors on the SharePoint server under windows application log: ‘Failed to connect to SERVER.com:XXXX Failed to initialize session with document engine: Unable to resolve Contentdistributor‘.

    Potential Cause:

    During installation on development servers a self-signed certificate can be created for communication between FAST and SharePoint.  It turns out that the self-signed certificate is only valid for one year and when it expires the above problems will occur.  Unfortunately there is no mechanism making it obvious to the user that the certificate has expired, hence the potential for confusion.

    Steps to validate this as a cause:

    • On the FAST Server, open the Microsoft Management Console (MMC). On Run, type mmc or mmc.exe and hit <enter>.
    • Add the Snap-in for Certificates and choose local computer. When the Certificate store opens, check the FASTSearchCert certificate under Certificates (Local Computer) -> Personal -> Certificates.
    • If the FASTSearchCert's Expiration date is before today’s date, then the certificate has expired.

    You have just validated the root cause for your problem. Please try the solution below,

    Solution:

    The fix is to generate and deploy new self-signed certificate and this can be achieved with the following steps:

    1. On the FAST ADMIN Server, make sure the FAST Search for SharePoint & FAST Search for SharePoint Monitoring windows services are stopped. Admin server is identified by having the “FAST Search for SharePoint Sam Admin” service
    2. Open the Microsoft FAST Search Server for 2010 for SharePoint PowerShell on the FAST server as an Administrator.
    3. Stop services using the nctrl command “nctrl stop” (see Nctrl.exe reference for more).
      1. Possible Issue restarting FAST shut down using net stop fastsearchservice or net stop fastsearchmonitoring (see http://support.microsoft.com/kb/2443446).
    4. Navigate to the FAST directory (i.e. <FASTSearchFolder>\installer\scripts)
    5. Run the following command: .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true
    6. You may be asked to enter a password. Type a password and hit <enter>. Note - Keep this password handy, as it will be required in a later step. This will generate a new certificate, valid for one year in the following folder: <FASTSearchFolder>\data\data_security\cert\FASTSearchCert.pfx
    7. This certificate will need to be exported and imported to any other the FAST Search servers (if SharePoint is not hosted on the same server as FAST).
      1. On 2<sup>nd</sup> and additional FAST servers follow steps 1-4
      2. Instead of step 5 run the following command

                                                                   i.      .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $false -thumbprint "numbers and letters specific to your certificate"

                                                                 ii.      You may be asked to enter a password. Please enter the password from step 6 and hit <enter>.

    1. Start the FAST Search on all FAST servers using the nctrl command “nctrl start”.
      1. This may take a couple mins, but if FAST Search for SharePoint QRProxy doesn’t start try #2 as described here: http://joelblogs.co.uk/2011/09/20/certificate-revocation-list-check-and-sharepoint-2010-without-an-internet-connection/
    2. This certificate will need to be copied to any other the SharePoint server (if SharePoint is not hosted on the same server as FAST).
    3. Similarly copy the <FASTSearchFolder>\installer\scripts\SecureFASTSearchConnector.ps1 script from the FAST server to the SharePoint Server (if SharePoint is not hosted on the same server as FAST).
    4. Now the certificate needs to be installed on the SharePoint server using PowerShell
    5. On the SharePoint Server, Start –> All Programs –> Microsoft SharePoint 2010 Products –> right click on SharePoint 2010 Management Shell and Run as administrator.
    6. Navigate to the location where the SecureFASTSearchConnector.ps1 script (this script may need to be copied from the FAST server as mentioned in step 10) exist.
    7. Run the following command (userName should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) windows service): .\SecureFASTSearchConnector.ps1 –certPath “path of the certificate\FASTSearchCert.pfx” –ssaName “name of your content SSA” –username “domain\username”
    8. You may be asked to enter a password. Please enter the password from step 6 and hit <enter>. Your PowerShell command prompt should look like this.

    Assuming there were no errors when running the PowerShell scripts the SharePoint server certificate has been deployed and will be valid for another year. The content sources should begin working normally again and crawls should complete successfully.

    Thanks to the following reference article on SharePoint Malarkey.


    Thank you in advance, Peter

    • Proposed as answer by Peter D Mendez Thursday, January 24, 2013 6:30 PM
    • Unproposed as answer by DesiHero-007 Thursday, April 10, 2014 7:06 AM
    Thursday, January 24, 2013 4:58 PM
  • This is an old thread, however I'm hoping someone will read it and be able to help.  We are having this issue on our SharePoint 2010 server.  We have followed all of the steps outlined and we are still getting the error.  This is in a QA environment and not our Production environment.  We have updated these certs in the past with no issues.  However, this time nothing seems to be getting rid of the errors and we cannot successful run crawls.  

    Any help is greatly appreciated.

    Monday, May 22, 2017 4:23 PM