none
FAST Search Server Errors

    Question

  • Last night we moved the storage location  for our SharePoint 2010 server, running in Hyper-V. We didn't change any of the networking and it's running on the same host server. SharePoint appears to be working successfully this morning, with the exception of the FAST server connection. Searches in SharePoint return an error that the FAST Search was unreachable. We're getting these errors in the FAST Search Event log. Anyone seen this before?

    Log Name:      FAST Search
    Source:        FAST Search Node Controller
    Date:          9/17/2013 10:26:00 AM
    Event ID:      1000
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      FASTSERVER.mydomain.com
    Description:
    Process indexer was not running, restarting it

    Log Name:      FAST Search
    Source:        FAST Search Node Controller
    Date:          9/17/2013 10:26:00 AM
    Event ID:      1000
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      FASTSERVER.mydomain.com
    Description:
    Starting indexer

    Log Name:      FAST Search
    Source:        FAST Search Content Distributor
    Date:          9/17/2013 10:26:05 AM
    Event ID:      1000
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      FASTSERVER.mydomain.com
    Description:
    Session 1379230421: op_set[1,1]: Failed to update document status in next subsystem. Triggering recovery...

    Log Name:      FAST Search
    Source:        FAST Search FSearch
    Date:          9/17/2013 10:26:17 AM
    Event ID:      1000
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      FASTSERVER.mydomain.com
    Description:
    Data Access Error: Failed to read all of generation file, url=C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data\FASTSERVER.mydomain.com_13050/generation.2.dat, read=4, expected=104

    Log Name:      FAST Search
    Source:        FAST Search Content Distributor
    Date:          9/17/2013 10:26:20 AM
    Event ID:      1000
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      FASTSERVER.mydomain.com
    Description:
    Session 1379230411: op_set[1,1]: Failed to update document status in next subsystem. Triggering recovery...

    Log Name:      FAST Search
    Source:        FAST Search Dictionary Producer
    Date:          9/17/2013 10:26:20 AM
    Event ID:      1000
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      FASTSERVER.mydomain.com
    Description:
    Committing a graceful exit with status code 1 because '(_numPages == (unsigned int)_wordNumIndexSize)' [common\pagedictcounts.cpp:106]. A monitoring oracle will restart me.


    Orange County District Attorney

    Tuesday, September 17, 2013 5:30 PM

All replies

  • Hi,

    From your description, you receive the above event log which indicates search indexer fails and restart unexpectedly after you moved the storage location of SharePoint 2010 server VM.

    According to event logs, it seems that SharePoint fails to access generation file generation.2.dat and the indexer exists with the error status (_numPages == (unsigned int)_wordNumIndexSize) in [common\pagedictcounts.cpp:106]. Before we can look deeper into this issue, please make sure you have checked following things which may result in similar issues:

    1. Have you installed latest CU and updates for SharePoint 2010 server? As I know, there is an known issue which is corrected by that is corrected in one of 2012 CU. 

    =============

    After you install the cumulative update package, you must run the SharePoint Products and Technologies Configuration Wizard (PSConfigUI.exe). This wizard upgrades and updates the farm to the latest version. This step is required to make sure that you have full SharePoint functionality. The farm will run if you do not run PSConfigUI.exe. However, the farm will have reduced functionality. For example, the search function will be unable to index content.

    =============

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;2597150

    1. What do you mean when you say "moving the storage location for our SharePoint 2010 server"? Did you simply move VHD file from one disk/volume/SAN to another? If this is what you have done, please also check the consistent of VHD file and the disk\volume in the VHD file.  A bit error or lack of space in file system will bring indexer failure with exact error message.

    It is also possible that indexer exists because of the FAST dictionary normalization.

    To further investigate the issue, could you please also provide the errors in ContentAPILog.log and ULS log which give more information about reason of indexer failure? If the issue is urgent and has made impact on your business, you can also contact Microsoft Support for fast help.


    Miles LI TechNet Community Support

    Thursday, September 19, 2013 10:45 AM
    Moderator
  • Thanks for the note back Niles. I've opened a support case on this. I'll post back what they find out.

    Orange County District Attorney

    Thursday, September 19, 2013 2:27 PM
  • Did you ever get an answer from support? I'm running into these same errors after an unexpected shutdown. SP2010 and FAST are patched at current levels. I can't reset the index on FAST due to an error "Could not retrieve 'NumDocs' from indexing engine. Unfortunately this is the only thread I can find with the errors mentioned above. This thread is pretty old so hopefully it will get a reply.
    • Edited by Mr_SHeister Wednesday, February 5, 2014 3:33 PM
    Wednesday, February 5, 2014 3:33 PM
  • Yes, I was able to finally resolve my problem. Microsoft support worked with me and dug pretty deep into my configuration between FAST and SharePoint. They were able to figure out what my problem was from some logs and information that was beyond me. I'll add their Resolution / Workaround below from my case which may help you out, I hope. Good luck!


    RESOLUTION/WORKAROUND AND SOME TROUBLESHOOTING STEPS
    ================================================
    ->Checked QR Server log and found below

    [2013-09-19 10:24:43.984] VERBOSE    qrserver HTTP server running on port 13280.
    [2013-09-19 10:24:43.984] VERBOSE    qrserver http://OCDAVFAST000.da.ocgov.com:13280/
    [2013-09-19 10:26:35.178] WARNING    qrserver Error received while searching tcp/localhost:13050. Error: [3] All partitions are down
    [2013-09-19 10:26:37.756] WARNING    qrserver Error received while searching tcp/localhost:13050. Error: [3] All partitions are down
    [2013-09-19 10:27:54.378] WARNING    qrserver Error received while searching tcp/localhost:13050. Error: [3] All partitions are down
    [2013-09-19 10:29:37.714] WARNING    qrserver Error received while searching tcp/localhost:13050. Error: [3] All partitions are down
    [2013-09-19 10:33:26.895] WARNING    qrserver Error received while searching tcp/localhost:13050. Error: [3] All partitions are down
    [2013-09-19 10:35:25.992] WARNING    qrserver Error received while searching tcp/localhost:13050. Error: [3] All partitions are down

    ->checked search-1.log and found below


    [2013-09-16 07:46:10.064] WARNING    searchctrl-search active_index_set (13052): Problem starting fsearch 0_1379229840272075000
    [2013-09-16 07:46:10.064] INFO       searchctrl-search fdispatch (13052): Unable to start fsearch. Search node has problems, stopping fdispatch.
    [2013-09-16 07:46:11.095] VERBOSE    searchctrl-search node_monitor: Indexer is unavailable, will continue attempting to reconnect.


    ->Checked fsearch.log

    [2013-09-19 10:37:39.905] VERBOSE    fsearch Loading rc-file: 'C:\FASTSE~1\var\searchctrl\0/etc/fsearchrc-1379229840272075000-13050'
    [2013-09-19 10:37:39.905] VERBOSE    fsearch Enabling low fragmentation heap succeeded.
    [2013-09-19 10:37:39.905] VERBOSE    fsearch Checking for persisted data in 'C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data\ocdavfast000.da.ocgov.com_13050'...
    [2013-09-19 10:37:39.905] WARNING    fsearch Data Access Error: Failed to read all of generation file, url=C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data\ocdavfast000.da.ocgov.com_13050/generation.2.dat, read=4, expected=104
    [2013-09-19 10:37:39.920] VERBOSE    fsearch ...no persisted data found
    [2013-09-19 10:37:39.920] VERBOSE    fsearch Reading index configuration from C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data\index.cf
    [2013-09-19 10:37:39.920] VERBOSE    fsearch Checking dataset version file C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data/version.txt
    [2013-09-19 10:37:39.920] VERBOSE    fsearch Dataset status: version=1.1, state='Ok'
    [2013-09-19 10:37:39.920] VERBOSE    fsearch Maximum cache memory usage (without result cache) : 6807290 B (6.49 MB)
    [2013-09-19 10:37:39.920] VERBOSE    fsearch Number of documents indexed: 698
    [2013-09-19 10:37:39.920] VERBOSE    fsearch Total attributevector (incl. disabled) memory usage is 0 B (0.00 MB)
    [2013-09-19 10:37:39.936] VERBOSE    fsearch Active attributevector memory usage is 2800 B (0.00 MB)
    [2013-09-19 10:37:39.936] VERBOSE    fsearch Bad magic for C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data/merged/bcatcontent/dictionary.pidx2
    [2013-09-19 10:37:39.936] VERBOSE    fsearch Bad magic for C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data/merged/msynthcat/dictionary.pidx2
    [2013-09-19 10:37:39.936] VERBOSE    fsearch Bad magic for C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data/merged/anchortext/dictionary.pidx2
    [2013-09-19 10:37:39.936] VERBOSE    fsearch Bad magic for C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data/merged/assocqueries/dictionary.pidx2
    [2013-09-19 10:37:39.936] ERROR      fsearch Cannot find master synthesized catalog.No synthetic indexes are loaded!
    [2013-09-19 10:37:39.936] CRITICAL   fsearch Pending generation fault, failed to open data set url=C:\FASTSearch\data\data_index\0\index_1379229840272075000\index_data, partition=0
    [2013-09-19 10:37:42.905] VERBOSE    fsearch Loading rc-file: 'C:\FASTSE~1\var\searchctrl\0/etc/fsearchrc-1379229840272075000-13050'
    [2013-09-19 10:37:42.952] VERBOSE    fsearch Enabling low fragmentation heap succeeded.

    ->Went to Fast Server and  stopped fast services.
    ->Renamed searchctrl and data_index folder to old.
    ->Started fast services
    ->Waited for some time
    ->Performed query on localhost:13280 and able to get results
    ->Performed search on SharePoint site and able to get results
    ->Ran securefastconnector to replace certificate however again it gave connection could not be validated.
    ->Went to MMC and removed old Fast Cert
    ->Trusted new cert after adding it in Trusted root authorities
    ->Ran command again and it validated the conection
    ->Ran  Ping-SPEnterpriseSearchContentService  -Hostname "ocdavfast000.da.ocgov.com" and it returend true for new cert
    ->Stopped crawl and it got stopped successfully


    Orange County District Attorney

    Wednesday, February 5, 2014 3:50 PM
  • Thanks for the info. I'll give it go and let you know what happens.
    Wednesday, February 5, 2014 7:33 PM
  • Just wanted to check in and thank you for the information. Renaming the folders and restarting the service stopped the errors I was getting and I was able to clear the collection and start rebuilding the index. Previously I was unable to clear the collection because the services kept crashing. Thanks again!
    Monday, February 10, 2014 10:52 PM