Documents gets public available very slowly RRS feed

  • General discussion

  • Hi, I'm indexing 8.5 million database records into FAST ESP, the whole process is taking 10 - 12 hours, right now we have 8.3 documents indexed, but only 1.9 million documents available for search.

    I tried to tweak rtsearchrc.xml to get low latency on indexing process, I increase the number of partition, indexing threads and size of partitions, after all these changes the documents still takes a lot of time to get public available, someone has a tip or any kind of guidance on how to make the documents public available a little more faster?

    Tuesday, May 8, 2012 7:29 PM

All replies

  • After JDBCConnector finishes pushing all content, ESP still takes more than 1 day to make all 8.5 million docs public. Sometimes the 8.5 million docs never get public available.
    Thursday, May 10, 2012 1:34 PM
  • Hi,

    You can split the Indexing process in two parts: 1. Indexing de documents; 2. Making public the documents.

    Here are the steps:

    1. in Console, execute "indexeradmin suspendindexing -a":     This command will allow only index of the documents
    2. Run your connector (JDBC Connector?
    3. When your connector finish, then you can run "indexeradmin resumeindexing -a".    This command will make public the documents

    This works specially with the initial load of documents (records).

    Thursday, May 10, 2012 6:08 PM
  • Hello Rogério,

    Felix's information has worked for customer's to increase the time required to feed data to the index.  But if you are running into feeding problems or if your indexer is being overloaded this won't necessary help.

    Also I would not recommend modifying the number or thread or changing the number of partitions as this can cause more problems.  Example: if your indexer is getting overloaded increase the number of partitions will just add more disk i/o load.

    During feeding are you seeing errors in the following components configserver, contentdistributor, procservers, indexingdispatcher, indexer, indexingproducer logs?  You can review the %FASTSEARCH%\var\log\syslog\all.log to get a general state of the server.

    Also can you provide the output of your rc -r?

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer |US Customer Service & Support

    Customer Service & Support                       Microsoft| Services

    Thursday, May 17, 2012 3:51 AM