locked
Document status in feeding RRS feed

  • Question

  • I have a lot of documents, which are still in queues and needs to be feed to ESP.There is considerable delay when these becomes available in search(indexed). this mostly becuase of more documents.

    So I like to know more

    How many documents are present in queues which are waiting for to be fed into the system,so I can calculate when particular document will be available in search(indexed)?

    In short I want to have clear understanding which are being fed to esp at this second and which will be going in next batch etc.


    Harvail

    Thursday, February 23, 2012 6:57 AM

Answers

  • Hello,

    I am not aware of any logging that Fast Search can provide to provide document level granularity.  Once a batch is submitted to the feeding chain the documents in the batch are processed as fast a possible.  Call backs are the only acknowlegement of the that status of the batch during feeding. 

    There is no way to identify a specific document is to be feed/indexed/searchable faster then other documents. 

    If you are utilzing the crawler (crawleradmin --force --refetchuri) it is possible to request a URL to be added to the top of the crawler queue.  But that is the extent of the request.

    If this is more an issue with feeding performance.  I would recommend opening a support case to address this type of issue.

    Hope this helps,

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                         Microsoft| Services


    Sunday, April 1, 2012 5:07 PM

All replies

  • Hi,

    It's more or less impossible to tell when your document will be indexed since this dependent on a lot of factors--feeding rate, I/O capacity of your system, etc.

    You could investigate the use of Callbacks in the Content API, these are messages passed from the FAST Document Processing, Indexing, and Search subsystems to the feeding program that will tell you when a particular document has passed a certain "gate" in the indexing process (e.g. completed document processing, stored to FIXML, live (e.g. searchable). This will at least give the feeding program an idea of the status of content after it's fed to FAST.

    The FAST Content Integration Guide has more information on Callback handling using the FAST Content API.

    -Gora

    Thursday, March 1, 2012 9:16 AM
  •  

    Hi Harvail,

    If am not wrong, easiest way to find the status of

  • Pending documents in queue.
  • Feeding rate per sec
  • Index rate  per sec
  • Failed document

    Use monitoring Tool/clarity, where you can see status for all.

    Under below navigation bars

    1) Indexing overview

    2) Content overview

    BR

    FAST ESP JI

  • Proposed as answer by FAST_ESP_JI Wednesday, March 7, 2012 4:12 AM
Wednesday, March 7, 2012 4:09 AM
  • Hi, I was more looking for which particular document is feed by which batch to indexer and what time etc.More Granular level info. More like the output of doclog but doclog only provides info for the document processing pipeline not of indexer :( In some conditions I want to particular document to be indexed at high priority etc.

    Harvail

    Wednesday, March 7, 2012 5:51 AM
  • Hello,

    I am not aware of any logging that Fast Search can provide to provide document level granularity.  Once a batch is submitted to the feeding chain the documents in the batch are processed as fast a possible.  Call backs are the only acknowlegement of the that status of the batch during feeding. 

    There is no way to identify a specific document is to be feed/indexed/searchable faster then other documents. 

    If you are utilzing the crawler (crawleradmin --force --refetchuri) it is possible to request a URL to be added to the top of the crawler queue.  But that is the extent of the request.

    If this is more an issue with feeding performance.  I would recommend opening a support case to address this type of issue.

    Hope this helps,

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                         Microsoft| Services


    Sunday, April 1, 2012 5:07 PM