none
exceed maximum memory usage RRS feed

  • Question

  • Hi,

    We're having a problem when trying to index a large number of records in a ESP 5.4 installation with  3 Index/search servers.


    according to the KB: http://support.microsoft.com/kb/2523632  partitions were modified with this values​​:

    Average size  of the documents:  562 bytes per doc

    16GB RAM = aprox 19.700  M docs per server

     

     rtsearchrc.xml config:

     docsDistributionPst="100,100,100,100"
     docsDistributionMax=""
     docsDistributionMaxMB="-1"
     triggers="10000,100000,2200000"

     

    all documents( 37 Millions aprox ) are successfull indexed but we have this error and the search are unavaileble:

    Loading attribute vectors will exceed maximum memory usage, set to 1887436800 B (1800.00 MB). fsearch will not load this index. Please reduce index size or number of navigator fields and sortable fields in the index profile


     

    anyone have some idea how to solve this? 

     

    Thanks

    Thursday, November 17, 2011 1:28 PM

Answers

  • Hi,

     

    I believe you have already opened a support case for this question.  This will be limited by your bytes per document.  Feel free to show us your status details page from your indexer.

     

    That page has a “Partition information” with current attribute vector memory usage of each partition.  If those values exceed 3GB, with 5 million documents per partition, then the bytes per document have increased since initial assessment. This memory usage is going to be the limiting factor for partition sizes, and is determined by the use of navigators and fullsort fields. With more physical memory, more partitions could be added. With your current system having 16gb ram, and using 64-bit Window Operating system, it appears that 15 million documents across 5 partitions is what you should expect to be able to handle.

     

    Also, regarding your docsDistributionPst setting of "100,100,100,100", this keeps three partitions empty for new content, and 100% of the content will be in the fourth partition.  We would generally recommend keeping only two partitions empty to index new content, like so:
    docsDistributionPst="100,100,100,50"

     

    And if you were using five partitions, we would generally recommend the below setting for an equal document distribution:
    docsDistributionPst="100,100,100,50,33"

     

    The KB article that you mentioned also shows that as an example.  Let us know if you have more questions.

     

    Thanks!

    Rob Vazzana | Sr Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                          Microsoft | Services

    Friday, November 18, 2011 10:33 PM
    Moderator

All replies

  • Hi,

     

    I believe you have already opened a support case for this question.  This will be limited by your bytes per document.  Feel free to show us your status details page from your indexer.

     

    That page has a “Partition information” with current attribute vector memory usage of each partition.  If those values exceed 3GB, with 5 million documents per partition, then the bytes per document have increased since initial assessment. This memory usage is going to be the limiting factor for partition sizes, and is determined by the use of navigators and fullsort fields. With more physical memory, more partitions could be added. With your current system having 16gb ram, and using 64-bit Window Operating system, it appears that 15 million documents across 5 partitions is what you should expect to be able to handle.

     

    Also, regarding your docsDistributionPst setting of "100,100,100,100", this keeps three partitions empty for new content, and 100% of the content will be in the fourth partition.  We would generally recommend keeping only two partitions empty to index new content, like so:
    docsDistributionPst="100,100,100,50"

     

    And if you were using five partitions, we would generally recommend the below setting for an equal document distribution:
    docsDistributionPst="100,100,100,50,33"

     

    The KB article that you mentioned also shows that as an example.  Let us know if you have more questions.

     

    Thanks!

    Rob Vazzana | Sr Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                          Microsoft | Services

    Friday, November 18, 2011 10:33 PM
    Moderator
  • Thanks for your information, very clear and usefull.

     

    we can configure ESP as you say.

     

    Now we are trying to shrink  the doc size on varios ways,  in order to have more capacity.  And for other way i started to work on a POC to do something similar to indexing on demand.

     

    The point is we have a lot of database records to be indexed, but many of them are historical, often are not consulted.

    the initial idea is to have indexed the last 3 years and the rest of the information is indexed only when users request it.




    What do you think this approach?   ... anyone try this?

     

     

    thanks!

     

     

     

    Saturday, November 19, 2011 3:57 PM
  • Glad to help!  Regarding your question about  what approach to use for your historical content that is updated only when users request it, it sounds like Archive Indexing would accomplish this.  The ESP Configuration guide documents Archive indexing on page 66.  Below is a small excerpt:

     

    About archive indexing

    Archive indexing improves the performance of typical archive systems.

    A typical archive system has an increasing archive size and few document updates. Archive indexing makes

    it easier to scale up the data volume, by indexing the content node by node. When the last node is getting

    full, you can add a new node. In addition, you can disable automatic reindexing to reduce the disk requirements.

     

    Feel free to review the guide and see if this will meet with your business needs.

     

    Thanks!

    Rob Vazzana | Sr Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                          Microsoft | Services

    Monday, November 21, 2011 6:27 PM
    Moderator
  • Hi Rob Va,

    Our system got exactly the same error with enerode1890. I'm going to adjust value for docsDistributionMaxMB attribute. My question is Does this change cause any down time for FAST system?

    Monday, January 21, 2013 3:25 AM