none
Exceeding Memory Usage... Again RRS feed

  • Question

  • Hi,

    I read the KB article http://social.technet.microsoft.com/Forums/en-US/fastinternetesp/thread/c397f351-1dbd-45e6-8b85-11bc28870ce8 and accordingly I increased the "docsDistributionMaxMB" paramter in rtsearchrc.xml from 1024 to 3072 (I'm running FAST ESP 5.3 SP4 with latest patches on Red Hat Linux 5.3, 64-bit arch, with 47GB RAM). I'm getting the following error:

    Loading attribute vectors will exceed maximum memory usage, set to 1887436800 B (1800.00 MB). fsearch will not load this index. Please reduce index size or number of navigator fields and sortable fields in the index profile

    The value--1800.00 MB--that is being reported in the error message above is exactly 90% of 2GB. According to the knowledgebase article quoted above:

    "ESP processes are 32-bit. Therefore, each process is limited in how much memory it can address: 2 gigabytes (GB) on 32-bit systems and 4 GB on 64-bit systems...."

     So it would appear that fsearch isn't making use of the full 4GB at its disposal. FYI, here is the output of the "uname -a" command on my linux system confirming that I am running on 64-bit server:

    Linux some_server 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

    Is there anyway to get fsearch to use the full 4GB it can access?

    Thanks

    Monday, February 20, 2012 5:47 PM

Answers

  • I got my question answered by support:

     

    docsDistributionMaxMB is *not* a search configuration parameter. It is indexer only, and determines how large the partitions created will be.


    Fsearch is configured through maxmemusage (in bytes) in fsearch.addon. This does have a default of 1800MB, and can be increased.

    So I set maxmemusage=3221225472 in etc/config_data/RTSearch/webcluster/fsearch.addon and I don't receive the error anymore. Note that I can set my maxmemusage above 2GB since I'm running FAST 5.3 on a 64-bit server.

    Hope this helps others.

    -Gora

    • Marked as answer by G-Man-UK Thursday, February 23, 2012 11:30 AM
    Thursday, February 23, 2012 11:30 AM

All replies

  • As a follow-up to this, my rtsearchrc.xml file has the following setting for docsDistributionMaxMB:

    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
      <!-- General Options -->
      <options
        xmlEncoding=""
        smallestIndexDelay="0"
        docsDistributionPst="100,100,100,50,33"
        docsDistributionMax="10000000,10000000,10000000,10000000,10000000"
        docsDistributionMaxMB="2048"
        numberDocsPerFixml="200"
        fsearchCachePstDist="5,20,5,20,4,20,6,4,4,6,4,2"
        maxMBCacheSize="300"

    ....

    It would appear that FAST just ignores values greater than 1800MB, is that correct? Is there any way to allow fsearch to use the full 4GB?

    Thanks,

    Gora

    • Marked as answer by G-Man-UK Thursday, February 23, 2012 11:28 AM
    • Unmarked as answer by G-Man-UK Thursday, February 23, 2012 11:28 AM
    Tuesday, February 21, 2012 10:30 AM
  • I got my question answered by support:

     

    docsDistributionMaxMB is *not* a search configuration parameter. It is indexer only, and determines how large the partitions created will be.


    Fsearch is configured through maxmemusage (in bytes) in fsearch.addon. This does have a default of 1800MB, and can be increased.

    So I set maxmemusage=3221225472 in etc/config_data/RTSearch/webcluster/fsearch.addon and I don't receive the error anymore. Note that I can set my maxmemusage above 2GB since I'm running FAST 5.3 on a 64-bit server.

    Hope this helps others.

    -Gora

    • Marked as answer by G-Man-UK Thursday, February 23, 2012 11:30 AM
    Thursday, February 23, 2012 11:30 AM
  • You may be able to split your index into more partitions, lowering the amount of data pr partitions, thus allowing more data to be indexed on the same node (and using more of your available memory by having more fsearch processes).   However, you can't add unlimited number of partitions.

    Another option may be to lower how much memory you need for your attribute vector (e.g. if we are talking strings, is there a more compact way of entereing the same information - using a more compact ID), replace a string with an int etc.

    But these are limited work-arounds, not really changing the nature of the problem.

    Thursday, February 23, 2012 9:53 PM
  • I got my question answered by support:

     

    docsDistributionMaxMB is *not* a search configuration parameter. It is indexer only, and determines how large the partitions created will be.


    Fsearch is configured through maxmemusage (in bytes) in fsearch.addon. This does have a default of 1800MB, and can be increased.

    So I set maxmemusage=3221225472 in etc/config_data/RTSearch/webcluster/fsearch.addon and I don't receive the error anymore. Note that I can set my maxmemusage above 2GB since I'm running FAST 5.3 on a 64-bit server.

    Hope this helps others.

    -Gora

    Hello Gora, that will work but keep in mind that any index-profile update will re-write the file fsearch.addon and any change will be lost. 

    This KB explains that the file you need to change is the fsearch.addon template, which is located in   %FASTEARCH%/etc/bliss-templates/FsearchAddonWriter/fsearch.addon.tmpl

    http://support.microsoft.com/kb/2018358   Modifications to fsearch.addon are deleted after index profile updates

    Nicolas MohamedLagash Systems

    Thursday, July 26, 2012 12:17 PM
  • Hi G-Man-UK,

    I have the same error message with you. But I can not find maxmemusage attribute in etc/config_data/RTSearch/webcluster/fsearch.addon

    Please help.

    Monday, January 21, 2013 6:58 AM
  • Hi

    I would like to clarify a few things regarding this maxmemusage parameter.

    The maxmemusage is not controlling the memory the fsearch process will use. Its main purpose is to prevent an fsearch to load a huge partition index putting a risk its stability.

    What is the memory limit of the fsearch process?

    On Linux 64-bit OS, the fsearch should have most of the 4 GB address space available.

    On Windows 64-bit OS, the fsearch also has the full 4 GB address space. This is due to the fact that fsearch alongside fixmlindex have the flag /LARGEADDRESSAWARE on (http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx#physical_memory_limits_windows_server_2008_r2 ).

    Those are the only two ESP processes with this linkage flag, all the others (including the indexer) cannot address more than 2 GB of memory.

    What is a huge partition index for a fsearch process?

    The responsibility of creating a partition index falls to the indexer which has plenty of mechanisms to limit the "size" of partition index (docsDistributionMaxMB is one of them).

    Upon startup the fsearch process

    1. loads in memory all navigators’ indexes

    The size of the navigation indexes depends on the number of documents the partition index has + the number of navigators defined in your index profile.

    [2013-02-01 08:41:16.819] VERBOSE    fsearch Number of documents indexed: 16

    [2013-02-01 08:41:16.823] VERBOSE    fsearch Active attributevector memory usage is 4521 B (0.00 MB)

    2. Evaluate the max memory cache for the main partition indexes.

    [2013-02-01 08:41:17.404] VERBOSE    fsearch Maximum cache memory usage: 822083575 B (784.00 MB)

    The global cache size is defined in rtsearchrc.xml parameter maxMBCacheSize and is distributed across all fsearch depending on how many documents their corresponding partition index has.

    The sum of #1 + #2 is the value tested against the maxmemusage (default 1800Mb).

    [2013-02-01 08:41:17.404] VERBOSE    fsearch Total attributevector + cache memory usage is 822088096 B (784.00 MB)

    Please note that the navigation indexes are actually loaded, the index cache is actually a Maximum cache usage (potential usage).

    In my example I have 4521 + 822083575 = 822088096 bytes. With only 16 documents indexed, you can imagine that 784 Mb is a max memory usage not the actual one.

    Also you can easily understand that the fsearch process memory usage is not limited to the navigation indexes and indexes cache alone, it also has to allocate memory to cache search results amongst other things.

    Thus allowing your fsearch process to go up to 3221225472 bytes (3 Gb) could harm your search experience (fsearch crash) although it might work. Yes it will solve the error "will not load this index" but IMHO it should be considered as a temporary solution.

    For long-term solution, like @Bjorn stated, the usual recommendation in those cases is :

    1. To review your document distribution. Things like adding a new partition or column should be considered.

    2. To review your index profile. Make sure all defined navigators are actually used in the query side.

    Hope this helps.


    Friday, February 1, 2013 9:06 AM