none
Lemmatization is Not Working in FAST ESP. RRS feed

  • Question

  • Hi All,

    I am looking for suggestion & advice to rectify lemmatization configuration in FAST ESP.

    I have configured lemmatization functionality on multinode FAST ESP Environment , but it’s not working as expected.
    While debugging I can see “Message=No query transformation” .

    QTF[Name=FastQT_Lemmatizer:
    Action=nop
    Query=No change to query
    Custom=
    Message=No query transformation
    MessageID=8]

    While configuration I have performed below steps.

    1) Configured <lemmatizationconfig.xml> file for "query_expansion". (rename $FASTSEARCH/etc/LemmatizationConfigQueryExpansion.xml to lemmatizationconfig.xml)
    2) Restart all prcserver
    3) Restarted all QrServers.
    4) And did reefeeding for all the documents.

    But still I am no getting expected output. I am passing query with lemmatization property “qtf_lemmatize= true” . but still FastQT_Lemmatizer: Action=nop,
    I am not sure why …

    Could you please suggest , why FastQT_Lemmatizer: Action coming “NOP” ??
    Did I missed some setting ??

    Thanks
    Thursday, September 4, 2014 10:26 AM

All replies

  • Hi Himanshu As I am aware that LemmatizationConfig.xml is being used by Lemmatizer stage at the time of "Document Processing" and LemmatizationConfigQueryExpansion.xml is being used for expanding the queries at the time of query processing. As you mentioned that you renamed $FASTSEARCH/etc/LemmatizationConfigQueryExpansion.xml to lemmatizationconfig.xml, it means that there is no configuration for lemmatization on "Query Processing Side",that's why I think it is not returning the correct transformation at the time of query. Your parameter "qtf_lemmatize= true" is coorect, otherwise it would give this below message in Query Transformation. <QUERYTRANSFORM NAME="FastQT_Lemmatizer" ACTION="nop" QUERY="" CUSTOM="No change to query" MESSAGE="Lemmatization turned off for current query" MESSAGEID="16" INSTANCE="lemmatizer" /> Can you revert back to default one and verify that below MESSAGE and MESSAGEID is coming or not to start from scratch to investigate the issue. <QUERYTRANSFORM NAME="FastQT_Lemmatizer" ACTION="Modified the query" QUERY="check" CUSTOM="Modified FQL tree" MESSAGE="Term rerouted (lemmatization by document expansion)" MESSAGEID="4" INSTANCE="lemmatizer" /> Thanks Ashish

    Ashish Gupta Click "Vote As Helpful"! if you think that post is helpful in responding your question click "Mark As Answer, if you think that this is your answer for your question.

    Saturday, September 6, 2014 5:28 PM
  • Ai Ashish,

    Thanks for your mail.

    In Current system configuration composite-fields are not set to be lemmas=Yes. That why i am not trying to perform Lemmatization at "document Processing" because i also don't want to change any thing in index-profile and then re-feeding the data.

    So i have only one option to perform lemmatization on "Query Processing Side", because lemmatization on Query side is independently of the index profile and configure the query transformer for any field in the index.

    kindly let me know your thoughts.

    Monday, September 8, 2014 9:24 AM
  • Could you check the qtf-config (the query transformation pipeline) if its configured correctly. i.e

    %FASTSEARCH%\etc\config_data\QRServer\<search cluster name>\etc\qrserver\qtf-config.xml

    You can as well try to add debug mode when issuing the query to see if it will give more details:


    http://yourqrserver:15100/cgi-bin/xsearch..........&qtf_packfql:debug=1

    Monday, September 8, 2014 5:44 PM
  • Yes there is problem in index-profile, in index-profile Composite field is not a lemma=Yes field. But Other fields like Title,body,headings etc are lemmatization=Yes field.
    Thats a reason Lemmatization functionality is not working.
    As I don’t want any cold update in system so I am not changing composite-field. So I am trying Lemmatization at Query exption. IF I am trying Q=Title:award&=1
    Then it is working fine , but my search scope got reduce.
    To overcome Search Scope problem I thought to create New field “lemmaContent” and use attribute Merger stage to merger/Store all composite-field data in to new “lemmaContent” field.
    Then I has performed Q=Lemmacontent:award&qtf_lemmatize=1
    But its not working also 
    I thought if I create same field as TITLE so it will work ,because Q=Title:award&qtf_lemmatize=1 is working Fine.
    TITLE Field..
    <field name="title" fullsort="yes" tokenize="auto" result="dynamic" fallback-ref="title" wildcard="full" lemmatize="yes">
    New field I have created
    <field name="lemmacontent" fullsort="yes" tokenize="auto" result="dynamic" fallback-ref="lemmacontent" wildcard="full" lemmatize="yes" />

    Kindly suggest this approach is correct or not …. If wrong then what is the alternate solution .. with cold update …
    Tuesday, September 9, 2014 5:07 PM
  • one more thing .... please let me know if i want to create custome Lemmatiation Dictionary ...what is the format... i am using lingustistudio.. if i m trying to import.. it is giving zero KB file.

    please let me know format

    Tuesday, September 9, 2014 5:09 PM
  • *without cold update !!
    Wednesday, September 10, 2014 2:05 AM