none
FAST ESP: Inconsistent results: Searching on a field having double data type. RRS feed

  • Question

  • Hi,
    We facing an issue in Fast related to double data type.

    We have a field name define as sales in our Index profile
    having data type as double.

    When we perform search on field Sales we are getting some inconsistent results.

    Search criteria:
    Sales value                     Result
    29,320,991,000             29,320,991,000 And 29,321,000,000
    29,321,000,000             29,320,991,000 And 29,321,000,000
    Range(29,320,991,000   29,320,991,000 and 29,321,000,000
    and 29,320,991,000)

    Issue :only the value to be searched must be returned but
    also getting some near by values.

    Observations:
    We tried different combinations where ‘9’ was inserted intentionally and have
    observed that FAST is converting or rounding off the values, if the length
    ofsearch term is > 7.

    For Example:
    Sales value      Result
    29999999       30000000
    10009999       10010000

    Please help us understand the issue and suggest if any solution present for thr same.

    Monday, June 6, 2011 7:16 AM

Answers

  • Hi Nitin,

     

    You may want to consider storing the data in both a float/double field and a string field.  You can use the string field for exact searches and the other field for navigation and/or range searches where the level of precision is not as critical.  You would likely have to add some logic to your front end to help make it easier for your users, but it is an option.

     

    Thanks,

    Patrick

     


    Patrick Schneider | Microsoft | Enterprise Search Group | Support Escalation Engineer | http://www.microsoft.com/enterprisesearch
    Wednesday, June 8, 2011 12:36 PM

All replies

  • Hi Nitin,

     

    Doubles in ESP are stored with 23 bits of precision, the same as floats, and I suspect that you are not able to get the necessary precision to search the double field the way you are trying to.  I've included information about the definition of float and double fields below.

     

    Thanks,

    Patrick

     

    Float:

    Floating point value where 23 bits are used for the mantissa and 8 bits used for the exponent (using base 2 for the exponent).  The target field must be configured as float in the Index Profile.

    Min value: 1e-39
    Max value: 1e38

     

    Double:

    Floating point value where 23 bits are used for the mantissa and 8 bits used for the exponent (using base 10 for the exponent).  The target field must be configured double as double in the Index Profile. Min value: double range (signed) 1E-128 - 1E127 (23 bits precision)

    Min value: 1e-128
    Max value: 1e127

    Note! If you submit a double value via the corresponding methods in the Content API, the values will be converted to float within the document processing framework. In this way you may lose some precision for the exponent, and the indexed value may not be identical to the submitted value. If you submit the value as a string instead, full float precision will be achieved.

     


    Patrick Schneider | Microsoft | Enterprise Search Group | Support Escalation Engineer | http://www.microsoft.com/enterprisesearch
    Monday, June 6, 2011 11:33 AM
  • Thanks Patrick.

    We tried by converting the double sales field to float. But their also we got same results.

    I have a query that :

    In document framework double gets converted to float and stored if not passed as string. Does the same conversion occur when we send a fql query to QR server.

    eg:-

    sales:range(double(29320991000), double(29320991000),from="GE",to="LE")

    sales:range(29320991000, 29320991000,from="GE",to="LE")

    sales:double(29320991000)

    sales:29320991000

    Tuesday, June 7, 2011 3:25 AM
  • Hi Nitin,

     

    I have verified that doubles are limited to 6 digits of precision of ESP.  Floats are slightly different, but you will find that the 6 digits of precision are more or less the same for them as well.  Some of this is due to the fact that ESP is 32 bit code, and some was due to design specifications that required consistent results across multiple platforms.

     

    Doubles being converted to floats during document processing if not submitted as a string are limits in document processing and do not apply to FQL queries.

     

    Thanks,

    Patrick

     


    Patrick Schneider | Microsoft | Enterprise Search Group | Support Escalation Engineer | http://www.microsoft.com/enterprisesearch
    Tuesday, June 7, 2011 12:58 PM
  • Thanks Patrick:)

    Dont know how to apporach this problem. Can you suggest a work arround for this.

     

    Wednesday, June 8, 2011 6:16 AM
  • Hi Nitin,

     

    You may want to consider storing the data in both a float/double field and a string field.  You can use the string field for exact searches and the other field for navigation and/or range searches where the level of precision is not as critical.  You would likely have to add some logic to your front end to help make it easier for your users, but it is an option.

     

    Thanks,

    Patrick

     


    Patrick Schneider | Microsoft | Enterprise Search Group | Support Escalation Engineer | http://www.microsoft.com/enterprisesearch
    Wednesday, June 8, 2011 12:36 PM
  • Thanks Patrick:).
    Wednesday, June 15, 2011 7:19 AM