none
FAST JDBC Connector and "ChangeDBIncludeFields" not working RRS feed

  • Question

  • When I leave the "ChangeDBIncludeFields" value blank, I get the desired behavior -- i.e. only records who have had some value change in one of the fields gets fed through to the pipeline.  But when I try to use a subset of fields for change detection, I get an error.

    Here's the XML that I'm using for the change detection:

    <parameter name="ChangeDBIncludeFields" type="string">
      <description><![CDATA[A semi colon separated list of field names that are to be included in the checksum computations. <br>Default: (none)]]>
      </description>
      <value><![CDATA[ProductReviewID;ProductID;ReviewerName;EmailAddress;Rating;Comments]]></value>
    </parameter>

    Here's the XML for the actual SQL statement:

    <parameter name="JDBCSQL" type="string">
      <value><![CDATA[SELECT ProductReviewID,ProductID,ReviewerName,ReviewDate,EmailAddress,Rating,Comments,ModifiedDate FROM AdventureWorks2008R2.Production.ProductReview]]></value>
    </parameter>

    Here's the error I'm getting:

    20:03:22,999 ERROR [CCTKDocumentFeeder] com.fastsearch.esp.cctk.changedetection.ChangeDetectorException: Field missing in input for checksum computation: ProductReviewID
    20:03:22,999 WARN  [CCTKDocumentFeeder] Publisher 0:Publish Exception!!
    20:03:22,999 WARN  [CCTKDocumentFeeder] Warning when crawling the source system: Error when adding document:1

    The "Error when adding document:1" comes from the first row in the result set has a ProductReviewID of 1, and that's the field configured as the primary key to be used for the doc id in FAST.

    Any suggestions on how to troubleshoot this, or what might be wrong?

    As a side note, I know the DB is set for Latin1, and my configuration file is set for UTF-8.  But I don't know if that would impact this, nor do I know where a list of valid codes for the JDBCCharacterEncoding parameter.


    Garth Grimm
    Avery Ranch Consulting
    www.averyranchconsulting.com

    Friday, February 17, 2012 2:17 AM

All replies

  • Garth,

    Just checking, do you also have ChangeDBExcludeFields  set?  That parameter takes precedence if both ChangeDBExcludeFields and ChangeDBIncludeFields have non-empty values.


    Igor Veytskin

    Monday, February 20, 2012 8:36 PM
    Moderator
  • No.  Just the ChangeDBIncludeFields has values.

    Garth Grimm
    Avery Ranch Consulting
    www.averyranchconsulting.com

    Monday, February 20, 2012 9:32 PM
  • Having the same issue. anyone have an idea? 
    Tuesday, April 15, 2014 3:14 AM
  • I found the solution. switch on the ExportToXML to true in your file

    locate the logs file and read the actual column value which will be something like

    

    [PropSetId]:[columnName]:[type] in my case it was

    

    4cc9f20A-N887-5C64-3289-C4946F3DF889:updatedate:64

    

    now insert this value 

    

    <parametername=" ChangeDBIncludeFields"type="string">
      <description><![CDATA[A semi colon separated list of field names that are to be included in the checksum computations. <br>Default: (none)]]>
      </description>
      <value>
    4cc9f20A-N887-5C64-3289-C4946F3DF889:updatedate:64</value>
    </parameter>

    Run the crawl and it will not complain anymore.

    

    hope this helps someone.

    • Proposed as answer by Omer Zubair Wednesday, April 16, 2014 4:16 AM
    Wednesday, April 16, 2014 4:16 AM