Answered by:
FAST ESP 5.3: logtransformer error

Question
-
We are using FAST ESP 5.3 sp4 as the enterprise search system. It contains English and Japanese data. Starting from December 16th, following error message has been showed in the Log page hourly.
ERROR The lql job had problems running
ERROR Unable to persist results. ERROR: invalid byte sequence for encoding "UTF8": 0x00
I have stopped logtransformer, deleted datasources.xml and restarted logtransformer but it's still not working. Please advice what else I can try to fix the issue.
Any help will be highly appreciated.
Felix
Friday, January 10, 2014 9:39 AM
Answers
-
I previously had seen a similar issue, which was caused by encoding in the querylog...specifically queries were encoded using gb2312 ( not UTF8) which was making the logtransformer chock and SBC reporting were failing.
Changing the encoding in the querylogs from ‘gb2312’ to UTF-8 and reprocessing the queries resulted to SBC reports working fine.
We also confirmed that the search profile query handling ‘encoding’ parameter is set to UTF-8.- Marked as answer by Rob Va - MSFT Friday, May 9, 2014 9:03 PM
Wednesday, February 5, 2014 3:14 PM
All replies
-
Hi Felix,
I am not sure if you use Query Reports. But if Query Reporting is not being used, we can stop logtransformer.
I would wait for others to pour some light on this specific error you are encountering.
neoHypocrite
Tuesday, January 28, 2014 8:44 AM -
I previously had seen a similar issue, which was caused by encoding in the querylog...specifically queries were encoded using gb2312 ( not UTF8) which was making the logtransformer chock and SBC reporting were failing.
Changing the encoding in the querylogs from ‘gb2312’ to UTF-8 and reprocessing the queries resulted to SBC reports working fine.
We also confirmed that the search profile query handling ‘encoding’ parameter is set to UTF-8.- Marked as answer by Rob Va - MSFT Friday, May 9, 2014 9:03 PM
Wednesday, February 5, 2014 3:14 PM -
Try this:
It seems that some of the files are still corrupted and
logtransformer could not able to parse them for reasons.I would recommend you following steps:
a) Check that all META files in
$FASTSEARCH/components/logtransformer/data/download have a size greater than
zero, and remove any that do not.b) Run 'nctrl stop logtransformer'
c) Backup and delete
$FASTSEARCH/components/logtransformer/data/datastore/datasources.xmld) Backup and delete
$FASTSEARCH/components/logtransformer/data/datastore/logmerger-statistics.xmle) Backup and delete
$FASTSEARCH/components/logtransformer/data/datastore/querylog/crateindex.xmlf) Run 'nctrl start logtransformer'
Then this:
Run following commands to download missing query logs:
a. $FASTSEARCH/components/logtransformer/bin>
launch.bat downloaderb. $FASTSEARCH/components/logtransformer/bin>
launch.bat logmergerThen this:
To process the missing periods, use queryreportgenerator (adjust the
start and end times) Example below:$FASTSEARCH/esp4j/bin> launch.bat
queryreportgenerator -s "2011-05-01 00:00:00" -e "2011-06-07
00:00:00" -D –LPlease use a small date interval. Date-Time format:
Year-Month-Day Hour:Minute:Second.
Wednesday, April 16, 2014 11:22 AM -
Hi Felix,
Is this issue still occurring? This error is actually related to Postgres, which is the database used by FAST. There is a patch to upgrade the Postgress version, and there is an adminserver patch that updates the adminserver component, which is responsible for displaying the reports that should be applied. Can you confirm that you have the below patches in place?
esp.5.3.sp4.adminserver.patch10
esp.5.3.sp4.adminstorage.patch01Let us know your findings.
Thanks!
Rob Vazzana | Sr Support Escalation Engineer | US Customer Service & Support
Customer Service & Support Microsoft| Services
- Proposed as answer by Rob Va - MSFT Wednesday, April 23, 2014 5:39 PM
Wednesday, April 23, 2014 5:35 PM