locked
Log files growing 20Gb each day - Running Out Of Disk Space RRS feed

  • Question

  • This folder "C:\esp\components\logtransformer\data\datastore\querylog" is growing 20Gb each day

    I'm Afraid I Will Run Out of Disk Space soon.

    Can I delete the files older than 2 days (crate0, crate1..) ?

    If I remove the old folders (crate0,1,2...) I need to edit crateindex.xml?


    mglauco

    Friday, February 24, 2012 4:08 PM

Answers

  • Hello Marcelo,

    The %FASTSEARCH%components/logtransformer/data/datastore/querylog/<crate0
    through 8>/ directories. Is that data that is used to generate the query reporting.  If you delete this data you will not be able to generate reports. 

    As previously mentioned you can disable the reporting by stoping the logtransformer component:

    Nctrl stop logtransformer

    To prevent it from running again you can remove it from %FASTSEARCH%\etc\NodeConf.xml
    Remove <proc>logtransformer</proc> from <startorder>

    To address space issues you can

    • How to enable and configure LogTransformer clean up job?
      -This process will automatically delete/clean up LogTransformers reporting ($FASTSEARCH/components/logtransformer/data/datastore) and logging ($FASTSEARCH/components/logtransformer/data/download/*.bak)
      a. Rename $FASTSEARCH/esp4j/etc/lqljobs/delete.xml.disabled to delete.xml
      b. Modify the value - to identify how many days of data to keep
                    <entry>
                    <key>daysToKeep</key>
                    <value>7</value> (default is 7 days)
                    </entry>
                    ....
                    <cron-expression>0 30 0 * * ?</cron-expression> (by default it is set to run at 12:30am everyday)
      c. Re-start the logtransformer process:
          -nctrl stop logtransformer
          -ps -ef logtransformer (validate that no hanging process is running)
          -nctrl start logtransformer

    Please let me know if this addresses your issue.

     

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                         Microsoft| Services



    Sunday, April 1, 2012 4:49 PM

All replies

  • If you don't need the query reporting functionality, you can just turn off the logtransformer. It's just reading the FAST query logs, and dumping them into an internal SQL database that the query reporting feature uses (at least this is true in FAST ESP 5.3).


    Thursday, March 1, 2012 9:11 AM
  • Hi Mqlauco,

    I am agreeing with G-Man-UK :)
    I like to add some more steps.

    Check where you installed logtransformer Then.

    1) Run the downloader command to harvest query log files from the QR-server nodes. 
        bin\launch.bat downloader

    2) Run the logmerger to read through all the log files and convert them into a datastore .    
        bin\launch.bat logmerger

    After performing the above steps your log files data will store in PostgreSQL database & you can see query reports in SBC.

    From this database any time you can retrieve required information, below is the sample query.
     
    3) Run the interactive LQL processor to generate the reports for SBC.
        bin\launch.bat ilql >
    select * from querylog for period “start date time “ to “End date time”

    Now with out any worry   you can delete the log files !!! help it will help you .

    BR

    FAST ESP JI

    • Proposed as answer by FAST_ESP_JI Wednesday, March 7, 2012 7:50 AM
    Wednesday, March 7, 2012 7:50 AM
  • Hello Marcelo,

    The %FASTSEARCH%components/logtransformer/data/datastore/querylog/<crate0
    through 8>/ directories. Is that data that is used to generate the query reporting.  If you delete this data you will not be able to generate reports. 

    As previously mentioned you can disable the reporting by stoping the logtransformer component:

    Nctrl stop logtransformer

    To prevent it from running again you can remove it from %FASTSEARCH%\etc\NodeConf.xml
    Remove <proc>logtransformer</proc> from <startorder>

    To address space issues you can

    • How to enable and configure LogTransformer clean up job?
      -This process will automatically delete/clean up LogTransformers reporting ($FASTSEARCH/components/logtransformer/data/datastore) and logging ($FASTSEARCH/components/logtransformer/data/download/*.bak)
      a. Rename $FASTSEARCH/esp4j/etc/lqljobs/delete.xml.disabled to delete.xml
      b. Modify the value - to identify how many days of data to keep
                    <entry>
                    <key>daysToKeep</key>
                    <value>7</value> (default is 7 days)
                    </entry>
                    ....
                    <cron-expression>0 30 0 * * ?</cron-expression> (by default it is set to run at 12:30am everyday)
      c. Re-start the logtransformer process:
          -nctrl stop logtransformer
          -ps -ef logtransformer (validate that no hanging process is running)
          -nctrl start logtransformer

    Please let me know if this addresses your issue.

     

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                         Microsoft| Services



    Sunday, April 1, 2012 4:49 PM