none
FAST ESP : Clearing Logs RRS feed

  • Question

  • Hi,

    Lot of logs get generated over a period of time in the system.

    In our system when er are trying to look at the archive logs following error comes:

    Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 16 bytes) in <FastDiretory>/www/admin/logs/index.php on line 348

     

    Is there a mechanism in fast that periodically removes the older logs Or we have to remove them manually?

    Thanks


    • Edited by Nitin D Monday, September 5, 2011 5:38 AM
    Monday, September 5, 2011 5:34 AM

Answers

  • Hi Nitin,

     

    You can configure ESP to rotate logs, and specifically, the rotatesize parameter in the LoggerConfig.xml should be helpful, as it allows one to specify the size in bytes that the log file will grow to before it is rotated.   The ESP Operations Guide describes more about this on page 132, and I have copied the related details below:

     

    About configuring logging

    The FAST ESP loggers can be configured to direct messages to various destinations, depending on the log level of the messages or the programs generating them.  The logging configuration file $FASTSEARCH/etc/LoggerConfig.xml controls how much logging output is produced and where. The default LoggerConfig.xml defines a few standard log destinations:

    • "server": The system log, which is collected by the log server on the admin node and viewable through the FAST ESP administrator interface.

    • "file": Local log files in $FASTSEARCH/var/log, with log files rotated (archived) based on a size threshold.

    • "dailyfile": Local log files as above, but rotated once each night.

    • "stdout": The standard output of the process.

    The <programs> section directs the messages generated by the FAST ESP components to particular log destinations. The <program name="default"> entry specifies log destinations and log level thresholds that apply to all programs. Other entries apply to the named "program" only, and those settings override those in the default entry.

    For example:

    <program name="fsearch">

    <output destination="server" threshold="warning"/>

    <output destination="file"/>

    <output destination="stdout"/>

    </program>

    This specifies that the "fsearch" program logs to the named destinations "server", "file" and "stdout". Only warning, error and critical messages are logged to the log server, while the two other destinations receive all messages regardless of level.  To use daily log rotation instead of size-based rotation for the local log files, replace <output destination="file"/> with <output destination="dailyfile"/>.

     

    To control the amount of space consumed by logs, adjust the log rotation settings in the <filedestination> entries.

     

    Note: If logs are rotated by size (the "file" destination), there is an absolute bound on the size consumed by the log. If logs are rotated daily, individual log files could grow arbitrarily large.  To use alternate settings for some programs, create new log destinations with the new settings. For example, to keep fewer archived log generations for programs generating high volumes of log messages:

    . . .

    <filedestination name="boundedfile">

    <directory>$FASTSEARCH/var/debuglog</directory>

    <backups>1</backups>

    <rotatesize>500000000</rotatesize>

    </filedestination>

    . . .

    <program name="mycomponent">

    <output destination="dailyfile" threshold="info"/>

    <output destination="boundedfile"/>

    </program>

    . . .

    This will log info, warning, error, and critical level messages to "dailyfile" logger, ensuring that (with the default settings) a 10 day record of the operation of the component is kept. Debug and verbose level messages are logged to an alternate log which is limited in size.

     ------

    Let us know your results, and if you have any questions.

     

    Thanks!

    Rob Vazzana | Sr Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                          Microsoft | Services

    Tuesday, September 20, 2011 7:04 PM
    Moderator

All replies

  • How about creating a scheduled windows command line job which stops the log, moves the log file to a different directory and starts the logging immediately?
    Chandra Sekhar
    • Edited by Chandra.Sekhar Thursday, September 8, 2011 10:10 PM I typed rename instead of move
    • Proposed as answer by Chandra.Sekhar Thursday, September 8, 2011 10:10 PM
    Thursday, September 8, 2011 10:08 PM
  • Hi Nitin,

     

    You can configure ESP to rotate logs, and specifically, the rotatesize parameter in the LoggerConfig.xml should be helpful, as it allows one to specify the size in bytes that the log file will grow to before it is rotated.   The ESP Operations Guide describes more about this on page 132, and I have copied the related details below:

     

    About configuring logging

    The FAST ESP loggers can be configured to direct messages to various destinations, depending on the log level of the messages or the programs generating them.  The logging configuration file $FASTSEARCH/etc/LoggerConfig.xml controls how much logging output is produced and where. The default LoggerConfig.xml defines a few standard log destinations:

    • "server": The system log, which is collected by the log server on the admin node and viewable through the FAST ESP administrator interface.

    • "file": Local log files in $FASTSEARCH/var/log, with log files rotated (archived) based on a size threshold.

    • "dailyfile": Local log files as above, but rotated once each night.

    • "stdout": The standard output of the process.

    The <programs> section directs the messages generated by the FAST ESP components to particular log destinations. The <program name="default"> entry specifies log destinations and log level thresholds that apply to all programs. Other entries apply to the named "program" only, and those settings override those in the default entry.

    For example:

    <program name="fsearch">

    <output destination="server" threshold="warning"/>

    <output destination="file"/>

    <output destination="stdout"/>

    </program>

    This specifies that the "fsearch" program logs to the named destinations "server", "file" and "stdout". Only warning, error and critical messages are logged to the log server, while the two other destinations receive all messages regardless of level.  To use daily log rotation instead of size-based rotation for the local log files, replace <output destination="file"/> with <output destination="dailyfile"/>.

     

    To control the amount of space consumed by logs, adjust the log rotation settings in the <filedestination> entries.

     

    Note: If logs are rotated by size (the "file" destination), there is an absolute bound on the size consumed by the log. If logs are rotated daily, individual log files could grow arbitrarily large.  To use alternate settings for some programs, create new log destinations with the new settings. For example, to keep fewer archived log generations for programs generating high volumes of log messages:

    . . .

    <filedestination name="boundedfile">

    <directory>$FASTSEARCH/var/debuglog</directory>

    <backups>1</backups>

    <rotatesize>500000000</rotatesize>

    </filedestination>

    . . .

    <program name="mycomponent">

    <output destination="dailyfile" threshold="info"/>

    <output destination="boundedfile"/>

    </program>

    . . .

    This will log info, warning, error, and critical level messages to "dailyfile" logger, ensuring that (with the default settings) a 10 day record of the operation of the component is kept. Debug and verbose level messages are logged to an alternate log which is limited in size.

     ------

    Let us know your results, and if you have any questions.

     

    Thanks!

    Rob Vazzana | Sr Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                          Microsoft | Services

    Tuesday, September 20, 2011 7:04 PM
    Moderator