none
AIX 6/7.1 Agent filling /var/opt/microsoft/scx/log/scx.log - Agent Version 1.5.1-184

    Question

  • Hello,

    We currently have a support case open with Microsoft but I wanted to check here as well.  A few days after installing ~140 AIX agents running AIX 6, 6.1, and 7.1 we starting noticing the scx.log growing quite large and filling disks.

    We don't receive any errors or other alerts that indicate a problem with the agent itself and running the commands below return output just fine.

    bash-4.2$ . /opt/microsoft/scx/bin/tools/setup.sh

    bash-4.2$ omicli ei root/scx SCX_Agent

    The log will start out like this typically:

    * Build number: 1.5.1-184 Labeled_Build
    * Process id: 12779762
    * Process started: 2017-04-18T14:52:29,501Z
    *
    * Log format: <date> <severity>     [<code module>:<line number>:<process id>:<thread id>] <message>

    2017-04-18T14:52:29,501Z Info       [scx.core.providers:205:12779762:1] SCX Provider Module loaded
    2017-04-18T14:54:29,753Z Error      [NetworkInterfaceInfo:1665:12779762:1] Invalid seleced speed: 242- interface: en0
    2017-04-18T14:54:29,753Z Error      [NetworkInterfaceInfo:1665:12779762:1] Invalid seleced speed: 242- interface: en1
    2017-04-18T15:12:29,262Z Warning    [scx.core.common.pal.system.process.processinstance:2565:12779762:1029] Module is empty

    Then we will start seeing these messages below repeatedly back and forth:

    2017-04-18T17:04:28,448Z Warning    [scx.core.common.pal.system.process.processinstance:2565:32309590:515] Module is empty.
    2017-04-18T17:04:28,520Z Warning    [scx.core.common.pal.system.process.processinstance:1913:32309590:1] Module path empty.
    2017-04-18T17:04:28,520Z Warning    [scx.core.common.pal.system.process.processinstance:1921:32309590:1] No module available.
    2017-04-18T17:04:28,520Z Warning    [scx.core.common.pal.system.process.processinstance:1913:32309590:1] Module path empty.

    Has anyone ever encountered this before?  For the time being we've stopped all of our AIX agents while we investigate and for what it's worth we have over 1000 RHEL systems running without any issues.

    Thanks in advance

    Tuesday, April 18, 2017 5:10 PM

Answers

  • Hello,

    We resolved this with MS support.  We were in the process of migrating AIX systems from IBM Tivoli Monitoring (ITM) and noticed almost identical messages in those logs as well.

    Apparently on AIX there are some system processes that the monitoring agents can't read which is what these errors are indicating.

    Our solution was to turn down the logging level to ERROR only on all of our agents using these commands during our install script:

    scxadmin -log-set provider FILE:/var/opt/microsoft/scx/log/scx.log:=ERROR

    scxadmin -log-set provider FILE:/var/opt/microsoft/scx/log/scxadmin.log:=ERROR

    Hopefully this helps others.

    • Marked as answer by millern4 Thursday, April 27, 2017 12:06 PM
    Thursday, April 27, 2017 12:06 PM

All replies

  • Hi,

    Can you please verify if the Network Interface is supported ? -- 

    https://github.com/Microsoft/pal/blob/ddec487978123d1dc7cb6de25d56afc8ac214ed4/source/code/scxsystemlib/networkinterface/networkinterface.cpp

    Line Number : 1467 has list of supported drivers.

    Thanks,

    Nirbhay Singh

    Thursday, April 27, 2017 7:17 AM
  • Hello,

    We resolved this with MS support.  We were in the process of migrating AIX systems from IBM Tivoli Monitoring (ITM) and noticed almost identical messages in those logs as well.

    Apparently on AIX there are some system processes that the monitoring agents can't read which is what these errors are indicating.

    Our solution was to turn down the logging level to ERROR only on all of our agents using these commands during our install script:

    scxadmin -log-set provider FILE:/var/opt/microsoft/scx/log/scx.log:=ERROR

    scxadmin -log-set provider FILE:/var/opt/microsoft/scx/log/scxadmin.log:=ERROR

    Hopefully this helps others.

    • Marked as answer by millern4 Thursday, April 27, 2017 12:06 PM
    Thursday, April 27, 2017 12:06 PM