locked
Unix Log Monitoring regular expression not picking up alerts RRS feed

  • Question

  • Hi,

    We are moving our unix monitoring to SCOM 2012 SP1 rollup 4.

    What I have got working is indvidual alert logging of Unix Log alerts by exporting the MP and changing the <IndividualAlerts> value to true and removing the suppression xml section then reimporting the MP.

    What I am trying to do is use the regular expression to peform the suppression of specific event (such as event codes).

    The expression is:

    ((?i:warning)(?!(.*1222)|(.*1001)))

    ie Search the log for "warning" (not case sensitive) then check if events 1222 or 1001 exist if so return no match, if they dont exist then return true. 

    I use the built in test function in SCOM when creating the rule and the tests come back as expected but when I inject test lines into the unix log, no alerts get generted.

    I suspect it could be the syntax not being accepted on the system (its running RedHat 6 )

    I have tested this with regex tools and works.

    When I try and test it on the server i get:

    [root@bld02 ~]# grep ((?i:Warning)(?!(.*1222)|(.*1001))) /var/log/messages
    -bash: !: event not found
    [root@bld02 ~]# tail /var/log/messages
    Nov 13 15:07:26 bld02 root: SCOM Test Warning Event ID 1001 Round 18
    Nov 13 15:07:29 bld02 root: SCOM Test Warning Event ID 1000 Round 18
    Nov 13 15:07:35 bld02 root: SCOM Test Warning Event ID 1002 Round 18

    So I am expecting 2 alerts to be generated.

    SCOM tests to show expression working:

    Test 1 Matching

    Test 2 to exclude

    Need some help with this, Thankyou in advance :)



    • Edited by Martin Sustaric Wednesday, November 13, 2013 5:52 AM updated case
    Wednesday, November 13, 2013 4:19 AM

Answers

  • Hello,

    Here's an example of modifying the MP to exclude particular events.  Firstly, I created a log file rule using the MP template that is fairly inclusive - matching the string Warning (with either a lower or upper case W).

    I then exported the MP, and modified the rule.  I set the IndividualAlerts = true and removed the AlertSuppression element, so that every matched line will fire a unique alert.  You don't have to remove the AlertSuppression, but you should use Individual alerts so that the exclusion logic doesn't exclude concurrent events that you actually want to match.

    Implementing the exclusion logic involves the addition of a System.ExpressionFilter definition in the rule. This will use a conditional evaluation of the //row element of the data item.  Here's an example of a dataitem matching an individual row:

    <DataItem type="System.Event.Data"time="2013-11-15T10:33:14.8839662-08:00"sourceHealthServiceId="667FF365-70DD-6607-5B66-F9F95253B29F">
     <EventOriginId>{86AB962D-2F44-29FD-A909-B99FF6FEB2C5}</EventOriginId>
     <PublisherId>{EC7EA4B1-0EA5-7E8E-701F-82FEF3367BC4}</PublisherId>
     <PublisherName>WSManEventProvider</PublisherName>
     <EventSourceName>WSManEventProvider</EventSourceName>
     <Channel>WSManEventProvider</Channel>
     <LoggingComputer/>
     <EventNumber>0</EventNumber>
     <EventCategory>3</EventCategory>
     <EventLevel>0</EventLevel>
     <UserName/>
     <RawDescription>Detected Entry: warning 1002</RawDescription>
     <CollectDescription Type="Boolean">true</CollectDescription>
    <EventData>
    <DataItem type="SCXLogProviderDataSourceData"time="2013-11-15T10:33:14.8839662-08:00"sourceHealthServiceId="667FF365-70DD-6607-5B66-F9F95253B29F">
    <SCXLogProviderDataSourceData>
     <row>warning 1002</row>
     </SCXLogProviderDataSourceData>
     </DataItem>
     </EventData>
     <EventDisplayNumber>0</EventDisplayNumber>
     <EventDescription>Detected Entry: warning 1002</EventDescription>
     </DataItem>

    Here is the rule in the MP XML.  The <ConditionDetection>...</ConditionDetection> content was what I added to do the exclusion filtering:

          <Rule ID="LogFileTemplate_66b86eaded094c309ffd2631b8367a32.Alert" Enabled="false" Target="Unix!Microsoft.Unix.Computer" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100">
            <Category>EventCollection</Category>
            <DataSources>
              <DataSource ID="EventDS" TypeID="Unix!Microsoft.Unix.SCXLog.VarPriv.DataSource">
                <Host>$Target/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$</Host>
                <LogFile>/tmp/test</LogFile>
                <UserName>$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$</UserName>
                <Password>$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$</Password>
                <RegExpFilter>warning</RegExpFilter>
                <IndividualAlerts>true</IndividualAlerts>
              </DataSource>
            </DataSources>
            <ConditionDetection TypeID="System!System.ExpressionFilter" ID="Filter">
    		<Expression>
    		  <RegExExpression>
    		    <ValueExpression>
    		      <XPathQuery Type="String">//row</XPathQuery>
    		    </ValueExpression>
    		    <Operator>DoesNotContainSubstring</Operator>
    		    <Pattern>1001</Pattern>
    		  </RegExExpression>
    		</Expression>
            </ConditionDetection>
            <WriteActions>
              <WriteAction ID="GenerateAlert" TypeID="Health!System.Health.GenerateAlert">
                <Priority>1</Priority>
                <Severity>2</Severity>
                <AlertName>Log File Alert:  ExclusionExample</AlertName>
                <AlertDescription>$Data/EventDescription$</AlertDescription>
              </WriteAction>
            </WriteActions>
          </Rule>

    I traced this with the Workflow Analyzer as I tested, which shows the logic being applied.  Here is the exclusion happening:

    Here's more info on the definition of an ExpressionFilter: http://msdn.microsoft.com/en-us/library/ee692979.aspx

    And more information on Regular Expressions in MPs: http://support.microsoft.com/kb/2702651/en-us

    You can also have multiple Expressions in the ExpressionFilter joined by OR or AND operators.

    Also, if you are comfortable with the MP authoring, you can just skip the step of creating the rules in the MP template and just author your own MP with the VSAE tool: http://social.technet.microsoft.com/wiki/contents/articles/18085.scom-2012-authoring-unixlinux-log-file-monitoring-rules.aspx


    www.operatingquadrant.com

    Friday, November 15, 2013 8:00 PM

All replies

  • Hello,

    Unfortunately, the "Test Log File Expression" utility runs on the Windows computer, and uses a different RegEx parser (.NET) than is used in the actual log file parsing.  The log files are parsed on the UNIX/Linux computer using standard POSIX RegEx.  I'm pretty sure that POSIX RegEx doesn't support negative lookaheads (?!)

    Is it possible to define inclusive RegExp for your desired alerting events?  Alterantively, if you export the MP and modify it, you could add a System.ExpressionFilter condition detection module to the rule to do additional .NET RegEx parsing before generating the alert. 

    -Kris


    www.operatingquadrant.com

    Wednesday, November 13, 2013 8:38 PM
  • Hi,

    Thanks for explaining why the regex isnt working, is it possible to update the regex engin on a unix/linux system so that this could work?

    Also were moving unix monitoring from our old CA NSM system to SCOM and the rules have exclusions. Do you have any links or can explain how to add a System.ExpressionFilter condition detection module to the rule to do additional .NET RegEx parsing before generating the alert?

    Thanks again Martin.

    Wednesday, November 13, 2013 10:16 PM
  • I can put together an example of the MP modification...give me a day or two to do that.

    -Kris


    www.operatingquadrant.com

    Wednesday, November 13, 2013 10:47 PM
  • Thanks for your help, I haven't been successful adding the necessary xml will wait for your reply.
    Thursday, November 14, 2013 8:37 AM
  • Hello,

    Here's an example of modifying the MP to exclude particular events.  Firstly, I created a log file rule using the MP template that is fairly inclusive - matching the string Warning (with either a lower or upper case W).

    I then exported the MP, and modified the rule.  I set the IndividualAlerts = true and removed the AlertSuppression element, so that every matched line will fire a unique alert.  You don't have to remove the AlertSuppression, but you should use Individual alerts so that the exclusion logic doesn't exclude concurrent events that you actually want to match.

    Implementing the exclusion logic involves the addition of a System.ExpressionFilter definition in the rule. This will use a conditional evaluation of the //row element of the data item.  Here's an example of a dataitem matching an individual row:

    <DataItem type="System.Event.Data"time="2013-11-15T10:33:14.8839662-08:00"sourceHealthServiceId="667FF365-70DD-6607-5B66-F9F95253B29F">
     <EventOriginId>{86AB962D-2F44-29FD-A909-B99FF6FEB2C5}</EventOriginId>
     <PublisherId>{EC7EA4B1-0EA5-7E8E-701F-82FEF3367BC4}</PublisherId>
     <PublisherName>WSManEventProvider</PublisherName>
     <EventSourceName>WSManEventProvider</EventSourceName>
     <Channel>WSManEventProvider</Channel>
     <LoggingComputer/>
     <EventNumber>0</EventNumber>
     <EventCategory>3</EventCategory>
     <EventLevel>0</EventLevel>
     <UserName/>
     <RawDescription>Detected Entry: warning 1002</RawDescription>
     <CollectDescription Type="Boolean">true</CollectDescription>
    <EventData>
    <DataItem type="SCXLogProviderDataSourceData"time="2013-11-15T10:33:14.8839662-08:00"sourceHealthServiceId="667FF365-70DD-6607-5B66-F9F95253B29F">
    <SCXLogProviderDataSourceData>
     <row>warning 1002</row>
     </SCXLogProviderDataSourceData>
     </DataItem>
     </EventData>
     <EventDisplayNumber>0</EventDisplayNumber>
     <EventDescription>Detected Entry: warning 1002</EventDescription>
     </DataItem>

    Here is the rule in the MP XML.  The <ConditionDetection>...</ConditionDetection> content was what I added to do the exclusion filtering:

          <Rule ID="LogFileTemplate_66b86eaded094c309ffd2631b8367a32.Alert" Enabled="false" Target="Unix!Microsoft.Unix.Computer" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100">
            <Category>EventCollection</Category>
            <DataSources>
              <DataSource ID="EventDS" TypeID="Unix!Microsoft.Unix.SCXLog.VarPriv.DataSource">
                <Host>$Target/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$</Host>
                <LogFile>/tmp/test</LogFile>
                <UserName>$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$</UserName>
                <Password>$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$</Password>
                <RegExpFilter>warning</RegExpFilter>
                <IndividualAlerts>true</IndividualAlerts>
              </DataSource>
            </DataSources>
            <ConditionDetection TypeID="System!System.ExpressionFilter" ID="Filter">
    		<Expression>
    		  <RegExExpression>
    		    <ValueExpression>
    		      <XPathQuery Type="String">//row</XPathQuery>
    		    </ValueExpression>
    		    <Operator>DoesNotContainSubstring</Operator>
    		    <Pattern>1001</Pattern>
    		  </RegExExpression>
    		</Expression>
            </ConditionDetection>
            <WriteActions>
              <WriteAction ID="GenerateAlert" TypeID="Health!System.Health.GenerateAlert">
                <Priority>1</Priority>
                <Severity>2</Severity>
                <AlertName>Log File Alert:  ExclusionExample</AlertName>
                <AlertDescription>$Data/EventDescription$</AlertDescription>
              </WriteAction>
            </WriteActions>
          </Rule>

    I traced this with the Workflow Analyzer as I tested, which shows the logic being applied.  Here is the exclusion happening:

    Here's more info on the definition of an ExpressionFilter: http://msdn.microsoft.com/en-us/library/ee692979.aspx

    And more information on Regular Expressions in MPs: http://support.microsoft.com/kb/2702651/en-us

    You can also have multiple Expressions in the ExpressionFilter joined by OR or AND operators.

    Also, if you are comfortable with the MP authoring, you can just skip the step of creating the rules in the MP template and just author your own MP with the VSAE tool: http://social.technet.microsoft.com/wiki/contents/articles/18085.scom-2012-authoring-unixlinux-log-file-monitoring-rules.aspx


    www.operatingquadrant.com

    Friday, November 15, 2013 8:00 PM
  • Hi Kris,

    This worked like a charm, just on a side note once you add the condition detection to the XML and the import the MP, you can then add/edit the filters in the gui. The condition field is added in the gui and allows you to edit the conditions from here:

    Thanks again for your help!

    Martin.


    Monday, November 18, 2013 12:03 AM
  • Hi Martin, I was just talking to Kris offline and wondering how is this Linux monitoring going for you. I have just put something like this in place, but I'm getting a funny side effect when I disable IndividualAlerts. It is as if the Eventlog provider will always analyze the whole set of lines and will alert on all the lines that have previously triggered the rule whenever the file is updated, matching the expression+condition or not.

    Have you experienced anything like that?

    Thank you.


    MCITP, MCSE, MCTS

    Monday, December 1, 2014 8:25 PM
  • Hey Jose,

    I haven't seen this behavior. But I did have an issue recently with my MP that had these filters configured. You might want to see if this affected you (I suspect it was caused by upgrade). See: https://social.technet.microsoft.com/Forums/systemcenter/en-US/9852bc5a-10f9-40e5-a379-c77026690c6d/console-errors-verification-failed-with-1-errors-and-when-deleting-overrides?forum=operationsmanagermgmtpacks

    A quick test to see is to export the MP which has the unix logs rules that have the filter condition then re import them. If it imports OK then your not affected. If it errors then you might have the same issue I had.



    Cheers,

    Martin

    Blog: http://sustaslog.wordpress.com  LinkedIn:   

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Monday, December 1, 2014 10:27 PM
  • Hey Martin, thanks for the reply. No, mine seems to be working fine with the overrides. However, still getting alerts every time the file is updated, as many alerts as the times the criteria is matched in the file. Would you mind sharing one of you rules' XML for comparison? I don't think they would be too different, but just wanted to make sure.

    Here's what I've got:

    <Rules>
          <Rule ID="Test.Extended.Log.File.Linux.Rule" Enabled="false" Target="Unix!Microsoft.Unix.Computer" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100">
            <Category>EventCollection</Category>
            <DataSources>
              <DataSource ID="EventDS" TypeID="Unix!Microsoft.Unix.SCXLog.VarPriv.DataSource">
                <Host>$Target/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$</Host>
                <LogFile>/etc/testlog</LogFile>
                <UserName>$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/UserName$</UserName>
                <Password>$RunAs[Name="Unix!Microsoft.Unix.ActionAccount"]/Password$</Password>
                <RegExpFilter>error</RegExpFilter>
                <IndividualAlerts>true</IndividualAlerts>
              </DataSource>
            </DataSources>
            <ConditionDetection TypeID="System!System.ExpressionFilter" ID="Filter">
              <Expression>
                <RegExExpression>
                  <ValueExpression>
                    <XPathQuery Type="String">//row</XPathQuery>
                  </ValueExpression>
                  <Operator>DoesNotContainSubstring</Operator>
                  <Pattern>1001</Pattern>
                </RegExExpression>
              </Expression>
            </ConditionDetection>
            <WriteActions>
              <WriteAction ID="GenerateAlert" TypeID="Health!System.Health.GenerateAlert">
                <Priority>1</Priority>
                <Severity>2</Severity>
                <AlertName>Log File Alert:  ExclusionExample</AlertName>
                <AlertDescription>
                  Description: $Data/EventDescription$
                  Raw Data:
                  $Data///SCXLogProviderDataSourceData$
                </AlertDescription>
              </WriteAction>
            </WriteActions>
          </Rule>
        </Rules>


    MCITP, MCSE, MCTS

    Tuesday, December 2, 2014 1:33 PM
  • Hey Jose,

    Here is an example I am using:

          <Rule ID="LogFileTemplate_8d97498787e44bb08d640e79a58c4919.Alert" Enabled="false" Target="Unix!Microsoft.Unix.Computer" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100">
            <Category>EventCollection</Category>
            <DataSources>
              <DataSource ID="EventDS" TypeID="Unix!Microsoft.Unix.SCXLog.VarPriv.DataSource">
                <Host>$Target/Property[Type="Unix!Microsoft.Unix.Computer"]/PrincipalName$</Host>
                <LogFile>/u01/app/oracle/diag/rdbms/pfinarc/PFINARC1/trace/alert_PFINARC1.log</LogFile>
                <UserName>$RunAs[Name="Unix!Microsoft.Unix.PrivilegedAccount"]/UserName$</UserName>
                <Password>$RunAs[Name="Unix!Microsoft.Unix.PrivilegedAccount"]/Password$</Password>
                <RegExpFilter>ORA-</RegExpFilter>
                <IndividualAlerts>true</IndividualAlerts>
              </DataSource>
            </DataSources>
            <ConditionDetection ID="Filter" TypeID="System!System.ExpressionFilter">
              <Expression>
                <RegExExpression>
                  <ValueExpression>
                    <XPathQuery Type="String">//row</XPathQuery>
                  </ValueExpression>
                  <Operator>DoesNotMatchRegularExpression</Operator>
                  <Pattern>27037|00245|00227|227|245|00202</Pattern>
                </RegExExpression>
              </Expression>
            </ConditionDetection>
            <WriteActions>
              <WriteAction ID="GenerateAlert" TypeID="Health!System.Health.GenerateAlert">
                <Priority>1</Priority>
                <Severity>2</Severity>
                <AlertName>Log File Alert:  XX XXXXXXXXX01 - alert_PFINARC1.log DB Log Monitoring Rule - Critical</AlertName>
                <AlertDescription>$Data/EventDescription$</AlertDescription>
              </WriteAction>
            </WriteActions>
          </Rule>

    One difference I do see is in the filter operation:

    Yours: "<Operator>DoesNotContainSubstring</Operator>"

    Mine: "<Operator>DoesNotMatchRegularExpression</Operator>"

    I tend to use regular expression as I seem to have less issues with the result (also when using the scom log wizard to create a rule it uses regular expression so trying to keep it all consistent). Might be the cause?  


    Cheers,

    Martin

        

    Blog:http://sustaslog.wordpress.comLinkedIn:

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.



    Tuesday, December 2, 2014 11:13 PM