locked
Notification throttling and alert content on SCOM 2007 RRS feed

  • Question

  • Hi,

    Our SCOM 2007 server is managed by a monitoring team so I don't have much control over it.I am trying to go through the SCOM alerts in my mailbox to see if I can minimize the amount of alerts we receive. The system has been (mis)configured by other team and I am trying to give them some ideas on how to ensure that only relevant alerts are sent to end users.

    One example is Warning level alert that is sent every 7 minutes for entire hour. I was wondering if I could just get 1 Warning level notification an hour for this alert unless it is escalating to Critical of course.

    From my experience with Nagios and Zabbix this would be trivial, however SCOM administrator assures me that is not possible.

    Another example is that some alerts show information like 'Memory Pages Per Second is too High' but email does not include values for threshold or how much parameter was over the thershold.According to SCOM Admin, 'SCOM can't provide you with thresholds for the alert if it is not hardcoded in the alert name'.

    Again, providing alert data in Nagios alerts is trivial.

    I have doubts that such complex system can' do these simple tasks. Am I being unreasonable or SCOM admin does not know his product well enough?

    Monday, April 15, 2013 9:17 AM

Answers

  • I'm saying there's ways to tune the email notifications to send only what you need, with performance related alerts it's typically better to configure a subscription to only send if the resolution state has been unchanged for X minutes.  This avoids an email everytime the monitor's threshold gets crossed to healthy / non-healthy, or tweak the thresholds where it doesn't 'flap' every 7 minutes.  The email content should be displayed if the 'alert description' is configured to include it and that field is included in the notification channel.  Review, and have your SCOM Admin review the following links... with some thought, and focused development.  The flexibility is there.

    http://technet.microsoft.com/en-us/library/hh212698.aspx
    http://blog.tyang.org/2010/07/19/enhanced-scom-alerts-notification-emails/
    http://blogs.technet.com/b/kevinholman/archive/2007/12/12/adding-custom-information-to-alert-descriptions-and-notifications.aspx

    $Data[Default='Not Present']/Context/DataItem/AlertDescription$ <-- variable that's need to include the below (from "Available Memory (MB) Monitor)

    The threshold for the Memory\Available MBytes performance counter has been exceeded. The value that exceeded the threshold is: 32.


    B. Wright

    • Proposed as answer by B. Wright Monday, April 15, 2013 10:08 AM
    • Marked as answer by Yog Li Thursday, April 25, 2013 9:22 AM
    Monday, April 15, 2013 10:08 AM
  • Agreed, there have been some oversights that have been committed by Microsoft that omitted the actual value in the alert (alertdesctiption$) that comes from some of the monitors in sealed management packs in their catalog.  Since those are 'sealed', then the only way to overcome this, that I have found, is build subscriptions to email notify those specific monitors' alerts through your own custom notification channel that adds those variables to the 'format' of the body of the notification utilizing the two values below - take note of the 'footnotes' it can be confusing, some of their 'default' "values" are actually 'number of samples' but the alert text implies the 'value' of the counter itself.

    *Value (Perf Counter Value):                  $Data/Context/Value$ 
    **Last Sampled Value                            $Data/Context/SampleValue$

    *Value will show the actual performance value for simple and avg monitors.  It will show number of samples for consecutive threshold monitors.   
    **Last Sampled Value works to show the last value evaluated in a consecutive sample value monitor


    B. Wright

    • Proposed as answer by B. Wright Monday, April 15, 2013 1:23 PM
    • Marked as answer by Yog Li Thursday, April 25, 2013 9:22 AM
    Monday, April 15, 2013 1:23 PM

All replies

  • What alert are you receiving "One example is Warning level alert that is sent every 7 minutes for entire hour. I was wondering if I could just get 1 Warning level notification an hour for this alert unless it is escalating to Critical of course."

    It sounds like this is an alert from an event based rule, and means that the event is being written to the app or system log and SCOM is trapping it. 

    Workarounds: 

    If you don't care about the event, disable the monitor for the given systems

    Demote the severity to informational, and ask that the subscription not send notifications for informational alerts.

    Custom notification alert sending can be developed to not send during x hours, batch send, or otherwise. 

    If the monitor/rule is enabled, and alert will be sent to the console, but does not NEED to be sent in email.  The notification subscription feature is highly flexible.


    B. Wright

    Monday, April 15, 2013 9:28 AM
  • Hi,

     Thank you for your input!

    The alert is for  Windows performance counter 'RAM Utilization Threshold Exceeded'

    Are you saying that I can tune notification subscription to ensure that I don't receive more than one Warning notification an hour?

    Also , I have just spoken to our SCOM admin asking why we can't see threhold and data in email notification, i.e. for p

    'Warning (Windows Core OS) SCOM Alert on XXXXXXXXXXXXX: Memory Pages Per Second is too High.'

    The email content does not have any data on what is the threshold of this parameter and what is the latest value.

    The only way to get this information according to SCOM admin is to logon to SCOM console and investigate the settings.This sounds unflexible to say it mildly.

     

    Monday, April 15, 2013 9:53 AM
  • I'm saying there's ways to tune the email notifications to send only what you need, with performance related alerts it's typically better to configure a subscription to only send if the resolution state has been unchanged for X minutes.  This avoids an email everytime the monitor's threshold gets crossed to healthy / non-healthy, or tweak the thresholds where it doesn't 'flap' every 7 minutes.  The email content should be displayed if the 'alert description' is configured to include it and that field is included in the notification channel.  Review, and have your SCOM Admin review the following links... with some thought, and focused development.  The flexibility is there.

    http://technet.microsoft.com/en-us/library/hh212698.aspx
    http://blog.tyang.org/2010/07/19/enhanced-scom-alerts-notification-emails/
    http://blogs.technet.com/b/kevinholman/archive/2007/12/12/adding-custom-information-to-alert-descriptions-and-notifications.aspx

    $Data[Default='Not Present']/Context/DataItem/AlertDescription$ <-- variable that's need to include the below (from "Available Memory (MB) Monitor)

    The threshold for the Memory\Available MBytes performance counter has been exceeded. The value that exceeded the threshold is: 32.


    B. Wright

    • Proposed as answer by B. Wright Monday, April 15, 2013 10:08 AM
    • Marked as answer by Yog Li Thursday, April 25, 2013 9:22 AM
    Monday, April 15, 2013 10:08 AM
  • Thank you , this helps a lot.

    It seems that there are values like  $Data/Context/Value$  that can be used as fields in Notification, however I am enabel to find any parameter for a threshold.I  understand that Alert Description may include this data but this does not happen in all cases i.e. like in case above

    Alert Description:

    The threshold for the Memory\Pages\sec performance counter has been exceeded.

    Monday, April 15, 2013 12:23 PM
  • Agreed, there have been some oversights that have been committed by Microsoft that omitted the actual value in the alert (alertdesctiption$) that comes from some of the monitors in sealed management packs in their catalog.  Since those are 'sealed', then the only way to overcome this, that I have found, is build subscriptions to email notify those specific monitors' alerts through your own custom notification channel that adds those variables to the 'format' of the body of the notification utilizing the two values below - take note of the 'footnotes' it can be confusing, some of their 'default' "values" are actually 'number of samples' but the alert text implies the 'value' of the counter itself.

    *Value (Perf Counter Value):                  $Data/Context/Value$ 
    **Last Sampled Value                            $Data/Context/SampleValue$

    *Value will show the actual performance value for simple and avg monitors.  It will show number of samples for consecutive threshold monitors.   
    **Last Sampled Value works to show the last value evaluated in a consecutive sample value monitor


    B. Wright

    • Proposed as answer by B. Wright Monday, April 15, 2013 1:23 PM
    • Marked as answer by Yog Li Thursday, April 25, 2013 9:22 AM
    Monday, April 15, 2013 1:23 PM
  • The reason that the threshold is not included in the alert data is because threshold is an "input" (a $Config variable) in the the workflow, not an "output" (property bag). alert data can only include the property bag value that has been output from the workflow. Therefore it can't be configured as part of the alert description.

    One workaround is to re-write the monitor type to output the threshold. However, this is not an efficient way because if the data source module is shared between multiple rules and monitors, by taking threshold as an input in the data source means this data source cannot be shared by other perf collection rules and when used by other monitors, it would break cook down if an override is created to change the threshold.

    If it's too much trouble for you to actually go to SCOM console and check what's the configured threshold, your SCOM SME would have to write a custom module for you.

    Wednesday, May 1, 2013 10:48 AM