none
Agent Managed Server Weekly Rebuild: How to Stop Alerts ? RRS feed

  • Question

  • Hi All, 

    Wanted to put this to the community as im not coming up with much from my searches. 

    We have a number of agent managed servers which are completely rebuilt on a weekly basis and i am looking for a way to prevent the heartbeat and failed to connect from generating an alert at the time of rebuild.

    if i put it into Mmode as soon as the server is trashed the agent goes to not monitored and the subsequent heartbeat failures are produced. 

    any thoughts would be much appreciated 



    Richard Scott


    • Edited by frikkieg Tuesday, November 19, 2019 3:13 PM
    Tuesday, November 19, 2019 3:13 PM

All replies

  • Hi,

    Just out of curiosity, why do these servers even need to be monitored if they’re being rebuilt weekly?

    It sounds to me like this is some kind of ”staging environment”, when the servers are rebuilt, do they have the same names & IP addresses?

    I would not monitor these servers in these conditions, you’ll make it very difficult for yourself I’m afraid.

    Best regards,
    Leon

     


    Blog: https://thesystemcenterblog.com LinkedIn:


    • Edited by Leon Laude Tuesday, November 19, 2019 3:44 PM
    Tuesday, November 19, 2019 3:28 PM
  • If these servers are rebuilt as frequent as every week i would probably not monitor them with SCOM.

    Even if you had the machines in maintenance mode, once you have rebuilt it it may not necessarily pick up the same record in SCOM could even end up an orphaned agent unless you delete it from the console and push the agent back on it


    Website: www.walshamsolutions.com Technical Blog: https://www.walshamsolutions.com/technical-blog Personal Blog: https://www.walshamsolutions.com/personal-blog Twitter: Dwalshampro

    Tuesday, November 19, 2019 3:34 PM
  • these servers are business critical web servers that need to be rebuilt rather than patched for extenuating circumstances. i need to monitor them for any normal 'errors' on a day to day basis. not monitoring them is not an option 

    so far every time the 'test' server has been rebuilt scom has successfully reconnected the agent without any issues ,   if the agent did not reconnect it is simple on a monday AM to repair the scom agent and get it back into SCOM so i am not to concerned from that point of view.

    I have experience with other monitoring tools when a similar situation and used to just uninstall the agent and reinstall after but SCOM still shouts at me when the agent is uninstalled

    ideally i suppose i need the ability to delete the agent from scom but the rebuild commands are executed locally and i cannot introduce the SCOM Powershell Module .

    i am also concerned to the status of performance data to an agent that has been deleted does this remove itself from the DW or would i still be able to query it  ?


    Richard Scott

    Tuesday, November 19, 2019 3:54 PM
  • In general data is purged from the SCOM database after 2-3 days of being marked deleted. We do not delete these immediately, because that is an expensive operation and best reserved for when grooming runs at night.

    This lets the discovered data remain, if you bring the agent back in within a day or two.

    You can automate deleting/repairing the SCOM agents, but I cannot guarantee that there will not be any raised alerts.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, November 19, 2019 4:15 PM
  • Maybe if you use something like Orchestrator and have a runbook that checks certain errors on the temp web servers you have and that could then create an alert in SCOM to say its down etc

    You can put those machines in maintenance mode before rebuilding too, but you'll still have to reinstall the SCOM agent and the historical data may still get discarded.


    Website: www.walshamsolutions.com Technical Blog: https://www.walshamsolutions.com/technical-blog Personal Blog: https://www.walshamsolutions.com/personal-blog Twitter: Dwalshampro

    Tuesday, November 19, 2019 6:09 PM
  • We have quite a number of Citrix servers which are rebuilt on a nightly or weekly basis. SCOM never has an issue automatically monitoring them again after the rebuild. Below is the script we use to maintenance mode them. It's set to run as a scheduled task on a management server 5 minutes or so before the rebuild starts.

    Create a group in SCOM containing whatever you want to go into maintenance mode. Either manually or dynamically if you can group them by name or some such. Put that in $GroupDisplayName. Make sure $DurationInMin covers however long the servers will be offline.

    $GroupDisplayName = "Weekly Maintenance Servers"
    $Reason = "PlannedOther"
    $Comment = "Weekly maintenance"
    $DurationInMin = 30
     
    Import-Module OperationsManager
    New-SCOMManagementGroupConnection -ComputerName localhost
     
    ForEach ($Member in (Get-ScomGroup -DisplayName $GroupDisplayName).GetRelatedMonitoringObjects()) {
        If ($Member.InMaintenanceMode -eq $false) {
            $Member.ScheduleMaintenanceMode([datetime]::Now.ToUniversalTime(),([datetime]::Now.ToUniversalTime()).AddMinutes($DurationInMin),"$Reason","$Comment" ,"Recursive")
        }
    }

    You could list objects in the script rather than using a group, but with a group modifying it is easy using the SCOM console, and you don't have to worry about classes in the script.

    Hope this helps.

    Steve

    Wednesday, November 20, 2019 4:22 AM
  • To answer the original post actual  question : that's probably because you are not putting in maintenance mode the health service along with the computer. 

    Read here : https://support.microsoft.com/en-gb/help/942866/a-computer-agent-unexpectedly-generates-heartbeat-alerts-after-you-put

    Wednesday, November 20, 2019 7:57 AM
  • Also on a side note, SquaredUp initiated an interesting tool quite a while a go : Bootstrap Discovery Tool .

    It allows to pre-discover things in scenarios where servers are destroyed/rebuilt on a regular basis (and agent is deleted from SCOM), so monitoring is effective almost immediatly instead of waiting hours for every object to be discovered/monitored.

    Wednesday, November 20, 2019 8:04 AM
  • i am also concerned to the status of performance data to an agent that has been deleted does this remove itself from the DW or would i still be able to query it  ?


    Richard Scott

    Hi Richard,

    the answer is No, you cannot find any data in the Datawarehosue as soon as you delete an agent from SCOM.

    Regards,


    (Please take a moment to "Vote as Helpful" and/or "Mark as Answer" where applicable. This helps the community, keeps the forums tidy, and recognizes useful contributions. Thanks!) Blog: https://blog.pohn.ch/ Twitter: @StoyanChalakov

    Wednesday, November 20, 2019 11:30 AM
    Moderator