none
Exchange 2010 Failover Scipt Notification

    Question

  • I've noticed as I'm sure all of you have, that exchange 2010 will failover to the other node for many different reasons, but I was curious does anyone have a script that will alert you of the failover?

    I hate finding out from my end users that Droids won't work or exchange seems slow. 

     

    I would like an alert telling me that it's failed over.

    Monday, August 15, 2011 3:35 PM

Answers

  • Hi again rholland,

    Try the following script and see if it helps you.
    You are probably not interested in the Get-MailboxDatabaseCopyStatus part, but I think it is useful as even if a database is in the “primary” server it does not mean it or its copies are healthy.

     

    Function sendEmail ([String] $body)
    {
    	$MailMessage = New-Object System.Net.Mail.MailMessage
    	$MailMessage.From = "exchange@letsexchange.com"
    	$MailMessage.To.Add("nuno@letsexchange.com")
    	$MailMessage.Subject = "DAG Not Healthy!"
    	$MailMessage.Body = $body
    	$MailMessage.Priority = "High"
    
    	$SMTPClient = New-Object System.Net.Mail.SMTPClient
    	$SMTPClient.Host = "HTCAS1.letsexchange.com"
    	$SMTPClient.Send($MailMessage)
    
    	#Schtasks.exe /Delete /TN "MonitorDAG" /F
    }
    
    [Bool] $bolFailover = $False
    [String] $errMessage = $null
    
    Get-MailboxDatabase | Sort Name | ForEach {
    	$db = $_.Name
    	$xNow = $_.Server.Name
    	$dbOwn = $_.ActivationPreference | ? {$_.Value -eq 1}
    
    	If ($xNow -ne $dbOwn.Key)
    	{
    		$errMessage += "`n$db on $xNow should be on $($dbOwn.Key)!"
    		$bolFailover = $True
    	}
    }
    
    $errMessage += "`n`n"
    
    Get-MailboxServer | Get-MailboxDatabaseCopyStatus | ForEach {
    	If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
    	{
    		$errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)"
    		$bolFailover = $True
    	}
    }
    
    If ($bolFailover) { sendEmail $errMessage }
    


    http://LetsExchange.blogspot.com
    • Marked as answer by rholland Tuesday, August 16, 2011 4:14 PM
    Monday, August 15, 2011 10:23 PM

All replies

  • Look up the event id for when the DAG failed over? Setup a alert\page for that event id?

    Monday, August 15, 2011 3:49 PM
  • not so easy, it can be different each time.
    Monday, August 15, 2011 3:58 PM
  • Yes, not easy...that is why you use monitoring software. What do you have for monitoring right now/

    See

     the Monitoring HA and SR technet article

    http://technet.microsoft.com/en-us/library/dd351258.aspx

    Monday, August 15, 2011 4:02 PM
  • well that's just it we don't have any monitoring software other than an event log manager, which works great, but I need a script to tell me when it's failed over.

     

    I'm not wanting to implement some huge system to monitor everything, just a script to inform me of fail-overs.

    Monday, August 15, 2011 4:41 PM
  • The link I posted above as details about the script. scehdule it run all the time, beware it gathers the data in real time, so may add additional load.

    Exchange 2010 includes a script called CollectOverMetrics.ps1, which can be found in the Scripts folder. CollectOverMetrics.ps1 reads DAG member event logs to gather information about database operations (such as database mounts, moves, and failovers) over a specific time period.

    Monday, August 15, 2011 5:52 PM
  • Thanks, I'll take a look
    Monday, August 15, 2011 6:47 PM
  • I think this is at least the second time I've seen this question posted, without anything that looks like an acceptable solution.

    I've seen references to monitoring cluster failover using WMI event receivers, but the examples were all in VB.

    I think a Powwershell solution using WMI events might be in order.

    Working............


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
    Monday, August 15, 2011 6:47 PM
  • Well I'm glad to hear from you because I've found nothing on the net nor on technet that specifically tackles this problem.
    Monday, August 15, 2011 6:58 PM
  • Hi rholland,

     

    You can schedule a script like this:

     

    Get-MailboxServer | Get-StorageGroupCopyStatus | ForEach {

          If ($_.Status.ToString() -notmatch "Mounted" -or $_.ContentIndexState.ToString() -notmatch "Healthy")

          {

                Write-Host "$($_.Name) - $($_.Status) - $($_.ContentIndexState) - OK" -ForegroundColor Green

          }

          Else

          {

                Write-Host "$($_.Name) - $($_.Status) - $($_.ContentIndexState) - FAILED" -ForegroundColor Red

          }

    }

     

    To run every 5 minutes, for example. If it goes into the Else, you can tell it to send you an e-mail with the error:

     

    $body = ""

    $body += "`n********************************"

    $body += "`n*                              *"

    $body += "`n*   WARNING: DAG Not Healthy!  *"

    $body += "`n*                              *"

    $body += "`n********************************"

    $body += $Error # put the output of Get-StorageGroupCopyStatus, for example, here

     

    $MailMessage = New-Object System.Net.Mail.MailMessage

    $MailMessage.From = "exchange@company.com"

    $MailMessage.To.Add("rholland@company.com")

    $MailMessage.Subject = "DAG Not Healthy!"

    $MailMessage.Body = $body

    $MailMessage.Priority = "High"

     

    $SMTPClient = New-Object System.Net.Mail.SMTPClient

    $SMTPClient.Host = "mail.company.com"

    $SMTPClient.Send($MailMessage)

     

     

    You can also add the following to the end of the script to delete the schedule task, otherwise you will be receiving an e-mail non-stop: Schtasks.exe /Delete /TN "MonitorDAG" /F

     

    Other option is to use a script like this:

     

    Get-MailboxDatabase | Sort Name | ForEach {$db = $_.Name; $xNow = $_.Server.Name; $dbOwn = $_.ActivationPreference | ? {$_.Value -eq 1}; Write-Host $db "on" $xNow "should be on" $dbOwn.Key -NoNewline; If ($xNow -ne $dbOwn.Key) {Write-Host " WRONG" -ForegroundColor Red;} Else {Write-Host " OK" -ForegroundColor Green}}

     

    This one is a one-liner that tells you if each database is where it is supposed to be based on the ActivationPreference attribute. You can change it so that instead of printing WRONG, it will send you an e-mail!

     

    I will be writing such a script to do the exact same thing for me (maybe tomorrow if I have time). When I do, I’ll post it here.

     

    Hope this helps!


    http://LetsExchange.blogspot.com
    Monday, August 15, 2011 7:12 PM
  • Excellent, I appreciate it, and look forward to the new script.
    Monday, August 15, 2011 7:47 PM
  • The first line of the script doesn't work, "Get-MailboxServer | Get-StorageGroupCopyStatus | ForEach {", can't find the argument, Get-StorageGroupCopyStatus", this seems to be a 2007 field that wasn't brought over in exchange 2010.
    Monday, August 15, 2011 8:14 PM
  • Have you seen this script?

    CheckDatabaseRe​dundancy.ps1
    http://gallery.technet.microsoft.com/scriptcenter/8833b4db-8016-47e5-b747-951d28faafe7


    Martina Miskovic
    Monday, August 15, 2011 8:21 PM
  • not really want I'm looking for Martina, but thank you.  I'm more concerned about databases fail-over, not database consistency.
    Monday, August 15, 2011 8:29 PM
  • Sorry, it should be Get-MailboxDatabaseCopyStatus. Was thinking of Exchange 2007...
    http://LetsExchange.blogspot.com
    Monday, August 15, 2011 8:48 PM
  • Hi again rholland,

    Try the following script and see if it helps you.
    You are probably not interested in the Get-MailboxDatabaseCopyStatus part, but I think it is useful as even if a database is in the “primary” server it does not mean it or its copies are healthy.

     

    Function sendEmail ([String] $body)
    {
    	$MailMessage = New-Object System.Net.Mail.MailMessage
    	$MailMessage.From = "exchange@letsexchange.com"
    	$MailMessage.To.Add("nuno@letsexchange.com")
    	$MailMessage.Subject = "DAG Not Healthy!"
    	$MailMessage.Body = $body
    	$MailMessage.Priority = "High"
    
    	$SMTPClient = New-Object System.Net.Mail.SMTPClient
    	$SMTPClient.Host = "HTCAS1.letsexchange.com"
    	$SMTPClient.Send($MailMessage)
    
    	#Schtasks.exe /Delete /TN "MonitorDAG" /F
    }
    
    [Bool] $bolFailover = $False
    [String] $errMessage = $null
    
    Get-MailboxDatabase | Sort Name | ForEach {
    	$db = $_.Name
    	$xNow = $_.Server.Name
    	$dbOwn = $_.ActivationPreference | ? {$_.Value -eq 1}
    
    	If ($xNow -ne $dbOwn.Key)
    	{
    		$errMessage += "`n$db on $xNow should be on $($dbOwn.Key)!"
    		$bolFailover = $True
    	}
    }
    
    $errMessage += "`n`n"
    
    Get-MailboxServer | Get-MailboxDatabaseCopyStatus | ForEach {
    	If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
    	{
    		$errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)"
    		$bolFailover = $True
    	}
    }
    
    If ($bolFailover) { sendEmail $errMessage }
    


    http://LetsExchange.blogspot.com
    • Marked as answer by rholland Tuesday, August 16, 2011 4:14 PM
    Monday, August 15, 2011 10:23 PM
  • I've noticed as I'm sure all of you have, that exchange 2010 will failover to the other node for many different reasons, but I was curious does anyone have a script that will alert you of the failover?

    I hate finding out from my end users that Droids won't work or exchange seems slow. 

     

    I would like an alert telling me that it's failed over.


    Why does a failover cause Driods not to work or make Exchange seem slow?

     

    Monday, August 15, 2011 11:58 PM
    Moderator
  • for some reason the droids don't work with the failover, but iphones and other active-sync devices seem to work just fine.  and the reason exchange seems slow, our fail-over site is 20 plus miles away, so it's a bit slower.
    Tuesday, August 16, 2011 3:26 PM
  • Hi rholland,

    Have you tried the script? Is it what you were looking for?


    http://LetsExchange.blogspot.com
    Tuesday, August 16, 2011 3:47 PM
  • This one is from Nuno: http://msexchangeguru.com/2011/08/16/monitor-dags/
    Tuesday, August 16, 2011 4:09 PM
  • yes, it works awesome, thanks!
    Tuesday, August 16, 2011 4:13 PM
  • Anytime rholland! Glad to hear that!   :)


    http://LetsExchange.blogspot.com
    Tuesday, August 16, 2011 4:19 PM
  • You seem to have a good handle on these types of issues, do you happen to have a scripted that would check which OWA site is active, meaning the site hosting the databases allow you to connect directly to the mailbox via owa, but if they have failed over then you get a redirect page.  I'm looking for a scripted that could tell me if the site has been redirected.

     

    I understand there are two types of setups and we use both, one is redirect and the other is proxy.  In our DR scenario we use redirect.

    Tuesday, August 16, 2011 4:31 PM
  • Hi rholland,

     

    I think it might be possible. Let me just make sure I understand what you are looking for:

    1.        You have Exchange 2010 in two sites;

    2.        Those two sites are different AD sites;

    3.        In each site you have a CAS array;

    4.        Your users use a website like owa.company.com to login to OWA. When Exchange fails over to the other site and users try to login, they get the message asking them to use owa2.company.com instead, for example.

     

    Is all this correct?

    And you would like a script to detect when the users that use owa.company.com failover to the other site that uses owa2.company.com?

     

    Can’t you use proxying with integrated windows authentication on the secondary site? That way connections would get proxied and users would continue to use the same address.

    Do you have active users in both sites? Can you tell me more about your config?

     

    If you have multiple databases in one site, one of them might failover to the other site and only those users would get the redirect link. We would have to use a script similar to the one before to detect any changes for each database.


    http://LetsExchange.blogspot.com
    Tuesday, August 16, 2011 7:07 PM
  •  

    Is all this correct?    

    And you would like a script to detect when the users that use owa.company.com failover to the other site that uses owa2.company.com?

     

    YES THIS IS CORRECT

     

    Can’t you use proxying with integrated windows authentication on the secondary site? That way connections would get proxied and users would continue to use the same address.

    Do you have active users in both sites? Can you tell me more about your config?

     

    I COULD USE PROXY BUT THIS IS OUR DR SITE AND IT NEEDS TO BE INDEPENDENT FROM OUR MAIN SITE.  IN THE EVENT OUR MAIN SITE ISN'T AVAILABLE THEN MAIL FLOWS INTO OUR DR SITE AND USERS CAN ACCESS OWA, ACTIVESYNC, ETC.. FROM THE DR SITE.

     

    If you have multiple databases in one site, one of them might failover to the other site and only those users would get the redirect link. We would have to use a script similar to the one before to detect any changes for each database.

    i AGREE

     

    Since I have your first script I'm not certain I need the second, but it could prove useful.

     

    Tuesday, August 16, 2011 7:50 PM
  • Hi Rholland,

    I have updated the script to check the AD sites for the server that should be hosting the database (the one with an ActivationPreference of 1) and the AD site where the database is currently mounted.

    This way, if a database fails over to a server in a different AD site, you will know about it! If you just want to be informed in these situations and not when a database fails over to another server in the same site, simply remove the ELSE when I'm comparing the AD sites.

    Is this what you wanted? Hope it helps!

     

    Function sendEmail ([String] $body)
    {
    	$MailMessage = New-Object System.Net.Mail.MailMessage
    	$MailMessage.From = "exchange@letsexchange.com"
    	$MailMessage.To.Add("nuno@letsexchange.com")
    	$MailMessage.Subject = "DAG Not Healthy!"
    	$MailMessage.Body = $body
    	$MailMessage.Priority = "High"
    
    	$SMTPClient = New-Object System.Net.Mail.SMTPClient
    	$SMTPClient.Host = "HTCAS1.letsexchange.com"
    	$SMTPClient.Send($MailMessage)
    }
    
    
    
    Function getExchangeServerADSite ([String] $excServer)
    {
    	# We could use WMI to check for the domain, but I think this method is better
    	# Get-WmiObject Win32_NTDomain -ComputerName $excServer
    
    	$configNC=([ADSI]"LDAP://RootDse").configurationNamingContext
    	$search = new-object DirectoryServices.DirectorySearcher([ADSI]"LDAP://$configNC")
    	$search.Filter = "(&(objectClass=msExchExchangeServer)(name=$excServer))"
    	$search.PageSize = 1000
    	[Void] $search.PropertiesToLoad.Add("msExchServerSite")
    
    	Try {
    		$adSite = [String] ($search.FindOne()).Properties.Item("msExchServerSite")
    		Return ($adSite.Split(",")[0]).Substring(3)
    	} Catch {
    		Return $null
    	}
    }
    
    
    
    [Bool] $bolFailover = $False
    [String] $errMessage = $null
    
    Get-MailboxDatabase | Sort Name | ForEach {
    	$db = $_.Name
    	$curServer = $_.Server.Name
    	$ownServer = $_.ActivationPreference | ? {$_.Value -eq 1}
    
    	# Compare the server where the DB is currently active to the server where it should be
    	If ($curServer -ne $ownServer.Key)
    	{
    		# Compare the AD sites of both servers
    		$siteCur = getExchangeServerADSite $curServer
    		$siteOwn = getExchangeServerADSite $ownServer.Key
    		
    		If ($siteCur -ne $null -and $siteOwn -ne $null -and $siteCur -ne $siteOwn)
    		{
    			$errMessage += "`n$db on $curServer should be on $($ownServer.Key) (DIFFERENT AD SITE: $siteCur)!"	
    		}
    		Else
    		{
    			$errMessage += "`n$db on $curServer should be on $($ownServer.Key)!"
    		}
    
    		$bolFailover = $True
    	}
    }
    
    $errMessage += "`n`n"
    
    Get-MailboxServer | Get-MailboxDatabaseCopyStatus | ForEach {
    	If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
    	{
    		$errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)"
    		$bolFailover = $True
    	}
    }
    
    If ($bolFailover)
    {
    	sendEmail $errMessage
    	#Schtasks.exe /Delete /TN "MonitorDAG" /F
    }
    

     


    http://LetsExchange.blogspot.com
    Thursday, August 18, 2011 10:19 AM
  • this is awesome, thanks so much!
    Thursday, August 18, 2011 12:33 PM
  • Anytime! Hope it works for what you want   :)

    Once you test it, let us know how it went please.


    http://LetsExchange.blogspot.com
    Thursday, August 18, 2011 1:00 PM
  • Hello, if Nuno Mota is still here I have a question about the posted script (the one marked as the answer).  We have need for the exact same script at our company but I am running into the following error when I run the script from the Exchange Management Shell:

    "Exception calling "Send" with "1" argument(s): "Failure sending mail."
    At C:\Scripts\Failover.ps1:14 char:18
    +  $SMTPClient.Send <<<< ($MailMessage)
      + CategoryInfo  : NotSpecified : (:) [], MethodInvocationException
      + FullyQualifiedErrorId  : DotNetMethodException"

    How would I go about fixing this?

    Thanks.

    (edited:  Resolved the errors I posted before; still need to resolve this error)

    Wednesday, August 31, 2011 8:21 PM
  • Ryanb,

    here you go:  I had similar issue, so I removed the Sendmail function and tweeked out the code a bit.

    * Modified Alert message to be a little clearer

    * Added a line to automatically move databases to the preferred server:

    We are still on 2010RTM, waiting for SP2.... This is working great at our office, we have 24 database servers with 46 database copies, so it is alot to keep track of.  The primary site has 16 Database servers, 8 of which are running on Tier2 storage, the other 8 are on Tier3 storage, with 8 servers at a remote site running on Tier3 storage.  Of course you know what happens if a DB moves to the remote site... causes havic with OWA, hub transport and CAS.  The remote site will eventually be a DR site, I designing the OWA, CAS, Hub setup.  We don't use the Edge we have Ironports and Barracudas as our SMTP filtering.

    Here's the code that I run every 30 minutes to check DAG status:

    ++++++++++++++++++++++++++++++++

    #The following four lines is all you need to change is the text in quotes
    [String] $MySMTPServer = "smtp.localdomain.dom"
    [String] $MyAlertsfrom = "ExchangeAlerts@yourdomain.com"
    [String] $MyAlertsFailTo = "yourname@yourdomain.com, yourboss@yourdomain.com, yourgroup@yourdomain.com"
    [String] $MyAlertsGoodTo = "yourname@yourdomain.com"

    Function getExchangeServerADSite ([String] $excServer)
    {
     # We could use WMI to check for the domain, but I think this method is better
     # Get-WmiObject Win32_NTDomain -ComputerName $excServer

     $configNC=([ADSI]"LDAP://RootDse").configurationNamingContext
     $search = new-object DirectoryServices.DirectorySearcher([ADSI]"LDAP://$configNC")
     $search.Filter = "(&(objectClass=msExchExchangeServer)(name=$excServer))"
     $search.PageSize = 1000
     [Void] $search.PropertiesToLoad.Add("msExchServerSite")

     Try {
      $adSite = [String] ($search.FindOne()).Properties.Item("msExchServerSite")
      Return ($adSite.Split(",")[0]).Substring(3)
     } Catch {
      Return $null
     }
    }


    [Bool] $DAGNormal = $False
    [Bool] $DAGIndNorm = $False
    [String] $errMessage = "  -- DAG Current Mount State --`n"

    Get-MailboxDatabase | Sort Name | ForEach {
     $db = $_.Name
     $curServer = $_.Server.Name
     $ownServer = $_.ActivationPreference | ? {$_.Value -eq 1}

     # Compare the server where the DB is currently active to the server where it should be
     If ($curServer -ne $ownServer.Key)
     {
      # Compare the AD sites of both servers
      $siteCur = getExchangeServerADSite $curServer
      $siteOwn = getExchangeServerADSite $ownServer.Key
      
      If ($siteCur -ne $null -and $siteOwn -ne $null -and $siteCur -ne $siteOwn)
      {
       $errMessage += "`n$db on $curServer should be on $($ownServer.Key) (DIFFERENT AD SITE: $siteCur)!" 
      }
      Else
      {
       $errMessage += "`n$db on $curServer should be on $($ownServer.Key)!`r"
      }

      $DAGNormal = $True
     }
    }
    If ($DAGNormal)
    {
     $errMessage += "`n`n   ** ACTION: Running Script - DB Move to Preferred Server`n`n`n"
     #The next line check all databases moves database back to the preferred server
     Get-MailboxDatabase | Sort Name | FOREACH {$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};If ( $xNow -ne $dbOwn.Key){Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False }}
    }

    $errMessage += "  -- Copy Status - Index State --`n"

    Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | ForEach {
     If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
     {
      $errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)!"
      $DAGNormal = $True
      $DAGIndNorm = $True
     }
    }
    If ($DAGIndNorm)
    {
     $errMessage += "`n`n ** ACTION: Please manually verify Copy - Index State `n`n"
    }

    If ($DAGNormal)
    {
     Send-MailMessage -From "$MyAlertsfrom" -To "$MyAlertsFailTo" -Subject "DAG NOT Healthy!" -Body $errMessage -Priority High -SMTPserver "$MySMTPServer" -DeliveryNotificationOption onFailure
     #end script here
     exit
    }
    #If all is good send a message to yourself just as confirmation the script is running
    Send-MailMessage -From "$MyAlertsfrom" -To "$MyAlertsGoodTo" -Subject "DAG IS Normal!" -Body "Your Exchange Enviroment is Normal" -Priority High -SMTPserver "$MySMTPServer" -DeliveryNotificationOption onFailure
    ++++++++++++++++++++++++++++++++

     


    Tony
    Friday, November 18, 2011 2:30 AM
  • Hi Ryanb,

    When I first wrote that script, I was using the "old" System.Net.Mail.SMTPClient method to send e-mails. With Exchange 2010, like Tony suggested, you it's better to use the Send-MailMessage 2010 cmdlet.

    If you check my latest script (last updated in September) in the Microsoft Script Center Repository, I have changed it to use the Send-MailMessage cmdlet.

    However, it looks like Tony has improved my script, so you might as well go ahead and use the one he posted.


    http://LetsExchange.blogspot.com
    Friday, November 18, 2011 3:55 PM
  • However, it looks like Tony has improved my script, so you might as well go ahead and use the one he posted.


    http://LetsExchange.blogspot.com

    Nuno,

    I'm only adding branches here, your expertise built the tree...  Thank you for the compliment.

    Tony


    Tony
    Friday, November 18, 2011 10:01 PM
  • I'm using Task Scheduler to send myself an email upon any events with a source of FailoverClustering being logged on the mailbox servers' System log.
    Thursday, January 05, 2012 2:15 PM
  • Hello again,

    I had been running the script posted by Nuno for a little while, but at some point it started returning false-positives and I stopped using it.  This was quite a while ago so I don't remember what might have changed at the time to cause this.  Anyway I'm going back to re-implement the script now and running into some problems.  Using Nuno's script I get a DAG not healthy alert with this text:

    Local_DB1\Server1 - Status: Unknown - Index: Unknown
    Remote_DB1\Server1 - Status: Unknown - Index: Unknown

    (Those are not the actual DB/Server names).  The problem here is that in the EMC the databases show "Healthy"/"Mounted" copy statuses like they should.  I don't know why it's returning an Unknown status.

    If on the other hand I use Tony's script then it doesn't return anything at all.  I am guessing with this one that maybe I didn't change something in the quoted text that needs to be changed to match our organization. 

    Here is the version of Nuno's script that I am using (the script is being run on the exchange server itself, and the bolded parts are values that are different in the actual script - everything else is verbatim):

    --------------------------------------

    Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010

    Function sendEmail ([String] $body)
    {
        $MailMessage = New-Object System.Net.Mail.MailMessage
        $MailMessage.From = "localserver@company.com"
        $MailMessage.To.Add("ryanb@company.com")
        $MailMessage.Subject = "DAG Not Healthy!"
        $MailMessage.Body = $body
        $MailMessage.Priority = "High"

        $SMTPClient = New-Object System.Net.Mail.SMTPClient
        $SMTPClient.Host = "localhost"
        $SMTPClient.Send($MailMessage)

        Schtasks.exe /Delete /TN "Monitor DAG" /F
    }

    [Bool] $bolFailover = $False
    [String] $errMessage = $null

    Get-MailboxDatabase | Sort Name | ForEach {
        $db = $_.Name
        $xNow = $_.Server.Name
        $dbOwn = $_.ActivationPreference | ? {$_.Value -eq 1}

        If ($xNow -ne $dbOwn.Key)
        {
            $errMessage += "`n$db on $xNow should be on $($dbOwn.Key)!"
            $bolFailover = $True
        }
    }

    $errMessage += "`n`n"

    Get-MailboxServer | Get-MailboxDatabaseCopyStatus | ForEach {
        If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
        {
            $errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)"
            $bolFailover = $True
        }
    }

    If ($bolFailover) { sendEmail $errMessage }
    Else { sendEmail "DAG Ok" }

    -----------------------------------------------------------------------

    And here is the version of Tony's script I tried to use:

    -----------------------------------------------------------------------

    Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010

    #The following four lines is all you need to change is the text in quotes
    [String] $MySMTPServer = "localserver"
    [String] $MyAlertsfrom = "localserver@company.com"
    [String] $MyAlertsFailTo = "ryanb@company.com"
    [String] $MyAlertsGoodTo = "ryanb@company.com"

    Function getExchangeServerADSite ([String] $excServer)
    {
     $configNC=([ADSI]"LDAP://RootDse").configurationNamingContext
     $search = new-object DirectoryServices.DirectorySearcher([ADSI]"LDAP://$configNC")
     $search.Filter = "(&(objectClass=msExchExchangeServer)(name=$excServer))"
     $search.PageSize = 1000
     [Void] $search.PropertiesToLoad.Add("msExchServerSite")

     Try {
      $adSite = [String] ($search.FindOne()).Properties.Item("msExchServerSite")
      Return ($adSite.Split(",")[0]).Substring(3)
     } Catch {
      Return $null
     }
    }


    [Bool] $DAGNormal = $False
    [Bool] $DAGIndNorm = $False
    [String] $errMessage = "  -- DAG Current Mount State --`n"

    Get-MailboxDatabase | Sort Name | ForEach {
     $db = $_.Name
     $curServer = $_.Server.Name
     $ownServer = $_.ActivationPreference | ? {$_.Value -eq 1}

     # Compare the server where the DB is currently active to the server where it should be
     If ($curServer -ne $ownServer.Key)
     {
      # Compare the AD sites of both servers
      $siteCur = getExchangeServerADSite $curServer
      $siteOwn = getExchangeServerADSite $ownServer.Key
     
      If ($siteCur -ne $null -and $siteOwn -ne $null -and $siteCur -ne $siteOwn)
      {
       $errMessage += "`n$db on $curServer should be on $($ownServer.Key) (DIFFERENT AD SITE: $siteCur)!"
      }
      Else
      {
       $errMessage += "`n$db on $curServer should be on $($ownServer.Key)!`r"
      }

      $DAGNormal = $True
     }
    }
    If ($DAGNormal)
    {
     $errMessage += "`n`n   ** ACTION: Running Script - DB Move to Preferred Server`n`n`n"
     #The next line check all databases moves database back to the preferred server
     Get-MailboxDatabase | Sort Name | FOREACH {$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};If ( $xNow -ne $dbOwn.Key){Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False }}
    }

    $errMessage += "  -- Copy Status - Index State --`n"

    Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | ForEach {
     If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
     {
      $errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)!"
      $DAGNormal = $True
      $DAGIndNorm = $True
     }
    }
    If ($DAGIndNorm)
    {
     $errMessage += "`n`n ** ACTION: Please manually verify Copy - Index State `n`n"
    }

    If ($DAGNormal)
    {
     Send-MailMessage -From "$MyAlertsfrom" -To "$MyAlertsFailTo" -Subject "DAG NOT Healthy!" -Body $errMessage -Priority High -SMTPserver "$MySMTPServer" -DeliveryNotificationOption onFailure
     #end script here
     exit
    }
    #If all is good send a message to yourself just as confirmation the script is running
    Send-MailMessage -From "$MyAlertsfrom" -To "$MyAlertsGoodTo" -Subject "DAG IS Normal!" -Body "Your Exchange Enviroment is Normal" -Priority High -SMTPserver "$MySMTPServer" -DeliveryNotificationOption onFailure
    ++++++++++++++++++++++++++++++++

    ------------------------------------------------------------

    Can anyone tell me what I might be missing in either of these scripts?  Tony's script that automatically fixes the DAG would be the preferred one to use, but Nuno's original script would suffice too.  Actually is there a way to separate automatically fixing the DAG into its own script? We may want to run that at a different frequency than the script that checks the DAG status.

    Thanks.

    Tuesday, February 21, 2012 7:27 PM
  • @PWC-Ryanb

    The EMC is sometimes unreliable, it is best to use the EMS.

    run this command from EMS: That will report all your databases and current copy status.

    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus

    My script will send and email weather bad or good, if you are not getting the Good alert, the script is failing somewhere.

    Make sure your code is exactly as below, re-copy if needed.  Of course this has only been tested on Exchange 2010 RTM, not sure if it works in 2007 or 2010 SP1/SP2.

    #The following four lines is all you need to change
    [String] $MySMTPServer = "server.domain.local"
    [String] $MyAlertsfrom = "ExchangeAlerts@yourcompany.com"
    [String] $MyAlertsFailTo = "support1@yourcompany.com, support2@yourcompany.com, support3@yourcompany.com"
    [String] $MyAlertsGoodTo = "support1@yourcompany.com, support2@yourcompany.com"
    
    Function getExchangeServerADSite ([String] $excServer)
    {
    	# We could use WMI to check for the domain, but I think this method is better
    	# Get-WmiObject Win32_NTDomain -ComputerName $excServer
    
    	$configNC=([ADSI]"LDAP://RootDse").configurationNamingContext
    	$search = new-object DirectoryServices.DirectorySearcher([ADSI]"LDAP://$configNC")
    	$search.Filter = "(&(objectClass=msExchExchangeServer)(name=$excServer))"
    	$search.PageSize = 1000
    	[Void] $search.PropertiesToLoad.Add("msExchServerSite")
    
    	Try {
    		$adSite = [String] ($search.FindOne()).Properties.Item("msExchServerSite")
    		Return ($adSite.Split(",")[0]).Substring(3)
    	} Catch {
    		Return $null
    	}
    }
    
    
    [Bool] $DAGNormal = $False
    [Bool] $DAGIndNorm = $False
    [String] $errMessage = "  -- DAG Current Mount State --`n"
    
    Get-MailboxDatabase | Sort Name | ForEach {
    	$db = $_.Name
    	$curServer = $_.Server.Name
    	$ownServer = $_.ActivationPreference | ? {$_.Value -eq 1}
    
    	# Compare the server where the DB is currently active to the server where it should be
    	If ($curServer -ne $ownServer.Key)
    	{
    		# Compare the AD sites of both servers
    		$siteCur = getExchangeServerADSite $curServer
    		$siteOwn = getExchangeServerADSite $ownServer.Key
    		
    		If ($siteCur -ne $null -and $siteOwn -ne $null -and $siteCur -ne $siteOwn)
    		{
    			$errMessage += "`n$db on $curServer should be on $($ownServer.Key) (DIFFERENT AD SITE: $siteCur)!"	
    		}
    		Else
    		{
    			$errMessage += "`n$db on $curServer should be on $($ownServer.Key)!`r"
    		}
    
    		$DAGNormal = $True
    	}
    }
    
    If ($DAGNormal)
    {
    	$errMessage += "`n`n   ** ACTION: Running Script - DB Move to Preferred Server`n`n`n"
    	#The next line check all databases moves database back to the preferred server
    	Get-MailboxDatabase | Sort Name | FOREACH {$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};If ( $xNow -ne $dbOwn.Key){Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False }}
    }
    
    $errMessage += "  -- Copy Status - Index State --`n"
    
    Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | ForEach {
    	If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
    	{
    		$errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)!"
    		$DAGNormal = $True
    		$DAGIndNorm = $True
    	}
    }
    If ($DAGIndNorm)
    {
    	$errMessage += "`n`n ** ACTION: Please manually verify Copy - Index State `n`n"
    }
    
    If ($DAGNormal)
    {
    	Send-MailMessage -From "$MyAlertsfrom" -To "$MyAlertsFailTo" -Subject "DAG NOT Healthy!" -Body $errMessage -Priority High -SMTPserver "$MySMTPServer" -DeliveryNotificationOption onFailure
    	#end script here
    	exit
    }
    #If all is good send a message to yourself just as confirmation the script is running
    Send-MailMessage -From "$MyAlertsfrom" -To "$MyAlertsGoodTo" -Subject "DAG IS Normal!" -Body "Leviton Exchange Health is Normal" -Priority High -SMTPserver "$MySMTPServer" -DeliveryNotificationOption onFailure
    


    Tony

    Tuesday, February 21, 2012 8:45 PM
  • Thanks a lot for the speedy reply Tony.  Looking at the 2 scripts side-by-side I can't see where I went wrong but it is working now (well at the very least it is properly reporting the DAG is normal, we'll have to wait for a problem to arise to verify the other half, but I have faith in your script :)).  For the record this is on Exchange 2010 as well.  Also I checked with the EMS and the database copy statuses returned healthy/mounted as they should.

    Edit: Also I think I figured out how to separate the automatic correction of the DAG into a separate script (not too difficult really) so you can ignore my last question in my previous post.  Thanks again!

    • Edited by PWC-Ryanb Tuesday, February 21, 2012 9:12 PM
    Tuesday, February 21, 2012 9:05 PM
  • @PWC-Ryanb

    ------------------------------------------------------------

    "Actually is there a way to separate automatically fixing the DAG into its own script?"

     Thanks.

    Ryan, run this script from EMS, will check the databases and move to preferred.  Outputing the status of each database.

    Get-MailboxDatabase | Sort Name | FOREACH {
    	$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};
    		$quoteon=" On ";$quotesb=" Should be on ";If ( $xNow -ne $dbOwn.Key.Name){$stat=" MOVING..."; }ELSE{$stat=" OK" };
    		$OutP=$db+$quoteon+$xNow+$quotesb+$dbOwn.Key.Name+$stat; write-output $OutP; 
    		If ( $xNow -ne $dbOwn.Key)
    			{Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False }
    }


    Tony

    Tuesday, February 21, 2012 9:23 PM
  • Updated script to correct sending to multiple recipients - had to use array, otherwise would only send to first recipient:

    Run this script from a scheduled task and it will automatically check DAG status and move to preferred server if needed.  The script will not correct errors with index copies (it can be done, but is risky), but will identify the problem indexes in the email recommending you check them manually.

    #The following four lines is all you need to change
    [String] $MySMTPServer = "server.domain.local"
    [String] $MyAlertsfrom = "ExchangeAlerts@yourcompany.com"
    [ARRAY] $MyAlertsFailTo = "support1@yourcompany.com", "support2@yourcompany.com", "yourboss@yourcompany.com"
    [ARRAY] $MyAlertsGoodTo = "support1@yourcompany.com", "support2@yourcompany.com"
    
    Function getExchangeServerADSite ([String] $excServer)
    {
    	# We could use WMI to check for the domain, but I think this method is better
    	# Get-WmiObject Win32_NTDomain -ComputerName $excServer
    
    	$configNC=([ADSI]"LDAP://RootDse").configurationNamingContext
    	$search = new-object DirectoryServices.DirectorySearcher([ADSI]"LDAP://$configNC")
    	$search.Filter = "(&(objectClass=msExchExchangeServer)(name=$excServer))"
    	$search.PageSize = 1000
    	[Void] $search.PropertiesToLoad.Add("msExchServerSite")
    
    	Try {
    		$adSite = [String] ($search.FindOne()).Properties.Item("msExchServerSite")
    		Return ($adSite.Split(",")[0]).Substring(3)
    	} Catch {
    		Return $null
    	}
    }
    
    
    [Bool] $DAGNormal = $False
    [Bool] $DAGIndNorm = $False
    [String] $errMessage = "  -- DAG Current Mount State --`n"
    
    Get-MailboxDatabase | Sort Name | ForEach {
    	$db = $_.Name
    	$curServer = $_.Server.Name
    	$ownServer = $_.ActivationPreference | ? {$_.Value -eq 1}
    
    	# Compare the server where the DB is currently active to the server where it should be
    	If ($curServer -ne $ownServer.Key)
    	{
    		# Compare the AD sites of both servers
    		$siteCur = getExchangeServerADSite $curServer
    		$siteOwn = getExchangeServerADSite $ownServer.Key
    		
    		If ($siteCur -ne $null -and $siteOwn -ne $null -and $siteCur -ne $siteOwn)
    		{
    			$errMessage += "`n$db on $curServer should be on $($ownServer.Key) (DIFFERENT AD SITE: $siteCur)!"	
    		}
    		Else
    		{
    			$errMessage += "`n$db on $curServer should be on $($ownServer.Key)!`r"
    		}
    
    		$DAGNormal = $True
    	}
    }
    
    If ($DAGNormal)
    {
    	$errMessage += "`n`n   ** ACTION: Running Script - DB Move to Preferred Server`n`n`n"
    	#The next line check all databases moves database back to the preferred server
    	Get-MailboxDatabase | Sort Name | FOREACH {$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};If ( $xNow -ne $dbOwn.Key){Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False }}
    }
    
    $errMessage += "  -- Copy Status - Index State --`n"
    
    Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | ForEach {
    	If ($_.Status -notmatch "Mounted" -and $_.Status -notmatch "Healthy" -or $_.ContentIndexState -notmatch "Healthy")
    	{
    		$errMessage += "`n$($_.Name) - Status: $($_.Status) - Index: $($_.ContentIndexState)!"
    		$DAGNormal = $True
    		$DAGIndNorm = $True
    	}
    }
    If ($DAGIndNorm)
    {
    	$errMessage += "`n`n ** ACTION: Please manually verify Copy - Index State `n`n"
    }
    
    If ($DAGNormal)
    {
    	Send-MailMessage -From $MyAlertsfrom -To $MyAlertsFailTo -Subject "DAG NOT Healthy!" -Body $errMessage -Priority High -SMTPserver $MySMTPServer -DeliveryNotificationOption onFailure
    	#end script here
    	exit
    }
    #If all is good send a message to yourself just as confirmation the script is running
    Send-MailMessage -From $MyAlertsfrom -To $MyAlertsGoodTo -Subject "DAG IS Normal!" -Body "Exchange Health is Normal" -Priority High -SMTPserver $MySMTPServer -DeliveryNotificationOption onFailure
    

    Here is the auto fix portion if you want to run manually from EMS:

    Get-MailboxDatabase | Sort Name | FOREACH {
    	$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};
    		$quoteon=" On ";$quotesb=" Should be on ";If ( $xNow -ne $dbOwn.Key.Name){$stat=" MOVING..."; }ELSE{$stat=" OK" };
    		$OutP=$db+$quoteon+$xNow+$quotesb+$dbOwn.Key.Name+$stat; write-output $OutP; 
    		If ( $xNow -ne $dbOwn.Key)
    			{Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False }
    }


    Tony

    • Proposed as answer by PWC-Ryanb Monday, February 27, 2012 7:21 PM
    Wednesday, February 22, 2012 6:25 PM
  • I can't really understand what is happening now.  On Friday we tested to make sure the script was working.  I had it scheduled to run every hour all day and didn't receive a single e-mail from it (this is good, there was no failover so I shouldn't receive an e-mail).  Then after 5PM when everyone had gone home we tested the failover detection by manually switching our active database to another server.  It detected the failover and notified us properly, so assuming all was well we left it enabled.

    1 hour after that (and every hour until we disabled it) we got:

    ---------------------------------------------------
    -- DAG Current Mount State --
    -- Copy Status - Index State --

    Site1_DB1\Server1 - Status: Unknown - Index: Unknown!
    Site2_DB1\Server1 - Status: Unknown - Index: Unknown!

     ** ACTION: Please manually verify Copy - Index State
    ----------------------------------------------------

    Now I decided to manually test it.  From the EMS I ran "Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus" and the copy status for each database returned healthy/mounted as it should.  Then I forced the script to run through Task Scheduler and got the same Unknown status message above (keep in mind this was running every hour on Friday with no issue until after the failover, but wouldn't work again even after we changed everything back).

    Perhaps there's a problem in how I am calling the script from Task Scheduler?  Currently I have the Action set to:
    Action: Start a program
    Program/script: powershell.exe
    Arguments: "& 'C:\Scripts\Failover.ps1'"

    Thanks for all the help you've provided so far.  Oh and the auto-fix portion worked perfectly when we tested it so thanks for that too.

    Edit:  After a little more testing I'm more convinced it is a problem with the Task Scheduler call.  I verified copy status in the EMS again and everything was fine.  So I ran the script from EMS just by typing
    powershell.exe "& 'C:\Scripts\Failover.ps1'"
    And I got a notification that the DAG is normal (expected result).  I immediately tried forcing the script to run in Task Scheduler and got the unknown copy status message referenced above, and it should be calling the script with the exact same arguments.

    • Edited by PWC-Ryanb Monday, February 27, 2012 2:58 PM
    Monday, February 27, 2012 2:52 PM
  • Perhaps there's a problem in how I am calling the script from Task Scheduler?  Currently I have the Action set to:
    Action: Start a program
    Program/script: powershell.exe
    Arguments: "& 'C:\Scripts\Failover.ps1'"

    Thanks for all the help you've provided so far.  Oh and the auto-fix portion worked perfectly when we tested it so thanks for that too.

    Edit:  After a little more testing I'm more convinced it is a problem with the Task Scheduler call.  I verified copy status in the EMS again and everything was fine.  So I ran the script from EMS just by typing
    powershell.exe "& 'C:\Scripts\Failover.ps1'"
    And I got a notification that the DAG is normal (expected result).  I immediately tried forcing the script to run in Task Scheduler and got the unknown copy status message referenced above, and it should be calling the script with the exact same arguments.

    You need to start Powershell with the exchange tools, otherwise it will fail:  I had many problems with that too.  Now I just execute a cmd file that contains the following script startup:

    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command ". 'E:\Your-Exchange-System-folder\Bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto; E:\Your-Scripts-Folder\DAGMonitor.ps1"


    Tony

    • Proposed as answer by PWC-Ryanb Monday, February 27, 2012 7:21 PM
    Monday, February 27, 2012 4:40 PM
  • Everything seems to be in working order now.  You've been an immense help on this; thank you!
    Monday, February 27, 2012 7:30 PM
  • Thanks for the script, this has helped me a lot.

    I have SCOM monitoring Exchange however trying to get it to send me valuable information regarding the activation preference of my DB's is like trying to get blood from a stone.  Your script is rudimentary yet it does exactly what I need, it just goes to show, when you over complicate something you will probably not achieve what you set out to do (SCOM)

    Friday, March 16, 2012 10:25 AM
  • Nuno...Tony...you guys are awesome!

    Thanks so much for this script and tweaks.  It's a lifesaver.

    Jim

    Tuesday, May 08, 2012 6:47 PM
  • Aggree that this is exactly what I was looking for!!

    Thanks to all.

    Friday, September 07, 2012 3:26 PM