none
Exchange 2010 Errors Adding or removing Servers from DAG RRS feed

  • Question

  • hey everyone. I have a running EXCHANGE SP2 Server, and recovered my second  failed EXCHANGE 2010 (SP2 Install) server last night from AD.  using the Technet Recovery DAG Article. When i try to rejoin the second server to the DAG i get the 1st below error.  i then tried to just remove the existing server from the DAG and just recreate a new one with both servers, and recive the second below error.  At this point id like the best way to remove both KLDEX02 and exchange-2010 DAG and there networks and recreate from scratch or if anyone has any better ideas i'm all ears i use the EMC whenever i can not fully up to speed with the Shell so if commands are needed syntax would be great Thanks everyone!

    #1

     The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2012-01-09_14-27-00.593_add-databaseavailabiltygroupserver.log".
    [2012-01-09T14:27:01] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Node exchange-2010 is already joined to a cluster. ---> Microsoft.Exchange.Cluster.Replay.AmClusterNodeJoinedException: An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Node exchange-2010 is already joined to a cluster..

     

    #2

    [2012-01-07T01:39:02] The preceding log entry comes from a different process running on computer 'KLDEX02.KLDLABS.COM'. END
    [2012-01-07T01:39:02] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2012-01-07_01-39-02.212_add-databaseavailabiltygroupserver.log".
    [2012-01-07T01:39:02] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Node exchange-2010 is already joined to a cluster. ---> Microsoft.Exchange.Cluster.Replay.AmClusterNodeJoinedException: An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Node exchange-2010 is already joined to a cluster..
       at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterNodeJoinedException) ---
       at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
       --- End of stack trace on server (KLDEX02.KLDLABS.COM) ---
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(AmServerName serverName, String databaseName, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(AmServerName serverName, Nullable`1 dbGuid, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
    [2012-01-07T01:39:02] Updated Progress 'Done!' 100%.
    [2012-01-07T01:39:02] COMPLETED
    add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().


    Jack Galante Sr Systems Engineer
    Monday, January 9, 2012 3:31 PM

Answers

  • SOlVED.  it seems it had to be removed and readded from the cluster service also, and then it started functioning again correctly.  Thanks ALl!
    Jack Galante Sr Systems Engineer
    • Marked as answer by Terence Yu Friday, January 13, 2012 2:24 AM
    Thursday, January 12, 2012 7:09 PM

All replies

  • Hi,

    It looks as if both error messages are from the add dag server operation.

    What is the error message received when you try to remove the bad server?

    Leif

    Monday, January 9, 2012 3:37 PM
  • This is what i get when i try and remove the server from the DAG (the one that i did not recover)

     

    cmdlet Remove-DatabaseAvailabilityGroupServer at command pipeline position 1
    Supply values for the following parameters:
    Identity: klddag01
    MailboxServer: KLDEX02
    WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
    "C:\ExchangeSetupLogs\DagTasks\dagtask_2012-01-09_15-36-57.268_remove-databaseavailabiltygroupserver.log".
    The following servers in the Windows Failover Cluster are not in Active Directory: exchange-2010. This is usually the r
    esult of an incomplete membership change (add or remove) of the database availabilty group.
        + CategoryInfo          : InvalidArgument: (:) [Remove-DatabaseAvailabilityGroupServer], DagTaskServersInClusterNo
       tInAd
        + FullyQualifiedErrorId : AB081D8F,Microsoft.Exchange.Management.SystemConfigurationTasks.RemoveDatabaseAvailabili
       tyGroupServer


    Jack Galante Sr Systems Engineer
    Monday, January 9, 2012 3:39 PM
  • This is the Log file referenced from above post

    remove-databaseavailabiltygroupserver started on machine KLDEX02.
    [2012-01-09T15:36:57] remove-dagserver started
    [2012-01-09T15:36:57] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
    [2012-01-09T15:36:57] Option 'Identity' = 'klddag01'.
    [2012-01-09T15:36:57] Option 'MailboxServer' = 'KLDEX02'.
    [2012-01-09T15:36:57] Option 'ConfigurationOnly' = ''.
    [2012-01-09T15:36:57] Option 'WhatIf' = ''.
    [2012-01-09T15:36:57] Process: w3wp w3wp.exe:7652.
    [2012-01-09T15:36:57] User context = 'NT AUTHORITY\SYSTEM'.
    [2012-01-09T15:36:57]   Member of group 'Everyone'.
    [2012-01-09T15:36:57]   Member of group 'BUILTIN\Users'.
    [2012-01-09T15:36:57]   Member of group 'NT AUTHORITY\SERVICE'.
    [2012-01-09T15:36:57]   Member of group 'CONSOLE LOGON'.
    [2012-01-09T15:36:57]   Member of group 'NT AUTHORITY\Authenticated Users'.
    [2012-01-09T15:36:57]   Member of group 'NT AUTHORITY\This Organization'.
    [2012-01-09T15:36:57]   Member of group 'BUILTIN\IIS_IUSRS'.
    [2012-01-09T15:36:57]   Member of group 'LOCAL'.
    [2012-01-09T15:36:57]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
    [2012-01-09T15:36:57]   Member of group 'BUILTIN\Administrators'.
    [2012-01-09T15:36:57] Updated Progress 'Validating the parameters.' 2%.
    [2012-01-09T15:36:57] Working
    [2012-01-09T15:36:57] Looking up IP addresses for KLDDAG01.
    [2012-01-09T15:36:57]   KLDDAG01 = [ 10.0.0.7 ].
    [2012-01-09T15:36:57] Looking up IP addresses for KLDEX02.KLDLABS.COM.
    [2012-01-09T15:36:57]   KLDEX02.KLDLABS.COM = [ ::1, 10.0.0.11 ].
    [2012-01-09T15:36:57] Updated Progress 'Validating removal of mailbox server 'KLDEX02' from 'KLDDAG01'.' 0%.
    [2012-01-09T15:36:57] Working
    [2012-01-09T15:36:57] CheckClusterStateForDagServerRemoval entered. m_removeNode=False, m_destroyCluster=False
    [2012-01-09T15:36:57] Dumping the cluster by connecting to: <existing cluster handle>.
    [2012-01-09T15:36:57] The cluster's name is: KLDDAG01.
    [2012-01-09T15:36:57] Groups
    [2012-01-09T15:36:57]     group: Cluster Group [Cluster Main Group]
    [2012-01-09T15:36:57]         OwnerNode: KLDEX02.KLDLABS.COM
    [2012-01-09T15:36:57]         State: Online
    [2012-01-09T15:36:57]             Resource: Cluster Name [Online, type = Network Name, PossibleOwners = EXCHANGE-2010,KLDEX02 ]
    [2012-01-09T15:36:57]                 NetName = [KLDDAG01]
    [2012-01-09T15:36:57]             Resource: File Share Witness (\\klddata01.kldlabs.com\KLDDAG01.kldlabs.com) [Online, type = File Share Witness, PossibleOwners = EXCHANGE-2010,KLDEX02 ]
    [2012-01-09T15:36:57]             Resource: IPv4 Static Address 1 (Cluster Group) [Online, type = IP Address, PossibleOwners = EXCHANGE-2010,KLDEX02 ]
    [2012-01-09T15:36:57]                 Address = [10.0.0.7]
    [2012-01-09T15:36:57]                     EnableDhcp = [0]
    [2012-01-09T15:36:57]                     Network = [Cluster Network 1]
    [2012-01-09T15:36:57]     group: Available Storage [not a CMS]
    [2012-01-09T15:36:57]         OwnerNode: KLDEX02.KLDLABS.COM
    [2012-01-09T15:36:57]         State: Offline
    [2012-01-09T15:36:57] Nodes
    [2012-01-09T15:36:57]     node: EXCHANGE-2010.KLDLABS.COM [ state = Down ]
    [2012-01-09T15:36:57]     node: KLDEX02.KLDLABS.COM [ state = Up ]
    [2012-01-09T15:36:57] Subnets
    [2012-01-09T15:36:57]     Name(Cluster Network 2), Mask(10.10.2.0/24), Role(ClusterNetworkRoleInternalUse)
    [2012-01-09T15:36:57]         NIC 10.10.2.112 on Node EXCHANGE-2010 in State=Unavailable
    [2012-01-09T15:36:57]     Name(Cluster Network 1), Mask(10.0.0.0/24), Role(ClusterNetworkRoleInternalAndClient)
    [2012-01-09T15:36:57]         NIC 10.0.0.11 on Node KLDEX02 in State=Up
    [2012-01-09T15:36:57]         NIC 10.0.0.12 on Node EXCHANGE-2010 in State=Unavailable
    [2012-01-09T15:36:57] Trying to open the node on the cluster.
    [2012-01-09T15:36:57] Node.GetState( KLDEX02.KLDLABS.COM ) reports that it is Up.
    [2012-01-09T15:36:57] Successfully resolved the servers based on the stopped servers list.
    [2012-01-09T15:36:57] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
    [2012-01-09T15:36:57] The following servers are in the StoppedServers list:
    [2012-01-09T15:36:57] Verifiying that the members of database availability group 'KLDDAG01' are also members of the cluster.
    [2012-01-09T15:36:57] Verifying that the members of cluster 'KLDDAG01' are also members of the database availability group.
    [2012-01-09T15:36:57] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2012-01-09_15-36-57.268_remove-databaseavailabiltygroupserver.log".
    [2012-01-09T15:36:57] WriteError! Exception = Microsoft.Exchange.Management.Tasks.DagTaskServersInClusterNotInAd: The following servers in the Windows Failover Cluster are not in Active Directory: exchange-2010. This is usually the result of an incomplete membership change (add or remove) of the database availabilty group.
    [2012-01-09T15:36:57] Updated Progress 'Done!' 100%.
    [2012-01-09T15:36:57] COMPLETED
    remove-databaseavailabiltygroupserver explicitly called CloseTempLogFile().


    Jack Galante Sr Systems Engineer
    Monday, January 9, 2012 3:43 PM
  • Hi,

     

    It appears that the server should have been removed from the DAG before the recovery took place.

    See:

  • Performing a server recovery operation   If a Mailbox server that's a member of a DAG is lost, or otherwise fails and is unrecoverable and needs replacement, you can perform a server recovery operation using the Setup /m:RecoverServer switch. However, before you can perform the recovery operation, you must first remove the server from the DAG using the Remove-DatabaseAvailabilityGroupServer cmdlet with the ConfigurationOnly parameter.

    from: http://technet.microsoft.com/en-us/library/dd298065.aspx

    I don't know if you can perform this operation now

    Leif

Monday, January 9, 2012 3:44 PM
  • Yes i did that using the ConfigurationOnly parameter because the server was unavailable.
    Jack Galante Sr Systems Engineer
    Monday, January 9, 2012 4:09 PM
  • SOlVED.  it seems it had to be removed and readded from the cluster service also, and then it started functioning again correctly.  Thanks ALl!
    Jack Galante Sr Systems Engineer
    • Marked as answer by Terence Yu Friday, January 13, 2012 2:24 AM
    Thursday, January 12, 2012 7:09 PM