none
Can't add server to DAG cluster. RRS feed

  • Question

  • I have two servers in a cluster, but when I add our main mail server I get the following error. 

    All of the servers have two NIC's which the same configuration with two networks.

    I can create a DAG with this server alone, but when I add another one, I get this message. 

    All three servers have the same DNS settings.


    add-databaseavailabiltygroupserver started on machine EXCHANGE.
    [2009-12-25T01:52:25] add-dagserver started
    [2009-12-25T01:52:25] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
    [2009-12-25T01:52:25] Option 'Identity' = 'dage'.
    [2009-12-25T01:52:25] Option 'MailboxServer' = 'exchange'.
    [2009-12-25T01:52:25] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
    [2009-12-25T01:52:25] Option 'WhatIf' = ''.
    [2009-12-25T01:52:25] Process: w3wp w3wp.exe:1368.
    [2009-12-25T01:52:25] User context = 'NT AUTHORITY\SYSTEM'.
    [2009-12-25T01:52:25]   Member of group 'Everyone'.
    [2009-12-25T01:52:25]   Member of group 'BUILTIN\Users'.
    [2009-12-25T01:52:25]   Member of group 'NT AUTHORITY\SERVICE'.
    [2009-12-25T01:52:25]   Member of group 'CONSOLE LOGON'.
    [2009-12-25T01:52:25]   Member of group 'NT AUTHORITY\Authenticated Users'.
    [2009-12-25T01:52:25]   Member of group 'NT AUTHORITY\This Organization'.
    [2009-12-25T01:52:25]   Member of group 'BUILTIN\IIS_IUSRS'.
    [2009-12-25T01:52:25]   Member of group 'LOCAL'.
    [2009-12-25T01:52:25]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
    [2009-12-25T01:52:25]   Member of group 'BUILTIN\Administrators'.
    [2009-12-25T01:52:25] Updated Progress 'Validating the parameters.' 2%.
    [2009-12-25T01:52:25] Working
    [2009-12-25T01:52:25] Mailbox server: value passed in = exchange, mailboxServer.Name = EXCHANGE, mailboxServer.Fqdn = EXCHANGE.acsi.com
    [2009-12-25T01:52:25] LogClussvcState: clussvc is Stopped on EXCHANGE.acsi.com. Exception (if any) = none
    [2009-12-25T01:52:25] The IP addresses for the DAG are (blank means DHCP): 
    [2009-12-25T01:52:25] Looking up IP addresses for DAGE.
    [2009-12-25T01:52:28] Failure while trying to resolve DAGE: threw a SocketException: The requested name is valid, but no data of the requested type was found.
    [2009-12-25T01:52:28] Looking up IP addresses for exchange.
    [2009-12-25T01:52:28]   exchange = [ fe80::1898:2778:f04d:d646%11, fe80::4984:510e:1fea:fb92%12, 2002:c701:63d2::c701:63d2, 199.1.99.210, 10.10.1.210 ].
    [2009-12-25T01:52:28] Looking up IP addresses for EXCHANGE.acsi.com.
    [2009-12-25T01:52:28]   EXCHANGE.acsi.com = [ fe80::1898:2778:f04d:d646%11, fe80::4984:510e:1fea:fb92%12, 2002:c701:63d2::c701:63d2, 199.1.99.210, 10.10.1.210 ].
    [2009-12-25T01:52:28] DAG DAGE has 2 servers:
    [2009-12-25T01:52:28] DAG DAGE contains server EXCHANGE2.
    [2009-12-25T01:52:28] DAG DAGE contains server EXCHANGE3.
    [2009-12-25T01:52:28] Updated Progress 'Checking if Mailbox server 'EXCHANGE' is in a database availability group.' 4%.
    [2009-12-25T01:52:28] Working
    [2009-12-25T01:52:28] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(EXCHANGE.acsi.com) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK.
    [2009-12-25T01:52:28] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
    [2009-12-25T01:52:28] DumpClusterTopology: Opening remote cluster DAGE.
    [2009-12-25T01:52:28] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmGetFqdnFailedNotFoundException: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: The fully qualified domain name for node 'DAGE' could not be found.
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.<>c__DisplayClass1.<ResolveFqdn>b__0(Object , EventArgs )
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.RunADOperationEx(EventHandler ev)
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.ResolveFqdn(String shortNodeName)
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.GetFqdn(String shortNodeName)
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerName..ctor(String serverName)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)
    [2009-12-25T01:52:28] Dumping the cluster by connecting to: DAGE.
    [2009-12-25T01:52:28] DumpClusterTopology: Not dumping, since the cluster could not be contacted.
    [2009-12-25T01:52:28] Opening the cluster on nodes [exchange2, exchange3].
    [2009-12-25T01:52:28] Other mailbox servers in the DAG are already members of cluster 'DAGE'
    [2009-12-25T01:52:28] The server EXCHANGE does not belong to a cluster, and the other servers belong to DAGE.
    [2009-12-25T01:52:29] Successfully resolved servers based on the AD configuration.
    [2009-12-25T01:52:29] The following servers are in the StartedServers list:
    [2009-12-25T01:52:29] The following servers are in the StoppedServers list:
    [2009-12-25T01:52:29] Verifiying that the members of database availability group 'DAGE' are also members of the cluster.
    [2009-12-25T01:52:29] Verifying that the members of cluster 'DAGE' are also members of the database availability group.
    [2009-12-25T01:52:29] According to GetNodeClusterState(), the server EXCHANGE is NotConfigured.
    [2009-12-25T01:52:29] The CNO is currently Offline.
    [2009-12-25T01:52:29] The network name 'DAGE' is not online. Please check that the IP address configuration of the Database Availability Group is correct.
    [2009-12-25T01:52:29] InternalValidate() done.
    [2009-12-25T01:52:29] Updated Progress 'Adding server 'EXCHANGE' to database availability group 'DAGE'.' 6%.
    [2009-12-25T01:52:29] Working
    [2009-12-25T01:52:29] Updated Progress 'Adding server 'EXCHANGE' to the cluster.' 8%.
    [2009-12-25T01:52:29] Working
    [2009-12-25T01:53:12] The following log entry comes from a different process, run on machine 'exchange3.acsi.com'. BEGIN
    [2009-12-25T01:53:12] [2009-12-25T01:52:29] Opening a local AmCluster handle.
    [2009-12-25T01:52:29] Updated Progress 'Adding server 'exchange' to database availability group 'DAGE'.' 2%.
    [2009-12-25T01:52:29] Working
    [2009-12-25T01:52:29] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x0 )
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseContinue, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x800713bb )
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x800713bb )
    [2009-12-25T01:53:11] Found a matching exception: Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x0 )
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )

    [2009-12-25T01:53:12] The preceding log entry comes from a different process, run on machine 'exchange3.acsi.com'. END
    [2009-12-25T01:53:12] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2009-12-25_01-52-25.205_add-databaseavailabiltygroupserver.log".
    [2009-12-25T01:53:12] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server. ---> Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
       at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException) ---
       at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
       --- End of stack trace on server (exchange3.acsi.com) ---
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
    [2009-12-25T01:53:12] Updated Progress 'Done!' 100%.
    [2009-12-25T01:53:12] COMPLETED
    add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

    Friday, December 25, 2009 1:56 AM

All replies

  • HI,

    Try again after disabling IPv6 on both Nics of both nodes.

    http://support.microsoft.com/default.aspx/kb/929852

    Any event errors in event log ?

    Regards
    Chinthaka Shameera | MCITP: EA | MCSE: M | http://howtoexchange.wordpress.com/
    Friday, December 25, 2009 5:01 AM
  • The IPV6 didn't fix it.


    Here's the event log.

    Log Name:      Application
    Source:        MSExchange Configuration Cmdlet - Remote Management
    Date:          12/25/2009 12:34:56 AM
    Event ID:      4
    Task Category: General
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      EXCHANGE.acsi.com
    Description:
    (PID 1452, Thread 7) Task Add-DatabaseAvailabilityGroupServer writing error when processing record of index 0. Error: Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server. ---> Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
       at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException) ---
       at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
       --- End of stack trace on server (exchange3.acsi.com) ---
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchange Configuration Cmdlet - Remote Management" />
        <EventID Qualifiers="49152">4</EventID>
        <Level>2</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2009-12-25T05:34:56.000000000Z" />
        <EventRecordID>1087150</EventRecordID>
        <Channel>Application</Channel>
        <Computer>EXCHANGE.acsi.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>1452</Data>
        <Data>7</Data>
        <Data>Add-DatabaseAvailabilityGroupServer</Data>
        <Data>0</Data>
        <Data>Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server. ---&gt; Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
       at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception&amp; errorException, Boolean throwExceptionOnFailure)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String&amp; verboseLog)
       --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException) ---
       at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String&amp; verboseLog)
       at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.&lt;&gt;c__DisplayClass34.&lt;RpcsAddNodeToCluster&gt;b__33()
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
       --- End of stack trace on server (exchange3.acsi.com) ---
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient&amp; rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient&amp; rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String&amp; verboseLog)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()</Data>
        <Data>{e713d62a-d3db-46f7-a554-b6dd55a7ffd3}</Data>
      </EventData>
    </Event>
    Friday, December 25, 2009 5:37 AM
  • Is yours a parent domain child domain architecture.

    Are the domain names disjoint?

    Are you able to failover between two cluster nodes.

    Do you find any error in cluster validation logs.

    Check if any cluster resource is offline?



    Thx
    • Edited by Pankaj.Rai Friday, December 25, 2009 7:17 AM editing
    Friday, December 25, 2009 7:11 AM
  • Hi,

    1. Please verify if the primary DNS suffix of a member computer on which Exchange 2010 is installed is not the same as the DNS domain name, even though the primary DNS suffix of the domain controller is the same as the DNS domain name.

    Understanding Disjoint Namespace Scenarios
    http://technet.microsoft.com/en-us/library/bb676377.aspx

    2. Please try to run ipconfig /all on all the exchange server and then post here.

    3. Please check the MAPI Network Adapter Configuration against the settings in article below:

    Planning for High Availability and Site Resilience<!---->
    http://technet.microsoft.com/en-us/library/dd638104.aspx

    4. Please try to use replication network on exchange server to ping each other.
     <!---->
    5. Besides,please check "C:\ExchangeSetupLogs\DagTasks\dagtask_2009-12-25_01-52-25.205_add-databaseavailabiltygroupserver.log and then post here.

    Regards,
    Xiu
    Friday, December 25, 2009 9:00 AM
  • I can fail over between two nodes. I just can't add this one server to any cluster except one that it's in itself.

    There's only one domain, not a parent child.
    Friday, December 25, 2009 2:50 PM
  • 1. I don't think I have a disjoint. There is only one domain and every server has the same primary dns suffix.

    2. This is the server that cannot be added. Can only be added to a group itself:


    Windows IP Configuration

       Host Name . . . . . . . . . . . . : EXCHANGE
       Primary Dns Suffix  . . . . . . . : acsi.com
       Node Type . . . . . . . . . . . . : Hybrid
       IP Routing Enabled. . . . . . . . : No
       WINS Proxy Enabled. . . . . . . . : No
       DNS Suffix Search List. . . . . . : acsi.com

    Ethernet adapter Local Area Connection* 9:

       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter
       Physical Address. . . . . . . . . : 02-1E-C9-45-A7-EA
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes

    Ethernet adapter Local Area Connection 2:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet #2
       Physical Address. . . . . . . . . : 00-1E-C9-45-A7-EB
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 10.10.1.210(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 10.10.1.1
       DNS Servers . . . . . . . . . . . : 199.1.99.30
                                           199.1.99.10
                                           199.1.99.20
       NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter Local Area Connection:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
       Physical Address. . . . . . . . . : 00-1E-C9-45-A7-EA
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 199.1.99.210(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 
       DNS Servers . . . . . . . . . . . : 199.1.99.30
                                           199.1.99.10
                                           199.1.99.20
       NetBIOS over Tcpip. . . . . . . . : Enabled



    This is one of the two that is in a functional group:


    Windows IP Configuration

       Host Name . . . . . . . . . . . . : exchange2
       Primary Dns Suffix  . . . . . . . : acsi.com
       Node Type . . . . . . . . . . . . : Hybrid
       IP Routing Enabled. . . . . . . . : No
       WINS Proxy Enabled. . . . . . . . : No
       DNS Suffix Search List. . . . . . : acsi.com

    Ethernet adapter Local Area Connection* 9:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter
       Physical Address. . . . . . . . . : 02-11-43-04-82-5C
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 169.254.1.39(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.0.0
       Default Gateway . . . . . . . . . : 
       NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter Local Area Connection:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Server Connection
       Physical Address. . . . . . . . . : 00-11-43-04-82-5C
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 199.1.99.212(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       IPv4 Address. . . . . . . . . . . : 199.1.99.154(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 
       DNS Servers . . . . . . . . . . . : 199.1.99.30
                                           199.1.99.10
                                           199.1.99.20
       NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter Local Area Connection 2:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
       Physical Address. . . . . . . . . : 00-10-18-5A-7D-38
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 10.10.1.212(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 10.10.1.1
       DNS Servers . . . . . . . . . . . : 199.1.99.30
                                           199.1.99.20
                                           199.1.99.10
       NetBIOS over Tcpip. . . . . . . . : Enabled





    And the other one that's part of the group that's working:


    Windows IP Configuration

       Host Name . . . . . . . . . . . . : exchange3
       Primary Dns Suffix  . . . . . . . : acsi.com
       Node Type . . . . . . . . . . . . : Hybrid
       IP Routing Enabled. . . . . . . . : No
       WINS Proxy Enabled. . . . . . . . : No
       DNS Suffix Search List. . . . . . : acsi.com

    Ethernet adapter Local Area Connection* 10:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter
       Physical Address. . . . . . . . . : 02-0C-29-95-A5-BD
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 169.254.2.159(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.0.0
       Default Gateway . . . . . . . . . : 
       NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter Local Area Connection 2:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
       Physical Address. . . . . . . . . : 00-0C-29-95-A5-C7
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 10.10.1.214(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 10.10.1.1
       DNS Servers . . . . . . . . . . . : 199.1.99.30
                                           199.1.99.10
                                           199.1.99.20
       NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter Local Area Connection:

       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
       Physical Address. . . . . . . . . : 00-0C-29-95-A5-BD
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 199.1.99.214(Preferred) 
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 
       DNS Servers . . . . . . . . . . . : 199.1.99.30
                                           199.1.99.10
                                           199.1.99.20
       NetBIOS over Tcpip. . . . . . . . : Enabled

    3.  I have both networks setup as the replication network. Is this a problem?

    4. Every server can ping every other network address of these three servers.

    5. Here's the log:

    add-databaseavailabiltygroupserver started on machine EXCHANGE.
    [2009-12-25T01:52:25] add-dagserver started
    [2009-12-25T01:52:25] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
    [2009-12-25T01:52:25] Option 'Identity' = 'dage'.
    [2009-12-25T01:52:25] Option 'MailboxServer' = 'exchange'.
    [2009-12-25T01:52:25] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
    [2009-12-25T01:52:25] Option 'WhatIf' = ''.
    [2009-12-25T01:52:25] Process: w3wp w3wp.exe:1368.
    [2009-12-25T01:52:25] User context = 'NT AUTHORITY\SYSTEM'.
    [2009-12-25T01:52:25]   Member of group 'Everyone'.
    [2009-12-25T01:52:25]   Member of group 'BUILTIN\Users'.
    [2009-12-25T01:52:25]   Member of group 'NT AUTHORITY\SERVICE'.
    [2009-12-25T01:52:25]   Member of group 'CONSOLE LOGON'.
    [2009-12-25T01:52:25]   Member of group 'NT AUTHORITY\Authenticated Users'.
    [2009-12-25T01:52:25]   Member of group 'NT AUTHORITY\This Organization'.
    [2009-12-25T01:52:25]   Member of group 'BUILTIN\IIS_IUSRS'.
    [2009-12-25T01:52:25]   Member of group 'LOCAL'.
    [2009-12-25T01:52:25]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
    [2009-12-25T01:52:25]   Member of group 'BUILTIN\Administrators'.
    [2009-12-25T01:52:25] Updated Progress 'Validating the parameters.' 2%.
    [2009-12-25T01:52:25] Working
    [2009-12-25T01:52:25] Mailbox server: value passed in = exchange, mailboxServer.Name = EXCHANGE, mailboxServer.Fqdn = EXCHANGE.acsi.com
    [2009-12-25T01:52:25] LogClussvcState: clussvc is Stopped on EXCHANGE.acsi.com. Exception (if any) = none
    [2009-12-25T01:52:25] The IP addresses for the DAG are (blank means DHCP): 
    [2009-12-25T01:52:25] Looking up IP addresses for DAGE.
    [2009-12-25T01:52:28] Failure while trying to resolve DAGE: threw a SocketException: The requested name is valid, but no data of the requested type was found.
    [2009-12-25T01:52:28] Looking up IP addresses for exchange.
    [2009-12-25T01:52:28]   exchange = [ fe80::1898:2778:f04d:d646%11, fe80::4984:510e:1fea:fb92%12, 2002:c701:63d2::c701:63d2, 199.1.99.210, 10.10.1.210 ].
    [2009-12-25T01:52:28] Looking up IP addresses for EXCHANGE.acsi.com.
    [2009-12-25T01:52:28]   EXCHANGE.acsi.com = [ fe80::1898:2778:f04d:d646%11, fe80::4984:510e:1fea:fb92%12, 2002:c701:63d2::c701:63d2, 199.1.99.210, 10.10.1.210 ].
    [2009-12-25T01:52:28] DAG DAGE has 2 servers:
    [2009-12-25T01:52:28] DAG DAGE contains server EXCHANGE2.
    [2009-12-25T01:52:28] DAG DAGE contains server EXCHANGE3.
    [2009-12-25T01:52:28] Updated Progress 'Checking if Mailbox server 'EXCHANGE' is in a database availability group.' 4%.
    [2009-12-25T01:52:28] Working
    [2009-12-25T01:52:28] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(EXCHANGE.acsi.com) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK.
    [2009-12-25T01:52:28] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
    [2009-12-25T01:52:28] DumpClusterTopology: Opening remote cluster DAGE.
    [2009-12-25T01:52:28] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmGetFqdnFailedNotFoundException: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: The fully qualified domain name for node 'DAGE' could not be found.
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.<>c__DisplayClass1.<ResolveFqdn>b__0(Object , EventArgs )
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.RunADOperationEx(EventHandler ev)
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.ResolveFqdn(String shortNodeName)
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.GetFqdn(String shortNodeName)
       at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerName..ctor(String serverName)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)
    [2009-12-25T01:52:28] Dumping the cluster by connecting to: DAGE.
    [2009-12-25T01:52:28] DumpClusterTopology: Not dumping, since the cluster could not be contacted.
    [2009-12-25T01:52:28] Opening the cluster on nodes [exchange2, exchange3].
    [2009-12-25T01:52:28] Other mailbox servers in the DAG are already members of cluster 'DAGE'
    [2009-12-25T01:52:28] The server EXCHANGE does not belong to a cluster, and the other servers belong to DAGE.
    [2009-12-25T01:52:29] Successfully resolved servers based on the AD configuration.
    [2009-12-25T01:52:29] The following servers are in the StartedServers list:
    [2009-12-25T01:52:29] The following servers are in the StoppedServers list:
    [2009-12-25T01:52:29] Verifiying that the members of database availability group 'DAGE' are also members of the cluster.
    [2009-12-25T01:52:29] Verifying that the members of cluster 'DAGE' are also members of the database availability group.
    [2009-12-25T01:52:29] According to GetNodeClusterState(), the server EXCHANGE is NotConfigured.
    [2009-12-25T01:52:29] The CNO is currently Offline.
    [2009-12-25T01:52:29] The network name 'DAGE' is not online. Please check that the IP address configuration of the Database Availability Group is correct.
    [2009-12-25T01:52:29] InternalValidate() done.
    [2009-12-25T01:52:29] Updated Progress 'Adding server 'EXCHANGE' to database availability group 'DAGE'.' 6%.
    [2009-12-25T01:52:29] Working
    [2009-12-25T01:52:29] Updated Progress 'Adding server 'EXCHANGE' to the cluster.' 8%.
    [2009-12-25T01:52:29] Working
    [2009-12-25T01:53:12] The following log entry comes from a different process, run on machine 'exchange3.acsi.com'. BEGIN
    [2009-12-25T01:53:12] [2009-12-25T01:52:29] Opening a local AmCluster handle.
    [2009-12-25T01:52:29] Updated Progress 'Adding server 'exchange' to database availability group 'DAGE'.' 2%.
    [2009-12-25T01:52:29] Working
    [2009-12-25T01:52:29] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x0 )
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseContinue, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x800713bb )
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x800713bb )
    [2009-12-25T01:53:11] Found a matching exception: Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = EXCHANGE, dwStatus = 0x0 )
    [2009-12-25T01:53:11] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )

    [2009-12-25T01:53:12] The preceding log entry comes from a different process, run on machine 'exchange3.acsi.com'. END
    [2009-12-25T01:53:12] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2009-12-25_01-52-25.205_add-databaseavailabiltygroupserver.log".
    [2009-12-25T01:53:12] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server. ---> Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server 'EXCHANGE'. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
       at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException) ---
       at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
       at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
       at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
       --- End of stack trace on server (exchange3.acsi.com) ---
       at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
       at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
       at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
    [2009-12-25T01:53:12] Updated Progress 'Done!' 100%.
    [2009-12-25T01:53:12] COMPLETED
    add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().


    Friday, December 25, 2009 3:09 PM
  • Are both nodes in different subnets? If so, all subnets are added into DAG Networks? Any firewall between these two nodes?

    Amit Tank
    MVP: Exchange Server | MCTS: Microsoft Exchange Server 2010, Configuration | MCITP: EMA | MCSA: M
    Blog: http://ExchangeShare.WordPress.com | User Group: http:/MUC-UG.org.in


    Friday, December 25, 2009 3:20 PM
  • No. All three severs are part of two networks. Each server has two NIC's-- one for each network. No firewalls.



    The weird thing is that exchange2 and exchange3 servers are working fine. I just can't add 'exchange' server to that DAG. I can create a DAG with only 'exchange' in it, but then I can't add either of the other two servers.
    Friday, December 25, 2009 4:05 PM
  • A quick look at previous post shows first server "exchange" has Failover cluster virtual adapter shows media disconnected, any reason?

    Ethernet adapter Local Area Connection* 9:

       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . : 
       Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter
       Physical Address. . . . . . . . . : 02-1E-C9-45-A7-EA
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes

    Amit Tank
    MVP: Exchange Server | MCTS: Microsoft Exchange Server 2010, Configuration | MCITP: EMA | MCSA: M
    Blog: http://ExchangeShare.WordPress.com | User Group: http:/MUC-UG.org.in


    Friday, December 25, 2009 4:13 PM
  • I could be wrong, but I think it's showing as disconnected because it's not part of the cluster. I unistalled the Failover Cluster Feature, and that adapter went away. But I still can't add it to the cluster.
    Friday, December 25, 2009 4:36 PM
  • Yeh, I think it should show connected. Can you run cluster validation wizard and check? 

    Also try enabling virtual adapter in Device Manager after selecting show hidden device in View menu...

    Amit Tank
    MVP: Exchange Server | MCTS: Microsoft Exchange Server 2010, Configuration | MCITP: EMA | MCSA: M
    Blog: http://ExchangeShare.WordPress.com | User Group: http:/MUC-UG.org.in


    Friday, December 25, 2009 5:17 PM
  • It is enabled already.

    After running the validation check it says it's suitable for clustering. 

    I totally rebuilt one of the servers thinking that that might be be problem and it still does the same think.  I can't get Exchange1 to form a cluster with either Exchange2 or Exchange3. Exchange 2 and 3 can cluster with no problems.

    When I try to manually add it to the cluster from Exchange1, I get an access denied error. When I try to add it manually to the cluster from Exchange2 or 3, I get a cannot access error.

    Help! 
    Saturday, December 26, 2009 5:02 AM
  • Hi,

     

    Error 0x800713bb equates to ERROR_CLUSTER_NODE_UNREACHABLE.

     

    Please try to ping DAG name from the problematic computer to see if it can resolve to the IP address.

     

    Please run the following command to check if windows cluster services has been enabled.

     

    sc.exe qc clussvc

     

    Please run the following command to check the cluster node status.

     

    cluster.exe node /status

     

    Please try to run the command below to validate if we can add node to cluster.

     

    cluster.exe /cluster:DAGE /add /node: EXCHANGE

     

    Besides, please try to run Get-DatabaseAvailabilityGroup | fl and then post here.

     

    Appendix A - Cluster.exe Commands

    http://technet.microsoft.com/en-us/library/cc723245.aspx

     

    Regards,

    Xiu

     

    Monday, December 28, 2009 7:29 AM
  • I can ping the cluster and get replies.

    From sc.exe qc clussvc (from the node that can't connect) I get this:


    C:\Users\administrator.ACSI>sc.exe qc clussvc
    [SC] QueryServiceConfig SUCCESS

    SERVICE_NAME: clussvc
            TYPE               : 10  WIN32_OWN_PROCESS
            START_TYPE         : 4   DISABLED
            ERROR_CONTROL      : 1   NORMAL
            BINARY_PATH_NAME   : C:\Windows\Cluster\clussvc.exe -s
            LOAD_ORDER_GROUP   :
            TAG                : 0
            DISPLAY_NAME       : Cluster Service
            DEPENDENCIES       : RpcSS
                               : NetFT
                               : Srv
            SERVICE_START_NAME : LocalSystem




    From cluster.exe node /status (on the one connected to the cluster) I get this:

    C:\Users\administrator.ACSI>cluster.exe node /status
    Listing status for all available nodes:

    Node           Node ID Status
    -------------- ------- ---------------------
    exchange2            1 Up


    From cluster.exe /cluster:dag-100 /add /node: exchange I get:

    This is running it on exchange:

    Configuring node exchange
    ---------------------------------------
     12% Validating cluster state on node exchange.
     25% Getting current node membership of cluster DAG-100.
     37% Adding node exchange to Cluster configuration data.
     50% Validating installation of the Microsoft Failover Cluster Virtual Adapter o
    n node exchange.
     62% Validating installation of the Cluster Disk Driver on node exchange.
     75% Configuring Cluster Service on node exchange.This phase has failed for Clus
    ter object 'exchange' with an error status of -2147024891 (0x80070005).
    Cleaning up exchange.

    System error 5 has occurred (0x00000005).
    Access is denied.

    When I run the same command from the node that is already in the cluster I get:

    Configuring node exchange

    ---------------------------------------

     12% Validating cluster state on node exchange.This phase has failed for Cluster object 'exchange' with an error status of -2147019845 (0x800713BB).

    This phase has failed for Cluster object 'exchange' with an error status of -2147019845 (0x800713BB).

    Cleaning up exchange.

     

    System error 5051 has occurred (0x000013bb).



    I checked and I can ping the all of the servers and the cluster from this server.




    Here's the results from Get-DatabaseAvailabiltyGRoup:

    RunspaceId                             : 9afe4d3d-0544-4106-991f-575372448a15
    Name                                   : DAG-100
    Servers                                : {EXCHANGE2}
    WitnessServer                          : EXCHANGE3.ACSI.COM
    WitnessDirectory                       : C:\DAGFileShareWitnesses\DAG-100.acsi.com
    AlternateWitnessServer                 :
    AlternateWitnessDirectory              :
    NetworkCompression                     : InterSubnetOnly
    NetworkEncryption                      : InterSubnetOnly
    DatacenterActivationMode               : Off
    StoppedMailboxServers                  : {}
    StartedMailboxServers                  : {}
    DatabaseAvailabilityGroupIpv4Addresses : {}
    OperationalServers                     :
    PrimaryActiveManager                   :
    ThirdPartyReplication                  : Disabled
    ReplicationPort                        : 0
    NetworkNames                           : {}
    AdminDisplayName                       :
    ExchangeVersion                        : 0.10 (14.0.100.0)
    DistinguishedName                      : CN=DAG-100,CN=Database Availability Groups,CN=Exchange Administrative Group (F
                                             YDIBOHF23SPDLT),CN=Administrative Groups,CN=Aptech,CN=Microsoft Exchange,CN=Se
                                             rvices,CN=Configuration,DC=acsi,DC=com
    Identity                               : DAG-100
    Guid                                   : 7462fec3-72c4-46dd-a6e9-c18b39cd999f
    ObjectCategory                         : acsi.com/Configuration/Schema/ms-Exch-MDB-Availability-Group
    ObjectClass                            : {top, msExchMDBAvailabilityGroup}
    WhenChanged                            : 12/28/2009 1:30:39 AM
    WhenCreated                            : 12/28/2009 1:30:15 AM
    WhenChangedUTC                         : 12/28/2009 6:30:39 AM
    WhenCreatedUTC                         : 12/28/2009 6:30:15 AM
    OrganizationId                         :
    OriginatingServer                      : support.acsi.com
    IsValid                                : True

     

     





     

    Monday, December 28, 2009 7:03 PM
  • I can fail over between two nodes. I just can't add this one server to any cluster except one that it's in itself.

    There's only one domain, not a parent child.
    Is your domain running at Windows 2k3 w/SP2?
    Elie B. MCITP: EMEA|EA|SA Blog: http://blog.elieb.info
    Monday, December 28, 2009 7:52 PM
  • Yes. It's running at 2k3 level. But I do have one DC that is 2008.

    Just for the heck of it, I just turned the 2008 DC off and I still got the same message.

    Monday, December 28, 2009 8:06 PM
  • You can have Win2k8 as DC as well.
    Is your Win2k3 domain set as a GC?

    Elie B. MCITP: EMEA|EA|SA Blog: http://blog.elieb.info
    Monday, December 28, 2009 8:19 PM
  • Yes. our 2 Win2k3 are GC as well as our 2008 is also.
    Monday, December 28, 2009 9:37 PM


  • Hello Xiu,

    How did we calculate : 0x800713bb equates to ERROR_CLUSTER_NODE_UNREACHABLE

    Thank you
    Tuesday, December 29, 2009 12:56 AM
  • I have no idea. But I do see that error message when I try to add the node from a node that's already part of the cluster. I can however, ping and resolve the other server.  
    Tuesday, December 29, 2009 1:45 AM
  • hi,

    only for one time en error occured like yours and i didnt join the node to the DAG and i installed Cluster service manually. tried again and node joined to DAG successfully...

    but it occured only once... and i couldnt find anything about it on my logs...

    regards,


    Mumin CICEK | Exchange - MVP | www.cozumpark.com | www.mumincicek.com
    Tuesday, December 29, 2009 12:05 PM
  • Thanks. Let me give that a try and see what happens.

    Didn't work. I get access denied when trying to add the node to the cluster from the server that isbn't part of the cluster.

    What is denying access? How do you change the permission of the cluster?
    Tuesday, December 29, 2009 6:11 PM
  • Sorry if the disjoint namespace comment caused any false alarms. You'd know if you had a disjoint DNS namespace.

    Thanks for posting all of the logs/ipconfig output.
    Which OS are you running?

    Try stopping clussvc on the machine that's already in the DAG, waiting for more than a minute, start it again, and then re-try the Add operation.

    Try looking at the event logs. Including Applications/Services -> Microsoft -> Windows -> FailoverClustering.
    There is additional logging you can look at (cluster.exe log /gen /copyto:"c:\clusterlogs"), but to be honest it's pretty detailed and can take a fair bit of effort to decipher.

    -martin


    Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, December 30, 2009 1:53 AM
  • All three servers involved are running Server 2008R2.  Two out of the three are clean installs.

    Stopping and then starting clussvc didn't do anything.

    I did find something interesting in the additional logging you mentioned :  10.10.1.210 is the server that won't join the cluster.


    00000bcc.00001d1c::2009/12/30-03:36:43.998 INFO  [NM] Received request from client address 10.10.1.210.
    00000bcc.00001d1c::2009/12/30-03:36:44.107 INFO  [QUORUM] AddNode Gum Handler exchange
    00000bcc.00001d1c::2009/12/30-03:36:44.107 INFO  [QUORUM] unique node id for new node 2 is 0000001d-0000-0000-0000-000000000002
    00000bcc.00001d1c::2009/12/30-03:36:44.201 INFO  [QUORUM] unique node id for new node 2 is 0000001d-0000-0000-0000-000000000002
    00000bcc.00001d1c::2009/12/30-03:36:44.201 INFO  [MQ-exchange] Starting
    00000bcc.00001d1c::2009/12/30-03:36:44.201 INFO  [MQ-exchange] Clearing 0 unsent and 0 unacknowledged messages.
    00000bd8.00001f18::2009/12/30-03:36:44.606 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000bd8.00001cc0::2009/12/30-03:36:44.638 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000bcc.00001d1c::2009/12/30-03:36:45.589 INFO  [QUORUM] Node 1: after eviction cluster will still have quorum <QuorumConfig tag='9:9:606' set='(1 2)' weights='(1)'/>
    00000bcc.00001d1c::2009/12/30-03:36:45.589 INFO  [QM] EvictNode Gum Handler exchange
    00000bcc.00001d1c::2009/12/30-03:36:45.589 INFO  [QUORUM] EvictCount for node 2 is incremented to 30
    00000bd8.00001f18::2009/12/30-03:36:46.010 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000bd8.00001cc0::2009/12/30-03:36:46.042 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'

    Wednesday, December 30, 2009 3:43 AM
  • " Two out of the three are clean installs. "

    You may one of them is installed from image??

    Thursday, December 31, 2009 2:55 AM
  • No,

    all three were upgraded to 2008R2 from 2008SP2. Then I did a clean install on two of them to 2008R2. I thought the upgrade might be the problem. But I can put two of them in a DAG, but not the main mail server which won't join a cluster that the other two are in.
    Friday, January 1, 2010 12:17 AM
  • I know it is silly, but have you tried removing and readding Failover-Cluster to the first box? Is the firewall enabled with the appropriate exceptions (or entirely disabled?)?
    Brian Day, Overall Exchange & AD Geek
    MCSA 2000/2003, CCNA
    MCTS: Microsoft Exchange Server 2010 Configuration
    LMNOP
    Saturday, January 2, 2010 6:22 PM
  • I think I have removed and re-added Failover Clustering to each of the 3 servers involved no less than 10 times on each box. I tried all sorts of different scenarios and nothing seems to work.

    Firewall is completed disabled on each box
    Sunday, January 3, 2010 2:22 AM
  • Is there any hardware compatability issues? Just wondering that the two servers that can join the cluster are Intel and the one that won't join is AMD. Just a long shot but everything else is the same. Same OS, Same Exchange Version, etc.
    Tuesday, January 5, 2010 3:19 PM
  • I didn't see anyone ask yet - but this is Enterprise Windows OS on all nodes?  (I think your having added Failover Clustering precludes this, but I needed to ask)
    Tuesday, January 5, 2010 3:35 PM
  • Yes. All three are Enterprise 2008R2 and Exchange 2010 Enterprise.

    I let Exchange 2010 install Failover, and even did it manually myself.

    The weird thing again is, I can make a DAG in Exchange 2010. But our "Exchange1" server in by itself, but I can't add "Exchange2" or "Exchange3" But if I take out "Exchange1" I can put "Exchange2" and "Exchange3" in with no prolems.  Just don't know what is different on "Exchange1" that prevents it from being in the DAG when the other server(s) are in the there.
    Tuesday, January 5, 2010 4:06 PM
  • Here's a little update:

    I reinstalled the OS on all three servers. And reinstalled Exchange 2010. And I am still getting the same error that I can;t join the one server to the DAG.

    Is it possible that this can be because of a hardware incompatibility issue. They are all Dell servers one the two that can join the group are Intel and the one that won't is AMD. Just wondering is maybe this is the problem. Other than that, they are all configured exactly the same.

    Thursday, January 14, 2010 7:36 PM
  • I had the same problem, and found the solution for me:
    • When creating the DAG, a computer account with the same name is created in Active Directory. (e.g. when you create "MyOrg-DAG" availability group, a computer account named "MyOrg-DAG" is created in Active Directory)
    • For some reason, when adding the computers, the account being used to create/add to the DAG didn't have (or didn't register) the proper permissions, and this computer account became invalid.
    • I deleted the DAG in Exchange, and then the computer account in Active Directory.
    • I recreated the DAG, and everything worked perfectly...
    Friday, February 19, 2010 12:14 AM
  • Travis, the computer account should have been created when you tried to add the first server to the DAG, not when you created the DAG. The new-DAG operation just creates an empty container.

    Anyway, if anyone else is curious about the computer account, Tim wrote some good steps:
    http://blogs.technet.com/timmcmic/archive/2010/01/05/exchange-2010-pre-staging-the-cluster-name-object-cno-to-support-a-database-availability-group-dag.aspx

    -martin
    Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, February 19, 2010 12:23 AM
  • Travis, the computer account should have been created when you tried to add the first server to the DAG, not when you created the DAG. The new-DAG operation just creates an empty container.

    Anyway, if anyone else is curious about the computer account, Tim wrote some good steps:
    http://blogs.technet.com/timmcmic/archive/2010/01/05/exchange-2010-pre-staging-the-cluster-name-object-cno-to-support-a-database-availability-group-dag.aspx

    -martin
    Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
    Correct.  I understand that, but it still got created when I experienced the same issues as KVAB.
    Friday, February 19, 2010 5:46 AM
  • I am having exactly the same problem as KVAB...

    The only difference is that my servers are exactly alike and KVAB's are INTEL and AMD

    anybody knows how to fix it?

    Wednesday, March 24, 2010 3:58 PM
  • I have never been able to resolve this either. Both systems are the same OS, serverice pack, etc.
    Thursday, March 25, 2010 8:12 PM
  • Hi KVAB,

    Have you found a fix to  the disjointed namespace issue?

    I also cannot join  4 HP Proliant 380 G6 servers to the dag but if I build a desktop or VM guests they can join successfully. 

    I am speaking to Microsoft premier support and they are taking a long time to find out what is causing the issue.

    Thanks many

     

    Tuesday, July 13, 2010 3:07 AM
  • Could we confirm if there is no matching user or computer object in the entire directory i.e. user having same logon name as of Computer you are adding. If we do have it please do try to delete the matching user account and attempt to add the DAG member.

    Thursday, January 6, 2011 2:26 PM
  • CollinGG could you confirm if in case you had configured VM with the same name as of the node which had failed to join in as a DAG member?
    Thursday, January 6, 2011 2:37 PM
  • Same exact issue here, two servers in the DAG are fine, fail over as they should and everything is happy. Trying to add a third DAG member and get these errors. Mine are all three VMWare ESX Enterprise 4 VMs. Was this ever resolved? Any pointers?
    Monday, January 17, 2011 2:40 AM
  • Did any one got any resolution for this.

    I got the same issue. All the three servers are windows 2008 R2. Exchange 2010 SP1. Same hardware.

    only difference is that the node that i am trying to add is in a different site and sub net.

    Also getting following event in system log

    Log Name:      System
    Source:        Microsoft-Windows-DistributedCOM
    Date:          25/01/2011 12:33:17 PM
    Event ID:      10009
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      server.domain.com
    Description:
    DCOM was unable to communicate with the computer Server using any of the configured protocols.

    some body please help

     

     

    Tuesday, January 25, 2011 2:42 PM
  • I came across this issue.  My resolution was to modify the ISA/TMG rules to disable the enforcement of strict RPC compliance as described in the article below

    http://www.exchangemaster.net/index.php?option=com_content&task=view&id=143&Itemid=1&lang=en


    Tony www.activedir.org Blog: www.open-a-socket.com
    Tuesday, February 15, 2011 10:06 PM
  • Hi

    I m in the exact situation ,

    I cannot understand why it works between tow servers and the thrid no !

    i have three servers runnng the exact same OS 2K8R2 SP1 , the exacte Echange 2010 SP1 ,one is in 120.100.100.0 subnet the two others are in 100.100.100.0 subnet

    i tried to set up DAG between 100 to 120  it works for two servers ,but when i try to add the third one (in 100 subnet) it gives me the DNS disjoint namespace error .

    if i try to setup a dag with this server it works but when i try to add the seconde server (100 or 120 subnet ) no one can be added !!!

    any explanation ,ideas ? help ?

    plz

    Saturday, March 12, 2011 9:24 PM
  • Hi everybody !

     

    Problem Solved ..ouf

    thanks to the luck .after two weeks on this problem , ive started from scratch and changes everything in the infrastructures but still not working

    One last thing .. THe Machine name ..

    my faulty server was called exchange i changed that to MAilserv and ... surprise everything works great :)

    it seem's that exchange is "reserved" even it"s not mentioned anywhere ..

    So change the machine name ( move mailboxes to another server including arbitration ones, unistall exchange ) and try it out ..

     

    • Proposed as answer by Nabil Zahid Monday, March 14, 2011 5:55 PM
    Monday, March 14, 2011 5:55 PM
  • Hi.

     

    I was having the same problem.

     

    I created an internal DNS entry for the server, removed the cluster feature rebooted and then adding the server to the DAG worked fine.

    • Proposed as answer by Klesw Friday, July 22, 2011 1:42 PM
    • Unproposed as answer by Klesw Friday, July 22, 2011 1:42 PM
    Thursday, April 14, 2011 3:07 PM
  • I believe this related but if not, it still is worth noting.

    In my scenario, I was able to Create the DAG object using the Shell and specifying a static IP address.  AFter that I added the first server to the DAG which created the DAG computer object successfully in the Compuers OU. I saw no issues after doing this except i noticed that the Cluster name would not come online, thus in the Failover cluster manager the Cluster name was always showing as failed, and the IP address of the CNO was failed to because an A record for the DAG computer object was not created in DNS. 

    I spoke to Microsoft for two hours and the tech found some internal article that pointed him toward the following solution.  He looked at the properties of the DAG computer object in ADSIEdit, and noticed the DNSHostname attribute had no value.  He proceeded to add the FQDN of the DAG computer object as the value for this attirbute, then replicated the change to the other DCs in the site.  After doing that we were able to bring the computer name online in the FCM and once the cluster was online an A record for it was automatically added to DNS.  We do not know what prevented the DNShostname value from being populated to begin with but I suspect it is GPO related somehow.

    Documented steps by the Tech were:

    Resolution:

    1.  Open ADSI Edit

    2.  Browse to the computer object for the cluster (CNO)

    3.  Open the properties of the CNO

    4.  Add the FQDN of the cluster to the DNSHostname attribute

    5.  Use repadmin /syncall FQDN.Of.DC to push changes to other DCs in the same site

    6.  Use Windows Failover Cluster Manager to bring the name resource online

    Friday, July 22, 2011 1:56 PM
  • i got the same problem with KVAB
    Thursday, September 13, 2012 3:17 AM
  • http://blogs.technet.com/b/timmcmic/archive/2010/01/05/exchange-2010-pre-staging-the-cluster-name-object-cno-to-support-a-database-availability-group-dag.aspx

    This Solution work for me..

    Friday, August 22, 2014 12:22 AM