Sign in
 
HomeOnline201020072003LibraryForumsPartnersEHLO Blog
Exchange Server TechCenter > Exchange Server 2010 Forums > Exchange Server 2010 > Problem while adding DAG members

Proposed Answer Problem while adding DAG members

    •  
      Wednesday, February 03, 2010 10:09 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Dear all ..

      I have a newly installed 2 Windows 2008 R2 DCs and 3 Exchange 2010 servers running on win2k8R2 (1 cas and 2 mailbox)

      I've created DAG and assigned a static IP address for it.

      When I tried to add the servers to the DAG, it fails with this error: server-side database availability group administrative operation failed. Error: The operation failed with message: An error occurred while attempting a cluster operation. Error: Cluster API '"CreateCluster() failed with 0x4d0. Error: The network location cannot be reached. For information about network troubleshooting, see Windows Help"' failed. [Server: *SERVERNAME*] .............

      Any advice ??

      Please note that both servers are identical with 4 NICs, but I disabled 3 NICs and use only one NIC from each server connected to the core switch

       

    All Replies

    • Thursday, February 04, 2010 4:37 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Any help ???
       
    • Thursday, February 04, 2010 4:40 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Hi,

      Is the Failover-Clustering feature installed on the two mailbox servers already? Is the machine you're trying to create the DAG from one of those mailbox servers and if it isn't, are the failover clustering tools installed locally? Any chance you could do a network capture and look to see what IP & Port the box tries to contact the desired DAG node on?
      Brian Day, Overall Exchange & AD Geek
      MCSA 2000/2003, CCNA
      MCTS: Microsoft Enterprise Server 2010, Configuration
      MCITP: Enterprise Messaging Administrator 2010
      LMNOP
       
    • Thursday, February 04, 2010 5:06 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Dear Brian,

      Yes, Failover-Clustering feature installed on the two mailbox servers already. And I'm creating DAG from one of those mailbox servers.

      How to do a network capture to see IP and port for DAG ?

      I noticed that Cluster service is disabled, I dont know if that's ok.

      Regards,
      Ali
       
    • Friday, February 05, 2010 6:52 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Any help guys ???
       
    • Friday, February 05, 2010 5:53 PM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Since the Failover Clustering feature is install, but the server isn't a member of a cluster, having the cluster service disabled is the right thing.
      I presume this is happening when trying to add the first server to the Dag.
      When the first server to the Dag gets created it creates a Windows Failover Cluster. The static IP's assigned to the Dag are for the cluster. Are the static IP's on the same subnet as the servers?
       
    • Sunday, February 07, 2010 4:40 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Yes, the static IP's on the same subnet as the servers ...
       
    • Monday, February 08, 2010 10:44 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      ??????
       
    • Thursday, February 11, 2010 5:51 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      This sounds vaguely familiar. But the case I'm thinking of involves joining a node to an existing cluster. The join operation creates a DCOM object on one server in order to look up the name of the other server, and if this fails (sometimes due to authentication problems) the error that's bubbled up is something like the above.

      But you mentioned this is done during CreateCluster(). So this is different.

      It would be very helpful if you could post the dagtask log content. The part I'm particularly interested in is the bit with the progress -- knowing which stage fails with ERROR_HOST_UNREACHABLE should help with the analysis. But please post the entire log, since there may be a clue elsewhere. If you are concerned about posting machine names and domain names in a public forum, you can do a search&replace first, but please keep the names unique.

      -martin
      Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
       
    • Sunday, February 14, 2010 5:18 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Dear Martin,

      The log file is below: the same error while adding any of the two servers:

      add-databaseavailabiltygroupserver started on machine MBX1. [2010-02-14T05:08:06] add-dagserver started [2010-02-14T05:08:06] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters } [2010-02-14T05:08:06] Option 'Identity' = 'DAG1'. [2010-02-14T05:08:06] Option 'MailboxServer' = 'MBX1'. [2010-02-14T05:08:06] Option 'DatabaseAvailabilityGroupIpAddresses' = ''. [2010-02-14T05:08:06] Option 'WhatIf' = ''. [2010-02-14T05:08:06] Process: w3wp w3wp.exe:3044. [2010-02-14T05:08:06] User context = 'NT AUTHORITY\SYSTEM'. [2010-02-14T05:08:06] Member of group 'Everyone'. [2010-02-14T05:08:06] Member of group 'BUILTIN\Users'. [2010-02-14T05:08:06] Member of group 'NT AUTHORITY\SERVICE'. [2010-02-14T05:08:06] Member of group 'CONSOLE LOGON'. [2010-02-14T05:08:06] Member of group 'NT AUTHORITY\Authenticated Users'. [2010-02-14T05:08:06] Member of group 'NT AUTHORITY\This Organization'. [2010-02-14T05:08:06] Member of group 'BUILTIN\IIS_IUSRS'. [2010-02-14T05:08:06] Member of group 'LOCAL'. [2010-02-14T05:08:06] Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'. [2010-02-14T05:08:06] Member of group 'BUILTIN\Administrators'. [2010-02-14T05:08:06] Updated Progress 'Validating the parameters.' 2%. [2010-02-14T05:08:06] Working [2010-02-14T05:08:06] Mailbox server: value passed in = MBX1, mailboxServer.Name = MBX1, mailboxServer.Fqdn = MBX1.MYDOMAIN.COM [2010-02-14T05:08:06] LogClussvcState: clussvc is Stopped on MBX1.MYDOMAIN.COM. Exception (if any) = none [2010-02-14T05:08:06] The IP addresses for the DAG are (blank means DHCP): 10.10.10.65 [2010-02-14T05:08:06] Looking up IP addresses for DAG1. [2010-02-14T05:08:13] Failure while trying to resolve DAG1: threw a SocketException: The requested name is valid, but no data of the requested type was found. [2010-02-14T05:08:13] Looking up IP addresses for MBX1. [2010-02-14T05:08:13] MBX1 = [ ::1, 10.10.10.62 ]. [2010-02-14T05:08:13] Looking up IP addresses for MBX1.MYDOMAIN.COM. [2010-02-14T05:08:13] MBX1.MYDOMAIN.COM = [ ::1, 10.10.10.62 ]. [2010-02-14T05:08:13] DAG DAG1 has 0 servers: [2010-02-14T05:08:13] Updated Progress 'Checking if Mailbox server 'MBX1' is in a database availability group.' 4%. [2010-02-14T05:08:13] Working [2010-02-14T05:08:13] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(MBX1.MYDOMAIN.COM) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK. [2010-02-14T05:08:13] Ignoring previous error, as it is acceptable if the cluster does not exist yet. [2010-02-14T05:08:13] DumpClusterTopology: Opening remote cluster DAG1. [2010-02-14T05:08:13] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmGetFqdnFailedNotFoundException: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: The fully qualified domain name for node 'DAG1' could not be found. at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.<>c__DisplayClass1.<ResolveFqdn>b__0(Object , EventArgs ) at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.RunADOperationEx(EventHandler ev) at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.ResolveFqdn(String shortNodeName) at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.GetFqdn(String shortNodeName) at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerName..ctor(String serverName) at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output) [2010-02-14T05:08:13] Dumping the cluster by connecting to: DAG1. [2010-02-14T05:08:13] DumpClusterTopology: Not dumping, since the cluster could not be contacted. [2010-02-14T05:08:13] The new server (MBX1) is not a member of a cluster, nor are the other servers (if there are any other servers). [2010-02-14T05:08:13] The computer account DAG1 does not exist. [2010-02-14T05:08:13] According to GetNodeClusterState(), the server MBX1 is NotConfigured. [2010-02-14T05:08:13] InternalValidate() done. [2010-02-14T05:08:13] Updated Progress 'Adding server 'MBX1' to database availability group 'DAG1'.' 6%. [2010-02-14T05:08:13] Working [2010-02-14T05:08:13] Connecting to server 'MBX1.MYDOMAIN.COM' via WMI... [2010-02-14T05:08:13] Fetching the network adapters and including the ones without default gateways. [2010-02-14T05:08:13] MBX1.MYDOMAIN.COM has an address: 10.10.10.62/24 default gateway(s)=10.10.10.250 [valid] [2010-02-14T05:08:13] IP address derived that's suitable for clustering DHCP/IPv6: 10.10.10.0/24. [2010-02-14T05:08:13] The address 10.10.10.65 falls under the network (10.10.10.0/24). [2010-02-14T05:08:13] Updated Progress 'Forming cluster named 'DAG1' on server 'MBX1'.' 8%. [2010-02-14T05:08:13] Working [2010-02-14T05:08:13] Forming cluster named 'DAG1' on server 'MBX1'. The cluster address is/are [10.10.10.65], and the cluster network prefix length is/are [24]. [2010-02-14T05:08:13] The following log entry comes from a different process, run on machine 'MBX1'. BEGIN [2010-02-14T05:08:13] [2010-02-14T05:08:13] Updated Progress 'Forming cluster named 'DAG1' on server 'MBX1'.' 2%. [2010-02-14T05:08:13] Working [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = DAG1, dwStatus = 0x0 ) [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = , dwStatus = 0x0 ) [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = MBX1.MYDOMAIN.COM, dwStatus = 0x0 ) [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = , dwStatus = 0x0 ) [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 10, szObjectName = MBX1.MYDOMAIN.COM, dwStatus = 0x4d0 ) [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = MBX1.MYDOMAIN.COM, dwStatus = 0x0 ) [2010-02-14T05:08:13] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseWarning, dwPercentComplete = 10, szObjectName = MBX1.MYDOMAIN.COM, dwStatus = 0xc0000061 ) [2010-02-14T05:08:13] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2010-02-14_05-08-06.276_add-databaseavailabiltygroupserver.log". [2010-02-14T05:08:13] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: An error occurred while attempting a cluster operation. Error: Cluster API '"CreateCluster() failed with 0x4d0. Error: The network location cannot be reached. For information about network troubleshooting, see Windows Help"' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"CreateCluster() failed with 0x4d0. Error: The network location cannot be reached. For information about network troubleshooting, see Windows Help"' failed. ---> System.ComponentModel.Win32Exception: The network location cannot be reached. For information about network troubleshooting, see Windows Help --- End of inner exception stack trace --- at Microsoft.Exchange.Cluster.ClusApi.AmCluster.CreateExchangeCluster(String clusterName, AmServerName firstNodeName, String[] ipAddress, UInt32[] ipPrefixLength, IClusterSetupProgress setupProgress, IntPtr context, Exception& failureException, Boolean throwExceptionOnFailure) at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog) --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) --- at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception) at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog) at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass2e.<RpcsCreateCluster>b__2d() at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation) --- End of stack trace on server (MBX1.MYDOMAIN.COM) --- at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo) at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation) at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation) at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunCreateCluster(AmServerName rpcServerName, String clusterName, AmServerName firstNode, String[] ipaddrs, UInt32[] netmasks, String& verboseLog) at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.FormCluster() [2010-02-14T05:08:13] Updated Progress 'Done!' 100%. [2010-02-14T05:08:13] COMPLETED add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().
       
    • Tuesday, February 16, 2010 2:06 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Thanks for the log. Unfortunately I don't have any good ideas on why it failed. It might be that the DCOM object couldn't be created.

      Try using the cluadmin tools to form a cluster with the name 'DAG1' and see if it succeeds. I think it will fail the same way (it uses the same underlying code :), but the cluadmin tools may have a better (i.e. actionable) error message.

      And please reply with the results, regardless of whether it's successful or not.

      -martin
      Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
       
    • Tuesday, February 16, 2010 6:14 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote

      Hi Martin,

      It failed with this error:

      An error occurred while creating the cluster.
      An error occurred creating cluster 'DAG1'.
      The network location cannot be reached. For information about network troubleshooting, see Windows Help

       
    • Wednesday, February 17, 2010 1:20 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      The good news is that it doesn't look like an Exchange bug, since it also fails when using the Windows tools.
      The bad news is that I don't know what's going on.

      There's also a forum you could try: http://social.technet.microsoft.com/Forums/en-US/winserverClustering , but of course if you post there you'll have better luck with a response if you've done your homework and run the appropriate diagnostic tools first and post the appropriate logs (and sorry, I'm not sure what they would be).
      There should be lots of documentation on MSDN/Technet with creating a cluster and troubleshooting.

      Good luck!

      -martin
      Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
       
    • Wednesday, February 17, 2010 2:53 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote

      Just curious if where the FSW is located. I have seen something similar when trying to add the server to the DAG, but the FSW was not reachible or permissions were messed up on the FSW. and this happens more often when the FSW is not on another exchange server.

      just a thought.

      Mitch Roberson |MCITP:Enterprise Server Admin, Messaging |MCTS:OCS with Voice Achievement |MCT |MCSE 2000\2003 |MCSE Messaging 2000\2003
       
    • Wednesday, February 17, 2010 10:00 AM
       
       
      Sign In to Vote
      0
      Sign In to Vote
      Mitch: the FSW should only be relevant when adding the second server to the DAG (more precisely: when the number of servers becomes even). alibah is having is having adding the first server, which is when the cluster gets formed.

      Although we do test out the FSW at new-dag time, this is only a test and it should show up as a Warning instead of an Error.

      -martin
      Legal Stuff: This posting is provided "AS IS" with no warranties, and confers no rights.
       
    • Wednesday, February 17, 2010 10:56 PM
       
       Proposed Answer
      Sign In to Vote
      0
      Sign In to Vote
      I have seen it fail in adding the server if permissions are not correct for the FSW. Matter of fact it happens in one of the online labs. So if it cannot connect to the FSW share it seems to fail when you try to add the Server to the DAG. this only happens if you bypass the initial warning when setting up the DAG and do not pay attention to the warning's.

      Mitch Roberson |MCITP:Enterprise Server Admin, Messaging |MCTS:OCS with Voice Achievement |MCT |MCSE 2000\2003 |MCSE Messaging 2000\2003
      • Proposed As Answer by navarrobl Friday, July 09, 2010 10:53 PM
      •  
       
    • Friday, July 09, 2010 10:57 PM
       
       Proposed Answer
      Sign In to Vote
      0
      Sign In to Vote

      Hello, we had the same problem (Error: Cluster API '"CreateCluster() failed with 0x4d0.), and the cause was that the administrative shares were off in each Mailbox server.

      Luis Navarro.

      • Proposed As Answer by navarrobl Friday, July 09, 2010 10:57 PM
      •