This forum has migrated to Microsoft Q&A. Visit Microsoft Q&A to post new questions.

Remove From My Forums

Error when adding MBX Server to DAG

Question
0

Sign in to vote

Hello,

I'm running into the following eror when trying to add first member of an Exchange 2010 DAG:

[PS] C:\Windows\system32>Add-DatabaseAvailabilityGroupServer -Identity DAG1 -MailboxServer SERVER1

WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file

"C:\ExchangeSetupLogs\DagTasks\dagtask_2010-08-18_21-43-16.169_add-databaseavailabiltygroupserver.log".

A server-side database availability group administrative operation failed. Error: The operation failed with message: A

　error occurred while attempting a cluster operation. Error: Cluster API '"CreateCluster() failed with 0x35. Error: Th

　network path was not found"' failed. [Server: Server1.domain.corp]

　　　 + CategoryInfo: InvalidArgument: (:) [Add-DatabaseAvailabilityGroupServer], DagTaskOperationFailedException

　　　 + FullyQualifiedErrorId : 67131516,Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer

AD and DNS are working optimally and i've disabled all OS firewalls and AV software. Any ideas on what might be causing this?

Thanks

Thursday, August 19, 2010 5:47 AM

All replies

0

Sign in to vote

Hi,

The operations get executed in w3wp.exe, which is running as the machine account, and not you.

So you'll need to open Local Powershell and run:
add-databaseavailabiltygroupserver labdag1 labmbx03
and that should work. If it doesn't, please let us know what the new error code is.

By the way, please try to avoid using the Failover clustering GUI. While you may need to use it with the Beta bits, we're trying hard so that you should never have to do anything with it.

I hope this will help you.

Regards.

Shafaquat Ali.
M.C.I.T.P Exchange 2007/2010, M.C.I.T.P Windows Server 2008, M.C.T.S OCS Server 2007 R2, URL: http://blog.WhatDoUC.net Phone: +923008210320

Thursday, August 19, 2010 7:26 AM
0

Sign in to vote

This operation has nothing to do with w3wp.exe first of all. Can you please post the log entries from add-databaseavailabiltygroupserver.log ???

Additionally, are you trying to do it in a virtualized environment? If yes and you have cloned the disks then you must sysprep the operating system before it can be a part of the DAG.
Milind Naphade | MCTS:M (Exchange 2007 and 2010) | http://www.msexchangegeek.com

Thursday, August 19, 2010 8:19 AM
0

Sign in to vote

I'm doing everything through powershell. Here's are the logs from the dagtask log files:

add-databaseavailabiltygroupserver started on machine SERVER1.
[2010-08-18T21:27:21] add-dagserver started
[2010-08-18T21:27:21] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2010-08-18T21:27:21] Option 'Identity' = 'DAG1'.
[2010-08-18T21:27:21] Option 'MailboxServer' = 'SERVER1'.
[2010-08-18T21:27:21] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2010-08-18T21:27:21] Option 'WhatIf' = ''.
[2010-08-18T21:27:21] Process: w3wp w3wp.exe:680.
[2010-08-18T21:27:21] User context = 'NT AUTHORITY\SYSTEM'.
[2010-08-18T21:27:21]   Member of group 'Everyone'.
[2010-08-18T21:27:21]   Member of group 'BUILTIN\Users'.
[2010-08-18T21:27:21]   Member of group 'NT AUTHORITY\SERVICE'.
[2010-08-18T21:27:21]   Member of group 'CONSOLE LOGON'.
[2010-08-18T21:27:21]   Member of group 'NT AUTHORITY\Authenticated Users'.
[2010-08-18T21:27:21]   Member of group 'NT AUTHORITY\This Organization'.
[2010-08-18T21:27:21]   Member of group 'BUILTIN\IIS_IUSRS'.
[2010-08-18T21:27:21]   Member of group 'LOCAL'.
[2010-08-18T21:27:21]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
[2010-08-18T21:27:21]   Member of group 'BUILTIN\Administrators'.
[2010-08-18T21:27:21] Updated Progress 'Validating the parameters.' 2%.
[2010-08-18T21:27:21] Working
[2010-08-18T21:27:21] Mailbox server: value passed in = SERVER1, mailboxServer.Name = SERVER1, mailboxServer.Fqdn = SERVER1.domain.corp
[2010-08-18T21:27:21] LogClussvcState: clussvc is Stopped on SERVER1.domain.corp. Exception (if any) = none
[2010-08-18T21:27:21] The IP addresses for the DAG are (blank means DHCP): 192.168.35.100,192.168.2.102
[2010-08-18T21:27:21] Looking up IP addresses for DAG1.
[2010-08-18T21:27:24] Failure while trying to resolve DAG1: threw a SocketException: The requested name is valid, but no data of the requested type was found.
[2010-08-18T21:27:24] Looking up IP addresses for SERVER1.
[2010-08-18T21:27:24]   SERVER1 = [ fe80::105a:66ed:56d1:3456%15, 192.168.6.46, 192.168.2.100 ].
[2010-08-18T21:27:24] Looking up IP addresses for SERVER1.domain.corp.
[2010-08-18T21:27:24]   SERVER1.domain.corp = [ fe80::105a:66ed:56d1:3456%15, 192.168.6.46, 192.168.2.100 ].
[2010-08-18T21:27:24] DAG DAG1 has 0 servers:
[2010-08-18T21:27:24] Updated Progress 'Checking if Mailbox server 'SERVER1' is in a database availability group.' 4%.
[2010-08-18T21:27:24] Working
[2010-08-18T21:27:24] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(SERVER1.domain.corp) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK.
[2010-08-18T21:27:24] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2010-08-18T21:27:24] DumpClusterTopology: Opening remote cluster DAG1.
[2010-08-18T21:27:24] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmGetFqdnFailedNotFoundException: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: The fully qualified domain name for node 'DAG1' could not be found.
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.<>c__DisplayClass1.<ResolveFqdn>b__0(Object , EventArgs )
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.RunADOperationEx(EventHandler ev)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.ResolveFqdn(String shortNodeName)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.GetFqdn(String shortNodeName)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerName..ctor(String serverName)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)
[2010-08-18T21:27:24] Dumping the cluster by connecting to: DAG1.
[2010-08-18T21:27:24] DumpClusterTopology: Not dumping, since the cluster could not be contacted.
[2010-08-18T21:27:24] DumpClusterTopology: Opening local cluster.
[2010-08-18T21:27:24] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(SERVER1.domain.corp) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed. ---> System.ComponentModel.Win32Exception: There are no more endpoints available from the endpoint mapper
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByName(AmServerName serverName)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)
[2010-08-18T21:27:24] Dumping the cluster by connecting to: .
[2010-08-18T21:27:24] DumpClusterTopology: Not dumping, since the cluster could not be contacted.
[2010-08-18T21:27:24] The new server (SERVER1) is not a member of a cluster, nor are the other servers (if there are any other servers).
[2010-08-18T21:27:24] The computer account DAG1 does not exist.
[2010-08-18T21:27:24] GetNodeClusterState( SERVER1 ) failed with 53. Continuing anyway (assuming clustering is still installed though).
[2010-08-18T21:27:24] InternalValidate() done.
[2010-08-18T21:27:24] Updated Progress 'Adding server 'SERVER1' to database availability group 'DAG1'.' 6%.
[2010-08-18T21:27:24] Working
[2010-08-18T21:27:24] Connecting to server 'SERVER1.domain.corp' via WMI...
[2010-08-18T21:27:24] Fetching the network adapters and including the ones without default gateways.
[2010-08-18T21:27:24] SERVER1.domain.corp has an address: 192.168.6.46/24 default gateway(s)=192.168.6.1 [valid]
[2010-08-18T21:27:24] IP address derived that's suitable for clustering DHCP/IPv6: 192.168.6.0/24.
[2010-08-18T21:27:24] SERVER1.domain.corp has an address: fe80::105a:66ed:56d1:3456/64 default gateway(s)=192.168.6.1 [invalid]
[2010-08-18T21:27:24] SERVER1.domain.corp has an address: 192.168.2.100/24 default gateway(s)=<none> [valid]
[2010-08-18T21:27:24] IP address derived that's suitable for clustering DHCP/IPv6: 192.168.2.0/24.
[2010-08-18T21:27:24] Could not find any suitable networks for the address 192.168.35.100!
[2010-08-18T21:27:24] Ignoring IP address '192.168.35.100'. Can't find a matching network on server SERVER1.
[2010-08-18T21:27:24] The address 192.168.2.102 falls under the network (192.168.2.0/24).
[2010-08-18T21:27:24] Updated Progress 'Forming cluster named 'DAG1' on server 'SERVER1'.' 8%.
[2010-08-18T21:27:24] Working
[2010-08-18T21:27:24] Forming cluster named 'DAG1' on server 'SERVER1'. The cluster address is/are [192.168.2.102], and the cluster network prefix length is/are [24].
[2010-08-18T21:27:24] The following log entry comes from a different process, run on machine 'SERVER1'. BEGIN
[2010-08-18T21:27:24] [2010-08-18T21:27:24] Updated Progress 'Forming cluster named 'DAG1' on server 'SERVER1'.' 2%.
[2010-08-18T21:27:24] Working
[2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = DAG1, dwStatus = 0x0 )
[2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = , dwStatus = 0x0 )
[2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = SERVER1.domain.corp, dwStatus = 0x0 )
[2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseContinue, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 10, szObjectName = SERVER1.domain.corp, dwStatus = 0x35 )
[2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 10, szObjectName = SERVER1.domain.corp, dwStatus = 0x35 )

[2010-08-18T21:27:24] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2010-08-18_21-27-21.899_add-databaseavailabiltygroupserver.log".
[2010-08-18T21:27:24] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: An error occurred while attempting a cluster operation. Error: Cluster API '"CreateCluster() failed with 0x35. Error: The network path was not found"' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"CreateCluster() failed with 0x35. Error: The network path was not found"' failed. ---> System.ComponentModel.Win32Exception: The network path was not found
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Cluster.ClusApi.AmCluster.CreateExchangeCluster(String clusterName, AmServerName firstNodeName, String[] ipAddress, UInt32[] ipPrefixLength, IClusterSetupProgress setupProgress, IntPtr context, Exception& failureException, Boolean throwExceptionOnFailure)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog)
   --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) ---
   at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog)
   at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass2e.<RpcsCreateCluster>b__2d()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
   --- End of stack trace on server (SERVER1.domain.corp) ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunCreateCluster(AmServerName rpcServerName, String clusterName, AmServerName firstNode, String[] ipaddrs, UInt32[] netmasks, String& verboseLog)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.FormCluster()
[2010-08-18T21:27:24] Updated Progress 'Done!' 100%.
[2010-08-18T21:27:24] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

Thursday, August 19, 2010 3:47 PM
0

Sign in to vote

I just had this exact problem. The EX2K10 servers are running Windows 2008 Enterprise R2. If a dedicated replication network has been set up with static routes between each member server, you actually need to specify internal DNS servers that accept dynamic updates for that network interface as the DAG name needs to be registered. I was able to get beyond this step after taking the following action:

- Stop Cluster Service and set to Disabled for each member server.

- Add internal DNS servers to DAG interface on each server, and specify DNS suffix.

- Change binding order to ensure Public/Primary interface is at the top.

- Unselect all bindings for IPv4/IPv6 stacks on DAG interface.

Thursday, August 19, 2010 6:21 PM
0

Sign in to vote

Hi,

I have seen it fail in adding the server if permissions are not correct for the FSW. Matter of fact it happens in one of the online labs. So if it cannot connect to the FSW share it seems to fail when you try to add the Server to the DAG. this only happens if you bypass the initial warning when setting up the DAG and do not pay attention to the warning's.

Also check that your Administrative shares are enable for each mailbox server.

Regards.

Shafaquat Ali.
M.C.I.T.P Exchange 2007/2010, M.C.I.T.P Windows Server 2008, M.C.T.S OCS Server 2007 R2, URL: http://blog.WhatDoUC.net Phone: +923008210320

Thursday, August 19, 2010 6:23 PM
0

Sign in to vote

From the Logs below it seems to be a network issue.

2010-08-18T21:27:24] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(SERVER1.domain.corp) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed. ---> System.ComponentModel.Win32Exception: There are no more endpoints available from the endpoint mapper

How are the nodes of DAG connected to each other? VPN..etc?

Have you tried adding the nodes manually using Failover Cluster Manager?

Thursday, August 19, 2010 10:04 PM
0

Sign in to vote

You go the point Prakash :-) (I hope I am not making any mistake)

Okay, so error code 0x6d9 translates into EPT_S_NOT_REGISTERED and error code 0x35 into ERROR_BAD_NETPATH. According to this error it is a RPC endpoint unavailability problem which may also be a biproduct of network failures between DAG nodes. In this case I would run a netdiag on the servers in question and see if it returns any errors.

Make sure the server where you have specified your FSW (HT server in your case as you have not specified any specific server) is accessible from all DAG members. Can you try comparing SIDs of both servers with psgetsid.exe from sysinternals if possible?

Milind Naphade | MCTS:M (Exchange 2007 and 2010) | http://www.msexchangegeek.com

Friday, August 20, 2010 3:57 AM
0

Sign in to vote

In your log, I didn’t see the task for the Windows Failover Clustering component installation. Did you install it before on the SERVER1?

Please check the settings of the NIC, incorrect configuration will cause the similar issue

“Network Adapter Configuration for DAGs” section in Planning for High Availability and Site Resilience
James Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com

Monday, August 23, 2010 9:09 AM
0

Sign in to vote

How's the issue currently?
James Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com

Thursday, August 26, 2010 8:18 AM
0

Sign in to vote
i guess i know whats going on with this issue:

If you have a Site To Site VPN configured check the Rule if its working as its suppose to. This is what i had to do in one of the instance

Checkpoint R71, NAT within VPN communities should work by default, however had to push it manually. For that we will have to create a RULE/Configuration and push it through CheckPoint.

Let me know if it helps!!
- Proposed as answer by prakp [MSFT] Tuesday, March 27, 2012 10:03 AM
Sunday, August 29, 2010 3:14 AM