locked
Help i have errors RRS feed

  • Question

  • I am receiving errors on my exchange box, all of a sudden and need some help as to which way to look to solve. Almost all of these started happening about the same time. To begin with my PDC died, and i was forced to seize the fsmo roles to the secondary domain controller. I had a copy of the dhcp scope and activated it on the "new" primary domain controller and proceeded to clean up the AD and get rid of this orphaned domain controller. Everything ran fine for a while. However then these errors started popping up. I have a feeling that the exchange server did not like this action, but i cant seem to figure out how to correct it or how to locate WHAT is the real problem. the following are the few errors i am getting, and they might not be related, understand this is a "in" production exchange server that can, after a reboot, send and rec email, but after a few hours, the incoming stops.

    - System
    - Provider
    [ Name] MSExchangeADTopology
    - EventID 2142
    [ Qualifiers] 49156
    Level 2
    Task 3
    Keywords 0x80000000000000
    - TimeCreated
    [ SystemTime] 2014-09-09T20:53:18.000000000Z
    EventRecordID 2074501
    Channel Application
    Computer CS-MAIL.$$$$$.com
    Security
    - EventData
    Microsoft.Exchange.Directory.TopologyService.exe
    3544
    $$$$$.com

    No Suitable Directory Servers Found in Forest $$$$$.com Site Default-First-Site-Name.

    and

    - System
    - Provider
    [ Name] MSExchangeDiagnostics
    - EventID 1006
    [ Qualifiers] 49156
    Level 2
    Task 2
    Keywords 0x80000000000000
    - TimeCreated
    [ SystemTime] 2014-09-09T15:21:07.000000000Z
    EventRecordID 2072666
    Channel Application
    Computer CS-MAIL.$$$$$.com
    Security
    - EventData
    \\CS-MAIL\LogicalDisk(HarddiskVolume1)\Free Megabytes
    71.00
    15
    9/9/2014 3:05:00 PM
    None
    DatabaseDriveSpaceTrigger

    harddiskvolume1

    and 

    - System
    - Provider
    [ Name] Application
    - EventID 65535
    [ Qualifiers] 0
    Level 2
    Task 0
    Keywords 0x80000000000000
    - TimeCreated
    [ SystemTime] 2014-09-09T15:32:14.000000000Z
    EventRecordID 2072715
    Channel Application
    Computer CS-MAIL.$$$$$.com
    Security
    - EventData
    <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Channels.TcpConnectionResetError.aspx</TraceIdentifier><Description>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the ............(limited to 60k characters)

    and

    - System
    - Provider
    [ Name] Application
    - EventID 65535
    [ Qualifiers] 0
    Level 2
    Task 0
    Keywords 0x80000000000000
    - TimeCreated
    [ SystemTime] 2014-09-09T15:32:14.000000000Z
    EventRecordID 2072715
    Channel Application
    Computer CS-MAIL.$$$$.com
    Security
    - EventData
    <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Channels.TcpConnectionResetError.aspx</TraceIdentifier><Description>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'. The local IP address and port is [::]:6122. The remote IP address and port is [::1]:890.</Description><AppDomain>MSExchangeFrontendTransport.exe</AppDomain><ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/MessageTraceRecord"></ExtendedData><Exception><ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'.</Message><StackTrace> at System.ServiceModel.Channels.SocketConnection.ConvertTransferException(SocketException socketException, TimeSpan timeout, Exception originalException, TransferOperation transferOperation, Boolean aborted, String timeoutErrorString, TransferOperation timeoutErrorTransferOperation, SocketConnection socketConnection, TimeSpan remainingTime) at System.ServiceModel.Channels.SocketConnection.ConvertSendException(SocketException socketException, TimeSpan remainingTime) at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout) at System.ServiceModel.Channels.BufferedConnection.WriteNow(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, BufferManager bufferManager) at System.ServiceModel.Channels.BufferedConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout) at System.ServiceModel.Channels.ConnectionStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security.NegotiateStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.NegotiateStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.NegotiateStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout) at System.ServiceModel.Channels.FramingDuplexSessionChannel.CloseOutputSessionCore(TimeSpan timeout) at System.ServiceModel.Channels.TransportDuplexSessionChannel.CloseOutputSession(TimeSpan timeout) at System.ServiceModel.Channels.TransportDuplexSessionChannel.OnClose(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.OnClose(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout) at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.ServiceModel.Channels.ServiceChannelProxy.ExecuteMessage(Object target, IMethodCallMessage methodCall) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeChannel(IMethodCallMessage methodCall) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp;amp; msgData, Int32 type) at System.ServiceModel.ICommunicationObject.Close() at Microsoft.Exchange.Net.WcfUtils.DisposeWcfClientGracefully(ICommunicationObject client, Boolean skipDispose) at Microsoft.Exchange.Net.ServiceProxyPool`1.GetClient(Boolean useCache) at Microsoft.Exchange.Net.ServiceProxyPool`1.TryCallServiceWithRetry(Action`1 action, String debugMessage, WCFConnectionStateTuple proxyToUse, Int32 numberOfRetries, Boolean doNotReturnProxyOnSuccess, Exception&amp;amp; exception) at Microsoft.Exchange.Data.Directory.ServiceTopologyProvider.GetTopologyVersions(IList`1 partitionFqdns) at Microsoft.Exchange.Data.Directory.TopologyProvider.GetTopologyVersion(String partitionFqdn) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.CheckTopologyVersionForRebuild() at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partitionFqdn, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partitionFqdn) at Microsoft.Exchange.Data.Directory.ADDataSession.GetConnection(String preferredServer, Boolean isWriteOperation, String optionalBaseDN, ADObjectId&amp;amp; rootId, ADScope scope) at Microsoft.Exchange.Data.Directory.ADDataSession.GetReadConnection(String preferredServer, String optionalBaseDN, ADObjectId&amp;amp; rootId, ADRawEntry scopeDeteriminingObject, DualSearchMode dualSearchMode) at Microsoft.Exchange.Data.Directory.ADGenericReader.GetNextResultCollection(Type controlType, DirectoryControl&amp;amp; responseControl) at Microsoft.Exchange.Data.Directory.ADPagedReader`1.GetNextResultCollection() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.GetNextPage() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.&amp;lt;GetEnumerator&amp;gt;d__0.MoveNext() at Microsoft.Exchange.Data.Directory.ADDataSession.ReadBatch[TKey,TData](TKey[] keys, ADObjectId rootId, Converter`2 filterBuilder, QueryFilter commonFilter, HashInserter`1 hashInserter, HashLookup`2 hashLookup, IEnumerable`1 properties, Boolean includeDeletedObjects, List`1 results) at Microsoft.Exchange.Data.Directory.ADDataSession.ReadMultiple[TKey,TData](TKey[] keys, ADObjectId rootId, Converter`2 filterBuilder, QueryFilter commonFilter, HashInserter`1 hashInserter, HashLookup`2 hashLookup, IEnumerable`1 properties, Boolean includeDeletedObjects) at Microsoft.Exchange.Data.Directory.ADDataSession.ReadMultiple[TKey,TData](TKey[] keys, Converter`2 filterBuilder, HashInserter`1 hashInserter, HashLookup`2 hashLookup, IEnumerable`1 properties, Boolean includeDeletedObjects) at Microsoft.Exchange.Data.Directory.ADDataSession.ReadMultiple[TKey,TData](TKey[] keys, Converter`2 filterBuilder, HashInserter`1 hashInserter, HashLookup`2 hashLookup, IEnumerable`1 properties) at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientObjectSession.FindByProxyAddresses[TData](ProxyAddress[] keys, IEnumerable`1 properties) at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache`1.&amp;lt;&amp;gt;c__DisplayClass16.&amp;lt;FindAndCacheRecipients&amp;gt;b__13() at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.RunADOperation(ADOperation adOperation, Int32 retryCount) at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache`1.FindAndCacheRecipients(IList`1 proxyAddressList) at Microsoft.Exchange.Transport.Categorizer.ProxyHubSelector.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;GetRecipientDatabases&amp;gt;b__0() at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.RunADOperation(ADOperation adOperation, Int32 retryCount) at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.TryRunADOperation(ADOperation adOperation, Int32 retryCount) at Microsoft.Exchange.Transport.Categorizer.ProxyHubSelector.GetRecipientDatabases(IReadOnlyMailItem mailItem) at Microsoft.Exchange.Transport.Categorizer.ProxyHubSelector.TrySelectHubServers(IReadOnlyMailItem mailItem, IEnumerable`1&amp;amp; hubServers) at Microsoft.Exchange.Transport.FrontendProxyRoutingAgent.FrontendProxyAgent.OnProxyInboundMessageEventHandler(ProxyInboundMessageEventSource source, ProxyInboundMessageEventArgs e) at Microsoft.Exchange.Data.Transport.Smtp.SmtpReceiveAgent.Invoke(String eventTopic, Object source, Object e) at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session) at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.Invoke() at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state) at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.BeginInvoke(String topic, Object source, Object e, AsyncCallback callback, Object callbackState) at Microsoft.Exchange.Protocols.Smtp.DataInboundProxySmtpCommand.RaiseProxyInboundMessageEvent(ISmtpInSession session, AsyncCallback callback) at Microsoft.Exchange.Protocols.Smtp.DataInboundProxySmtpCommand.ContinueEndOfHeaders(IAsyncResult ar) at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExAsyncResult.InvokeCompleted() at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state) at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.BeginInvoke(String topic, Object source, Object e, AsyncCallback callback, Object callbackState) at Microsoft.Exchange.Protocols.Smtp.BaseDataSmtpCommand.RaiseEOHEvent(Object state, ISmtpInSession session, AsyncCallback callback, EndOfHeadersEventArgs eventArgs) at Microsoft.Exchange.Protocols.Smtp.DataInboundProxySmtpCommand.RawDataReceived(Byte[] data, Int32 offset, Int32 numBytes) at Microsoft.Exchange.Protocols.Smtp.SmtpInSession.ReadComplete(IAsyncResult asyncResult) at Transport.AgentFactories.FrontendAgent.FrontendAgentFactory.ReadComplete(IAsyncResult asyncResult) at Microsoft.Exchange.Net.LazyAsyncResult.WorkerThreadComplete(Object state) at Microsoft.Exchange.Net.LazyAsyncResult.Complete(IntPtr userToken) at Microsoft.Exchange.Net.LazyAsyncResult.ProtectedInvokeCallback(Object value, IntPtr userToken) at Microsoft.Exchange.Net.NetworkConnection.InvokeRecvCallback(LazyAsyncResultWithTimeout asyncResult, Object result) at Microsoft.Exchange.Net.NetworkConnection.ReadDataAvailable(IAsyncResult asyncResult) at System.Net.LazyAsyncResult.Complete(IntPtr userToken) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.ContextAwareResult.Complete(IntPtr userToken) at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped) at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP) </StackTrace><ExceptionString>System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'. ---&amp;gt; System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout) --- End of inner exception stack trace ---</ExceptionString><InnerException><ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>An existing connection was forcibly closed by the remote host</Message><StackTrace> at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)</StackTrace><ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)</ExceptionString><NativeErrorCode>2746</NativeErrorCode></InnerException></Exception></TraceRecord>

    - System
    - Provider
    [ Name] MSExchange ADAccess
    - EventID 2454
    [ Qualifiers] 49156
    Level 2
    Task 9
    Keywords 0x80000000000000
    - TimeCreated
    [ SystemTime] 2014-09-09T20:53:20.000000000Z
    EventRecordID 2074712
    Channel Application
    Computer CS-MAIL.$$$$$.com
    Security
    - EventData
    w3wp.exe (AutoDisc)
    8260
    Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Domain Controller. at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, String partitionFqdn, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.ADNotificationListener.TryIssueNotificationRequests(List`1 notificationRequest) at Microsoft.Exchange.Data.Directory.ADNotificationListener.ReissueNotificationRequests(Boolean throwOnFailure, Boolean reissueIfListening) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.<>c__DisplayClass8.<RebuildNonPartitionScopedConnectionPoolManager>b__6()

    Like i said, i assume that the exchange server cant "talk" to the domain controller. (yet AD is on the exchange server), how do i FIX this "while in production"?

    Any help in what direction to look to solve this will be greatly appreciated. 


    "Once you have eliminated the impossible, whatever remains, however improbable, must be the truth." ~Spock

    Tuesday, September 9, 2014 9:27 PM

All replies

  • In the TCP/IP properties on the Exchange server network card, what servers are designated as DNS servers?

    When you said you cleaned up AD, could you elaborate?

    Usually, there is the first step, rather well documented, where you use ntdsutil to remove the defunct domain controller. Even if that is performed correctly, there can still be references to the old domain controller, notably in the DNS zones (and also in Active Directory Sites and Services). Did you double-check these locations for references to the removed DC? In my experience, these references are never removed automatically and have to be removed manually.

    By the way, what version of Exchange do you have?

    If it's 2007 or 2010, you could run the Exchange Best Practice Analyzer. Does that list any configuration errors?

    That's what I would check, to start with. 


    Please mark as helpful if you find my contribution useful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Tuesday, September 9, 2014 11:10 PM
  • I think you can still get the 2003 ExBPA here if you need it:

    http://www.microsoft.com/en-us/download/details.aspx?id=22485

    Running DCDIAG to check out the DC wouldn't be a bad idea, nor would a quick "repadmin /replsummary".

    Also, have a look at AD Sites and Services and see if it all looks okay.

    Running Exchange on a DC is hardly ever a good idea.


    --- Rich Matheisen MCSE&I, Exchange MVP

    Wednesday, September 10, 2014 4:31 PM
  • I corrected the issue, well at least the main problem. I know this is going to sound weird, but had to due with an update to the exchange server. (surprise, surprise). 

    About a year ago i set this exchange up and needed to add a receive connector. (for some older printers/copiers that scan to email.) i set up and internal rec connector based on ip list. A couple of months later, the printers/copiers are upgrade, policies are changed, scan to email is stopped, ........although the new printer/copiers retain their static ip and the connector is not removed. 

    Apparently, the update is installed. (we had a major power outage that drained our UPSs and shut down the servers.) this started the update process. (which is something i now have insured does not happen again, which is another story). When the machines rebooted, my PDC was broken but the exchange server seemed fine.

    That exchange update "BROKE" something involving my transport service. about every 6 hours the server would shut off. I found an article online that mentioned, in passing, something about all send / rec connectors that were NOT default maybe being an issue and removing them.

    Apparently that was it. I went into to my exchange box and removed the custom rec connector and no more problem. I do not understand the MSExchange ADAccess / w3wp.exe error, but it no longer is present.

    I still have 

    System
    - Provider
    Name] MSExchangeDiagnostics
    - EventID 1006
    Qualifiers] 49156
    Level 2
    Task 2
    Keywords 0x80000000000000
    - TimeCreated
    SystemTime] 2014-09-09T15:21:07.000000000Z
    EventRecordID 2072666
    Channel Application
    Computer CS-MAIL.$$$$$.com
    Security
    - EventData
    \\CS-MAIL\LogicalDisk(HarddiskVolume1)\Free Megabytes
    71.00
    15
    9/9/2014 3:05:00 PM
    None
    DatabaseDriveSpaceTrigger

    harddiskvolume


    But will look for that in another thread.. 


    "Once you have eliminated the impossible, whatever remains, however improbable, must be the truth." ~Spock


    • Edited by Leehweht Monday, September 15, 2014 1:48 PM
    Monday, September 15, 2014 1:47 PM