none
Exchange 2013 faces a set of problem:Event 10028,4027,1015 .etc; Many Core services stopped and could not be start. RRS feed

  • Question

  • I had a set of 2013 DAG Exchange servers with windows 2012 OS in virtual environment( VMware vCenter ):

    A Domain controller, two mailboxes, a client access.

    Recently when I tried to access https://<CASServerName>/ecp I got the following error page:

    Then I  found some errors in Event Viewer:

    Log Name:      System
    Source:        Microsoft-Windows-DistributedCOM
    Date:          4/21/2014 2:43:10 PM
    Event ID:      10028
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          SYSTEM
    Computer:      zhuvm-fog-2928.qa2013dag.local
    Description:
    DCOM was unable to communicate with the computer ZHUVM-FOG-2927.qa2013dag.local using any of the configured protocols; requested by PID      95c (C:\Program Files\Microsoft\Exchange Server\V15\Bin\MSExchangeHMWorker.exe).
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-DistributedCOM" Guid="{1B562E86-B7AA-4131-BADC-B6F3A001407E}" EventSourceName="DCOM" />
        <EventID Qualifiers="0">10028</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8080000000000000</Keywords>
        <TimeCreated SystemTime="2014-04-21T06:43:10.813315700Z" />
        <EventRecordID>24185</EventRecordID>
        <Correlation />
        <Execution ProcessID="640" ThreadID="544" />
        <Channel>System</Channel>
        <Computer>zhuvm-fog-2928.qa2013dag.local</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="param1">ZHUVM-FOG-2927.qa2013dag.local</Data>
        <Data Name="param2">     95c</Data>
        <Data Name="param3">C:\Program Files\Microsoft\Exchange Server\V15\Bin\MSExchangeHMWorker.exe</Data>
        <Binary>3C5265636F726423313A20436F6D70757465723D286E756C6C293B5069643D3634303B342F32312F3230313420363A34333A31303A3831333B5374617475733D313732323B47656E636F6D703D323B4465746C6F633D313731303B466C6167733D303B506172616D733D313B7B506172616D23303A307D3E3C5265636F726423323A20436F6D70757465723D286E756C6C293B5069643D3634303B342F32312F3230313420363A34333A31303A3831333B5374617475733D313732323B47656E636F6D703D31383B4465746C6F633D313434323B466C6167733D303B506172616D733D313B7B506172616D23303A5A4855564D2D464F472D323932372E7161323031336461672E6C6F63616C7D3E3C5265636F726423333A20436F6D70757465723D286E756C6C293B5069643D3634303B342F32312F3230313420363A34333A31303A3831333B5374617475733D313732323B47656E636F6D703D31383B4465746C6F633D3332323B466C6167733D303B506172616D733D303B3E3C5265636F726423343A20436F6D70757465723D286E756C6C293B5069643D3634303B342F32312F3230313420363A34333A31303A3831333B5374617475733D31313030313B47656E636F6D703D31383B4465746C6F633D3332303B466C6167733D303B506172616D733D313B7B506172616D23303A5A4855564D2D464F472D323932372E7161323031336461672E6C6F63616C7D3E</Binary>
      </EventData>
    </Event>

    Log Name:      Application
    Source:        MSExchange ADAccess
    Date:          4/24/2014 9:58:02 AM
    Event ID:      4027
    Task Category: General
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      zhuvm-fog-2928.qa2013dag.local
    Description:
    Process MSExchangeHMWorker.exe (ExHMWorker) (PID=544). WCF request (Get Servers for qa2013dag.local) to the Microsoft Exchange Active Directory Topology service on server (TopologyClientTcpEndpoint (localhost)) failed. Make sure that the service is running. In addition, make sure that the network ports that are used by Microsoft Exchange Active Directory Topology service are not blocked by a firewall. The WCF call was retried 3 time(s). Error Details 
     System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:890/Microsoft.Exchange.Directory.TopologyService. The connection attempt lasted for a time span of 00:00:02.0436133. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:890.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:890
       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
       at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
       at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       --- End of inner exception stack trace ---
    
    Server stack trace: 
       at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
       at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
    
    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at System.ServiceModel.ICommunicationObject.Open()
       at Microsoft.Exchange.Net.ServiceProxyPool`1.GetClient()
       at Microsoft.Exchange.Net.ServiceProxyPool`1.TryCallServiceWithRetry(Action`1 action, String debugMessage, WCFConnectionStateTuple proxyToUse, Int32 numberOfRetries, Boolean doNotReturnProxyOnSuccess, Exception& exception)
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchange ADAccess" />
        <EventID Qualifiers="49156">4027</EventID>
        <Level>2</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2014-04-24T01:58:02.000000000Z" />
        <EventRecordID>3810658</EventRecordID>
        <Channel>Application</Channel>
        <Computer>zhuvm-fog-2928.qa2013dag.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>MSExchangeHMWorker.exe (ExHMWorker)</Data>
        <Data>544</Data>
        <Data>Get Servers for qa2013dag.local</Data>
        <Data>TopologyClientTcpEndpoint (localhost)</Data>
        <Data>3</Data>
        <Data>System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:890/Microsoft.Exchange.Directory.TopologyService. The connection attempt lasted for a time span of 00:00:02.0436133. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:890.  ---&gt; System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:890
       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
       at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
       at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       --- End of inner exception stack trace ---
    
    Server stack trace: 
       at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
       at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
    
    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
       at System.ServiceModel.ICommunicationObject.Open()
       at Microsoft.Exchange.Net.ServiceProxyPool`1.GetClient()
       at Microsoft.Exchange.Net.ServiceProxyPool`1.TryCallServiceWithRetry(Action`1 action, String debugMessage, WCFConnectionStateTuple proxyToUse, Int32 numberOfRetries, Boolean doNotReturnProxyOnSuccess, Exception&amp; exception)</Data>
      </EventData>
    </Event>

    Log Name:      Application
    Source:        MSExchangeDiagnostics
    Date:          4/24/2014 9:56:45 AM
    Event ID:      1015
    Task Category: General
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      zhuvm-fog-2928.qa2013dag.local
    Description:
    Unable to contact the active directory. Inner Exception System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException: The computer is not in a site.
       at System.DirectoryServices.ActiveDirectory.ActiveDirectorySite.GetComputerSite()
       at Microsoft.Exchange.Diagnostics.Service.MachineInformationSource.PollMachineInformation()
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchangeDiagnostics" />
        <EventID Qualifiers="49156">1015</EventID>
        <Level>2</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2014-04-24T01:56:45.000000000Z" />
        <EventRecordID>3810649</EventRecordID>
        <Channel>Application</Channel>
        <Computer>zhuvm-fog-2928.qa2013dag.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException: The computer is not in a site.
       at System.DirectoryServices.ActiveDirectory.ActiveDirectorySite.GetComputerSite()
       at Microsoft.Exchange.Diagnostics.Service.MachineInformationSource.PollMachineInformation()</Data>
      </EventData>
    </Event>

    Besides, there are many services in Exchange server stopped & could not be started:

    I observed no ERRORs in Domain Controller, the firewall of all servers had been turned off and IPv6 enabled.

    I  could not determine what happened to my server, it's really urgent issue for me!!!

    How could I fix these servers?



    • Edited by Lasia Thursday, April 24, 2014 2:13 AM
    Thursday, April 24, 2014 2:08 AM

Answers

  • Finally I got to know what a stupid mistake I made.

    Before this mess I used to add an extra DC for my domain, and created a new site & assigned it to the DC.

    Then, I uninstall the DC. What confused me is that the Site I assigned to the uninstalled machine still exist(Why didn't the Active Directory Uninstall Guide inform me about this????), as well as the deleted host.

    That comes to be the question.When I built a new machine to install the Mailbox role to replace the broken DAG environment, the prerequisite check module told me:

    "This computer doesn't belong to a valid Active Directory site. "

    Then I checked the sites, and find the extra one. So I deleted it, restarted every server & service. Everything works well now.


    • Marked as answer by Lasia Tuesday, May 13, 2014 9:09 AM
    • Edited by Lasia Tuesday, May 13, 2014 9:31 AM
    Tuesday, May 13, 2014 9:09 AM

All replies

  • Hello,

    What do you get when you manually start these core exchange services,? Please post the error tp forum.

    As core exchange services are not be started, you may get the above error.

    Please use telent command-line tool to check if 890 port is opened.


    Cara Chen
    TechNet Community Support


    • Edited by cara chen Tuesday, April 29, 2014 6:35 AM edit
    Friday, April 25, 2014 7:52 AM
  • Hi Cara,

    I have turned off the firewall; Besides, the Inbound rule "MSExchangeADTopology - WCF (TCP-In)" is enabled.But I got failure when I create a rule to allow any connection to port 890 and telnet it.

    When I start the services manually, I got the following Error message:

    Friday, April 25, 2014 9:35 AM
  • Hello,

    I recommend you use get-healthreport cmdlet to check if your server run normally.

    Besides, please check if there are other related logs in application log.


    Cara Chen
    TechNet Community Support

    Tuesday, April 29, 2014 6:40 AM
  • How to use this cmdlet?

    Should I import some modules? When I run it, I was told:

    get-healthreport : The term 'get-healthreport' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

    At line:1 char:1

    + get-healthreport

    + ~~~~~~~~~~~~~~~~

    + CategoryInfo : ObjectNotFound: (get-healthreport:String) [], CommandNotFoundException

    + FullyQualifiedErrorId : CommandNotFoundException

    • Edited by Lasia Monday, May 5, 2014 7:30 AM
    Monday, May 5, 2014 7:17 AM
  • Hello,

    According to the error, you have no permission to run the cmdlet, please run the get-rolegroupmember "organization management" cmdlet to check if you are a member of organization management role group, and run the get-rolegroupmember "server management" cmdlet to check if you are a member of server management role group.

    If you have no enough permission to run the cmdlet , please run add-rolegroupmember cmdlet to add proper permission.

    Here is an article for your reference.

    http://technet.microsoft.com/en-us/library/jj218724(v=exchg.150).aspx


    Cara Chen
    TechNet Community Support

    Monday, May 5, 2014 8:08 AM
  • I checked the user group in Domain Controller, the Administrator user I used to run the cmdlet was in both "organization management" & "server management" group.

    By the way, get-rolegroupmember was not recognized as well.

    Should I use the power shell for exchange? But I cannot find this in my Administrative Tools list.

    There are only Windows PowerShell (x86).....

    Wednesday, May 7, 2014 3:54 AM
  • Finally I got to know what a stupid mistake I made.

    Before this mess I used to add an extra DC for my domain, and created a new site & assigned it to the DC.

    Then, I uninstall the DC. What confused me is that the Site I assigned to the uninstalled machine still exist(Why didn't the Active Directory Uninstall Guide inform me about this????), as well as the deleted host.

    That comes to be the question.When I built a new machine to install the Mailbox role to replace the broken DAG environment, the prerequisite check module told me:

    "This computer doesn't belong to a valid Active Directory site. "

    Then I checked the sites, and find the extra one. So I deleted it, restarted every server & service. Everything works well now.


    • Marked as answer by Lasia Tuesday, May 13, 2014 9:09 AM
    • Edited by Lasia Tuesday, May 13, 2014 9:31 AM
    Tuesday, May 13, 2014 9:09 AM