Exchange errors: MSExchangeSA 9176, 9057, referencing old decommissioned DC/GC whose IP the new DC/GC is now using RRS feed

  • Question

  • We have noticed the following errors on our Exchange 2007 server:

    Log Name:      Application
    Source:        MSExchangeSA
    Date:          8/09/2014 1:15:55 PM
    Event ID:      9176
    Task Category: NSPI Proxy
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      EXCHANGE.fqdn
    NSPI Proxy can contact Global Catalog oldDC01.fqdn but it does not support the NSPI service. After a Domain Controller is promoted to a Global Catalog, the Global Catalog must be rebooted to support MAPI Clients.  Reboot oldDC01.fqdn as soon as possible. 

    Log Name:      Application
    Source:        MSExchangeSA
    Date:          8/09/2014 1:15:55 PM
    Event ID:      9057
    Task Category: NSPI Proxy
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      EXCHANGE.fqdn
    NSPI Proxy cannot contact any Global Catalog that supports the NSPI Service. New clients will be refused until a Global Catalog is available. After a Domain Controller is promoted to a Global Catalog, it must be rebooted to support MAPI Clients. 

    These errors started appearing on our Exchange server (EXCHANGE.fqdn) when two domain controllers (oldDC01.fqdn with AD/GC/DHCP/DNS and oldDC02.fqdn with AD/GC/DNS) were demoted, decommissioned and replaced by rebuilt ones with different names but the same IPs - this was done to prevent manual reconfiguration of every device with manually entered DNS servers.

    There were no errors when the servers were decommissioned and I have only found this error when troubleshooting Outlook clients not creating new profiles firstly in remote sites but now also in our primary site with Exchange.

    The following workaround was used in the client PCs:

    I've found the following also which leads me to think it could be a misconfiguration problem in AD or Exchange:

    Navigating in EMC to the Server Configuration node, right clicking EXCHANGE, clicking Properties and navigating to the System Settings tab reveals just the new servers under both the Domain controller servers and Global catalog servers fields.

    AD seems healthy according to dcdiag, not really sure where else to check now...

    Monday, September 8, 2014 7:01 AM

All replies

  • Hi,

    The above issue often occurs if the NSPI is not enabled on the global catalog server. To resolve this issue, restart the global catalog server and check the result, this enable the NSPI.

    Hope this can be helpful to you.

    Best regards,

    Amy Wang
    TechNet Community Support

    Tuesday, September 9, 2014 6:02 AM
  • Do you have the Exchange server hard-coded to use certain DC/GC machines?

    What does the Application Event Log event 2180 tell you about the DC/GCs Exchange sees?

    Also, just because the old DC/GCs are gone from the OU doesn't mean they're entirely gone from DNS *or* from the AD! Check AD Sites and Services and verify that the old machines have been cleanly removed. I've seen lots of DCPROMOs leave behind debris that caused replication problems. As an aside, have you run "repadmin /replsummary" (as a starting point) and verified that all the endpoints are what you think they are and that they're replicating properly?

    --- Rich Matheisen MCSE&I, Exchange MVP

    • Marked as answer by Amy.Wang Monday, September 22, 2014 9:15 AM
    • Unmarked as answer by AMS IT Thursday, September 25, 2014 10:37 PM
    Tuesday, September 9, 2014 9:39 PM
  • The section at the end of this blog post lists the places references to the old DC(s) may be hiding:


    Please ignore the first part (just variations on ways to remove the domain controller, which you have already done).

    I would concentrate on DNS. I have found that there are almost always references to the old domain controller here, regardless of the method used to removed it (even a clean dcpromo leaves references here).

    I'm not sure that you do have references to old DCs here, or that this is the cause of the problem, but it is one thing you could double-check, if nothing else, to rule it out and move on to something else.

    I have a link to Ace Fekay's excellent article on metadata cleanup (at the bottom of the post) but it looks likes he's in the process of migrating his blog, so I'm not sure what that will produce.

    Please mark as helpful if you find my contribution useful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Tuesday, September 9, 2014 10:27 PM
  • Hello, I have restarted all the DCs (which are all GCs) already and this hasn't helped. I have also restarted the Exchange server at least once since finding these errors.
    Wednesday, September 10, 2014 6:57 AM
  • I don't think it's hard-coded although it was configured before I started here.

    Where should I look to confirm if there is anything hardcoded or not?

    The DS Server value doesn't exist on the following key on my exchange server:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Exchange Provider  

    I've combed through all DNS trees twice now and haven't found anything related amiss (some old servers were still listed and I'm confident that's been cleaned up now). Nothing stood out in Sites and Services either when I looked through. I wanted to comb through ADSI Edit but didn't really have a starting point.

    I just ran repadmin /replsummary
    Below are the results:

    Replication Summary Start Time: 2014-09-10 17:00:16
    Beginning data collection for replication summary, this may take awhile:

    Source DSA          largest delta    fails/total %%   error
     ADL-DC-01                 05m:38s    0 /   5    0
     BNE-DC-03                 09m:12s    0 /  40    0
     BNE-DC-04                 12m:30s    0 /  10    0
     CNS-DC-01                 12m:30s    0 /  30    0
     FRE-DC-02                 05m:30s    0 /   5    0
     HOB-DC-01                 05m:34s    0 /   5    0
     HOB-DC-03                 12m:30s    0 /  10    0
     MEL-DC-01                 05m:36s    0 /   5    0
     NEW-DC-02                 05m:36s    0 /   5    0

    Destination DSA     largest delta    fails/total %%   error
     ADL-DC-01                 05m:15s    0 /  10    0
     BNE-DC-03                 12m:31s    0 /  15    0
     BNE-DC-04                    :20s    0 /  10    0
     CNS-DC-01                 05m:41s    0 /  30    0
     FRE-DC-02                 03m:22s    0 /  10    0
     HOB-DC-01                 05m:17s    0 /  10    0
     HOB-DC-03                 02m:07s    0 /  10    0
     MEL-DC-01                 08m:43s    0 /  10    0
     NEW-DC-02                 09m:20s    0 /  10    0

    Wednesday, September 10, 2014 7:03 AM
  • I'll have a look at this tomorrow and see how I go, thanks for the link.

    I think DNS might be an issue but it could be caused by something cached in Exchange. Possibly related to the fact that the new domain controller has the same IP as the old one.

    Wednesday, September 10, 2014 7:07 AM
  • This should get you the settings for all of your servers:

    Get-ExchangeServer -status | fl name, *controller*, *catalog*

    I wouldn't use ADSIEDIT for locating the information. I think it's fine for modifying stuff, but for general "poking around" I'd use LDP.exe. You have the ability to use LDAP queries and you'll see pretty much everything in one pane rather than having to click on individual properties.

    The repadmin looks okay.

    I asked earlier "What does the Application Event Log event 2180 tell you about the DC/GCs Exchange sees?" but you never supplied that information. That should point out pretty quickly which DC/GC machines the AD discovery process finds and what it thinks about the status of each of them.

    --- Rich Matheisen MCSE&I, Exchange MVP

    Wednesday, September 10, 2014 3:50 PM
  • Ah yes sorry forgot to check the event log. On our Exchange server there are no logs with Event ID 2180 in the Application log.

    I ran the Get-ExchangeServer command myself earlier to check and nothing out of the ordinary was there:

    Name                            : EXCHANGE
    StaticDomainControllers         : {}
    StaticConfigDomainController    :
    StaticExcludedDomainControllers : {}
    CurrentDomainControllers        : {BNE-DC-03.fqdn, HOB-DC-03.fqdn, BNE-DC-04.fqdn}
    CurrentConfigDomainController   : HOB-DC-03.fqdn
    StaticGlobalCatalogs            : {}
    CurrentGlobalCatalogs           : {BNE-DC-04.fqdn, HOB-DC-03.fqdn, BNE-DC-03.fqdn}

    BNE-DC-03 and BNE-DC-04 are our new domain controllers replacing BNE-DC-01 and BNE-DC-02 respectively. HOB-DC-03 is currently sitting in our Brisbane office (HQ) prior to being deployed to our Hobart depot.

    Thursday, September 11, 2014 1:16 AM
  • My bad. It's 2080 (not 2180). The event source s/b MSExchangeDSAccess.

    --- Rich Matheisen MCSE&I, Exchange MVP

    Thursday, September 11, 2014 7:05 PM
  • Ah yep I think I had looked at this one before too for clues but no help :(

    Process STORE.EXE (PID=4624). Exchange Active Directory Provider has discovered the following servers with the following characteristics: 
     (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version) 
    BNE-DC-03.fqdn CDG 1 7 7 1 0 1 1 7 1
    BNE-DC-04.fqdn CDG 1 7 7 1 0 1 1 7 1
    HOB-DC-03.fqdn CDG 1 7 7 1 0 1 1 7 1
    MEL-DC-01.fqdn CDG 1 7 7 1 0 1 1 7 1
    NEW-DC-02.fqdn CDG 1 7 7 1 0 1 1 7 1
    CNS-DC-01.fqdn CDG 1 7 7 1 0 1 1 7 1
    HOB-DC-01.fqdn CDG 1 7 7 1 0 1 1 7 1
    FRE-DC-02.fqdn CDG 1 7 7 1 0 1 1 7 1
    ADL-DC-01.fqdn CDG 1 7 7 1 0 1 1 7 1

    Thursday, September 11, 2014 10:22 PM
  • Just to be sure that the correct DC names are reported in the 9176 events with the MSExchangeSA source, have you removed the old PTR records from your DNS? You said you reused the IP addresses for the new DCs. Maybe the name you see in the event are taken from a PTR record?

    Are the DCs running 2008 or 2008 R2? Are any of the running 2003? There were some problems with the number of NSPI connections per client with 2008.

    I suppose one thing you could do is to get a copy of rpcdump.exe and run it in each GC. The GUID for NSPI is "f5cc5a18-4264-101a-8c59-08002b2f8426" and it should be present on each GC.

    Another thing would be to setup netmon and have a look at what's going on. That would at least get you the IP address of the GC.

    --- Rich Matheisen MCSE&I, Exchange MVP

    • Marked as answer by Amy.Wang Monday, September 22, 2014 9:14 AM
    • Unmarked as answer by AMS IT Thursday, September 25, 2014 10:37 PM
    Saturday, September 13, 2014 5:09 PM
  • Ah that got me excited, thought I may have missed it but just went through Reverse Lookup Zones for our ten subnets and there were no mentions to the old DCs. I recall going through and removing the Name Server references to them as well when they were demoted and had even found references to older servers that no longer existed in there...

    Just found references to both old servers BNE-DC-01 and BNE-DC-02 as nameservers (along with a few old ones) in an old subnet that's no longer in use. Not sure if that would have any effect on anything though.

    The DCs in question (BNE-DC-03 and BNE-DC-04) are both running 2012 R2. The rest around the country are a mix of 2012 R2, 2012, 2008 R2 and one 2008.

    I'll try run that command in the coming days. 

    Thanks for all your help and suggestions so far.

    Monday, September 15, 2014 10:44 PM