none
Exchange 2013 Content Catalog Index Failed All Databases

    Question

  • I have  a two node DAG I have successfully updated the database copies no bad copies. The content catalog index on all copies shows as failed. Update-DatabaseCopy -catalogonly does not work. /have deleted thedatabase abd remounted it shows healthy and returns to failed catalog soon after have strict targets client needs HA. cant failover the database as a result. The content database is failed even on servers with one copy only on DAG member server. Help
    Sunday, December 02, 2012 6:17 PM

Answers

  • Finally Solved

    Got assistance from Microsoft Support. It seems there is a bug in the RTM its looking for a AD Group that does not exist will probably be fixed later. The fix is:

    Create a new AD group named ‘ContentSubmitters’ and grant ‘Admistrators’ and ‘NetworkService’ full access to it

    Thereafter restart "Microsoft Exchange Search" & "Microsoft Exchange Search Host Controller" services

    This worked for me and the index went from crawling to healthy on all databasecopies.

    Thanks for all your help.

    Monday, January 28, 2013 10:36 AM
  • Hi Bright,

    Please try to manually rebuild the full-text index catalog:

    1. Stop the Microsoft Exchange Search Service on the Exchange server on which the problematic database is mounted.

    2. Delete the full-text index catalog directory

    3. Start the Microsoft Exchange Search Service

    Then, please reseed the catalog of the database copy again.

    Thanks,

    Da Xu

    • Marked as answer by Da XuModerator Monday, December 17, 2012 2:58 AM
    Tuesday, December 11, 2012 7:31 AM
    Moderator

All replies

  • If you need to move the databases while the catalogs are failed you can use the Skip Client Experience Checks parameter.
    Monday, December 03, 2012 12:50 AM
  • I need the content index to reflect healthy this is a client site and the infrastructure will be undergoing quality assurance how do I reset the index like it used to be done in 2010.

    Even after moving all nodes from the DAG the database index still reflects failed

    Monday, December 03, 2012 9:19 AM
  • Hi BrightM,

    Please run the Get-MailboxDatabaseCopyStatus databasename | fl *index* to check the detailed information of the error.

    If possible, please create a new database and put it in a dedicated disk to check the status.

    If you have any feedback on our support, please click here


    Frank Wang
    TechNet Community Support

    Tuesday, December 04, 2012 3:11 AM
  • The output I get is as follows:

    Content Index State            :     Failed

    ContentIndexErrorMessage  :     An internal error occurred for the database or its index.

    I tried t create a blank database but I it went healthy and then went failed soon after.

    Tuesday, December 04, 2012 3:21 PM
  • Hi BrightM,

    Two suggestions:

    1. Restart the Microsoft Exchange Search Indexer service on both member servers.

    2. Using Troubleshoot-CI.ps1 script.

    .\Troubleshoot-CI.ps1 -Database "database name" -Action DetectAndResolve

    Thanks,

    Da Xu

    Wednesday, December 05, 2012 5:02 AM
    Moderator
  • Ok I have reduced the problem to this

    I have a Mapi network replication allowed 192.168.101.0/24 Status UP for subnet and interfaces

    Ialso have  replication network replication allowed 10.10.10.0/30 Status Partitioned for subnet and Unreachable for interfaces

    On the replication network the transmission is good for sending and not receiving.

    How can I set the relevant static routes to resolve this problem.

    Thursday, December 06, 2012 5:40 PM
  • Hi BrightM,

    If the two replication networks are on same subnet (10.10.10.0/30), there is no need to add static route.

    Please make sure the replication network (10.10.10.0/30) has no gateway, lmhosts lookup disabled, no netbios, no dns servers, and is setup not to register the connection address in DNS.

    Also, please check your firewall setting.

    Da Xu

    Friday, December 07, 2012 3:09 AM
    Moderator
  • Da Xu

    This is correct I have managed to do this and my replication network is up. I still have an issue with the content index on the active copy which still reflects failed.

    Regards

    Bright

    Friday, December 07, 2012 11:29 AM
  • Hi

    .\Troubleshoot-CI.ps1 -Database "database name" -Action DetectAndResolve

    Tried this I get an error The troubleshooter failed with error:System.InvalidOperationException: Couldn't read registry key:Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID} At E:\Exchange Server 15\Scripts\CITSLibrary.ps1

    Bright

    Friday, December 07, 2012 11:58 AM
  • Hi Bright,

    Please try to manually rebuild the full-text index catalog:

    1. Stop the Microsoft Exchange Search Service on the Exchange server on which the problematic database is mounted.

    2. Delete the full-text index catalog directory

    3. Start the Microsoft Exchange Search Service

    Then, please reseed the catalog of the database copy again.

    Thanks,

    Da Xu

    • Marked as answer by Da XuModerator Monday, December 17, 2012 2:58 AM
    Tuesday, December 11, 2012 7:31 AM
    Moderator
  • Hi Bright,

    Is there any update about this issue?

    Thanks,

    Da Xu
    Thursday, December 13, 2012 8:04 AM
    Moderator
  • Hi

    .\Troubleshoot-CI.ps1 -Database "database name" -Action DetectAndResolve

    Tried this I get an error The troubleshooter failed with error:System.InvalidOperationException: Couldn't read registry key:Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID} At E:\Exchange Server 15\Scripts\CITSLibrary.ps1

    Bright

    Monday, December 17, 2012 11:55 AM
  • Still no resolution.

    1. Where is the content index directory for Exchange 2013 databases.

    Wednesday, January 02, 2013 11:30 AM
  • Hi,

    Check C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database....\

    Following also will help you to understand.

    How to Manually Rebuild the Full-Text Index Catalog
    http://www.msexchangegeek.com/2009/09/09/how-to-manually-rebuild-the-full-text-index-catalog/

    How to Rebuild the Full-Text Index Catalog
    http://technet.microsoft.com/en-us/library/aa995966(v=exchg.80).aspx

    ecsword

    Wednesday, January 02, 2013 8:16 PM
  • I wanted to follwup on this information.

    For Exchange 2013, the service name is Microsoft Exchange Search Host Controller that holds the locks on the indexing files.

    Also, I have this issue on all databases in two separate forests and cannot get the databases to failover, copy, or have a healthy content index status.

    When I follow the steps above I continue to get the failed content index status.

    Any additional information on how to resolve this issue would be great.

    If I need to start a new thread please let me know.

    Thursday, January 03, 2013 5:33 PM
  • I believe that the issue is that the location of

    HKLM\Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID}

    has moved to this location in exchange 2013

    HKLM\Software\Microsoft\ExchangeServer\V15\Search\IndexStatus\{GUID}

    The issue is that Troubleshoot-CI.ps1 calls upon CITSLibrary.ps1 and CITSLibrary.ps1 is looking in the registry for the old location. Correct me if I'm wrong but I dont think Microsoft got around to fixing the scripts. I looked at CITSLibrary.ps1 briefly but I dont know PS that well. Maybe someone can help us out here.

    Monday, January 14, 2013 11:08 AM
  • Hi Jared

    Move-ActiveMailBoxDatabase -SkipClientExperienceChecks

    The move works but I am unable to connect to the databases after the move and the indexes are still failed.

    Regards

    Bright

    Monday, January 14, 2013 1:08 PM
  • Hi  Ecsworld

    Ok these articles seem to refer to older versions of Exchange. 15 does not actually create the Catalog{GUID} folder. I don't know if this is the new design.

    Regards

    Bright



    Monday, January 14, 2013 1:12 PM
  • Hi all

    Thanks for all your helpful responses to date. Some additional clarification on the problem. The content index is on the active database and the index remains failed even when the database is removed from the DAG or if the DAG is deleted. And the server remains standalone. Exchange 2013/Windows 2012.

    Regards

    Bright


    Monday, January 14, 2013 1:47 PM
  • Hi Bright and others running into this issue,

    I am also experiencing the same behavior and am wondering if you are all running SCCM on your network?  Last night I re-installed all of my systems and was able to replicate until SCCM found the systems.  I'm currently working with the SCCM Admin to exclude the Exchange servers and will be re-installing once that has been completed.

    If someone can test this theory before I get to it please let me know.

    Thursday, January 17, 2013 5:59 PM
  • You just might  have something here let me try and remove sccm client agent from the dag members and see whether this might be the problem.
    Tuesday, January 22, 2013 1:33 PM
  • As a final and sad followup, I was able to have the SCCM server down for two days this week and re-installed the Exchange system.  Although I could manually replicate using "update-mailboxdatabasecopy db01\mbox1 -deleteexistingfiles" with out error after suspending the database, I was unable to get the content index state to "Healthy".

    For the time being, since we cannot get the replication to be healthy we have decided not to move forward with Exchange 2013 in this domain.  We are going to wait for about 6 months and try again.  Hopefully some of the weir issues we have seen will be taken care of by then.

    Let me know how it works out for you all.

    Thursday, January 24, 2013 3:35 PM
  • Finally Solved

    Got assistance from Microsoft Support. It seems there is a bug in the RTM its looking for a AD Group that does not exist will probably be fixed later. The fix is:

    Create a new AD group named ‘ContentSubmitters’ and grant ‘Admistrators’ and ‘NetworkService’ full access to it

    Thereafter restart "Microsoft Exchange Search" & "Microsoft Exchange Search Host Controller" services

    This worked for me and the index went from crawling to healthy on all databasecopies.

    Thanks for all your help.

    Monday, January 28, 2013 10:36 AM
  • Hi Bright,

    When you say "grant Administrators and NetworkService full access to it", what object is it?  Is this the database and log folders?

    Also do you add ContentSubmitters group to any object on the mailbox server?

    Thank you.

    Wednesday, January 30, 2013 2:29 PM
  • When you create the active directory security group called ContentSubmitters right click the group so created and properties security tab add those two groups and give them full control to the group.
    Thursday, February 14, 2013 5:05 PM
  • Ok - I've just done this is my POC environment (with CU1 applied) and it worked. My Journal Database now has a Healthy Status - What's going on, why are we having to create these groups and apply the relevant permissions.

    Not very happy that these issues are still occurring post CU1!

    Please advise if this was a fix in Exchange 2013 CU1 or there is something I'm missing...

    Ryan

    Friday, April 05, 2013 1:40 AM
  • Thanks for passing that on!  It seemed to do the trick.
    • Edited by Robert Tuck Friday, August 02, 2013 4:56 PM
    Friday, August 02, 2013 4:53 PM
  • Hello,

       I've been following this due to the exact issue.   However applied the ContentSubmitters group to no avail.   Are there any other solutions available?

      We have 2012 standard.  Two servers, one for DB's and one for CAS.   The databases are intermittent, one will fail today a different one will fail tomorrow.  A reboot always fixes the issue for about an hour.

    Please share any ideas!  Thanks!

    Wednesday, August 07, 2013 3:22 PM
  • Hello,

    We're having this same issue. I've tried adding the "ContentSubmitters" AD group to no avail. The "Microsoft Exchange Search Host Controller" service appears to be disabled on our server. Is this by design?

    We're running Exchange 2013 CU2 on a W12 server.

    Thanks!

    Thursday, September 12, 2013 2:53 PM
  • Hello,

    We're having this same issue. I've tried adding the "ContentSubmitters" AD group to no avail. The "Microsoft Exchange Search Host Controller" service appears to be disabled on our server. Is this by design?

    We're running Exchange 2013 CU2 on a W12 server.

    Thanks!


    The service should be set to startup type Automatic on a Multi-role Exchange 2013 server.

    Did my post help? Please use "Vote As Helpful", "Mark as answer" or "Propose as answer". Thank you!

    Thursday, September 26, 2013 12:47 PM
  • I was having the same issue. The NIC's configured in my DAG were bound in the wrong order so my Cluster Heartbeat network was first rather than my Exchange network. To resolve the issue I had to open my Network and Sharing Center from Control Panel. Click "Change Adapter Settings". Hit the "Alt" key to bring up the "File, Edit, View, Tools, Advanced, Help Menu. Click on "Advanced" and select "Advanced Settings". Set the correct Nic order. After this I restarted the "Microsoft Exchange Search" and the "Microsoft Exchange Search Host Controller" services from the services applet.  Now my content index shows healthy for my DAG.

    Hope this helps.

    Thursday, January 16, 2014 6:19 PM
  • First off Bright, thank you soo much for this info.  MS can't seem to pull their head out of their posteriors long enough to actually look in to issues like this.   

    Second, this is still an issue with SP1.  Unbelievable. 

    Tuesday, March 25, 2014 9:23 PM
  • I believe that the issue is that the location of

    HKLM\Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID}

    has moved to this location in exchange 2013

    HKLM\Software\Microsoft\ExchangeServer\V15\Search\IndexStatus\{GUID}

    The issue is that Troubleshoot-CI.ps1 calls upon CITSLibrary.ps1 and CITSLibrary.ps1 is looking in the registry for the old location. Correct me if I'm wrong but I dont think Microsoft got around to fixing the scripts. I looked at CITSLibrary.ps1 briefly but I dont know PS that well. Maybe someone can help us out here.

    Correct - The Troubleshoot-CI script doesn't work on 2013.

    BTW, everyone else, please see this relevant KB:

    KB 2807668

    Content Index status of all or most of the mailbox databases in the environment shows "Failed"



    Mike Crowley | MVP
    My Blog -- Planet Technologies

    Thursday, May 01, 2014 8:03 PM
  • Hi Bright,

    not sure how you got to create a AD group and add Admistrators’ and ‘NetworkService’ to it as members.

    my DC won't allow you to add those, just Domain Admins.

    Friday, May 23, 2014 8:59 AM
  • Great info, one question I have is, whom do you put into the security group ContentSubmitters?

    Thanks 

    Thursday, May 28, 2015 7:29 AM
  • No, I think he meant to give them "full access" to the AD object, not make them members of the group.

    Still, is this supposed to be a "Global" group, "Universal?"

    Does it need to reside where the other Exchange groups reside, or can it live anywhere in AD?


    tnjman

    Sunday, May 31, 2015 8:04 PM
  • I was unsure of this too until i realized that you do not add Administrators and NetworkService as members of the group, instead you go to the Security Tab and give them full access.

    Charles Herrington

    Monday, March 14, 2016 12:47 PM
  • you can also rebuild the search foundation to see
    1. goes to E:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data and renamed the folder data to data_old
    2. execute this cmdlets now:

    PS C:\Windows\system32> cd "E:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer"

    PS E:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer> .\installconfig.ps1 -action I -dataFolder

    "e:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

    Monday, September 26, 2016 2:50 AM