none
Exchnage 2013 CU 14 Database Indexing failed.

    Question

  • Hi!

    After installing EX2013CU14 the database indexing if failed (all databases).

    I tried all solution what i found:

    Stop MSExchangeFastSearch and HostControllerService, remove index and start the service but doesn't work.

    I tried also the ContentSubmitter group solution but doesn't work.

    Get this error in event viewer:

    Watson report about to be sent for process id: 28160, with parameters: E12IIS, c-RTL-AMD64, 15.00.1236.003, M.E.Search.Service, M.E.Data.Directory, M.E.D.D.ScopeSet.GetOrgWideDefaultScopeSet, System.ArgumentNullException, 301, 15.00.1236.000.
    ErrorReportingEnabled: False 



    Wednesday, October 05, 2016 6:10 AM

All replies


  • Hi

    If you install single server so you must have one default database over there right, plz ensure following things:-

    • can you be able to send/receive mails to local users?

    • OWA is working fine ?

    • All services are up and running?

    You can see failed content index of the databases by this PowerShell:-

    Get-MailboxDatabaseCopyStatus * | ft -auto

    Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Failed"}

    Now you can see the results with failed contents

    You can also check the application event logs to further dig out the issue.

    Check Following Event IDs:-

    • Event ID: 1012

    • Event ID: 1009

    check Exchange Server Information Store service may not encounter with some error or service is up and running.

    Try to update the database by this powershell:-

    Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Failed"} | Update-MailboxDatabaseCopy -CatalogOn

    run this command twice and check if failed contents cleared.

    To rebuild a failed content index you need to first stop the search services on your Exchange server. This can be cause searches on other healthy databases.

    Rebuilding process can also create a significant load on the server.

    Do the Followings:-

    Stop these services:-

    • Microsoft Exchange Search Host Controller

    • Microsoft Exchange Search

    Now do the PowerShell as:-

    Get-MailboxDatabase DB01 | select EdbFilePath

    it will give you your exchange data base path so go to this path and see there are two thing one is .EDB and other is a folder, content index is stored in a folder named for the GUID of the database, its like a long name.

    Now Delete this folder and do the same for other databases for same issue and restart the above stopped services:-

    you can also start service from PowerShell:-

    • start-service MSExchangeFastSearch
    • start-service HostControllerService

    Now the content index will be rebuild and it may take some time as per the database size.

    Now check with this command that your database contents are healthy or not ?

    Get-MailboxDatabaseCopyStatus * | ft -auto

    run this command two three times.

    Hope this will help you to resolve your problem.

    Kindly click "Mark as Answer" on the post that helps you, this can be beneficial to other community members reading this thread.

    Regards.

    H.shakir



    • Edited by H Shakir Wednesday, October 05, 2016 7:09 AM
    Wednesday, October 05, 2016 7:06 AM
  • Shakir gave you great information and I see you said you deleted the index prior, however did you actually kill the entire index folder?Also;

    1. is this a standalone server
    2. do you have more then one EDB and if so is the index fine on those EDB's?

    Search, Recover, & Extract Mailboxes, Folders, & Email Items from Offline Exchange Mailbox and Public Folder EDB's and Live Exchange Servers or Import/Migrate direct from Offline EDB to Any Production Exchange Server, even cross version i.e. 2003 --> 2007 --> 2010 --> 2013 --> 2016 with Lucid8's DigiScope http://www.lucid8.com/product/digiscope.asp


    Wednesday, October 05, 2016 3:12 PM
  • I tried Shakir's solutions but doesn't work for me.

    This is a standalone server, no DAG.

     
    Wednesday, October 05, 2016 6:16 PM
  • Hi ToniSlow,

    kindly provide exchange application events details here to further dig out your issue and also answer all questions in above posts.

    Regards.

    H.Shakir

    Wednesday, October 05, 2016 7:57 PM
  • I'm having the same issue after CU14....Users could not search for newly added public contacts. 

    Attempted to rebuild the indexes and I'm getting. 

    Watson report about to be sent for process id: 14568, with parameters: E12IIS, c-RTL-AMD64, 15.00.1236.003, M.E.Search.Service, unknown, M.E.D.D.A.SessionSettingsFactory.FromOrganizationIdWithoutRbacScopesServiceOnly, System.ArgumentNullException, a135, unknown.
    ErrorReportingEnabled: True

    I cannot rebuild my indexes and my users and completely without search on an 8000 member contact database.  Please help.  

    Also, my ContentIndexState is going back and forth every couple of seconds between Crawling and Failed status.

    Wednesday, October 05, 2016 8:32 PM
  • "Also, my ContentIndexState is going back and forth every couple of seconds between Crawling and Failed status."

    Same here (because theMSExchangeFastSearch service is restart after the ContentIndexState is going to Failed status),

    Need solution from MS.

    Wednesday, October 05, 2016 8:36 PM
  • Just a me too. I cannot update the content index of a database since upgrading from Ex2016CU2 -> Ex2016CU3

    Watson report about to be sent for process id: 29224, with parameters: E12IIS, c-RTL-AMD64, 15.01.0544.027, M.E.Search.Service, M.E.Data.Directory, M.E.D.D.ScopeSet.GetOrgWideDefaultScopeSet, System.ArgumentNullException, 37a5, 15.01.0544.027.
    ErrorReportingEnabled: True

    A bit more Information. I've rebuild content indexes before, Stop both Exchange Search Services, remove GUID directory, start Search Services, wait, success.

    Now with the new CU, after you restart the search services, the GUID dir increases in size, until the size is what I'd expect for my database size. However the Content index status never goes to Healthy. It looks like the Search Service cannot finalize something. At this point the Search Service starts to go into a crash and restart loop.

    This happens every 50-60 Seconds:

    The Microsoft Exchange Search service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

    • Edited by flds23 Thursday, October 06, 2016 9:26 AM
    Wednesday, October 05, 2016 9:01 PM
  • Hello,

    Do you see the process ID or event id every time remain same or get changed every time ? If its different its means that RPC client access service is restarting.

    Try to increase the connection time out...

    plz check service health also and replication status.

    if you still face the same issue, if everything is working fine so ignore this and go for the upcoming update/patch.

    You can stop specific logging and events by

    https://technet.microsoft.com/en-us/library/bb687437.aspx

    Regards.

    H.Shakir



    • Edited by H Shakir Thursday, October 06, 2016 6:19 AM
    Thursday, October 06, 2016 6:14 AM
  • Hello,

    I see different process ID every time in event viewer.

    The process is Microsoft.Exchange.Search.Service not the RPC client access.

    I test the service health with Test-ServiceHealth command:

    [PS] C:\Users\phadmin\Desktop>Test-ServiceHealth -Server LUNA


    Role                    : Mailbox Server Role
    RequiredServicesRunning : True
    ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeDelivery, MSExchangeIS, MSExchangeMailboxAssistant
                              s, MSExchangeRepl, MSExchangeRPC, MSExchangeServiceHost, MSExchangeSubmission, MSExchangeThro
                              ttling, MSExchangeTransportLogSearch, W3Svc, WinRM}
    ServicesNotRunning      : {}

    Role                    : Client Access Server Role
    RequiredServicesRunning : True
    ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeMailboxReplication, MSExchangeRPC, MSExchangeServi
                              ceHost, W3Svc, WinRM}
    ServicesNotRunning      : {}

    Role                    : Unified Messaging Server Role
    RequiredServicesRunning : True
    ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeServiceHost, MSExchangeUM, W3Svc, WinRM}
    ServicesNotRunning      : {}

    Role                    : Hub Transport Server Role
    RequiredServicesRunning : True
    ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeEdgeSync, MSExchangeServiceHost, MSExchangeTranspo
                              rt, MSExchangeTransportLogSearch, W3Svc, WinRM}
    ServicesNotRunning      : {}

    I dont 't need wait 3 month.


    • Edited by ToniSlow Thursday, October 06, 2016 7:07 AM
    Thursday, October 06, 2016 7:05 AM
  • I'm 100% with ToniSlow on this.  After updating to something that was supposed to be a helpful patch.....Microsoft has now disabled my entire search facility.  Did the Microsoft tech just really advise to squelch the error reporting, ignore the fact that an entire companies mail search and contact search will no longer work, and wait til the next release fixes it in December?  You have 3 users that have reported the exact same behavior.  So, it doesn't appear to be an isolated incident.  Would I be better off opening an incident with MS Business Critical Support to get someone to acknowledge the bug was introduced and get a hotfix out stat?  Please advise.
    Thursday, October 06, 2016 1:36 PM
  • Same issue  from another topic here

    Please check the noderunner.exe processor usage.  It's very high.


    • Edited by ToniSlow Thursday, October 06, 2016 2:00 PM
    Thursday, October 06, 2016 1:59 PM
  • noderunner and ParserServer all running at very high processor usage.
    Thursday, October 06, 2016 2:24 PM
  • You should consider opening cases with Microsoft Support.

    Blog:    Twitter:   

    Thursday, October 06, 2016 2:28 PM
    Moderator
  • Yes - please open a case so this can be investigated.

    Cheers,

    Rhoderick

    Microsoft Senior Exchange PFE

    Blog: http://blogs.technet.com/rmilne  Twitter:   LinkedIn:   Facebook:   XING:

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Friday, October 07, 2016 4:04 PM
  • How can we open a case?
    Saturday, October 08, 2016 5:31 AM
  • https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wf=0&wfName=productselection&locale=en-us&gprid=16662&productname=Microsoft+Exchange+Server+2013 

    And choose your version and next screen details of the problem.

    May be there is single support fee you have to pay to Microsoft.

    Thanks

    Prabodha

    Saturday, October 08, 2016 10:44 AM
  • same here, exchange 2016 cu3.

    I installed cu3 and then finalized migration to 2016 only to find that public folders can't be searched...

    in my case the search service crashes every minute or so and it started doing so right after I tried to rebuild the indexes (but I suspect it was happening before, I'm not sure).

    I can't find any indication of the nature of tha problem. 

    Saturday, October 08, 2016 10:54 AM
  • I can't open a case, bacause every option is need to paid.

    Shame.



    • Edited by ToniSlow Saturday, October 08, 2016 4:28 PM
    Saturday, October 08, 2016 3:11 PM
  • I can't open a case, bacause every option is paid.

    Shame.


    If its a true bug, you will be reimbursed. By the way, what version of .Net is installed?


    Blog:    Twitter:   

    Saturday, October 08, 2016 4:07 PM
    Moderator
  • I check the server manager:

    .Net 3.5 inclued 2.0 installed.

    .Net 4.5 installed

    Also installed this for .Net 4.6 and 4.6.1 via windows update.

    You think it's a .Net bug?

    Try to remove the latest .Net update what I installed?


    • Edited by ToniSlow Saturday, October 08, 2016 6:31 PM
    Saturday, October 08, 2016 6:28 PM
  • I check the server manager:

    .Net 3.5 inclued 2.0 installed.

    .Net 4.5 installed

    Also installed this for .Net 4.6 and 4.6.1 via windows update.

    You think it's a .Net bug?

    Try to remove the latest .Net update what I installed?



    No, I wouldn't remove anything unless Microsoft support directs. I was just curious.

    Blog:    Twitter:   

    Saturday, October 08, 2016 6:32 PM
    Moderator
  • Hi,

    Please use only .Net 4.5 and yes there is a bug in .Net 4.6, I tested it already in my test environment. You can remove and install only .Net 4.5 and it will work perfectly.

    Thanks

    Prabodha

    Sunday, October 09, 2016 6:51 AM
  • Hi,

    Please use only .Net 4.5 and yes there is a bug in .Net 4.6, I tested it already in my test environment. You can remove and install only .Net 4.5 and it will work perfectly.

    Thanks

    Prabodha

    I wouldn't do that unless recommended by Microsoft Support. 4.6.1 is fully supported with CU13 and beyond.

    4 If you're upgrading from a previously installed Exchange cumulative update, we strongly recommend that you install the latest cumulative update before .NET Framework 4.6.1 and its related post-release fixes.

    5 .NET Framework 4.6.1 requires post-release fixes if you want to install it on a server running a supported version of Exchange. The following are the .NET Framework 4.6.1 post-release fixes needed for Exchange.

    https://technet.microsoft.com/en-us/library/ff728623(v=exchg.150).aspx


    Blog:    Twitter:   

    Sunday, October 09, 2016 1:40 PM
    Moderator
  • I have .Net 3.5 and 4.5 but no 4.6, so the problem is not directly linked to having 4.6 installed.

    In my case my W2012R2 has only the features required by Exchange 2016, nothing else.

    Sunday, October 09, 2016 1:41 PM
  • Ok I tried these steps:

    Remove .Net 4.6, repair .Net4.5 and reindex the databases but doesn't work.

    Install .Net 4.6 again and all patch via Windows Update and install the .Net4.6 hotfix for Windows Server 2012R2 then reindex the databases but doesn't work.

    No more idea...


    • Edited by ToniSlow Sunday, October 09, 2016 6:41 PM
    Sunday, October 09, 2016 6:39 PM
  • A client of ours is experiencing the same thing.
    We installed CU14 last month & didn't see this issue until we moved the database. Now we are seeing the exact same symptoms:

    - Error 4999 in event viewer > application logs:

    Watson report about to be sent for process id: 45484, with parameters: E12IIS, c-RTL-AMD64, 15.00.1236.003, M.E.Search.Service, M.E.Data.Directory, M.E.D.D.ScopeSet.GetOrgWideDefaultScopeSet, System.ArgumentNullException, 301, 15.00.1236.000.
    ErrorReportingEnabled: True 

    - The Exchange Search service keeps stopping/starting itself every 5-60 seconds & the mailbox

    - The ContentIndexState keeps changing between "Crawling & Failed"

    I am theorizing that the issue has been there all along, but didn't start rearing it's ugly head until after the db move.

    Unfortunately, we are stuck on this issue as well.


    Monday, October 10, 2016 2:17 AM
  • A similar thread on Exchange 2016 CU1 concludes that the problem was caused by Exchange using instructions not available on older processors:

    https://social.technet.microsoft.com/Forums/office/en-US/6097a282-5bc3-4591-838b-cac9c96b9155/exchange-server-2016-cu1-noderunnerexe-crashing?forum=Exch2016GD

    My Exchange VM has Opteron 6238 cores. Is this true for the others experiencing the problem?

    Monday, October 10, 2016 6:23 AM
  • Ok, I opened an SR.

    Will post here any useful info I get from it.

    Monday, October 10, 2016 6:37 AM
  • Ok I tried these steps:

    Remove .Net 4.6, repair .Net4.5 and reindex the databases but doesn't work.

    Install .Net 4.6 again and all patch via Windows Update and install the .Net4.6 hotfix for Windows Server 2012R2 then reindex the databases but doesn't work.

    No more idea...


    OK, as I stated above, please do not do that unless directed by Microsoft Support. The goal in troubleshooting is to no further harm.

    Blog:    Twitter:   

    Monday, October 10, 2016 10:36 AM
    Moderator
  • OK, I understand.

    Do nothing, waiting for MS.

    Monday, October 10, 2016 11:06 AM
  • OK, I understand.

    Do nothing, waiting for MS.

    Did you open a ticket? 

    Blog:    Twitter:   

    Monday, October 10, 2016 11:18 AM
    Moderator
  • Yes,

    I'm waiting for the first contact.

    Should be real soon.

    Monday, October 10, 2016 11:47 AM
  • Please keep us posted on your progress.
    Monday, October 10, 2016 6:11 PM
  • Nothing useful so far.

    just one find worth mentioning:

    the issue appears related to a specific db. if that one is dismounted search appears to be stable.

    at least that's what I concluded a few minutes ago.

    Monday, October 10, 2016 7:57 PM
  • Thanks for the info.
    I am also interested in the "answer"  ...before we install CU14.

    Side-note--  seems like I have not received email notifications of changes on this forum topic since Oct 6.

    ==

    Tuesday, October 11, 2016 5:58 PM
  • I have the same problem with CU14
    Wednesday, October 12, 2016 8:28 AM
  • I have the same issue with CU14
    Wednesday, October 12, 2016 2:28 PM
  • Hi

    We also opened a case with MS.

    In our case we noticed difference after CU14 installation. users complained that they are unable to search items past patch date.

    Then we made new DB and moved mailboxes to new database, still same issue.

    Then we deleted indexing and whole hell got loose.

    Same errors as people before and search service crashes every 29-39 second while trying to crawl. For us its good that we have multiple servers in DAG and we are letting that 1 node crash as keeping that node for rest databases as passive.

    At the moment we uploaded all logs to MS and waiting for feedback.

    Wednesday, October 12, 2016 4:41 PM
  • Hi,

    just received the proposed "solution" from support:

    install a new server (they say CU3) and move there you data.

    Apparently it's indeed an error (as if it could have been something else) but there's no time frame for a fix.

    :-(

    Friday, October 14, 2016 10:20 AM
  • Hi,

    just received the proposed "solution" from support:

    install a new server (they say CU3) and move there you data.

    Apparently it's indeed an error (as if it could have been something else) but there's no time frame for a fix.

    :-(

    You wouldn't happen to have gotten a transcript of that conversation, would you?...
    Friday, October 14, 2016 5:34 PM
  • Please Microsoft.....This can't possibly be the solution......... to build a new server....and only upgrade it to 11 Cumulative update versions ago and stop there,  and then migrate your entire environment to the new server, remap the users and rebuild every desktop .ost file.......

    Everyone here is experiencing the same problem.  Database is clean but not indexing new searches.  If I remove and reindex the search......then the search service crashes every several seconds while trying to crawl. New and moved databases hasn't fixed this issue and the issue is reported by many users.  

    Please Microsoft don't turn your back on your customers and your partner's customers.  My Client purchased Exchange server because it was the Microsoft standard email server.  Now, I have to tell them that to get the functionality of searching their mail and contacts back, they have to invest in thousands of dollars of retrofitting?  I don't understand this logic.

    Please direct me to the Microsoft Whitepaper that warned me 3 years ago to stop upgrading my Exchange Cumulative updates or I would lose critical functionality in the search engine.   

    Monday, October 17, 2016 4:33 PM
  • Please Microsoft.....This can't possibly be the solution......... to build a new server....and only upgrade it to 11 Cumulative update versions ago and stop there,  and then migrate your entire environment to the new server, remap the users and rebuild every desktop .ost file.......

    Everyone here is experiencing the same problem.  Database is clean but not indexing new searches.  If I remove and reindex the search......then the search service crashes every several seconds while trying to crawl. New and moved databases hasn't fixed this issue and the issue is reported by many users.  

    Please Microsoft don't turn your back on your customers and your partner's customers.  My Client purchased Exchange server because it was the Microsoft standard email server.  Now, I have to tell them that to get the functionality of searching their mail and contacts back, they have to invest in thousands of dollars of retrofitting?  I don't understand this logic.

    Please direct me to the Microsoft Whitepaper that warned me 3 years ago to stop upgrading my Exchange Cumulative updates or I would lose critical functionality in the search engine.   

    I believe he is referring to 2016 CU3, not Exchange 2013 CU3. I would highly urge you to open a case with Microsoft Support. Dont rely on the forums for an answer.

    Blog:    Twitter:   

    Monday, October 17, 2016 5:50 PM
    Moderator
  • Hi,

    Andy is right, I was referring to 2016 CU3.

    and no, it does not solve the problem. New server did exactly as the other, indexes do not update.

    Now I'm told there was a misunderstanding, try CU2. So that's what I'll do tomorrow.

    Monday, October 17, 2016 7:07 PM
  • Hello,

    same problem with 2016 CU3.

    Some DBs working, but most indexes do not update.

    Tuesday, October 18, 2016 6:36 AM
  • Hi

    We also opened a case with MS.

    In our case we noticed difference after CU14 installation. users complained that they are unable to search items past patch date.

    Then we made new DB and moved mailboxes to new database, still same issue.

    Then we deleted indexing and whole hell got loose.

    Same errors as people before and search service crashes every 29-39 second while trying to crawl. For us its good that we have multiple servers in DAG and we are letting that 1 node crash as keeping that node for rest databases as passive.

    At the moment we uploaded all logs to MS and waiting for feedback.

    Hi

    Update from our side is after we uploaded dmp files then our case were escalated to engineers . Seems there is a deeper "bug" in search engine.

    Tuesday, October 18, 2016 9:14 AM
  • Hi all! We have exactly the same problem at our customer. The funny thing is that they had some problems with the search results in OWA so we had proposed them an update from Exchange 2016 RTM to CU3 update first of all. After that we had stopped the search services, deleted the index folder and restart services to create index files again. It was 4 days ago, after that i found that contentindexstate is changing between failed and crawling and I have found event id 4999 and all the other problems that others had already mentioned before in this forum.

    Moreover we have another Exchange 2016 with CU3 installed without problems, contentindexstate is healthy and search services works fine.

    Please, if anyone knows the answer to the problem don't hesitate to share with us!

    Tuesday, October 18, 2016 11:05 AM
  • I just cannot believe it. After being struck by the DFS update that basically broke it, now we migrate from Exchange 2010 to Exchange 2013 and... THIS!

    We have only one database, and the indexing service has been working for hours until the crash-restart loop begun: EventId 4999, then some events about the Microsoft Exchange Search Service Starting and indexing started and crashing again 2 seconds later.

    Any advice will be VERY welcome.

    Tuesday, October 18, 2016 2:47 PM
  • just a fast update:

    installing a new 2016 CU3 did not help, a new db clould not be searched just as the one in the original server.

    so I removed the new 2016 CU3 and reinstalled from scratch a 2016 CU2. I then moved a small mailbox and indeed it's searchable. I can't be sure it will stay ok after 100 Gb go there, but there's hope.

    clearly this is not a practical solution in more complex situations, but in simple installations it may be doable.

    Tuesday, October 18, 2016 5:10 PM
  • Ok,

    I managed to move all contents from the crippled server to a new Exchange 2016 CU2 install and now search is ok.

    Clearly this is a rollback to the pre CU3 setup, so it's not a solution, but at least there is a way to get back your search.

    Since not all databases loose seach installing CU3 (in my case one survived, the other did not) I'd say it's somethng that depends on the db, or on events during the setup. Strange is the fact that moving data to a new db does not help, as moving data to a new CU3 server. In both cases indexes would not get updated.

    Wednesday, October 19, 2016 10:42 AM
  • How much data did you migrated and how long it was taken and what about your hardware specs?

    I have to get a solution for a single server, single database with ~500GB of data in 150-170 mailboxes. The search function is partially working (not all results shown) in OWA but I can't wait for the next CU. I hope there will be a hotfix in a few days but if not I have to prepare with a plan B. So I'm curious what can I do later, 'cause there's no light at the end of the tunnel now.

    Anyone with new information from Microsoft support?

    Wednesday, October 19, 2016 11:44 AM
  • How much data did you migrated and how long it was taken and what about your hardware specs?

    I have to get a solution for a single server, single database with ~500GB of data in 150-170 mailboxes. The search function is partially working (not all results shown) in OWA but I can't wait for the next CU. I hope there will be a hotfix in a few days but if not I have to prepare with a plan B. So I'm curious what can I do later, 'cause there's no light at the end of the tunnel now.

    Anyone with new information from Microsoft support?

    I have 100 users, 50gb in mailboxes, 100gb in public folders.

    First server in 4 cores 20gb, second (CU2) server is a 2 core 20gb vm. It's holding only one db (that happens to contain public folder mailboxes only). My plans are to wait for a while and then move the rest of the data, connectors and, all and get rid of the crippled CU3. I will then add 2 cores.

    moved 98Gb in 18 hours total, mostly during the night. no complains about performance, a few received requests to restart outlook, as usual.

    Wednesday, October 19, 2016 6:33 PM
  • Thanks for the info! This speed is similar as we have migrated ~470GB of mail data with an own imapsync script from a 5 years old linux server in march. It was taken for 4 days, from friday evening to thuesday morning. I have made a few Exchange to Exchange migrations too, they were all very slow. And you can't speed it up, it's not using the most of the resources to let the mail server usable under migration process. It's good but I think MS has to make some freedom for me to decide what I want. A fast migration or usable mail server while I'm moving data. I'll choose the fast migration with receiving mails only. But now we have no days for that because of an annoying bug. I hope MS will fix it quickly.
    Thursday, October 20, 2016 2:44 PM
  • Anyone with new information?
    Monday, October 24, 2016 9:23 AM
  • We are experiencing exactly the same issue. Users couldn't search anything in public folders database past the CU14 installation date. I tried rebuilding index on that database but it failed and the Search service keeps restarting exactly as described above by other users. New info?- Yet another company experience this issue and wait for solution. Now we cannot search anything in PF DB as index is empty due to unsuccessful rebuild attempt.

    Monday, October 24, 2016 10:52 AM
  • Same issue here with 2016 CU3. I just finished migrating all Mailboxes and Public Folders (130GB & 240GB) to a new installed server running CU2 - took me about 2 days. Luckily nobody worked on the weekend.

    Monday, October 24, 2016 8:48 PM
  • We have the same problem as well.  Have had to disable affected indexes so that the search service doesn't crash.

    You can use Outlook advanced search on the problem folders even when normal search doesn't work.

    • Proposed as answer by mikeyk16 Wednesday, October 26, 2016 2:41 PM
    • Unproposed as answer by mikeyk16 Wednesday, October 26, 2016 2:41 PM
    Tuesday, October 25, 2016 12:19 PM
  • We are having the same issue.   What I did to start addressing the issue is disable the indexes on all databases on the server that is having issues. (We are running Exchange Server 2013 CU14 in DAG.)  I then enabled each index one by one and checked the event log and also kept refreshing the Microsoft Exchange Services until I found the database that is causing the errors/flapping of the Microsoft Exchange Search Service.   Once i found it, i disabled that index and left the others to crawl until they were healthy.   So far all my indexes are now healthy except for the one that was causing the search service to flap, which is still disabled currently.  I will report back on this post as to how i fixed the issue with the DB.  I am examining all mailboxes in that DB to see if i can identify any issues.  Ill report back tonight if i have a total resolution.  Hope this helps you guys.   
    Wednesday, October 26, 2016 2:48 PM
  • We are having the same issue.   What I did to start addressing the issue is disable the indexes on all databases on the server that is having issues. (We are running Exchange Server 2013 CU14 in DAG.)  I then enabled each index one by one and checked the event log and also kept refreshing the Microsoft Exchange Services until I found the database that is causing the errors/flapping of the Microsoft Exchange Search Service.   Once i found it, i disabled that index and left the others to crawl until they were healthy.   So far all my indexes are now healthy except for the one that was causing the search service to flap, which is still disabled currently.  I will report back on this post as to how i fixed the issue with the DB.  I am examining all mailboxes in that DB to see if i can identify any issues.  Ill report back tonight if i have a total resolution.  Hope this helps you guys.   

    Move mailboxes from that DB perhaps...


    Blog:    Twitter:   

    Wednesday, October 26, 2016 3:15 PM
    Moderator
  • I have Ex2013CU14 single server - very simple setup with 11gb single database. Started to Follow MikeyK16 with a view to moving the mailboxes on to a new database one at a time until I find the duff mailbox.

    Set IndexEnable =$False on the mailbox database and public folder database (both showing IndexStatus Failed) stop the search services, delete the index files and start the search services.

    Create a new mailboxdatabase, wait a few minutes IndexStaus is Healthy on it. Restart the Information store service and index status is immediately Failed - Check the other databases and IndexStatus is still Disabled. So this is an "empty", new mailbox database. Stop the search, delete the Index files and restart the search - very quickly moves to failed.

    So not a mailbox corruption issue, not a search index corruption issue? Why does it fail on restart of the store service?

    ..ps Just re-enabled the index service on the original mailboxdatabase and its now healthy???

    • Edited by Viffer Saturday, October 29, 2016 9:59 AM More information
    Saturday, October 29, 2016 9:57 AM
  • That is exactly what I am doing.  I stood up a new DB and am moving users in the problematic DB to the new one, mailbox by mailbox, until i find the one causing the issue.  Best case scenario, the move requests to the new DB will fix the issue on its own.  Ill follow up once I have more info.
    Saturday, October 29, 2016 5:20 PM
  • MikeyK16 - For me its not a dodgy mailbox problem - how come the service fails on a new database with no mailboxes in it?

    Monday, October 31, 2016 9:54 AM
  • MikeyK16 - For me its not a dodgy mailbox problem - how come the service fails on a new database with no mailboxes in it?

    I agree, it's not a faulty mailbox. When trying to solve the problem I created a new db in a new 2016 CU3 server (see my previous posts) and it had the problem since the beginning, with a single mailbox created from scratch.

    In my environment one db was ok, the other was broken, and all new db that I created where broken. So it has to do with the db, but not because the db is corrupt, or if it is it's not something that can the fixed.

    Monday, October 31, 2016 7:34 PM
  • I've had the same issue as the OP. Error 4999, followed by two 1001 error reports. Search service crashing and restarting every minute or so.
    Problems started to occur after Exchange Standard 2016 CU3 installation.
    For me, users started to notice missing search results in Public Folders. After some attempts to fix things, every index on all databases started to fail.

    I disabled search indexing on the Databases that had Public Folders in them:

    Set-MailboxDatabase "Public Folder Database" -IndexEnabled $false

    After that, i stopped Search service + Search Host controller service, deleted all index folders in all Database locations, and started the search services again.
    Everything went back to Crawling, and eventually Healthy, as it supposed to do.
    As a test, i enabled indexing for a Public Folder Database, and promptly got error 4999 thrown back in my face along with corrupted indexes.

    For now, things are back to Crawling (without indexing of PF'S, off course).

    Could any of you guys trie and do the same?
    So (if necessary), move all PF mailboxes to a new database, disable indexing on that database, and reseed all indexes?

    Regards,

    Martijn

    • Proposed as answer by AdrianYardstcik Wednesday, January 18, 2017 4:44 PM
    Wednesday, November 02, 2016 9:15 AM
  • Hi

    I have a 2013 Env that I upgraded to CU14. and am facing the same issue tried all the fixes as mentioned in the blog but its still not working. I guess at this point there are 2 options available.

    Build a new Exchange server with CU13 and migrate to it.

    Raise a case with MS in the hope they have an immediate fix for the issue.

    Regards

    Dominic

    Wednesday, November 02, 2016 3:50 PM
  • Same here. 48 DBs on Exchange 2016 CU3. 0 Users in the DBs. Clustered. ~2 indices corrupted per week.

    Klaus


    Postmaster

    Thursday, November 03, 2016 2:06 PM
  • I can confirm this issue in Exchange 2013 CU14, Public Folder Searches; applied the patch this past weekend after waiting a month for feedback.  I stumbled across this forum after a lengthy search for the problem.  I haven't been able to find official recognition of this issue.  I've opened a case with MS and will report back any new information. 
    Thursday, November 03, 2016 9:57 PM
  • I contacted MS support regarding the public folder search issue, spent approx an hour explaining the situation and demonstrating the problem. Search index was verified healthy (I rebuilt them during pre escalation troubleshooting) and all other configuration checked OK.  

    MS confirmed that there is nothing publicly available regarding this problem and after insisting on a response, I received an email confirming the issue.  In a nutshell, their response was to wait for CU15.  They did not suggest a workaround / recovery plan.  I will send anyone that wants a copy of this confirmation by email.  The cost of my case was refunded.

    I did come up with a work around that may work for some people, it's simple but blunt.  In my case I need to search public folders, in addition to mailboxes, so I engaged cache mode in outlook, marked the public folder as a favorite and then added the public folder favorite to the local outlook cache.  Search is working.  Here are the steps:

    Switch to Outlook’s Folder List.
    Expand Public Folders and find the folder you want indexed.
    Right-click the folder and click Add to Favorites….
    Set any options or rename the favorite and click the Add button.
    Right-click on Public Folders and click Properties for “Public
    Folders”….
    Click the Advanced… button.
    Click the Advanced tab.
    Check Use Exchange Cached Mode and Download Public Folder
    Favorites.
    Click OK to close the Advanced property sheet.
    Click OK to close the Properties property sheet.
    Select your favorite folders or otherwise synchronize your client to
    download all items. This can take quite some time and is not recommended
    over remote connections.

    Now to script and deploy. 

    Credit to https://blogs.msdn.microsoft.com/heaths/2006/02/16/fast-searching-in-public-folders/ for the configuration steps.  I only used the steps above in my fix.  Now to script this.
    Friday, November 04, 2016 2:36 AM

  • MS confirmed that there is nothing publicly available regarding this problem 

    I also received a statement from a Microsoft employee (albeit via personal connections).

    This problem is a known issue, it's not clear when a fix will be available. Most probably a part of CU15 / CU4

    For me, i'm installing a new Exchange 2016 from scratch with CU2 and will move databases to it.

    Friday, November 04, 2016 2:32 PM
  • I contacted MS support regarding the public folder search issue, spent approx an hour explaining the situation and demonstrating the problem. Search index was verified healthy (I rebuilt them during pre escalation troubleshooting) and all other configuration checked OK.  

    MS confirmed that there is nothing publicly available regarding this problem and after insisting on a response, I received an email confirming the issue.  In a nutshell, their response was to wait for CU15.  They did not suggest a workaround / recovery plan.  I will send anyone that wants a copy of this confirmation by email.  The cost of my case was refunded.

    I did come up with a work around that may work for some people, it's simple but blunt.  In my case I need to search public folders, in addition to mailboxes, so I engaged cache mode in outlook, marked the public folder as a favorite and then added the public folder favorite to the local outlook cache.  Search is working.  Here are the steps:

    Switch to Outlook’s Folder List.
    Expand Public Folders and find the folder you want indexed.
    Right-click the folder and click Add to Favorites….
    Set any options or rename the favorite and click the Add button.
    Right-click on Public Folders and click Properties for “Public
    Folders”….
    Click the Advanced… button.
    Click the Advanced tab.
    Check Use Exchange Cached Mode and Download Public Folder
    Favorites.
    Click OK to close the Advanced property sheet.
    Click OK to close the Properties property sheet.
    Select your favorite folders or otherwise synchronize your client to
    download all items. This can take quite some time and is not recommended
    over remote connections.

    Now to script and deploy. 

    Credit to https://blogs.msdn.microsoft.com/heaths/2006/02/16/fast-searching-in-public-folders/ for the configuration steps.  I only used the steps above in my fix.  Now to script this.
    You must not be using Outlook 2016 then! 

    Blog:    Twitter:   

    Friday, November 04, 2016 2:47 PM
    Moderator
  • I have the same problems,

    as workaround i have disabled the context indexing for the public folder Database on my two Exchange 2013 CU 14 DAG Servers.

    after disabling the Search Service no more crash.

    For me helps this temporary.

    Monday, November 07, 2016 7:07 AM
  • Has anyone rolled back to CU13? According to posts here and here, CU14 has to be completely uninstalled from the server before installing CU13. We have a customer where we've deployed eight CU14 servers and were about to proceed with migrations until we discovered this issue. The plan now is to uninstall CU14 and deploy CU13 on all servers. Would it be recommended to also re-install the base OS to ensure no artifacts are left behind from the CU14 install?


    Tuesday, November 08, 2016 7:09 PM
  • Has anyone rolled back to CU13? According to posts here and here, CU14 has to be completely uninstalled from the server before installing CU13. We have a customer where we've deployed eight CU14 servers and were about to proceed with migrations until we discovered this issue. The plan now is to uninstall CU14 and deploy CU13 on all servers. Would it be recommended to also re-install the base OS to ensure no artifacts are left behind from the CU14 install?


    Hi,

    to rollback you need to install a new CU13 server and move all workloads to it, then remove the old one. There is no safe way to downgrade in-place. If you can uninstall CU14 then I suppose you removed all load (or plan to do so), so at this point I'd reinstall the OS. This is what I did, and there were no issues.

    Uninstalling CU14 restarting and installing CU13 is probably ok, and a bit faster, but not much if you save the patched and prepared image of a fresh OS, so why run the risk?

    Thursday, November 10, 2016 8:11 AM
  • On this part---    must not be using Outlook 2016 --
      

    Just to verify we understand that statement.

    Outlook 2016  +  Exchange 2016  =  will use the Exchange Index / Search.

    Outlook 2016  +  Exchange 2013  =  will use the Windows desktop search.

    Correct ?

    Thanks.

    ==

    Thursday, November 10, 2016 2:48 PM
  • I can confirm the issue was related to the public folders.  We had one public folder from years ago that wasn't being used.  I deleted the public folder and once the indexing was complete, i no longer saw the flapping or 4999 errors.   You can also disable indexing for the PF.  Hope this helps.
    Wednesday, November 30, 2016 3:43 PM
  • On this part---    must not be using Outlook 2016 --
      

    Just to verify we understand that statement.

    Outlook 2016  +  Exchange 2016  =  will use the Exchange Index / Search.

    Outlook 2016  +  Exchange 2013  =  will use the Windows desktop search.

    Correct ?

    Thanks.

    ==

    There is a bug in Outlook 2016. Even if you check download PF favorites and cache them, it does not actually work and no caching is done.

    Blog:    Twitter:   

    Wednesday, November 30, 2016 3:49 PM
    Moderator
  • what is latest on it ? What is solution / workaround ?

    Temporary i needed disable search service for database where public folder mailbox is stored

    The Microsoft Exchange Search service terminated unexpectedly.

    Get-MailboxDatabase DatabasewithPublicFolderMailbox | set-mailboxdatabase -IndexEnabled $false

    Friday, December 02, 2016 10:01 AM
  • Hey Guys.

    Wanted to let you know that we got a feedback from microsoft (after long time monitoring and gathering logs)

    That cause for indexing crashes has been found and patch implemented but wont be released as independent download/patch :(

    For that we need to wait for next CU for both exchange 13 and 16.

    Friday, December 02, 2016 11:41 AM
  • Thanks for the update.

    Thomas Stensitzki - MCSM, MCM, MCSE, MCSA, MCITP - Blog: http://justcantgetenough.granikos.eu/

    Monday, December 05, 2016 8:25 AM
  • After installing CU14 same problem on two Exchange 2013.
    Tuesday, December 06, 2016 8:18 AM
  • Do you know when Exchange 2013 CU15 is expected to be released? We are on Exchange 2013 CU9 right now and have a maintenance window to upgrade in 2 weeks. I am wondering if I should just go up to CU13 or wait till CU15.
    Tuesday, December 06, 2016 9:05 PM
  • Do you know when Exchange 2013 CU15 is expected to be released? We are on Exchange 2013 CU9 right now and have a maintenance window to upgrade in 2 weeks. I am wondering if I should just go up to CU13 or wait till CU15.

    Typically 6-8 weeks after the previous CU. So maybe this month.


    Blog:    Twitter:   

    Tuesday, December 06, 2016 9:58 PM
    Moderator
  • I found a solution to the issue, it has to do with the indexing on the public folders database (cannot complete the indexing on that specific database and crashes the search service). I found the database that the public folders resides on was always crawling or in a failed state. If you have a public folders mailbox on a database that does not need indexing turn the indexing off on that database then restart the search services. If you have the public folders mailbox on a database that needs searching/indexing create a new database and migrate the public folders mailbox to it (using the new move request) and then turn indexing off, then restart the search services and it will be fixed. 

    This way you don't have to wait till CU 15 whenever it may come out.


    Thursday, December 08, 2016 12:20 AM
  • I´ve the same problem with CU14 / Ex2013. We have two DB´s. One for Mailboxes, one for public folders.
    "Get-MailboxDatabaseCopyStatus | FL Name,*Index*" says "ContentIndexState failed" on both DB´s. One minute later the same command says "Healthy" for the Mailbox-DB and "Crawling" for the Public folder Database and so on.
    You can also watch this with the performance monitor: MSExchange Search Indexes --> Crawler: Mailboxes Remaining. You can see crawling ist starting and aborting after a short time.
    ... Waiting für Patch / CU15 ...
    Tuesday, December 13, 2016 10:21 AM
  • disable indexing on the public folder database then re-index the database with your mailboxes.
    Tuesday, December 13, 2016 7:26 PM
  • Exchange 2013 CU15 and Exchange 2016 CU4 have been released.

    Issues with PF indexing have been fixed.

    Tuesday, December 13, 2016 10:37 PM
  • Haven't tried this one yet, this note on the KB for the failing indexing kind of scares me: "After you install this update on either version of Exchange Server, you should move the public folder mailbox to a new database to make sure that all items are indexed correctly."

    Really? Isn't enough just deleting the GUID folder to force indexes to be rebuilt?

    Wednesday, December 14, 2016 9:36 AM
  • After installing the CU15, stop HostControllerService and MSExchangeFastSearch, delete the GUID folder and restart services to force index rebuild work for me. 

    Monday, December 19, 2016 1:25 PM
  • Merci beaucoup Olivier, it worked. The index has been rebuilt and searches are working again.

    • Edited by apsicsm Wednesday, January 04, 2017 6:29 PM typo
    Wednesday, January 04, 2017 6:28 PM
  • A similar thread related to Exchange 2016 fixed the situation by change the database activation preference.

    https://social.technet.microsoft.com/Forums/en-US/614068c5-205c-4ee6-ac10-c880cda441dc/content-index-problems-after-exchange-2016-cu1-in-dag-environment?forum=Exch2016Adm

    Maybe this helps in your scenario as well.

    Cheers,
    Thomas


    Thomas Stensitzki - MCSM, MCM, MCSE, MCSA, MCITP - Blog: http://justcantgetenough.granikos.eu/

    Wednesday, January 11, 2017 9:34 AM
  • I've had the same issue as the OP. Error 4999, followed by two 1001 error reports. Search service crashing and restarting every minute or so.
    Problems started to occur after Exchange Standard 2016 CU3 installation.
    For me, users started to notice missing search results in Public Folders. After some attempts to fix things, every index on all databases started to fail.

    I disabled search indexing on the Databases that had Public Folders in them:

    Set-MailboxDatabase "Public Folder Database" -IndexEnabled $false

    After that, i stopped Search service + Search Host controller service, deleted all index folders in all Database locations, and started the search services again.
    Everything went back to Crawling, and eventually Healthy, as it supposed to do.
    As a test, i enabled indexing for a Public Folder Database, and promptly got error 4999 thrown back in my face along with corrupted indexes.

    For now, things are back to Crawling (without indexing of PF'S, off course).

    Could any of you guys trie and do the same?
    So (if necessary), move all PF mailboxes to a new database, disable indexing on that database, and reseed all indexes?

    Regards,

    Martijn

    Martin,

    Although this is not a fix for the actual problem with the Indexing service, I must say that this was the ONLY acceptable solution for us. I must have read at least 25 articles affiliated with this issue and nothing worked. I was only able to get indexing to properly crawl by performing what you said.

    I have only 1 exchange server(no DAG) . I noticed that some Admins posted that they moved all the mailboxes to a new server. I personally will not accept that as a solution... it's a ridiculous amount of work. MS needs to find a fix; this cannot be placed on the customers.

    What you did here is a great workaround... until we get a CU that will fix this, what I consider, critical issue. KUDOS for the good work and thank you for sharing this with us. 

    AdrianP




    Wednesday, January 18, 2017 4:42 PM
  • For anyone having this issue on Exchange 2013:

    We had the same issue, indexing worked on databases without public folder mailboxes but crashed when indexing databases with public folder mailboxes. Simple test is to disable indexing on the databases with public mailboxes. 

    Set-MailboxDatabase "Database Name" -IndexEnabled $False #Repeat for all Databases with Public Folders

    Get-MailboxDatabaseCopyStatus #Check Status of indexing

    If this resolves your issue you need KB 3202691 - Included in CU 15


    https://support.microsoft.com/en-us/help/3202691/public-folders-indexing-doesn-t-work-correctly-after-you-apply-latest-cumulative-updates-for-exchange-server

    https://support.microsoft.com/en-us/help/3197044/cumulative-update-15-for-exchange-server-2013

    Pete B

    Friday, March 10, 2017 11:22 AM