Asked by:
content index failed

Question
-
Hello,
For a few days now, our search index has stopped working.
I skimmed a lot of other forum and topic before coming to try my luck here.So to summarize, we do not have any search index on Outlook / Owa anymore. The value "contentindexstate" is failed with the message An internal error occurred for the database or its index.
We are in Exchange version 2016 CU 12 without DAG.
We have events MSExcahngeFastSearch 1006/1010/1009 and MsExcahnge Common 4999.We have tried well on all the classic solutions:
-
Recreate the index (stop / start service search and search host controler with delete the folder * .Single)
-
Add ContentSubmitters group
-
Re-create the Hostcontroller nodes with modifications to the WcfConfigurator files to remove the value <AuthorizedRole> ContentSubmitters </ AuthorizedRole>
This is to do a little tour of the situation.
If you have ideas that could help us, we thank you in advance. -
All replies
-
I would start by looking at your APPLICATION event log and filter it to only show critical and error events. Look for any issues regarding the databases since its very possible there are problems within the DB. Obviously you want to clear any such issues up asap. You should also do the same for the SYSTEM event log, but look for disk related issues since that could be a cause as well
Search, Recover, Export Mailboxes, Contacts, Calendars, Tasks from ALL versions of Exchange Offline EDB's, On-Premises Exchange Databases & Office 365. Export, Migrate/Recover into On-Premises Exchange Server, Office 365 with Lucid8's DigiScope
- Proposed as answer by Niko.ChengMicrosoft contingent staff, Moderator Monday, May 6, 2019 7:07 AM
-
Hey Niko ,
Thanks for your answer.
Here critical and error log events, we have in Application event log :
- Msexchange Common 4999 :
Watson report about to be sent for process id: 39844, with parameters: E12IIS, c-RTL-AMD64, 16.16.0120.013, NodeRunner#ContentEngineNode1, M.C.NlpBase.LanguageAndEncodingDetection, M.C.N.L.ILaED.IdentifyLanguage, S.Reflection.TargetInvocationException, da5f-dumptidset, 16.16.0120.013.
ErrorReportingEnabled: False- .NET Runtime 1023 :
Application : NodeRunner.exe
Version du Framework : v4.0.30319
Description : le processus a été arrêté en raison d'une erreur interne dans le runtime .NET à l'adresse IP 00007FF80E7A19D8 (00007FF80E720000) avec le code de sortie 80131506.- Application Error 1000
Nom de l’application défaillante NodeRunner.exe, version : 16.16.120.13, horodatage : 0x593aa40e
Nom du module défaillant : clr.dll, version : 4.7.3394.0, horodatage : 0x5c5371fc
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00000000000819d8
ID du processus défaillant : 0x27d8
Heure de début de l’application défaillante : 0x01d5070f94adea6d
Chemin d’accès de l’application défaillante : E:\Exchange2016\Bin\Search\Ceres\Runtime\1.0\ResourceProfile\contentengine\NodeRunner.exe
Chemin d’accès du module défaillant: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
ID de rapport : 11c2b1a2-7303-11e9-814b-005056bf12b4
Nom complet du package défaillant :
ID de l’application relative au package défaillant :- MsExchangeDiagnostics 1006
The performance counter '\\EXCHANGE\MSExchange Assistants - Per Database(msexchangemailboxassistants-total)\Quarantined Assistant Count Total' sustained a value of '5 771,00', for the '10' minute(s) interval starting at '10/05/2019 08:40:00'. Threshold breached since '30/04/2019 10:20'. None Trigger Name:AssistantsQuarantinedTrigger. Instance:msexchangemailboxassistants-total
Side System Event :
- Schannel 36887
Une alerte irrécupérable a été reçue du point de terminaison distant. Le code d’alerte irrécupérable défini par protocole de TLS est 46.
I have already tried to find information on these events but without sucess.
I don't know if those events talk to you.The last thing I noticed lately is that the noderunner contentengine loads in memory up to about 512Mo and crash. Which gives the event 4999.
I have already tried to increase the memory of the server and modify the config file noderunnner not to limit the memory or set the limit to 2048MO.
thank you in advance if you can help me on this incident.
-
- is this on a physical box or a VM
- Also check the links below
- https://social.technet.microsoft.com/Forums/windows/en-US/54cec6d8-4edc-4596-ada8-6ea9fd249bbf/event-id-1006-the-performance-counter-msexchange-assistants-per-databaseevent-dispatchers?forum=Exch2016GD
- https://blogs.technet.microsoft.com/manjubn/2014/04/14/exchange-2013-search-not-working-3-noderunner-exe-process-terminated-with-an-outofmemoryexception/
Search, Recover, Export Mailboxes, Contacts, Calendars, Tasks from ALL versions of Exchange Offline EDB's, On-Premises Exchange Databases & Office 365. Export, Migrate/Recover into On-Premises Exchange Server, Office 365 with Lucid8's DigiScope
-
-
OK since its a virtual server I would suggest you look at the memory allocation as I suspect you are experiencing a paging issue. Try reserving more of true memory from the host in order to avoid using a paging type memory
Search, Recover, Export Mailboxes, Contacts, Calendars, Tasks from ALL versions of Exchange Offline EDB's, On-Premises Exchange Databases & Office 365. Export, Migrate/Recover into On-Premises Exchange Server, Office 365 with Lucid8's DigiScope
-
Hi ,
Thanks for your help.
I will try to find a way to do that.We try both of 2 links but no one works.
To complete, we noticed that the process noderunner contentengine crash when it arrives at 500MO memory used.
The server on its side is not loaded in memory (envirion 50% of free)
Regards,
-
Hi QD,
Have you tried Troy's suggestion? How about the result ?
If the issue still cannot be resolved, I'd recommend you open a ticket to help you dig this issue deeply.
Best Regards,
Niko Cheng
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.
Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams. -
Hi Niko ,
I try to isolate the virtual machine on a server alone. We still have crash for noderunner
Best, option we have is to wait for CU 13 or migrate to O365.
We opened a support ticket with our software assurance. They could not help us any more than with solutions we had already tested.
Regards,
-
Hi ITQD,
In my opinion, migrate to Office 365 would be the best option.
Best Regards,
Niko Cheng
Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.
Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.