none
[WORKAROUND] Windows Search Service Terminated Unexpectedly; Faulting application: SearchIndexer.exe

    Question

  • A number of users with laptops have reported that searches in Outlook either don't work or are incomplete.  All of these users are using Outlook in cached mode, and almost all of them were imaged recently.  I got my hands on one of these machines and discovered that the Windows Search service seems to tank randomly.  The 'SearchIndexer.exe' process seems to crash also.  It appears to be an issue with the machines as a whole versus the profile since I encountered the same problem when I logged on as myself.  (This was the first time I logged onto this machine.)  On Friday I logged on as myself for the first time, build the OST & left it on over the weekend to index.  I checked the machine this morning and found the System event log full of errors :

    • The Windows Search service terminated unexpectedly.  It has done this 5785 time(s). (Event ID: 7034; Source: Service Control Manager; OpCode: Info)
    • The Windows Search service entered the running state.
    • Wash, Rinse & Repeat (more details below)

    I went through the following to no avail:

    Every time it gets up to a point (like 27kish) then breaks and starts the stop/start service process again.  Its hard to say what its choking on since I'm not aware of a way to 'parse' the mail file or enable super-verbose logging to see what its having trouble with.

    The only indexed locations are 'Offline Files' for my user, and 'Microsoft Office Outlook' for my user as well.  We're not indexing any drives, libraries or network locations, even via the symlink 'hack'.  This is a new profile so there's not much else here in terms of files to index.

    At the same time of setting up the 'problematic' machine on Friday, I imaged & logged into 3 laptops as myself to build and index the OST.  All three machines finished indexing over 57k items without a problem, and there are no issues in the Application or System event logs.

    This is Windows 7 with just about every update available prior to the release of SP1 and Office Enterprise 2007 with SP2.  (Outlook 12.0.6539.5000 SP2 MSO 12.0.6535.5002)

    Short messing around with the 'Indexing Service' (the Windows Feature), uninstalling/reinstalling WDS reimaging this machine, I don't know what else to try.

    ------------------------------

    APPLICATION LOG
    Outlook Search has encountered an error and is temporarily disabling indexing for store C:\Users\jgp2750\AppData\Local\Microsoft\Outlook\outlook.ost (error=0x800706ba).
    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Outlook" />
      <EventID Qualifiers="32768">36</EventID>
      <Level>3</Level>
      <Task>0</Task>
      <Keywords>0x80000000000000</Keywords>
      <TimeCreated SystemTime="2011-04-18T14:32:45.000000000Z" />
      <EventRecordID>218711</EventRecordID>
      <Channel>Application</Channel>
      <Computer>FQDN</Computer>
      <Security />
      </System>
    - <EventData>
      <Data>C:\Users\jgp2750\AppData\Local\Microsoft\Outlook\outlook.ost</Data>
      <Data>0x800706ba</Data>
      </EventData>
      </Event>

    Faulting application name: SearchIndexer.exe, version: 7.0.7600.16385, time stamp: 0x4a5bcdd0
    Faulting module name: NLSData0001.dll, version: 6.1.7600.16385, time stamp: 0x4a5bda85
    Exception code: 0xc0000005
    Fault offset: 0x00260a0d
    Faulting process id: 0xb38
    Faulting application start time: 0x01cbfdd577062774
    Faulting application path: C:\Windows\system32\SearchIndexer.exe
    Faulting module path: C:\Windows\System32\NLSData0001.dll
    Report Id: b967dd6d-69c8-11e0-99bd-0024d60226ee
    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Application Error" />
      <EventID Qualifiers="0">1000</EventID>
      <Level>2</Level>
      <Task>100</Task>
      <Keywords>0x80000000000000</Keywords>
      <TimeCreated SystemTime="2011-04-18T14:32:45.000000000Z" />
      <EventRecordID>218710</EventRecordID>
      <Channel>Application</Channel>
      <Computer>FQDN</Computer>
      <Security />
      </System>
    - <EventData>
      <Data>SearchIndexer.exe</Data>
      <Data>7.0.7600.16385</Data>
      <Data>4a5bcdd0</Data>
      <Data>NLSData0001.dll</Data>
      <Data>6.1.7600.16385</Data>
      <Data>4a5bda85</Data>
      <Data>c0000005</Data>
      <Data>00260a0d</Data>
      <Data>b38</Data>
      <Data>01cbfdd577062774</Data>
      <Data>C:\Windows\system32\SearchIndexer.exe</Data>
      <Data>C:\Windows\System32\NLSData0001.dll</Data>
      <Data>b967dd6d-69c8-11e0-99bd-0024d60226ee</Data>
      </EventData>
      </Event>


    Faulting application name: SearchIndexer.exe, version: 7.0.7600.16385, time stamp: 0x4a5bcdd0
    Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
    Exception code: 0xc0000005
    Fault offset: 0x0000004c
    Faulting process id: 0x10c8
    Faulting application start time: 0x01cbfdd546f303ec
    Faulting application path: C:\Windows\system32\SearchIndexer.exe
    Faulting module path: unknown
    Report Id: 87c5842d-69c8-11e0-99bd-0024d60226ee
    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Application Error" />
      <EventID Qualifiers="0">1000</EventID>
      <Level>2</Level>
      <Task>100</Task>
      <Keywords>0x80000000000000</Keywords>
      <TimeCreated SystemTime="2011-04-18T14:31:22.000000000Z" />
      <EventRecordID>218698</EventRecordID>
      <Channel>Application</Channel>
      <Computer>FQDN</Computer>
      <Security />
      </System>
    - <EventData>
      <Data>SearchIndexer.exe</Data>
      <Data>7.0.7600.16385</Data>
      <Data>4a5bcdd0</Data>
      <Data>unknown</Data>
      <Data>0.0.0.0</Data>
      <Data>00000000</Data>
      <Data>c0000005</Data>
      <Data>0000004c</Data>
      <Data>10c8</Data>
      <Data>01cbfdd546f303ec</Data>
      <Data>C:\Windows\system32\SearchIndexer.exe</Data>
      <Data>unknown</Data>
      <Data>87c5842d-69c8-11e0-99bd-0024d60226ee</Data>
      </EventData>
      </Event>


    SYSTEM LOG
    The Windows Search service terminated unexpectedly.  It has done this 5831 time(s).
    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
      <EventID Qualifiers="49152">7034</EventID>
      <Version>0</Version>
      <Level>2</Level>
      <Task>0</Task>
      <Opcode>0</Opcode>
      <Keywords>0x8080000000000000</Keywords>
      <TimeCreated SystemTime="2011-04-18T14:34:52.973976600Z" />
      <EventRecordID>74749</EventRecordID>
      <Correlation />
      <Execution ProcessID="520" ThreadID="4604" />
      <Channel>System</Channel>
      <Computer>FQDN</Computer>
      <Security />
      </System>
    - <EventData>
      <Data Name="param1">Windows Search</Data>
      <Data Name="param2">5831</Data>
      </EventData>
      </Event>

    The Windows Search service entered the running state.
    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
      <EventID Qualifiers="16384">7036</EventID>
      <Version>0</Version>
      <Level>4</Level>
      <Task>0</Task>
      <Opcode>0</Opcode>
      <Keywords>0x8080000000000000</Keywords>
      <TimeCreated SystemTime="2011-04-18T14:35:30.537732600Z" />
      <EventRecordID>74750</EventRecordID>
      <Correlation />
      <Execution ProcessID="520" ThreadID="1364" />
      <Channel>System</Channel>
      <Computer>FQDN</Computer>
      <Security />
      </System>
    - <EventData>
      <Data Name="param1">Windows Search</Data>
      <Data Name="param2">running</Data>
      <Binary>57005300650061007200630068002F0034000000</Binary>
      </EventData>
      </Event>

     




    Monday, April 18, 2011 2:39 PM

Answers

  • I've got a positive update for this issue here.  Credit goes to my colleage - he's a rockstar!

    Random nlsdataxxxx.dll files in the system32 directory are getting corrupted in our environment.  The corruption causes searchindexer.exe behavior from crashing to cpu “spinning”.   A system file scanner check (sfc /scannow) discovered these errors.  Replacing the corrupted DLL’s with good DLL’s resolves the indexing problem.

    For example in corrupted file nldata002a.dll, comparing the good and bad, the bad DLL has bogus data from E000 to F000.

    It is also disturbing that sfc says the source files are also corrupt.

    Currently my solution is to run “sfc /scannow” and review the log at c:\windows\logs\cbs\cbs.log to determine the corrupt file.  I then replace the corrupt file with a file from a good machine. 

    Note: the corrupt file will have the same size, version, timestamp as the good file.  It may be easier just to script the replacement of all the NLSDataxxxx.dll files.

    Still unknown as to why the files are getting corrupted on certain machines and not others.

     

    The CBS.log was too large for pastebin.com so here's a link to it for further analysis: http://www.juli.us/support/forums/technet_IndexSearch_CBS.log

    For now we're going to take ownership of the NlsData*.dll files, then replace them with good copies, stored on the network, which were taken from a known working machine.  The areas highlighted in bold above are big concerns.  Its one thing for a file to get corrupt, and another for the source file(s) to get corrupt also.  But for a block of data within a file to get corrupt - that's very specific.  (We used WinDiff to compare both DLL's and a little ways down you see data, but it's zero'd out!  No screen shot at the moment)  This is pretty scary. 

    We've got a 'fix' but we don't know the cause, so we'll always have this gray cloud looming over us.  I'm open to other ideas, but I think I'm going to close this thread by the end of the week if there isn't any more input.

    Here's a good resource for reading .CBS logs: http://support.microsoft.com/kb/928228
    • Marked as answer by JuliusPIV Friday, March 30, 2012 4:54 PM
    Tuesday, May 03, 2011 9:12 PM

All replies

  • I am having essentially the same problem. 
    Monday, April 18, 2011 11:19 PM
  • Hi,

    Thanks for the post!

    It seems a known issue in Outlook 2007, please refer to the following two  KB articles to check if it works:

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;927676

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;930010 

    Since it's related to the Office issue, I recommend also ask this question in Office forum.

    Thanks for your understanding and cooperation!

    Regards,

    Miya


    This posting is provided "AS IS" with no warranties, and confers no rights. | Please remember to click "Mark as Answer" on the post that helps you, and to click "Unmark as Answer" if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
    Wednesday, April 20, 2011 7:46 AM
  • Morning - thank you for the responses all.
    Miya Yao:
      I didn't see any '3036' Event ID's in the Application or System logs, mostly:
    • Event ID 1000 SearchIndexer.exe
    • Event ID 36 - Outlook Search has encountered an error and is temporarily disabling indexing for store C:\Users\%username%\AppData\Local\Microsoft\Outlook\Outlook.ost (error=0x80000ffff)
    • Event ID 7034 The Windows Search service terminated unexpectedly.  It has done this 5674 time(s).
    • Event ID 7036 The Windows Search service entered the running state.
      This is a regular user's machine, not an IT machine so the 'Exchange System Manager tool' (or tools) shouldn't be present on the machine.  Despite that however I ran 'fixmapi', noticed my first 3036 Event ID (The content source <mapi://{S-1-5-21-1234567890-987654321-135792468-01928}/> cannot be accessed.) in the Application Event log & rebooted.  After a the reboot, I rebuilt the index, noticed another 3036 Event during the process, but it ultimately continues to fail around/after indexing 26k items.
      I could post in the Office suite, but it seems that there's may be an underlying issue with the Windows Search system.  If you think its better to post there, I'll do so.

    Microsoft Provided a Windows Search Diagnostics tool (v6.1.6511.1) to help troubleshoot.  I don't dare paste the log here since its uber long (but I've made it available on pastebin) so here are the highlights:
    --BEGIN
      Running Search Version Check:...
      Search Version Check: Passed
      Running Setup Check: Verify that Search is installed.  Check Search Version.  Check if the registry settings are ...
      Setup Check: Failed
      C:\ProgramData\Microsoft\Search\Data\ missing or has wrong permissions
      Running Event Log Check: Scan the event log for errors....
      Event Log Check: Passed
      Running Outlook Check: Check if Outlook is running and verify its configuration....
      Outlook Check: Passed
      Running Indexing Statistics: Run common queries and report the results...
      Indexing Statistics: Failed
      Running Indexing Progress: Periodically check the progress of indexed files....
      Indexing Progress: Failed
      Running CPU and Memory Usage: Check the computer's CPU and memory usage....
      CPU and Memory Usage: Passed
      Running Disk Space Check: Check disk space usage of all Search related processes and files....
      Disk Space Check: Passed
      Running Filter Check: Check whether Search filters with known issues are installed...
      Running Battery Check: Check whether the system is running on battery/AC power...
      Battery Check: Failed
      *** Search Validation Filaed ***
      Please click the log file link below for details
    --EOF
     The utility also has options to start iDNA and ETW Traces:
    • iDNA Trace fails (tttracer.exe exits with a status of 1) and it creats a zero byte file.
    • ETW Trace appears to succeed (it launches) and runs until I terminate it (logman.exe exits with a status of 0).  It creates a .ETL file, which I can't access; apparently I'm missing a suite of tools that include MSSLogToDatabase, MSSLogToText (among others? I'm sure) to properly extract (or read) the data contained within.
    I'll likely be sending the logs to them for further analysis but we'll see.
    Open to ideas from all.  Thanks again.



    Wednesday, April 20, 2011 1:24 PM
  • I've got a positive update for this issue here.  Credit goes to my colleage - he's a rockstar!

    Random nlsdataxxxx.dll files in the system32 directory are getting corrupted in our environment.  The corruption causes searchindexer.exe behavior from crashing to cpu “spinning”.   A system file scanner check (sfc /scannow) discovered these errors.  Replacing the corrupted DLL’s with good DLL’s resolves the indexing problem.

    For example in corrupted file nldata002a.dll, comparing the good and bad, the bad DLL has bogus data from E000 to F000.

    It is also disturbing that sfc says the source files are also corrupt.

    Currently my solution is to run “sfc /scannow” and review the log at c:\windows\logs\cbs\cbs.log to determine the corrupt file.  I then replace the corrupt file with a file from a good machine. 

    Note: the corrupt file will have the same size, version, timestamp as the good file.  It may be easier just to script the replacement of all the NLSDataxxxx.dll files.

    Still unknown as to why the files are getting corrupted on certain machines and not others.

     

    The CBS.log was too large for pastebin.com so here's a link to it for further analysis: http://www.juli.us/support/forums/technet_IndexSearch_CBS.log

    For now we're going to take ownership of the NlsData*.dll files, then replace them with good copies, stored on the network, which were taken from a known working machine.  The areas highlighted in bold above are big concerns.  Its one thing for a file to get corrupt, and another for the source file(s) to get corrupt also.  But for a block of data within a file to get corrupt - that's very specific.  (We used WinDiff to compare both DLL's and a little ways down you see data, but it's zero'd out!  No screen shot at the moment)  This is pretty scary. 

    We've got a 'fix' but we don't know the cause, so we'll always have this gray cloud looming over us.  I'm open to other ideas, but I think I'm going to close this thread by the end of the week if there isn't any more input.

    Here's a good resource for reading .CBS logs: http://support.microsoft.com/kb/928228
    • Marked as answer by JuliusPIV Friday, March 30, 2012 4:54 PM
    Tuesday, May 03, 2011 9:12 PM
  • I've got a positive update for this issue here.  Credit goes to my colleage - he's a rockstar!

    Random nlsdataxxxx.dll files in the system32 directory are getting corrupted in our environment.  The corruption causes searchindexer.exe behavior from crashing to cpu “spinning”.   A system file scanner check (sfc /scannow) discovered these errors.  Replacing the corrupted DLL’s with good DLL’s resolves the indexing problem.

    For example in corrupted file nldata002a.dll, comparing the good and bad, the bad DLL has bogus data from E000 to F000.

    It is also disturbing that sfc says the source files are also corrupt.

    Currently my solution is to run “sfc /scannow” and review the log at c:\windows\logs\cbs\cbs.log to determine the corrupt file.  I then replace the corrupt file with a file from a good machine. 

    Note: the corrupt file will have the same size, version, timestamp as the good file.  It may be easier just to script the replacement of all the NLSDataxxxx.dll files.

    Still unknown as to why the files are getting corrupted on certain machines and not others.

    The CBS.log was too large for pastebin.com so here's a link to it for further analysis: http://www.juli.us/support/forums/technet_IndexSearch_CBS.log

    For now we're going to take ownership of the NlsData*.dll files, then replace them with good copies, stored on the network, which were taken from a known working machine.  The areas highlighted in bold above are big concerns.  Its one thing for a file to get corrupt, and another for the source file(s) to get corrupt also.  But for a block of data within a file to get corrupt - that's very specific.  (We used WinDiff to compare both DLL's and a little ways down you see data, but it's zero'd out!  No screen shot at the moment)  This is pretty scary. 

    We've got a 'fix' but we don't know the cause, so we'll always have this gray cloud looming over us.  I'm open to other ideas, but I think I'm going to close this thread by the end of the week if there isn't any more input.

    Here's a good resource for reading .CBS logs: http://support.microsoft.com/kb/928228

    I had the same issue with nlslexicons0416.dll being corrupt. I copied from another computer and Windows Search immediately started.

    Wednesday, March 28, 2012 4:03 PM