none
ContentIndexState Failed After Activating Passive Copy

    Question

  • I am having sporadic issues with the indexes failing on some of my databases.  Running get-mailboxdatabasecopystatus -server server | ft Name,*index* shows that all of the indexes are healthy.  Test-replicationhealth indicates all tests passing as well.  Everything looks OK until I try to activate a passive copy.  Typically, the copy will activate for a few seconds, then activate one of the other copies and I will see the index as failed on the one I initially tried to activate.  Sometimes more than one standby copy will indicate that it has a failed index after this.  I've already tried the script to reset the index a few times and the problem keeps returning.  Some of the databases that are failing don't even have any active users on them yet.

    We are running a three node DAG with a copy of the databases on each server.  Two of the DAG members are in one location (on the same network segment) and the other is offsite, but connected by a LAN quality connection (10Gb).  All have Service Pack 3 installed.  We are in the process of migrating users off of 2007.  Any help is appreciated.  Thanks in advance!



    Thursday, May 23, 2013 9:31 PM

All replies

  • Any luck with a manual reseed?
    Friday, May 24, 2013 1:55 AM
  • Hi,

    Can you check the server eventvwr and post the error message?

    Regards


    Ganga Enterprise Messaging Administrator-Exchange 2010. Windows 2008 Active Directory Configuring.

    Friday, May 24, 2013 2:12 AM
  • Any luck with a manual reseed?

    Yes - Manual reseed with -catalogonly fixes the issue, but the issue eventually returns.  It seems I can activate the other copies without a problem for a week or so.  If I let them go longer and try to do it I'll get the content index failure again.  I've also tried reseeding both the database and index, but no luck.
    Friday, May 24, 2013 3:07 AM
  • Hi,

    Can you check the server eventvwr and post the error message?

    Regards


    Ganga Enterprise Messaging Administrator-Exchange 2010. Windows 2008 Active Directory Configuring.

    Here's the warning I saw just before the failure today...

    Log Name:      Application
    Source:        MSExchange Search Indexer
    Date:          5/23/2013 3:54:43 PM
    Event ID:      118
    Task Category: General
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      XCHMS1W.lakeco.org
    Description:
    Exchange Search Indexer has temporarily disabled indexing of the mailbox database FC1 (GUID = e96d02a6-e367-48a1-a163-d6764f389f5d) due to an error (Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to set CI watermark (hr=0x80004005, ec=1142)
    Diagnostic context:
        Lid: 1494    ---- Remote Context Beg ----
        Lid: 44215 
        Lid: 60049   StoreEc: 0x8004010F
        Lid: 49469 
        Lid: 65341   StoreEc: 0x8004010F
        Lid: 56125 
        Lid: 47933   StoreEc: 0x8004010F
        Lid: 32829 
        Lid: 49213   StoreEc: 0x8004010F
        Lid: 48573 
        Lid: 64957   StoreEc: 0x8004010F
        Lid: 14638   StoreEc: 0x476    
        Lid: 1750    ---- Remote Context End ----
        Lid: 12018   StoreEc: 0x476    
        Lid: 9266    StoreEc: 0x476    
       at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
       at Microsoft.Mapi.ExRpcAdmin.CiSetWaterMark(Guid mdbGuid, Guid instanceGuid, Boolean isHighWatermark, UInt64 watermark)
       at Microsoft.Exchange.Search.PropertyStore.set_LowWaterMark(UInt64 value)
       at Microsoft.Exchange.Search.PropertyStore.SetLowWaterMark(PropertyStore ps, UInt64 lowWatermark, Object unused1, Object unused2, Object unused3, Object& unusedOut1, Object& unusedOut2)
       at Microsoft.Exchange.Search.PropertyStore.DoRetriableOperation[InParameter1Type,InParameter2Type,InParameter3Type,InParameter4Type,OutParameter1Type,OutParameter2Type,ReturnType](Guid mdbGuid, InParameter1Type inParameter1, InParameter2Type inParameter2, InParameter3Type inParameter3, InParameter4Type inParameter4, OutParameter1Type& outParameter1, OutParameter2Type& outParameter2, OperationDelegate`7 operationDelegate)
       at Microsoft.Exchange.Search.NotificationQueue.RemoveCompletedNotifications()
       at Microsoft.Exchange.Search.NotificationQueue.MaintenanceProcedure()), and Operations Manager would never alert it.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchange Search Indexer" />
        <EventID Qualifiers="32772">118</EventID>
        <Level>3</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-05-23T20:54:43.000000000Z" />
        <EventRecordID>437154</EventRecordID>
        <Channel>Application</Channel>
        <Computer>XCHMS1W.lakeco.org</Computer>
        <Security />
      </System>
      <EventData>
        <Data>FC1</Data>
        <Data>e96d02a6-e367-48a1-a163-d6764f389f5d</Data>
        <Data>Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to set CI watermark (hr=0x80004005, ec=1142)
    Diagnostic context:
        Lid: 1494    ---- Remote Context Beg ----
        Lid: 44215 
        Lid: 60049   StoreEc: 0x8004010F
        Lid: 49469 
        Lid: 65341   StoreEc: 0x8004010F
        Lid: 56125 
        Lid: 47933   StoreEc: 0x8004010F
        Lid: 32829 
        Lid: 49213   StoreEc: 0x8004010F
        Lid: 48573 
        Lid: 64957   StoreEc: 0x8004010F
        Lid: 14638   StoreEc: 0x476    
        Lid: 1750    ---- Remote Context End ----
        Lid: 12018   StoreEc: 0x476    
        Lid: 9266    StoreEc: 0x476    
       at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
       at Microsoft.Mapi.ExRpcAdmin.CiSetWaterMark(Guid mdbGuid, Guid instanceGuid, Boolean isHighWatermark, UInt64 watermark)
       at Microsoft.Exchange.Search.PropertyStore.set_LowWaterMark(UInt64 value)
       at Microsoft.Exchange.Search.PropertyStore.SetLowWaterMark(PropertyStore ps, UInt64 lowWatermark, Object unused1, Object unused2, Object unused3, Object&amp; unusedOut1, Object&amp; unusedOut2)
       at Microsoft.Exchange.Search.PropertyStore.DoRetriableOperation[InParameter1Type,InParameter2Type,InParameter3Type,InParameter4Type,OutParameter1Type,OutParameter2Type,ReturnType](Guid mdbGuid, InParameter1Type inParameter1, InParameter2Type inParameter2, InParameter3Type inParameter3, InParameter4Type inParameter4, OutParameter1Type&amp; outParameter1, OutParameter2Type&amp; outParameter2, OperationDelegate`7 operationDelegate)
       at Microsoft.Exchange.Search.NotificationQueue.RemoveCompletedNotifications()
       at Microsoft.Exchange.Search.NotificationQueue.MaintenanceProcedure()</Data>
      </EventData>
    </Event>

    Then this error...

    Log Name:      Application
    Source:        ExchangeStoreDB
    Date:          5/23/2013 3:54:44 PM
    Event ID:      122
    Task Category: Database recovery
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      XCHMS1W.lakeco.org
    Description:
    At '5/23/2013 3:54:41 PM' the Microsoft Exchange Information Store Database 'FC1' copy on this server experienced a corrupted search catalog. Consult the event log on the server for other "ExchangeStoreDb" and "MSExchange Search Indexer" events for more specific information about the failure. A successful failover restored service.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="ExchangeStoreDB" />
        <EventID Qualifiers="49156">122</EventID>
        <Level>2</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-05-23T20:54:44.000000000Z" />
        <EventRecordID>437155</EventRecordID>
        <Channel>Application</Channel>
        <Computer>XCHMS1W.lakeco.org</Computer>
        <Security />
      </System>
      <EventData>
        <Data>FC1</Data>
        <Data>e96d02a6-e367-48a1-a163-d6764f389f5d</Data>
        <Data>ContentIndex</Data>
        <Data>CatalogReseed</Data>
        <Data>FC1</Data>
        <Data>
        </Data>
        <Data>
        </Data>
        <Data>5/23/2013 3:54:41 PM</Data>
      </EventData>
    </Event>

    Friday, May 24, 2013 3:14 AM
  • Is is happening with all the databases on a particular mailbox server?

    OM (MCITP) | Blog

    Friday, May 24, 2013 3:24 AM
  • Is is happening with all the databases on a particular mailbox server?

    OM (MCITP) | Blog


    Not all of the databases will fail, but I don't see a pattern on which ones do and don't.  It doesn't seem to be constrained to one server either.  It happens on all of them from time to time.  However, it does seem to affect the two that are in the same site more often than the one that is remote.  The remote server seems to be the one that it fails over to when the content index fails.  The database copies on the remote server have the lowest activation preference as well (3).



    Friday, May 24, 2013 3:40 PM
  • We've been having exactly this same issue in our Exchange 2010 environment. Did you ever manage to figure out and resolve this?
    Wednesday, April 12, 2017 2:06 PM
  • catalogindex size will go up to 20% of the Mailbox DB Size after the search indexing process is completed.

    It could either be an issue with Search Index where you might have to reset the search index for that DB.

    If the search index has already been reset then look for any underlying issues with the storage.

    Check in the System log to see if you find any events related to NTFS/Storage.

    To Reset Search Index follow below:

    Exchange 2010 Database copy on server has content index catalog files in the following state: Failed


    Pavan Maganti ~ ( Exchange | 2003/2007/2010/E15(2013)) ~~ Please remember to click “Vote As Helpful&quot; if it really helps and &quot;Mark as Answer” if it answers your question, “Unmark as Answer” if a marked post does not actually answer your question. ~~ This Information is provided is &quot;AS IS&quot; and confers NO Rights!!

    Friday, April 21, 2017 12:06 PM