none
CU6 Issues RRS feed

  • Question

  • I just installed CU6 over SP1 at 8PM uk time 12/09/2014 and for the first time with no errors.

    However all databases are mounted byt content index status is unknown.

    Get-MailboxDatabaseCopyStatus | Ft Name,contentindexstate, Contentindexerrormessage -Autosize

    shows the following error message: Could not find registry value of Index Status for database

    And yes checking the key nothing is their (Sophos AV is off)

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

    I tried restarting services and recreating the index server to no avail. How on earth do I get contentindex back?

    I also have server 2 on a different lan subnet same domain 10mb fibre link, upgrading from sp1 to cu6 but it takes hours to perform the mailbox role transport service, about 3 hours before I finally get an error.

    On that server I need to check if the error match's this article [cant link too tired to try and make it! - DiscoverySearchMailbox issue maybe]

    On that server the keys exist.

    If I stick with server 1 for now does anyone know how to kick the index service back into touch or restore working keys for the registry.

    Saturday, September 13, 2014 9:05 AM

Answers

  • So far after recreating the registry keys I could not get the search index to work so I changed to healthy state to try to manually reseed to no avail.

    However following this article:

    blogs.technet.com/b/johnbai/archive/2013/07/16/exchange-2013-high-availability-fast-search-and-the-windows-registry.aspx

    I was able to change the index state keys to 3 so example:

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus]
    "{b7772c8c-2f3a-48d7-bbcb-75df0befdaf5}"="3,3,51539607564,2014-06-12 21:54:48Z,0,,0,0"
    "{9b169239-79e0-4325-9f03-dee6d1085738}"="3,3,51539607564,2014-09-15 16:02:09Z,0,,0,0"
    "{cb1c7d99-2f45-4cd2-b5ed-db2b22077692}"="3,3,4294967297,2014-09-15 16:05:51Z,0,,18,0"
    "{bcd21097-dd52-4533-9ddf-d8b47e4d0134}"="3,3,4294967297,2014-09-15 16:04:34Z,0,,18,0"
    "{2bf3d91c-cd12-4789-ac9a-a1ec063c6038}"="3,3,51539607564,2014-09-15 16:05:01Z,0,,0,0"
    "{2d08b685-44f4-455b-92c3-0211f331e15f}"="3,3,4294967297,2014-09-15 16:05:47Z,0,,0,0"

    The search is now performing a full reseed of all index's after I started the 2 search services.

    So far looking good!

    Wednesday, September 17, 2014 8:41 AM
  • I resolved my strange failure on server 2 by finding out that diagnostics did not work in powershell:

    [Diagnostics.PerformanceCounterCategory]::GetCategories() | Format-Table -auto

    when trying to run the exact script that failed:

    New-PerfCounters –definitionfilename “E:\EX2013CU6\setup\perf\mailboxtransportsubmissionserviceperfcounters.xml"

    I had to fix the

    lodctr /R to rebuild the performance counters.

    the diagnostics then run ok and install works.

    now all db are in unknown state!

    I HAD TO EXPORT THE REGRISTRY FROM SERVER 2 TO SERVER 1 FOR INDEX STATUS TO GET THIS FAR.

    Monday, September 15, 2014 12:12 PM
  • I renamed machinekeys to machinekeysold restarted search services and everything has sprung back to live.

    That's the bit I like about exchange 2013 when it works it works well.

    I believe a 3rd party app was creating issue which was also uninstalled . I need to reboot after index's are recreated but all looks good.

    Thursday, September 18, 2014 4:14 PM

All replies

  • Hi Wilson

    probably you can try running the below command to update 

    Update-MailboxDatabaseCopy -Identity DBNAME\ServerName -CatalogOnly -Force

    Also you can try below 
    How to Rebuild the Full-Text Index Catalog
    http://technet.microsoft.com/en-us/library/aa995966(EXCHG.80).aspx 
      
    Reseed the Search Catalog: Exchange 2010 Help
    http://technet.microsoft.com/en-us/library/ee633475.aspx

    Good Luck


    Remember to mark as helpful if you find my contribution useful or as an answer if it does answer your question.That will encourage me - and others - to take time out to help you Check out my latest blog posts on http://exchangequery.com

    Saturday, September 13, 2014 9:51 AM
  • Like I said index rebuilds don't work, the keys are missing.

    So it bombs straight out.

    reseeding is useless as server 2 takes 4 hours to get stage 4 of 9 mailbox role and bombs out on new-perfcounters.

    So its causing me no end of grief.

    nothing fancy setup yet every time I do one of these CU's it breaks mail and I have to spend hours to fix it. Now this ones got our mail down for almost a day now.

    Saturday, September 13, 2014 12:27 PM
  • whats the best number to call Microsoft tech support on to get this silly issue resolved anyone?
    Saturday, September 13, 2014 12:30 PM
  • Start here:

    https://support.microsoft.com/common/international.aspx?RDPATH=%2foas%2fdefault.aspx%3fgprid%3d16662


    Twitter!: Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

    Saturday, September 13, 2014 12:49 PM
    Moderator
  • Did you try this to rebuild database by database?. 
    http://blogs.technet.com/b/exchangesearch/archive/2013/10/28/rebuild-an-index-on-exchange-2013-for-specific-databases.aspx

    Try this as a final step if doesn't solve contact Microsoft support. 
    http://smallbusiness.support.microsoft.com/en-US/contact


    Thanks, MAS
    Please mark as helpful if you find my comment helpful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Saturday, September 13, 2014 12:49 PM
  • Yes tried all those, what I found is that the NOTHING works if the registry keys are missing.

    After install they went, I need to know how to recreate them.

    If you remove them and query the server will inform you they are missing. So any index build routing checks for the keys.

    On the support front it looks like mon-fri then.

    I still cant fathom this issue on server 2 either which after 4 hours of the setup fails on step 5 of 9: mailbox rule: mailbox service.

    Thows up some errors on remove-perfcounters and new-perfcounters.

    system.componentmodel.win32exception (0x80004005): the configuration registry key is invalid at Microsoft.exchange.configuration.tasks.task.throwexception

    could post screenshot but need to see how I valideate account!

    Saturday, September 13, 2014 1:06 PM
  • I resolved my strange failure on server 2 by finding out that diagnostics did not work in powershell:

    [Diagnostics.PerformanceCounterCategory]::GetCategories() | Format-Table -auto

    when trying to run the exact script that failed:

    New-PerfCounters –definitionfilename “E:\EX2013CU6\setup\perf\mailboxtransportsubmissionserviceperfcounters.xml"

    I had to fix the

    lodctr /R to rebuild the performance counters.

    the diagnostics then run ok and install works.

    now all db are in unknown state!

    I HAD TO EXPORT THE REGRISTRY FROM SERVER 2 TO SERVER 1 FOR INDEX STATUS TO GET THIS FAR.

    Monday, September 15, 2014 12:12 PM
  • Hi,

    Thanks for your response.

    "now all db are in unknown state", if there are corrupt entries for the machine name in the hosts file, this symptom will occur. I recommend you verify the records and do an ipconfig /flushdns to check the result.

    Hope this can be helpful to you.

    Best regards,

    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Amy Wang
    TechNet Community Support

    Tuesday, September 16, 2014 7:43 AM
    Moderator
  • So far after recreating the registry keys I could not get the search index to work so I changed to healthy state to try to manually reseed to no avail.

    However following this article:

    blogs.technet.com/b/johnbai/archive/2013/07/16/exchange-2013-high-availability-fast-search-and-the-windows-registry.aspx

    I was able to change the index state keys to 3 so example:

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus]
    "{b7772c8c-2f3a-48d7-bbcb-75df0befdaf5}"="3,3,51539607564,2014-06-12 21:54:48Z,0,,0,0"
    "{9b169239-79e0-4325-9f03-dee6d1085738}"="3,3,51539607564,2014-09-15 16:02:09Z,0,,0,0"
    "{cb1c7d99-2f45-4cd2-b5ed-db2b22077692}"="3,3,4294967297,2014-09-15 16:05:51Z,0,,18,0"
    "{bcd21097-dd52-4533-9ddf-d8b47e4d0134}"="3,3,4294967297,2014-09-15 16:04:34Z,0,,18,0"
    "{2bf3d91c-cd12-4789-ac9a-a1ec063c6038}"="3,3,51539607564,2014-09-15 16:05:01Z,0,,0,0"
    "{2d08b685-44f4-455b-92c3-0211f331e15f}"="3,3,4294967297,2014-09-15 16:05:47Z,0,,0,0"

    The search is now performing a full reseed of all index's after I started the 2 search services.

    So far looking good!

    Wednesday, September 17, 2014 8:41 AM
  • Clearing DNS resolver did nothing its been rebooted 7 times since Friday, 2 since Monday.

    Its local cache is consistant with the local primary onsite DNS, and all AD DC and both mail servers resolve by name and FQDN.

    I have stopped both search services and removed the keys so I am back in the state I was in after CU6.

    the keys just act as reference markers.

    So at the moment I am at:

    [PS] E:\Exchange\Scripts>Get-MailboxDatabaseCopyStatus | Ft Name,contentindexstate, Contentindexerrormessage -Autosize

    Name                      ContentIndexState ContentIndexErrorMessage
    ----                      ----------------- ------------------------
    OxfsStaff\SRV1                     Unknown Could not find registry value of Index Status for database {cb1c7d99-2f4...
    OxfPublicF\SRV1                     Unknown Could not find registry value of Index Status for database {b7772c8c-2f3...
    LonStaff\SRV1                     Unknown Could not find registry value of Index Status for database {2d08b685-44f...
    OxfDir\SRV1           Unknown Could not find registry value of Index Status for database {bcd21097-dd5...
    LonDir\SRV1           Unknown Could not find registry value of Index Status for database {2bf3d91c-cd1...

    This leaves me with trying to work out how to get the search indexer to re create the indexs.

    deleting the index folders just gets them recreated but at a 16mb never updating size.

    forcing a -catalogonly bombs out in secs with no errors. I get the same with the index keys or without.

    Something else somewhere is broken for the search index on this server for those database's, I am at maximum for standard for 5 databses so I can't create a test to see I it works fine on a new DB.



    • Edited by pwilson669 Wednesday, September 17, 2014 2:14 PM
    Wednesday, September 17, 2014 11:12 AM
  • These are the standard fast error messages that I can not shift:

    EventID 1010

    An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.FaultException: error CS1548: Cryptographic failure while signing assembly 'c:\Windows\Temp\yvv2twoi.djw\Microsoft.Exchange.TransportFlow.30.dll' -- 'Error signing assembly -- Unknown error (8013141c)'

     

    Server stack trace:

       at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]:

       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

       at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.PutFlow(String name, String serializedFlow)

       at Microsoft.Exchange.Search.Fast.FastManagementClient.<>c__DisplayClass2.<PerformFastOperation>b__1()

       at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)

    AND ALSO:

    An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.FaultException: error CS1548: Cryptographic failure while signing assembly 'c:\Windows\Temp\kmwjp5kk.gzx\Microsoft.Exchange.Search.Writer.36.dll' -- 'Error signing assembly -- Unknown error (8013141c)'

     

    Server stack trace:

       at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]:

       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

       at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.PutFlow(String name, String serializedFlow)

       at Microsoft.Exchange.Search.Fast.FastManagementClient.<>c__DisplayClass2.<PerformFastOperation>b__1()

       at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)

    As well as eventID 1006

    The FastFeeder component received a connection exception from FAST. Error details: System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Internal error while processing request (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:

    Microsoft.Ceres.InteractionEngine.Component.ProcessingEngineException: Internal error while processing request

       at Microsoft.Ceres.InteractionEngine.Component.CieProcessingEngine.LogAndRethrowException(Exception e)

       at Microsoft.Ceres.InteractionEngine.Component.CieProcessingEngine.ExecuteSearchFlow(String flowName, IEnumerable`1 inputData)

       at SyncInvokeExecuteSearchFlow(Object , Object[] , Object[] )

       at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)

       at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)

       at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)

       at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)

       at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)).

    Wednesday, September 17, 2014 2:13 PM
  • Using one of the lines out of the event logs "Error signing assembly -- Unknown error (8013141c)"

    I started doing some digging that pointed me at C:\Users\All Users\Microsoft\Crypto\RSA

    In machinekeys unlike my other mail server it has 1000+ of keys that keep getting generated while my other server only has 17 and are static.

    I have checked the security and all looks fine, but taking ownership from system back to system and applying inherit permissions here errors on some of the keys.

    Maybe CU6 install and something else has broken encryption signing on the fastsearch?

    Thursday, September 18, 2014 2:47 PM
  • I renamed machinekeys to machinekeysold restarted search services and everything has sprung back to live.

    That's the bit I like about exchange 2013 when it works it works well.

    I believe a 3rd party app was creating issue which was also uninstalled . I need to reboot after index's are recreated but all looks good.

    Thursday, September 18, 2014 4:14 PM
  • Hi,

    Thanks for your sharing.

    It's great to hear the good news.

    Best regards,

    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Amy Wang
    TechNet Community Support

    Friday, September 19, 2014 2:18 AM
    Moderator