none
Search Service is stuck in starting after updates were applied RRS feed

  • Question

  •  

    The day after applying updates; SP3 and cumulative updates, SharePoint 2007 search service stopped working. 

     
    Initially I checked Central Administration > Application Management > Content databases and noted that the property for Search Server was not selected so I pointed it to my server name (the only one in the list), when that had no effect,  I issued iisreset command. Next I restarted the server but that had no effect. [update to previous statement: I found out that pointing to a search server here is only necessary if you want Sharepoint to index HELP search database].


    Shared Services will not open the  Search Adminstration  page– it is frozen  in “loading” status. I cannot open Search settings page at all.
    In Windows Services, osearch (Office SharePoint Server Search)  and spsearch (Windows SharePoint Services Search Server) are set to Manual so I switched them to Automatice and started them both. Now osearch is frozen in “Starting” status.
    I also noted that the timer services stopped during the update – no timer service has been started since then.


    I am new SharePoint Admin and not sure what to do.  I started a preupgrade check to see if it would alert me of any more specific problems but it failed to finish.  I think I should run Products and Technologies wizard, but I am not sure if I would mess up my entire farm if I did that.  Right now it is only search that is not working and the site collection seems fine. 
    What concerns me most is the timer service is not running. 
    Can someone advise me on steps to get the search service aware of the database again so the services can start again?


    Windows event log errors:
    A database error occurred.

    Source: Microsoft OLE DB Provider for SQL Server
    Code: 18456 occurred 1 time(s)
    Description: Login failed for user 'xxx\username.

    Context: Application 'f866eea5-f180-4a9e-be01-dcf2f9c642a5'

    --
    A database error occurred.

    Source: Microsoft OLE DB Provider for SQL Server
    Code: 6005 occurred 1 time(s)
    Description: SHUTDOWN is in progress.

    Context: Application 'f866eea5-f180-4a9e-be01-dcf2f9c642a5'

    --
    Error importing registry hive from the configuration database and into the registry.

    Context: Application 'f866eea5-f180-4a9e-be01-dcf2f9c642a5'

    Details:
      (0x80040e14)
    Volume Shadow Copy Service error: Unexpected error RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...).  hr = 0x80070005.

    Operation:
       Initializing Writer

    Context:
       Writer Class Id: {8d5f38cd-fb7a-49ca-ae1b-21d1f017d8f9}
       Writer Name: OSearch VSS Writer
       Writer Instance ID: {c00a8cfb-7d28-4b47-a78a-4b0ca31887a9}

    --
    Unknown SQL Exception 64 occured. Additional error information from SQL Server is included below.

    A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)

    --
    Failed to initiate the upgrade sequence.
    An exception of type System.Data.SqlClient.SqlException was thrown.  Additional exception information: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
    System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
       at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
       at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
       at System.Data.SqlClient.TdsParserStateObject.ReadByte()
       at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
       at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
       at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
       at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
       at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
       at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
       at Microsoft.SharePoint.Utilities.SqlSession.ExecuteNonQuery(SqlCommand command)
       at Microsoft.SharePoint.Upgrade.SPDatabaseSequence.GetVersion(Guid id, Version defaultVersion, SqlSession session, SPDatabaseSequence sequence)
       at Microsoft.SharePoint.Upgrade.SPDatabaseSequence.GetVersion(Guid id, Version defaultVersion)
       at Microsoft.SharePoint.Upgrade.SPDatabaseWssSequence.get_BuildVersion()
       at Microsoft.SharePoint.Upgrade.SPDatabaseWssSequence.InternalCanUpgrade(String sqlstrLegacyBuildVersion)
       at Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence.get_CanUpgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.CanUpgrade(Object o)
       at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.get_CanUpgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.ReflexiveCanUpgrade(Object o)
       at Microsoft.SharePoint.Upgrade.SPManager.NeedsUpgrade(Object o, Boolean bRecurse)
       at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.get_NeedsUpgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.Initialize()
       at Microsoft.SharePoint.PostSetupConfiguration.UpgradeBootstrapTask.Run()
       at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

    --
    Configuration of SharePoint Products and Technologies failed.  Configuration must be performed in order for this product to operate properly.  To diagnose the problem, review the extended error information located at D:\Program Files\Microsoft Office Servers\12.0\LOGS\PSCDiagnostics_11_2_2011_18_47_15_228_507141731.log, fix the problem, and run this configuration wizard again.


    This is the error that gets thrown when user attempts to search:
    A database error occurred.

    Source: Microsoft OLE DB Provider for SQL Server
    Code: 2812 occurred 17 time(s)
    Description: Could not find stored procedure 'dbo.proc_MSS_GetSampleExtremes'.
    Query machine 'SERVERNAME' has been taken out of rotation due to this error: The system cannot find the file specified.   0x80070002.  It will be retried in 15 seconds. Component: f866eea5-f180-4a9e-be01-dcf2f9c642a5

    ULS log error:
    Excerpt from log:
    [SPManager] [ERROR] [11/2/2011 7:08:45 PM]: Upgrade [SearchSharedDatabase Name=SharePoint_SSP1_Search_DB Parent=SPDatabaseServiceInstance] failed.
    [SPManager] [ERROR] [11/2/2011 7:08:45 PM]: This SqlTransaction has completed; it is no longer usable.
    [SPManager] [ERROR] [11/2/2011 7:08:45 PM]:    at System.Data.SqlClient.SqlTransaction.ZombieCheck()
       at System.Data.SqlClient.SqlTransaction.Rollback(String transactionName)
       at Microsoft.Office.Server.Data.TransactionalSqlSession.Rollback()
       at Microsoft.SharePoint.Upgrade.SPDatabaseAction.Rollback()
       at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()
    [SPManager] [ERROR] [11/2/2011 7:08:45 PM]: Action 12.2.508.0 of Microsoft.Office.Server.Search.Upgrade.SearchDatabaseSequence failed to rollback.
    [SPManager] [ERROR] [11/2/2011 7:08:45 PM]:    at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.Upgrade(Obj

    PSCDiagnostics log error:
    11/02/2011 18:47:27  7  ERR            Task upgradebootstrap has failed with an unknown exception
    11/02/2011 18:47:27  7  ERR            Exception: System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
       at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
       at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
       at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
       at System.Data.SqlClient.TdsParserStateObject.ReadByte()
       at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
       at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
       at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
       at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
       at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
       at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
       at Microsoft.SharePoint.Utilities.SqlSession.ExecuteNonQuery(SqlCommand command)
       at Microsoft.SharePoint.Upgrade.SPDatabaseSequence.GetVersion(Guid id, Version defaultVersion, SqlSession session, SPDatabaseSequence sequence)
       at Microsoft.SharePoint.Upgrade.SPDatabaseSequence.GetVersion(Guid id, Version defaultVersion)
       at Microsoft.SharePoint.Upgrade.SPDatabaseWssSequence.get_BuildVersion()
       at Microsoft.SharePoint.Upgrade.SPDatabaseWssSequence.InternalCanUpgrade(String sqlstrLegacyBuildVersion)
       at Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence.get_CanUpgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.CanUpgrade(Object o)
       at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.get_CanUpgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.ReflexiveCanUpgrade(Object o)
       at Microsoft.SharePoint.Upgrade.SPManager.NeedsUpgrade(Object o, Boolean bRecurse)
       at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.get_NeedsUpgrade()
       at Microsoft.SharePoint.Upgrade.SPManager.Initialize()
       at Microsoft.SharePoint.PostSetupConfiguration.UpgradeBootstrapTask.Run()
       at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

     




    • Edited by dms.WebDev Monday, November 14, 2011 9:55 PM
    Friday, November 4, 2011 11:08 PM

Answers

  • I found the solution to my original problem.  The cause of the problem may have been a combination of or one or the other: a SQL permissions issue or a space constraint on our VMWare. PSConfig failed because it did not have permission to access the database as the account the 2007 Microsft Office Servers SP3 was run as. 

    The steps to solve this in my particular situation were:

    1.  review sql logs looking for permissions type errors

    2.  Increase SharePoint production OS drive by 10 GB

    3.  add "sysadmin" level permission to Builtin/Administrator account on SQL Server housing SharePoint production databases

    4.  log on to production SharePoint server as serverfarm account

    5. run elevated command prompt to stop any updates in progress (as was indicated by a 2nd attempt to reapply SP3)

                       stsadm -o setproperty -pn command-line-upgrade-running -pv no

    6. reapply SP3 updates from server via elevated command prompt as serverfarm account.  This resulted in "files already applied" (a good sign)

    7. run psconfig: psconfig -cmd upgrade -inplace b2b -wait -force

    Once psconfig finished successfully, search service and timer service began working immediately as if they were never turned off. All alerts were sent at once. There was nothing to reconfigure;  all scopes, crawls etc were still intact! Woo hoo!

     

    I don't think I can mark my own answer as answer, would someone kindly do that for me?

    thank you,

    Doreen


    dms

    • Edited by dms.WebDev Monday, December 5, 2011 4:01 PM
    • Marked as answer by Wayne Fan Friday, December 9, 2011 1:00 PM
    Wednesday, November 30, 2011 11:16 PM

All replies

  • Looks like your SSP gone bad. Delete the SSP. Create a new one. Then run the PS Config Wizard.

    http://technet.microsoft.com/en-us/library/cc263248(v=office.12).aspx


    Regards, Vishwas
    Monday, November 7, 2011 1:12 AM
  • Thank you for the reply - after trying some things yesterday (disabling osearch allowed search administration to appear but not search settings, starting psconfig failed) - I have come to that conclusion myself.  Will give it a try to create a new ssp (yikes!) and let you know how it goes. 

    Doreen


    dms
    Wednesday, November 9, 2011 2:53 PM
  • Vishwas B,

    The link you provided describes how to delete Shared Services via stsadm....do I really want to delete shared services before I create a new one?  Is there a best practise for the steps to create a replacement Shared Services?  I haven't done this before and I don't want to blow up my production server.  I will search for some documentation but if you have some advice on how to proceed, I would appreciate it.

    Doreen


    dms
    Wednesday, November 9, 2011 10:09 PM
  • Hi,

    I would suggest you to first create a new SSP. Then mark your new SSP as the default one then try to delete the existing one.


    Thanks, Rahul Rashu
    Thursday, November 10, 2011 11:20 AM
  • Disable the Office search service from services.msc

    Reboot

    Run the P S Config Wizard

    Set the search service back to automatic.... and let us know how that goes.

    N ya some times bad SSP do not allow creation of new one so changing association wouldn't even come in to picture. That's the reason I suggested to delete the old one before creating new one.
    Regards, Vishwas
    Thursday, November 10, 2011 5:20 PM
  • You did not tel l how this went

    "

    Disable the Office search service from services.msc

    Reboot

    Run the P S Config Wizard

    Set the search service back to automatic.... "

    -------------------------------------------------------------------------------

    You need to create new web applications for SSP and Mysite (you can change the url back to previous my site url once your new ssp is created)

    And databases for SSPAdmin, SSP, Search would be recreated. Do not worry about them


    Regards, Vishwas
    Saturday, November 12, 2011 12:24 AM
  • Hi Vishwas,

    One of the first things I tried was disabling osearch in services.msc, rebooting the server and running psconfig.  After disabling osearch, I was able to open Search Administration in Central administration but search settings still would not open.  Psconfig failed to run.  Once I started osearch again (it defaulted back to manual), osearch stayed in "starting" mode forever and still is. Also Search administration would not open again in Central Administration - it displays a perpetual "loading...".  

    Thank you for the feedback. I will give it a shot in my development environment today and let you know how it goes.


    dms
    Monday, November 14, 2011 4:00 PM
  • I found the solution to my original problem.  The cause of the problem may have been a combination of or one or the other: a SQL permissions issue or a space constraint on our VMWare. PSConfig failed because it did not have permission to access the database as the account the 2007 Microsft Office Servers SP3 was run as. 

    The steps to solve this in my particular situation were:

    1.  review sql logs looking for permissions type errors

    2.  Increase SharePoint production OS drive by 10 GB

    3.  add "sysadmin" level permission to Builtin/Administrator account on SQL Server housing SharePoint production databases

    4.  log on to production SharePoint server as serverfarm account

    5. run elevated command prompt to stop any updates in progress (as was indicated by a 2nd attempt to reapply SP3)

                       stsadm -o setproperty -pn command-line-upgrade-running -pv no

    6. reapply SP3 updates from server via elevated command prompt as serverfarm account.  This resulted in "files already applied" (a good sign)

    7. run psconfig: psconfig -cmd upgrade -inplace b2b -wait -force

    Once psconfig finished successfully, search service and timer service began working immediately as if they were never turned off. All alerts were sent at once. There was nothing to reconfigure;  all scopes, crawls etc were still intact! Woo hoo!

     

    I don't think I can mark my own answer as answer, would someone kindly do that for me?

    thank you,

    Doreen


    dms

    • Edited by dms.WebDev Monday, December 5, 2011 4:01 PM
    • Marked as answer by Wayne Fan Friday, December 9, 2011 1:00 PM
    Wednesday, November 30, 2011 11:16 PM
  • I found the solution to my original problem.  The cause of the problem may have been a combination of or one or the other: a SQL permissions issue or a space constraint on our VMWare. PSConfig failed because it did not have permission to access the database as the account the 2007 Microsft Office Servers SP3 was run as. 

    The steps to solve this in my particular situation were:

    1.  review sql logs looking for permissions type errors

    2.  Increase SharePoint production OS drive by 10 GB

    3.  add "sysadmin" level permission to Builtin/Administrator account on SQL Server housing SharePoint production databases

    4.  log on to production SharePoint server as serverfarm account

    5. run elevated command prompt to stop any updates in progress (as was indicated by a 2nd attempt to reapply SP3)

                       stsadm -o setproperty -pn command-line-upgrade-running -pv no

    6. reapply SP3 updates from server via elevated command prompt as serverfarm account.  This resulted in "files already applied" (a good sign)

    7. run psconfig: psconfig -cmd upgrade -inplace b2b -wait -force

    Once psconfig finished successfully, search service and timer service began working immediately as if they were never turned off. All alerts were sent at once. There was nothing to reconfigure;  all scopes, crawls etc were still intact! Woo hoo!

    I don't think I can mark my own answer as answer, would someone kindly do that for me?

    thank you,

    Doreen


    dms

    Very helpfull - It solved my problem


    Friday, June 8, 2012 5:24 PM
  • I had the same issue and it turned out to be that SP3 for MOSS 2007 was applied without SP3 for WSS being applied first. Following your instructions and installing SP3 for WSS fixed the issue. 
    Monday, December 30, 2013 5:37 PM