none
Getting unknown errors in central administration. Site backups always fail. OWSTimer Errors.

    Question

  • Originally we had a 1 server standalone MOSS2007 install. (All service packs applied)

    Then we added another server to the farm to act as a web server.

    We decided to take the web server offline. So that ideally we would be back to our 1 server standalone setup. All our sharepoint sites still work fine after we did this, but things on the central administration side are horribly broken.

    1. The web server still appears in farm topology even though it is offline, no longer exists. It also appears as "Windows SharePoint Services Database".

    2. Trying to go into "Servers in farm" under Operations causes an unknown error.

    3. Trying to go into "Timer job status" under Operations causes an unknown error.

    The error log displays the following each time I try to access "Servers in farm" or "Timer job status"

    OWSTIMER.EXE (0x199C) The following error occured while trying to read the ConnectionString property.  Object reference not set to an instance of an object.     

     

    Now before I try to do some serious reconfiguration I would like to backup my sites. If I use the "Perform a backup" feature in central administration, it tells me the backup job has been scheduled, but after waiting for hours it still hasn't started, and doesn't look like it ever will.

    I then tried to do a backup using STSADM, however the backup just stops halfway through at "Shared Search Index", no errors, it just hangs. Here is the cmdline dump:

    C:\Program Files\Common Files\microsoft shared\Web Server Extensions\12\BIN>stsadm -o backup -directory C:\SPBackupFiles -backupmethod full -item  SharedServices


    Verbose: Using directory: C:\SPBackupFiles\spbr0000\.
    Verbose: The backup/restore process included the following objects:
        Farm\
            [Configuration Database]\
            Windows SharePoint Services Web Application\
            [WSS_Administration]\
                [Web Application]\
                    SharePoint_AdminContent_7f7dd5bf-d912-4e5c-9346-5f1ea4a2ac9e\
            SharedServices\
                [SharePoint - 80]\
                    WSS_Content\
                [SharedServices_DB]\
                Project Server Application\
                    PWA\
                        ProjectServer_Published\
                        ProjectServer_Draft\
                        ProjectServer_Archive\
                        ProjectServer_Reporting\
                [UserProfileApplication]\
                [SessionStateSharedApplication]\
                [Shared Search Index]\
                    [SharedServices_Search_DB]\
            Global Search Settings\
            SSO\
            Windows SharePoint Services Help Search\
                [Search instance]\
                    [WSS_Search_IMV003]\

    Verbose: Adding SharedServices to Backup list.
    Verbose: Adding SharePoint - 80 to Backup list.
    Verbose: Adding WSS_Content to Backup list.
    Verbose: Adding SharedServices_DB to Backup list.
    Verbose: Adding Project Server Application to Backup list.
    Verbose: Adding PWA to Backup list.
    Verbose: Adding ProjectServer_Published to Backup list.
    Verbose: Adding ProjectServer_Draft to Backup list.
    Verbose: Adding ProjectServer_Archive to Backup list.
    Verbose: Adding ProjectServer_Reporting to Backup list.
    Verbose: Adding UserProfileApplication to Backup list.
    Verbose: Adding SessionStateSharedApplication to Backup list.
    Verbose: Adding Shared Search Index to Backup list.
    Verbose: Adding SharedServices_Search_DB to Backup list.
    Progress: Starting Backup.
    Start Time: 10/8/2010 12:04:18 PM.
    Verbose: Requested by xxxx\xxxx.
    Verbose: Backup/Restore Settings:
            Backup
            Backup Method: Full
            Top Component: SharedServices
            Directory: C:\SPBackupFiles
            Progress updated: 5
            Backup threads created: 1

    Verbose: Adding SharedServices to Backup list.
    Verbose: Adding SharePoint - 80 to Backup list.
    Verbose: Adding WSS_Content to Backup list.
    Verbose: Adding SharedServices_DB to Backup list.
    Verbose: Adding Project Server Application to Backup list.
    Verbose: Adding PWA to Backup list.
    Verbose: Adding ProjectServer_Published to Backup list.
    Verbose: Adding ProjectServer_Draft to Backup list.
    Verbose: Adding ProjectServer_Archive to Backup list.
    Verbose: Adding ProjectServer_Reporting to Backup list.
    Verbose: Adding UserProfileApplication to Backup list.
    Verbose: Adding SessionStateSharedApplication to Backup list.
    Verbose: Adding Shared Search Index to Backup list.
    Verbose: Adding SharedServices_Search_DB to Backup list.
    Verbose: Starting OnPrepareBackup event.
    Verbose: Starting object: Farm.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: Configuration Database.
    Verbose: Saving SPPersistedObject State
    Error: Object Configuration Database failed in event OnPrepareBackup. For more i
    nformation, see the error log located in the backup directory.
            NullReferenceException: Object reference not set to an instance of an ob
    ject.
    Verbose: Starting object: Windows SharePoint Services Web Application.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: WSS_Administration.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: Web Application.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: SharePoint_AdminContent_7f7dd5bf-d912-4e5c-9346-5f1ea4
    a2ac9e.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: SharedServices.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: SharePoint - 80.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: WSS_Content.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: SharedServices_DB.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: Project Server Application.
    Verbose: Starting object: IMVPWA.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: IMVProjectServer_Published.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: IMVProjectServer_Draft.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: IMVProjectServer_Archive.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: IMVProjectServer_Reporting.
    Verbose: Saving SPPersistedObject State
    Verbose: Starting object: UserProfileApplication.
    Verbose: Starting object: SessionStateSharedApplication.
    Verbose: Starting object: Shared Search Index.

     

    Can anyone offer advice on how to fix the issue with the unknown errors in central administration as well as the errors backing up?

    Friday, October 8, 2010 6:36 PM

Answers

  • I suggest you to setup new SharePoint server and attach Content databases from old one. I guess you screwed up Configuration Database when you tried to join WFE to standalone server.
    Oleg
    Wednesday, October 13, 2010 4:12 PM

All replies

  • What steps you perform to take that server out of farm?

    thanks

    -ws


    SharePoint administrator, MCTS,MCITP
    Friday, October 8, 2010 7:11 PM
  • The Standalone SharePoint setup is not Web Farm. How could you add WFE to standalone server??? For upgrade standalone server to Web farm you shold use different process.

    To restore Central Administration try to re-install Central Administration.

    1. Run Configuration Wizard and uncheck "this server hosts Central Administration site". Complete a wizard.

    2.Run Configuration Wizard again and check checkbox "this server hosts Central Administration site".

    It will re-create Central Administration site and database.


    Oleg
    Friday, October 8, 2010 7:13 PM
  • The Standalone SharePoint setup is not Web Farm. How could you add WFE to standalone server??? For upgrade standalone server to Web farm you shold use different process.

    To restore Central Administration try to re-install Central Administration.

    1. Run Configuration Wizard and uncheck "this server hosts Central Administration site". Complete a wizard.

    2.Run Configuration Wizard again and check checkbox "this server hosts Central Administration site".

    It will re-create Central Administration site and database.


    Oleg

    Hi Oleg. I ran the configuration wizard and followed your instructions. And there is absolutely no change in behaviour. Same errors when trying to access server farms settings and timer status. Also my old web server still appears in farm topology.

    As for how I removed the web server, I simply uninstalled Sharepoint from it. From my understanding once I did this the standalone server would realize it's no longer online and remove it from the farm topology.

    Friday, October 8, 2010 8:17 PM
  • yes you need to Sharepoint config wizards to remove that server from the farm. but in this case just follow these steps server will removed from the farm: http://www.mylifeinaminute.com/2008/07/23/manually-removing-servers-in-moss-2007/

     

    for unknown error you should turn off the custom error and see whats the actual error. how to turn off the the costom error follow these steps for Central admin Web app. http://stevepietrek.com/2007/07/28/turn-off-sharepoint-custom-error-messages/

     

    hope this helps

    thanks

    -ws


    SharePoint administrator, MCTS,MCITP
    Friday, October 8, 2010 8:27 PM
  • yes you need to Sharepoint config wizards to remove that server from the farm. but in this case just follow these steps server will removed from the farm: http://www.mylifeinaminute.com/2008/07/23/manually-removing-servers-in-moss-2007/

     

    for unknown error you should turn off the custom error and see whats the actual error. how to turn off the the costom error follow these steps for Central admin Web app. http://stevepietrek.com/2007/07/28/turn-off-sharepoint-custom-error-messages/

     

    hope this helps

    thanks

    -ws


    SharePoint administrator, MCTS,MCITP

    I tried manually removing the Server from the DB as in your first link. I found it's GUID in the Objects table, however there is no reference to it in the dependencies table.

     

    I turned off custom errors and turned on callstack information. This is the error I get when going to "Servers in farm":

    Object reference not set to an instance of an object.   at Microsoft.SharePoint.Administration.SPDatabase.get_Server()
       at Microsoft.SharePoint.ApplicationPages.FarmServersPage.get_ConfigurationDatabaseServer()
       at ASP._admin_farmservers_aspx.__Render__control8(HtmlTextWriter __w, Control parameterContainer)
       at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
       at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
       at System.Web.UI.Control.Render(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
       at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
       at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)
       at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)
       at System.Web.UI.HtmlControls.HtmlForm.Render(HtmlTextWriter output)
       at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.HtmlControls.HtmlForm.RenderControl(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
       at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
       at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
       at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
       at System.Web.UI.Control.Render(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
       at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
       at System.Web.UI.Page.Render(HtmlTextWriter writer)
       at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
       at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
       at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

     

    And the error I get when looking at timer status (they both error out when calling get_Server)

     

    Object reference not set to an instance of an object.   at Microsoft.SharePoint.Administration.SPDatabase.get_Server()
       at Microsoft.SharePoint.Administration.SPNativeConfigurationProvider.GetProperty(Guid id, Int32 key, Object arguments, Object& value)
       at Microsoft.SharePoint.Library.SPRequestInternalClass.GetTimerRunningJobs(Guid gServiceId, Guid gVServerId, UInt32& pdwColCount, UInt32& pdwRowCount, Object& pvarDataSet)
       at Microsoft.SharePoint.Library.SPRequest.GetTimerRunningJobs(Guid gServiceId, Guid gVServerId, UInt32& pdwColCount, UInt32& pdwRowCount, Object& pvarDataSet)
       at Microsoft.SharePoint.Administration.SPRunningJobCollection.EnsureRunningJobsData()
       at Microsoft.SharePoint.Administration.SPRunningJobCollection.Undirty()
       at Microsoft.SharePoint.SPBaseCollection.System.Collections.IEnumerable.GetEnumerator()
       at Microsoft.SharePoint.ApplicationPages.RunningJobsDataSourceView.AddService(SPService service, DataTable table)
       at Microsoft.SharePoint.ApplicationPages.TimerJobDataSourceView.FillDataTable(DataTable table, DataSourceSelectArguments selectArguments)
       at Microsoft.SharePoint.WebControls.DataTableDataSourceView.Select(DataSourceSelectArguments selectArguments)
       at Microsoft.SharePoint.WebControls.AdministrationDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments)
       at System.Web.UI.DataSourceView.Select(DataSourceSelectArguments arguments, DataSourceViewSelectCallback callback)
       at System.Web.UI.WebControls.DataBoundControl.PerformSelect()
       at System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
       at System.Web.UI.WebControls.GridView.DataBind()
       at System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
       at System.Web.UI.WebControls.CompositeDataBoundControl.CreateChildControls()
       at System.Web.UI.Control.EnsureChildControls()
       at System.Web.UI.Control.PreRenderRecursiveInternal()
       at System.Web.UI.Control.PreRenderRecursiveInternal()
       at System.Web.UI.Control.PreRenderRecursiveInternal()
       at System.Web.UI.Control.PreRenderRecursiveInternal()
       at System.Web.UI.Control.PreRenderRecursiveInternal()
       at System.Web.UI.Control.PreRenderRecursiveInternal()
       at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

    Friday, October 8, 2010 8:51 PM
  • Do you have Standalone single server installation?
    Oleg
    Friday, October 8, 2010 8:58 PM
  • Do you have Standalone single server installation?
    Oleg

    Yes, the server was installed using standalone single server install.
    Friday, October 8, 2010 9:19 PM
  • I'm running out of ideas on how to fix this. Does anyone have any other suggestions?

    Perhaps deleting the instance of the web server out of the DB's Objects table might work?

    Tuesday, October 12, 2010 4:58 PM
  • Another update.

    I went into the config database and removed all the web server object from the objects table as well as any other objects that specified it as its parent. On the plus side the web server no longer appears in my farm topology list, however I still cant get into timer job status or servers in farm screens.

    Another thing I noticed after looking at "timer job definitions" is that none of the timer jobs have run ever since adding the new server. So it appears the timer service is screwed up as well. I reset the timer service by removing all XML files, setting the config files contents to 1 and restarting the timer and administration service, but to no avail. And yes, I've double checked that the timer service is running.

    So I still have the same issues, however now I've discovered that the timer service is not runnings its jobs.

    Log file spits out the following:

    10/12/2010 12:05:03.78 	OWSTIMER.EXE (0x0910)          	0x1BFC	Search Server Common     	MS Search Administration   	7hkp	High  	Processing content source 'Default'.	 
    10/12/2010 12:05:03.78 	OWSTIMER.EXE (0x0910)          	0x1BFC	Search Server Common     	MS Search Administration   	7hkq	High  	Content source 'Default' has no start addresses; skipping.	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	General            	8dyt	High  	The following error occured while trying to read the ConnectionString property. Object reference not set to an instance of an object.	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	General            	8e2s	Medium 	Unknown SPRequest error occurred. More information: 0x80004003	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	General            	8e04	High  	NullReferenceException in SPRequest: Object reference not set to an instance of an object.  at Microsoft.SharePoint.Administration.SPDatabase.get_Server()   at Microsoft.SharePoint.Administration.SPNativeConfigurationProvider.GetProperty(Guid id, Int32 key, Object arguments, Object& value)   at Microsoft.SharePoint.Library.SPRequestInternalClass.UpdateTimerRunningJobProgress(Guid gServiceId, Guid gVServerId, Guid gJobId, Int32 lCurrentTargetPercentDone)   at Microsoft.SharePoint.Library.SPRequest.UpdateTimerRunningJobProgress(Guid gServiceId, Guid gVServerId, Guid gJobId, Int32 lCurrentTargetPercentDone)	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	General            	8e05	High  	NullReferenceException in SPRequest. stackTrace at create: 	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	General            	8e06	High  	NullReferenceException in SPRequest. stackTrace at release: 	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	Timer             	8nbb	High  	The following exception was thrown when updating the progress of job definition Microsoft.SharePoint.Search.Administration.SPSearchJobDefinition, id 96ff679c-071d-47cd-98d7-e579c3dd44b8. Object reference not set to an instance of an object.	 
    10/12/2010 12:05:03.80 	OWSTIMER.EXE (0x0910)          	0x1BFC	Windows SharePoint Services  	General            	8dyt	High  	The following error occured while trying to read the ConnectionString property. Object reference not set to an instance of an object.	 
    

    Tuesday, October 12, 2010 6:06 PM
  • I suggest you to setup new SharePoint server and attach Content databases from old one. I guess you screwed up Configuration Database when you tried to join WFE to standalone server.
    Oleg
    Wednesday, October 13, 2010 4:12 PM