locked
SharePoint 2013 Search, High CPU usage RRS feed

  • Question

  • I have a sharepoint 2013 site with 100+ subwebs and about 3.5GB documentations. The site performance is normal if I do not setup any enterprise search service.

    But once I setup a enterprise search service, the MSSQL and w3wp progresses will spend near 100% CPU for a long time. After the CPU usage turn down, once I click any link on the web page then the CPU usages will grow up to 100% for 15-20 minutes again.

    I made the following step to reduce the resource usages on the server.
    1. Reduce SharePoint logging, only log down Error events and Unexpected events.
    2. Optimize MS-SQL server configuration, give 2 BG minimun memory to the database server, and boost the SQL Server's priority.
    3. In the SharePoint search center, limit the crawl impact to 1 document at a time.

    But the problem is still not fixed.

    Check with the logs, 99% items are similar to:

    12/20/2012 02:33:00.60  w3wp.exe (0x15D4)                        0x146C                                0x6FB7003                      7t61 High     [Forced due to logging gap, cached @ 12/20/2012 02:33:00.53, Original Level: Verbose] {0} 
    12/20/2012 02:33:00.60  w3wp.exe (0x15D4)                        0x146C SharePoint Server              Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity 
    12/20/2012 02:33:34.36  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Asp Runtime                    aj1kl High     [Forced due to logging gap, cached @ 12/20/2012 02:33:34.31, Original Level: Verbose] SPRequestModule.PreRequestExecuteAppHandler cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:34.36  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:34.46  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:34.40, Original Level: Verbose] SQL connection time: 0.0505 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:34.46  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:34.73  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:34.46, Original Level: Verbose] SQL connection time: 0.1201 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:34.73  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Web Controls                   90i3 High     [Forced due to logging gap, Original Level: Verbose] PersonalActions::CreateChildControls() - Was called. cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.44  w3wp.exe (0x13A4)                        0x214C Web Content Management         Publishing                     aftdl High     [Forced due to logging gap, cached @ 12/20/2012 02:33:34.73, Original Level: Verbose] TaxonomyNavigationContext: Checking for target URL query parameters cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.44  w3wp.exe (0x13A4)                        0x214C Web Content Management         Publishing                     8vy8 High     [Forced due to logging gap, Original Level: Medium] CrossListQueryCache::GetSiteData() - Query: {0} Url: {1}. {2} cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.53  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.47, Original Level: Verbose] SQL connection time: 0.0542 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.53  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.62  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.58, Original Level: Verbose] SQL connection time: 0.0712 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.62  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.70  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          General                        g3ql High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.67, Original Level: Verbose] GetUriScheme(/aboutus/Pages/home.aspx) cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.70  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.76  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.75, Original Level: Verbose] SQL connection time: 0.024 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.76  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.84  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.81, Original Level: Verbose] SQL connection time: 0.0375 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.84  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.90  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       ahjqp High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.89, Original Level: Verbose] SQL connection time: 0.0545 cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.90  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          Database                       8acb High     [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity cea0ec9b-b1d5-600d-f3ef-49eb3271ba12
    12/20/2012 02:33:35.97  w3wp.exe (0x13A4)                        0x214C SharePoint Foundation          General                        g3ql High     [Forced due to logging gap, cached @ 12/20/2012 02:33:35.95, Original Level: Verbose] GetUriScheme(/depts/people/Pages/Home.aspx) cea0ec9b-b1d5-600d-f3ef-49eb3271ba12

    It is real bad that, when this problem appears, even if I reboot the OS, disable the SharePoint Search server, once I access my web site again, the CPU will group up to 100% quickly. (the problem become worse).

    That means I have to restore my content database everytime once I use the enterprise search.

    Any idea?

    Thursday, December 20, 2012 10:56 AM

Answers

  • We've had the same problem on several SP 2013 installations, with both new and upgraded content databases. The only solution we've found is to delete the publishing page that is reported in the logfiles and recreate it. In your case that means deleting "/depts/people/Pages/Home.aspx".

    Deleting the page has solved the problem (for now) in our installations. For a upgraded solution we got the problem also on a seccond page a bit later, so we deleted and recreated that aswell.

    We couldn't find anything wrong with the pages themselves. They worked as expected when we opened them, the only thing they had in common is that all were publishing pages in different Pages libraries.

    ElinK


    ElinK

    • Marked as answer by Claud _ Thursday, April 11, 2013 6:58 AM
    Friday, March 8, 2013 1:58 PM

All replies

  • I run the following sql to get the top CPU usage

    SELECT TOP 10 qs.execution_count, total_CPU_inSeconds = qs.total_worker_time/1000000, total_elapsed_time_inSeconds = qs.total_elapsed_time/1000000, SUBSTRING(st.text, 0 , 60) as cmd from sys.dm_exec_query_stats as qs CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as st cross apply sys.dm_exec_query_plan (qs.plan_handle) as qp ORDER BY qs.total_worker_time desc

    The result shows:

    Execution_count, Total_CPU_inSeconds, Total_elapsed_time_inSeconds, Cmd
    12558 300 301 CREATE PROCEDURE dbo.proc_NavStructDirtyPageDependents( ....
    12558 117 118 CREATE PROCEDURE dbo.proc_NavStructDirtyPageDependents( ....
    2108 116 117 CREATE PROCEDURE dbo.proc_GetWebNavStruct( @SiteId uni....
    12558 56 56 CREATE PROCEDURE dbo.proc_NavStructDirtyPageDependents( ....
    945 40 40 CREATE PROCEDURE dbo.proc_NavStructDeleteNodeByIds( @S....
    945 18 18 CREATE PROCEDURE dbo.proc_NavStructDeleteNodeByIds( @S....
    2108 17 17 CREATE PROCEDURE dbo.proc_GetWebNavAcls( @SiteId uniqu....
    588 7 7 CREATE PROCEDURE [dbo].[profilesynch_US_GetChangedLanguageA....
    23 3 3 SELECT TOP 10 qs.execution_count, total_CPU_inSec....
    7120 3 3 (@LFFP uniqueidentifier,@SITEID uniqueidentifier,@IU int,@L....

    Press F5 to refresh the result in each second, the execution_count of "CREATE PROCEDURE dbo.proc_NavStructDirtyPageDependents" increases 30+ in every second.

    What happen on SharePoint13? I just have 100+ sites and 300+ pages there. My web site compeletly crashed due to this stupid and unacceptable performance.


    • Edited by Claud _ Saturday, December 22, 2012 2:43 PM
    Saturday, December 22, 2012 2:41 PM
  • Hi,

    I have exactly the same problem, do you find any solution ?

    Kouilb


    http://kouilb.wordpress.com

    Thursday, March 7, 2013 2:27 PM
  • We've had the same problem on several SP 2013 installations, with both new and upgraded content databases. The only solution we've found is to delete the publishing page that is reported in the logfiles and recreate it. In your case that means deleting "/depts/people/Pages/Home.aspx".

    Deleting the page has solved the problem (for now) in our installations. For a upgraded solution we got the problem also on a seccond page a bit later, so we deleted and recreated that aswell.

    We couldn't find anything wrong with the pages themselves. They worked as expected when we opened them, the only thing they had in common is that all were publishing pages in different Pages libraries.

    ElinK


    ElinK

    • Marked as answer by Claud _ Thursday, April 11, 2013 6:58 AM
    Friday, March 8, 2013 1:58 PM
  • Yes, after deleting page recorded in logs the server is OK.

    But we don't know why this page break our farm.

    Kouilb.


    http://kouilb.wordpress.com

    Friday, March 8, 2013 2:40 PM
  • I am seeing the exact same behaviour in a new environment. (WIN2008R2,SP2013, SQL2008R2).

    However, I am unable to delete pages referenced in the ULS log as some of the pages are SharePoint System pages (_layouts15/settings.aspx, forms/AllItems.aspx). After a few hours the environment is very slow and eventually it will stop working all together.

    Is this a SQL or SharePoint 2013 related issue?

    Anyone found a reason for this behaviour or even a solution?

    /Nicolaj

    Tuesday, March 12, 2013 4:56 PM
  • Hello,

    We had the same problem for several times now, we had to delete the page to fix the problem, and then restored the page. The pages caused the problem were created by Admin, and edited by web author, and we also used "Add Link" to add the page link to our global navigation.

    Anyone know why it happened, is a bug in SharePoint?

    Wednesday, June 26, 2013 3:29 PM
  • We have this exact problem at the moment, the fix suggested won't work for us as multiple pages/documents/system files are referenced in the logs. Oddly enough this issue was kicked off by a search crawl.
    Friday, October 18, 2013 9:56 AM
  • Hi Claud,
    you might want to check this Microsoft Support Article, that covers your issue:

    Outdated database statistics decrease SharePoint Server performance, cause time-outs, and generate run-time errors https://support.microsoft.com/en-in/kb/3103194

    Regards,
    Christian Heim (MSFT)

    Thursday, August 18, 2016 12:27 PM