none
SCCM 2012 Super Busy Every Hour

    Question

  • Hi,

    SCCM 2012 SP1 with CU 2. Windows 2008 R2 Standard with SP1. 16GB RAM. 8CPU running VMWare.

    I noticed that approximately every hour (but not exactly at the same hour) to a maximum of 2 hours, my SCCM reach a minimum of 85% CPU.  I have AV exclusions set and I even turned off Endpoint on the server to find out if EP is the cause of the spike but it isn't.  Then I thought of SQLServer process.  It's using high memory 9GB on an average but even with it's high memory use, the CPU can be at 4%.  Then every hour, something happens that triggers the spike.

    I already adjusted the Summarization schedule to every 8hours but it is still happening.  How do I find out what is it in SCCM that's pegging the CPU at 85% every hour?

    Pls. guide or advice. Thank you!

    Wednesday, September 25, 2013 3:41 PM

Answers

  • Thanks for all your replies!

    UPDATE: These are the things that helped so far bring down the CPU level.  From every hour high CPU alert of 85% to approximately 5 High CPU alerts in a day - which is pretty quiet to us.

    - I reviewed all my collections and changed the Collection schedule to weekly at night *instead* of leaving whatever default time it set when I created each collection.

    - All Collections I consider critical were changed to "Use Incremental Updates for This Collection".

    - I changed the Software Update Summarization schedule from hourly to every 12 hours

    - with the helped of smsdbmon.log (which I learned new from this post) it helped me figure out which collections are running or busy at certain time.

    I forwarded the SQL-related suggestions to my DBA and it may take time before I can come back here and update this post what happens to that.  Most suggestions here are very informative and helpful to us.  I will update this post as soon as I have more info.  In the meantime the above suggestions helped a lot!

    Thank you all for your time and effort on this.

    • Marked as answer by SwissMiss123 Thursday, September 26, 2013 5:15 PM
    Thursday, September 26, 2013 5:14 PM

All replies

  • Hard to say, could be collections or other "maintenance"-ish that runs on the server. Use process monitor or something like that to find what process that runs on the server at those times. If you have SCOM you could see it in there.

    Ps. the SQL Server should be using a lot of RAM. Make sure that your RAM limitation (high and low) are set correctly.


    Tim Nilimaa | Blog: http://infoworks.tv | Twitter: @timnilimaa

    Wednesday, September 25, 2013 3:47 PM
  • Thank you.  Let me review my collections again and maintenance.
    Wednesday, September 25, 2013 3:53 PM
  • We'd need more info to help you with this. Considering you are running on VMWare this doesn't suprise me. First of all having 8CPU's on a VMWare server can actually make performance worse rather than better. It's because of the wait time for physical CPU's. Next you are running on shared storage I'm sure which could be an issue. You'd need to provide client count, more info about your CM environment, more info about your VM host and storage.

    Even after all of that high CPU on a VMware server i fine. Mine generally run 85% or more at all times. You really only need to be concerned if you are hitting 99% or more.

    Lastly, running CM on virtual hardware is almost always a nightmore for people.


    John Marcum | http://myitforum.com/myitforumwp/author/johnmarcum/

    Wednesday, September 25, 2013 3:55 PM
  • Thanks John.  So it sounded like we are at a normal range? The CPU when it's busy is usually around the 80s.  Thanks also for the feedback.  At least that give me an idea that nothng sounds extra ordinary.

    In our environment we have 300 Clients, 1 SCCM Server with all the roles installed in it including SQL, SUS and Endpoint. SCCM 2012 SP1 with CU 2. Windows 2008 R2 Standard with SP1. 16GB RAM. 8CPU running VSphere client 5.

    I adjusted most of my schedule for Collection membership, hope this helps.  It sounded like maybe I just need to change my threshold for VMWare alerts to 90% rather than 85%. 

    But I would still continue to work on this issue.  Any additional input is appreciate.  Thanks!

    Wednesday, September 25, 2013 4:13 PM
  • ONly 300 clients? I'd drop the CPU's to 4 and see what happens. It may actually work better.


    John Marcum | http://myitforum.com/myitforumwp/author/johnmarcum/

    Wednesday, September 25, 2013 4:16 PM
  • Yes, only 300.  Thanks for the suggestion.  I'll suggest that to my team and consider that.

    In the meantime, I will observe within 24 hours if my collection schedule adjustments made a difference.

    Pls. continue to give me suggestions/advice when possible. Thanks!

    • Edited by SwissMiss123 Wednesday, September 25, 2013 4:32 PM
    Wednesday, September 25, 2013 4:28 PM
  • I already adjusted the Summarization schedule to every 8hours but it is still happening.  How do I find out what is it in SCCM that's pegging the CPU at 85% every hour?

    What process is consuming so much? Have you already used taskmgr to get details?

    Torsten Meringer | http://www.mssccmfaq.de

    Wednesday, September 25, 2013 7:34 PM
  • Hi Torsten,

    It was the SQLServer process.  I double checked all my AV Exclusion for SQL and I have all the exclusions sets already.  I also noticed that SQLserver process is using 9GB memory on average but it looks like that's the expected memory use (based on my feedback from this post) so that's not something extraordinary....  Then every hour, something happens that triggers the spike.  I also turned off Endpoint, just to rule it out, and there is still a spike.  So I thought that SCCM must be doing something every hour that is causing the spike...

    Wednesday, September 25, 2013 8:48 PM
  • Either examine smsdbmon.log or use SQL trace to get an idea what's happening SQL-wise.

    Torsten Meringer | http://www.mssccmfaq.de

    Wednesday, September 25, 2013 9:01 PM
  • Note that there is a known performance impact of the built-in software update summarization task that has caused many folks to decrease the interval at which it runs.

    Jason | http://blog.configmgrftw.com

    Thursday, September 26, 2013 2:18 AM
  • Also, make sure that you have same amount of files for your DB that you have cores/vCPU's.

    Tim Nilimaa | Blog: http://infoworks.tv | Twitter: @timnilimaa

    Thursday, September 26, 2013 6:25 AM
  • I used to have a query that would tell what SQL is doing. I can't seem to find it but you can likely find one online. My guess is that if you check your disk queue lengths in perfmon you will see high latency which will translate into high CPU utilization. The root cause is hosting SQL shared storage that is not properly allocated. MS has a SQL performance team at CSS that can quickly identify these issues and tell you how to resolve them.


    John Marcum | http://myitforum.com/myitforumwp/author/johnmarcum/

    Thursday, September 26, 2013 2:09 PM
  • You can use this article as guide to helping track down the source of the issue: http://blogs.technet.com/b/umairkhan/archive/2013/09/03/configmgr-troubleshooting-high-cpu-utilization-on-the-sql-server-and-parallelism.aspx

    Jason | http://blog.configmgrftw.com

    Thursday, September 26, 2013 2:30 PM
  • John,

    Was your query something like this?

    SELECT
        --virtual file latency
        [ReadLatency] =
            CASE WHEN [num_of_reads] = 0
                THEN 0 ELSE ([io_stall_read_ms] / [num_of_reads]) END,
        [WriteLatency] =
            CASE WHEN [num_of_writes] = 0
                THEN 0 ELSE ([io_stall_write_ms] / [num_of_writes]) END,
        [Latency] =
            CASE WHEN ([num_of_reads] = 0 AND [num_of_writes] = 0)
                THEN 0 ELSE ([io_stall] / ([num_of_reads] + [num_of_writes])) END,
        --avg bytes per IOP
        [AvgBPerRead] =
            CASE WHEN [num_of_reads] = 0
                THEN 0 ELSE ([num_of_bytes_read] / [num_of_reads]) END,
        [AvgBPerWrite] =
            CASE WHEN [io_stall_write_ms] = 0
                THEN 0 ELSE ([num_of_bytes_written] / [num_of_writes]) END,
        [AvgBPerTransfer] =
            CASE WHEN ([num_of_reads] = 0 AND [num_of_writes] = 0)
                THEN 0 ELSE
                    (([num_of_bytes_read] + [num_of_bytes_written]) /
                    ([num_of_reads] + [num_of_writes])) END,
        LEFT ([mf].[physical_name], 2) AS [Drive],
        DB_NAME ([vfs].[database_id]) AS [DB],
        --[vfs].*,
        [mf].[physical_name]
    FROM
        sys.dm_io_virtual_file_stats (NULL,NULL) AS [vfs]
    JOIN sys.master_files AS [mf]
        ON [vfs].[database_id] = [mf].[database_id]
        AND [vfs].[file_id] = [mf].[file_id]
    -- WHERE [vfs].[file_id] = 2 -- log files
    -- ORDER BY [Latency] DESC
    -- ORDER BY [ReadLatency] DESC
    ORDER BY [WriteLatency] DESC;
    GO
    
    

    http://www.sqlskills.com/blogs/paul/how-to-examine-io-subsystem-latencies-from-within-sql-server/

    0-10ms fast, 11-20ms is OK, 21-50 is slow, > 50 ms is bad

    Thursday, September 26, 2013 2:39 PM
  • Hi Tim,

    What do you mean by, "make sure that you have same amount of files for your DB that you have cores/vCPU's".  Is this a setting or something so I can point it to my DBA?

    Thanks!

    Thursday, September 26, 2013 4:19 PM
  • check out Kent's blog, for info on configuring SQL for SCCM, including multiple files for the database...

    http://blog.coretech.dk/kea/system-center-2012-configuration-manager-sql-recommendations/


    just came across this thread also about "High CPU Usage on SCCM's SQL Instance"

    http://www.windows-noob.com/forums/index.php?/topic/7554-high-cpu-usage-on-sccms-sql-instance/

    "after SP1, CU1, the Software Summarization was set to run every hour, and had my SQL pegged. We went from 4 to 6 cores, same thing, bumped it to 8, same thing... If you're looking for where summarization is, right click on "All Software Updates" under Software Updates in Software Library, and you'll see summarization. It was set to every hour, so I changed that to 1 day, and it's much happier now."
    • Edited by fernsantos Thursday, September 26, 2013 4:30 PM
    Thursday, September 26, 2013 4:24 PM
  • Thanks for all your replies!

    UPDATE: These are the things that helped so far bring down the CPU level.  From every hour high CPU alert of 85% to approximately 5 High CPU alerts in a day - which is pretty quiet to us.

    - I reviewed all my collections and changed the Collection schedule to weekly at night *instead* of leaving whatever default time it set when I created each collection.

    - All Collections I consider critical were changed to "Use Incremental Updates for This Collection".

    - I changed the Software Update Summarization schedule from hourly to every 12 hours

    - with the helped of smsdbmon.log (which I learned new from this post) it helped me figure out which collections are running or busy at certain time.

    I forwarded the SQL-related suggestions to my DBA and it may take time before I can come back here and update this post what happens to that.  Most suggestions here are very informative and helpful to us.  I will update this post as soon as I have more info.  In the meantime the above suggestions helped a lot!

    Thank you all for your time and effort on this.

    • Marked as answer by SwissMiss123 Thursday, September 26, 2013 5:15 PM
    Thursday, September 26, 2013 5:14 PM