System Center Configuration Manager TechCenter >
System Center Configuration Manager Forums
>
Configuration Manager General
>
CPU Spike and SCCM SP2 Upgrade.
CPU Spike and SCCM SP2 Upgrade.
There is an intermittent spike with SQL service on the CPU every 2
seconds. 2 Seconds 100%, then back down for 2 seconds. I upgraded to
SP2 yesterday.I am running on a DL380, was running like a champ before SP2.
Server 2003 x64. SQL Server 2008Additionally I get this failure:
Faulting application CcmExec.exe, version 4.0.6487.2000, faulting
module CcmExec.exe, version 4.0.6487.2000, fault address
0x00077731.As well as:
The application, C:\WINDOWS\SysWOW64\CCM\CcmExec.exe, generated an
application error The error occurred on 10/23/2009 @ 08:18:41.617 The
exception generated was c0000005 at address 00477731 (CcmExec)The app appears to be running, with the spikes and fails
periodically.
Answers
- Thanks Surendar
1. No problems with DCOM errors now nor CCMEXEC crashing - this was the fix but a bit of a worry that it needed to be done manually
2. Manual upgrade and clean server build eventually showed software inventory thorugh the Resource Explorer but still not showing up via SQL Reports or via direct SQL query though. Worked this one out - v_GS_INSTALLED_SOFTWARE is actually pulled via Hardware inventory which is via smsdef.mof also - SP2 had unselected Asset Intelligence categories that were definitely selected before as I have 40+ custom reports that all broke. I finally worked out that this table was in fact populated via hardware inventory/smsdef.mof and traced it back to these Asset Intelligence categories being turned off! Anyway, I have selected the missing categories again and rerun Machine Policies\hardware inventory and it is all back again and all reports are working!
3. It was after 2 days I fixed the custom mof files as I didn't notice they had changed back to defaults
4. This is dangerous as all products/categories were sync'ed through which ruined my DEV updating process as previously only the updates we needed were showing which made drag/drop each month easy to add new patches to update lists. This is now not possible as all products/service packs etc. are showing in SCCM. My bad though as should have been more wary on this as it was in the readme! Only affected my DEV instance though, I caught it in TEST before any sync's happened :) Wish I knew how to clean them all out of SCCM and start again without deleting Software Update point and recreating all the update lists, tasks, Software Update Packages etc! No major though - only DEV!
So I am happy now - all is fixed again!
-----------------------------------------------------------------------
SUMMARY OF MY FIXES POST SCCM-SP2
-----------------------------------------------------------------------
DCOM\COM Security\Launch and Activation Permissions no longer included "SMS Admins" therefore I added them back in (Local Launch/Remote Activation) - reboot to stop DCOM errors and CCMEXEC crashing
Created new MP on another system, changed the default MP to point to the new MP, deleted original MP. Let it flow through for about 30 minutes then reversed (created new MP on original Site Server, assigned as default, then deleted secondary MP and deleted secondary Site System)
Re-applied any previous customisations to smsdef.mof and configuration.mof (as these were removed by the upgrade)
As per the readme, changed my WSUS SP2 Products and Classifications back to what we had before as SP2 set back to defaults
Reapplied Asset Intelligence categories as these were lost in the upgrade (as these update smsdef.mof when turning on/off)
Plenty of reboots in between at the end to ensure all was still clean
Hope this helps someone else out there!
Steve- Proposed As Answer byKiwifulla Thursday, November 05, 2009 1:18 AM
- Marked As Answer byYog Li - MSFTMSFT, ModeratorFriday, November 13, 2009 7:17 AM
- Hi Kiwifulla,
After testing my upgrade in a cloned VM lab of my live environment I was all ready to go as I had no errors at all.
Performed the upgrade last night in production and wham... got hit with the same problems as others in this thread.
If anyone can explain why a cloned live environment worked fine but my live environment didn't, you're doing better than me :)
Can I just clarify something though? On the DCOM security bits mentioned in this thread, where EXACTLY are you setting them?
I added it to the SMS Agent Host with the ID of {AD65A69D-3831-40D7-9629-9B0B50A93843} under Component Services>Computers>My Computer>DCOM Config.
I originally added Local Service and System to have Local Launch rights as that's what the event log errors were moaning about.
This cleared most errors but resulted in CCMExec.exe crashing constantly.
I updated the CCM Client > Still 1 crash but not as bad, but MP not working.
I then added SMS Admins and gave it permissions, still no better.
There were plenty of reboots in-between these
I then patched the box with Microsoft .NET Framework 3.5 Service Pack 1 and .NET Framework 3.5 Family Update for .NET versions 2.0 through 3.5 (KB951847)
Lo and belhold... everything is fine now?!?!?
Just my 2 cents for others having problems :)
http://www.microsoft.com/technet/security/Bulletin/MS08-067.mspx If you don't ever patch anything, for god sake make sure this patch is on.......- Marked As Answer byYog Li - MSFTMSFT, ModeratorFriday, November 13, 2009 7:17 AM
All Replies
- Hard to troubleshoot this in the forums. Start digging through server logs and post anything odd. Personally I would call PSS.
John Marcum | http://www.TrueSec.com/en/Training.htm | http://myitforum.com/cs2/blogs/jmarcum - Just to let you know that I am having the exact same symptoms after installing SCCM SP2.
Have you had any luck chasing this up?
Anthony Sheehy - .net I am having the same symptoms after installing SCCM SP2 too - 2 different SCCM SP2 environments but both corrupted from applying SCCM SP2:
env1/. Server 2003 SP2 x86, SCCM SP1 R2 (to SP2), local SQL 2005 db
env2/. Server 2008 SP2 x64, SCCM SP1 R2 (to SP2), remote SQL 2008 db
No software inventories anymore either, plus alll existing database v_GS_INSTALLED_SOFTWARE info completely gone (I think this relates to inventories corrupting or the MP not accepting the inventories sent)...
New DCOM errors too
As per http://social.technet.microsoft.com/Forums/en-US/configmgrsetup/thread/ce4dcc4e-cfa8-4128-9f2f-595bd4053865 I also see critical status for the SMS_MPCONTROL_MANAGER component. Reviewing the messages for that component reveals errors stating "MP Control Manager detected management point is not resonding to HTTP requests. The HTTP status code and text is 500, Internal Server Error".
Any help much appreciated - wasted 2 days/nights on this already :(- Hi Kiwifulla,
Can you post MP_Framework.log and M__SINV.log from SMS_CCM or CCM\Logs folder to troubleshoot further with your MP issues?
Surendar Reddy - Hi Surendar
I followed advice from other threads and deleted/recreated the MP (for both environments) which resolved the MP issues and the status's so far (6 hours now) are all green :) And no more crashing of CcmExec.exe!
However...the only thing that still doesn't work is the Software Inventory - as in, all the logs I have been through look OK (I can post them up but can you please confirm the exact logfile names as I have been through so many I lose track which ones are specific to Software Inventory as far as writing to the database goes).
I just built a new server including management appz (Forefront, SCOM R2, WUpdate etc), installed the latest client (via the Client folder source on the Site Server), ran all the inventories but again, only the hardware inv has come through. In Resource Explorer only hardware shows, and no dates or anything under software.
When I query the database directly, there is only one or two products showing under v_GS_INSTALLED_SOFTWARE (where pre-SP2 there were alot of products in there) however those few products are returned from clients that are currently offline - all live client software has disappeared from this table as though the most recent (delta?) has corrupted the data and therefore no information was retained.
A couple of other things to note that I have changed (re other threads)...
1/. DCOM\COM Security\Launch and Activation Permissions no longer included "SMS Admins" therefore I added them back in (Local Launch/Remote Activation) and this stopped all the DCOM errors that I kept getting after each reboot. (this is how our Prod SP1 is configured too)
2/. I have manually upgraded the SCCM Client on the Site Servers for each environment as one of them still was 4.00.6221.1000 even though the SP2 upgrade said it was successful for both environments?
3/. My pre-SP2 customisations to smsdef.mof and configuration.mof disappeared therefore I redid these tonight and all that hardware info has started coming back (e.g NIC drivers, more DNS info). These files were renamed to .backup at the time of the upgrade, then all customisations gone
4/. As per the readme, my WSUS SP2 Products and Classifications were all set back to defaults, therefore I changed back to what we had before SP2.
Thanks for any help :) I am due to upgrade Prod on Friday however I think at this stage I will reschedule!
Steve - Hi Kiwifulla,
For Software inventory MP_Sinv on your MP server which will show you the flow from clients and its status. please post it to see whats happening.
1. DCOM permission that you added is fine, however it is not reflect anything for your software inventory issue.
2. After manually upgrading client to SP2, is this client sending software inventory?
3. After upgrade its require 1 hour to complete Component Manager backing stage, during this time if you copy your custom mof files it will not success. however to confirm monitor dataldr.log for mof compile success (0x0).
4.After upgrade Products and Classifications will be set to default it is by design.
Please post MP_Sinv.log this can be found at SMS_CCM or CCM\Logs folder.
Surendar Reddy - Thanks Surendar
1. No problems with DCOM errors now nor CCMEXEC crashing - this was the fix but a bit of a worry that it needed to be done manually
2. Manual upgrade and clean server build eventually showed software inventory thorugh the Resource Explorer but still not showing up via SQL Reports or via direct SQL query though. Worked this one out - v_GS_INSTALLED_SOFTWARE is actually pulled via Hardware inventory which is via smsdef.mof also - SP2 had unselected Asset Intelligence categories that were definitely selected before as I have 40+ custom reports that all broke. I finally worked out that this table was in fact populated via hardware inventory/smsdef.mof and traced it back to these Asset Intelligence categories being turned off! Anyway, I have selected the missing categories again and rerun Machine Policies\hardware inventory and it is all back again and all reports are working!
3. It was after 2 days I fixed the custom mof files as I didn't notice they had changed back to defaults
4. This is dangerous as all products/categories were sync'ed through which ruined my DEV updating process as previously only the updates we needed were showing which made drag/drop each month easy to add new patches to update lists. This is now not possible as all products/service packs etc. are showing in SCCM. My bad though as should have been more wary on this as it was in the readme! Only affected my DEV instance though, I caught it in TEST before any sync's happened :) Wish I knew how to clean them all out of SCCM and start again without deleting Software Update point and recreating all the update lists, tasks, Software Update Packages etc! No major though - only DEV!
So I am happy now - all is fixed again!
-----------------------------------------------------------------------
SUMMARY OF MY FIXES POST SCCM-SP2
-----------------------------------------------------------------------
DCOM\COM Security\Launch and Activation Permissions no longer included "SMS Admins" therefore I added them back in (Local Launch/Remote Activation) - reboot to stop DCOM errors and CCMEXEC crashing
Created new MP on another system, changed the default MP to point to the new MP, deleted original MP. Let it flow through for about 30 minutes then reversed (created new MP on original Site Server, assigned as default, then deleted secondary MP and deleted secondary Site System)
Re-applied any previous customisations to smsdef.mof and configuration.mof (as these were removed by the upgrade)
As per the readme, changed my WSUS SP2 Products and Classifications back to what we had before as SP2 set back to defaults
Reapplied Asset Intelligence categories as these were lost in the upgrade (as these update smsdef.mof when turning on/off)
Plenty of reboots in between at the end to ensure all was still clean
Hope this helps someone else out there!
Steve- Proposed As Answer byKiwifulla Thursday, November 05, 2009 1:18 AM
- Marked As Answer byYog Li - MSFTMSFT, ModeratorFriday, November 13, 2009 7:17 AM
- Hi Kiwifulla,
After testing my upgrade in a cloned VM lab of my live environment I was all ready to go as I had no errors at all.
Performed the upgrade last night in production and wham... got hit with the same problems as others in this thread.
If anyone can explain why a cloned live environment worked fine but my live environment didn't, you're doing better than me :)
Can I just clarify something though? On the DCOM security bits mentioned in this thread, where EXACTLY are you setting them?
I added it to the SMS Agent Host with the ID of {AD65A69D-3831-40D7-9629-9B0B50A93843} under Component Services>Computers>My Computer>DCOM Config.
I originally added Local Service and System to have Local Launch rights as that's what the event log errors were moaning about.
This cleared most errors but resulted in CCMExec.exe crashing constantly.
I updated the CCM Client > Still 1 crash but not as bad, but MP not working.
I then added SMS Admins and gave it permissions, still no better.
There were plenty of reboots in-between these
I then patched the box with Microsoft .NET Framework 3.5 Service Pack 1 and .NET Framework 3.5 Family Update for .NET versions 2.0 through 3.5 (KB951847)
Lo and belhold... everything is fine now?!?!?
Just my 2 cents for others having problems :)
http://www.microsoft.com/technet/security/Bulletin/MS08-067.mspx If you don't ever patch anything, for god sake make sure this patch is on.......- Marked As Answer byYog Li - MSFTMSFT, ModeratorFriday, November 13, 2009 7:17 AM
- Hi Stevyb69
The DCOM security I added was via dcomcnfg (run box), Component Services\Computers\My Computer\Right-click and select Properties, click on COM Security tab, under Launch and Activation Permissions, add SMS Admins back in with Local Launch and Remote Activation... but...
I have now applied SP2 to our Prod server and had the same CCMEXEC crash again :( - this time I logged a Premier Support call which lasted hours!
Anyway, the DCOM permissions were already set correctly in the above mentioned path, but I did get other DCOM errors with a specific GUID that pointed to the SMS Agent Host (same path you specified) and once I added the Local Launch permission for Local Service and rebooted - the DCOM errors stopped.
Coincidentally, I too applied Microsoft .NET Framework 3.5 Service Pack 1 and .NET Framework 3.5 Family Update for .NET versions 2.0 through 3.5 (KB951847) BUT I added this BEFORE I applied SP2 (as this was the only pending Windows Update earier in the day - and I did a clean reboot an hour or so before applying SP2).
Microsoft spent hours running debugging on the CCMEXEC and trolling through logs/registry etc. and after all those hours and reboots, the server became more stable (as the CCMEXEC was still crashing after each reboot for the first hour or so). I mentioned the MP move and they said I could do this but wanted to wait for debug results first, therefore I didn't actually do anything with the MP.
I did have to add all my smsdef.mof and configuration.mof changes back in, as well as reenabling Asset Intelligence categories, and of course the WSUS Products and Classifications.
I now wonder if because all the clients were potentially sending hardware/software inventories based on the original customised mof files (with addtional classes enabled and the Asset Intelligence classes enabled) and with the server at one point (up to about an hour) by not having those additional classes enabled, if this pushed the CCMEXEC over the top as there were alot of MP errors rejecting hardware inventories, and software inventories turning up in the sinv.box\BADSinv folder. Perhaps once the server processed the new .mof files and the clients pulled down the new policy, then the new inventories sent were able to be processed correctly again? That and the dodgy DCOM fix/reboot!
Perhaps your cloned environment had less inventories to process as there was probably not as many clients in the VMware environment?
Who knows though - that's my theory :)
Also another thing to note is that I upgraded the CCM Client after the server had stabilised (about 2am!) so this didn't contribute to the fix.
Cheers
