locked
WSUS - Wsuspool high CPU & hung worker process requests RRS feed

  • Question

  • PROBLEM (BRIEF):
    Randomly and intermittently, the WSUSPOOL in IIS starts using high CPU and RAM.  Start seeing repeated worker process requests over and over from the same client(s) that seem to be hung.  Seems to be with newly imaged clients connecting for the first time.

    MORE INFORMATION:
    I'm running SERVER 2016 with WSUS (SSL) connected to SCCM 2016.  I checked the IIS logs and don't see anything blocked.  The problem will resolve itself after a day or so and stay resolved for several days.  Newly imaged workstations have no problem connected.  Then the problem will suddenly return again.

    TEMPORARY SOLUTION:
    I've found recycling the WSUSPOOL app pool can help a little with the issue but not for long.  The CPU usage will climb back up within 30 minutes.  I've also found that the problem will resolve itself after a day... but a few days later a new set of workstations may suddenly start exhibiting the same behavior.  

    WSUSPOOL (IP: 10.210.218.111):

    When the problem occurs, one or more clients will be stuck in the list of requests.  Meanwhile, existing client requests seem to be processed without any problem at all.  See behavior below for client 10.210.218.111

    WebsiteID URL Verb Client IP State Module Name Time Elapsed
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 596734
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 531672
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 467687
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 403109
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 339687
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 275703
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 211390
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 147484
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 834222
    716775307 /ClientWebService/client.asmx post fe80::7507:6f2c:3fd8:3ec9%10 ExecuteRequestHander ManagedPipelineHander 66609
    716775307 /ClientWebService/client.asmx post 10.210.218.111 ExecuteRequestHander ManagedPipelineHander 16734
    716775307 /ClientWebService/client.asmx post 10.210.35.126 SendResponse IIS Web Core 13890
    716775307 /ClientWebService/client.asmx post 10.212.29.229 ExecuteRequestHander ManagedPipelineHander 1375


    IIS LOGS (FILTERED ON CLIENT 10.210.218.111):

    2017-07-07 16:52:12 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 15
    2017-07-07 16:52:12 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 500 0 0 78
    2017-07-07 16:52:12 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 15
    2017-07-07 16:55:23 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 0
    2017-07-07 16:55:23 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 78
    2017-07-07 16:58:35 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 31
    2017-07-07 16:58:36 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 62
    2017-07-07 17:02:06 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 18251
    2017-07-07 17:02:41 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 35031
    2017-07-07 17:06:15 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 15
    2017-07-07 17:06:15 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 140
    2017-07-07 17:08:55 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 0
    2017-07-07 17:08:55 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 62
    2017-07-07 17:11:20 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 0
    2017-07-07 17:11:26 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 6578
    2017-07-07 17:14:39 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 15
    2017-07-07 17:14:39 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 78
    2017-07-07 17:17:51 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 15
    2017-07-07 17:17:51 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 78
    2017-07-07 17:21:04 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 718
    2017-07-07 17:21:06 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 1703
    2017-07-07 17:24:20 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 546
    2017-07-07 17:24:23 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 2406
    2017-07-07 17:27:37 10.107.23.18 POST /SimpleAuthWebService/SimpleAuth.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 1062
    2017-07-07 17:27:39 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 2984
    2017-07-07 17:27:43 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 3515
    2017-07-07 17:46:59 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 62
    2017-07-07 17:46:59 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 15
    2017-07-07 17:46:59 10.107.23.18 POST /ClientWebService/client.asmx - 8531 - 10.210.218.111 Windows-Update-Agent/10.0.10011.16384+Client-Protocol/1.40 - 200 0 0 62

    Friday, July 7, 2017 6:22 PM

Answers

  • I noticed that I also couldn't get the WSUS Server Cleanup to succeed either.  I decided to reinstall WSUS (remove the WSUS DB and the downloads) and start over.  Everything is working normally now.
    • Marked as answer by Travis Mathews Thursday, September 7, 2017 10:28 PM
    Thursday, September 7, 2017 10:28 PM

All replies

  • While my script doesn't look like it will fix your issues, this seems to be an issue with a fragmented database, along with possible SusClientId duplication.

    Have a peek at my Adamj Clean-WSUS script. It is the last WSUS Script you will ever need.

    http://community.spiceworks.com/scripts/show/2998-adamj-clean-wsus

    What it does:

    1. Remove all Drivers from the WSUS Database.
    2. Shrink your WSUSContent folder's size by declining superseded updates.
    3. Remove declined updates from the WSUS Database.
    4. Clean out all the synchronization logs that have built up over time (configurable, with the default keeping the last 14 days of logs).
    5. Compress Update Revisions.
    6. Remove Obsolete Updates.
    7. Computer Object Cleanup (configurable, with the default of deleting computer objects that have not synced within 30 days).
    8. Application Pool Memory Configuration to display the current private memory limit and easily increase it by any configurable amount.
    9. Run the Recommended SQL database Maintenance script on the actual SQL database.
    10. Run the Server Cleanup Wizard.

    It will email the report out to you or save it to a file, or both.

    Although the script is lengthy, it has been made to be super easy to setup and use. There are some prerequisites and instructions at the top of the script. After installing the prerequisites and configuring the variables for your environment, simply run:

    .\Clean-WSUS.ps1 -FirstRun

    and then

    .\Clean-WSUS.ps1 -InstallTask

    If you wish to view or increase the Application Pool Memory Configuration, you must run it with the required switch. See Get-Help .\Clean-WSUS.ps1 -Examples

    If you're having trouble, there's also a -HelpMe option that will create a log so you can send it to me for support.

    And on any client you image, run this after you image

    net stop bits
    net stop wuauserv
    reg delete "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v AccountDomainSid /f
    reg delete "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v PingID /f
    reg delete "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v SusClientId /f
    reg delete "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v SusClientIDValidation /f
    rd /s /q "%WinDir%\SoftwareDistribution"
    net start bits
    net start wuauserv
    wuauclt /resetauthorization /detectnow

    or before you sysprep, delete the 4 reg key entries. Sysprep does NOT take care of this.


    Adam Marshall, MCSE: Security
    http://www.adamj.org

    Sunday, July 9, 2017 4:42 AM
  • I also assume that your WSUS Server is a VM? You should assign it as many vCPUs as possible without crossing CPU affinity (if you have dual processors with 12 cores each, assign 12 vCPUs to WSUS). RAM is less important than vCPUs.

    Adam Marshall, MCSE: Security
    http://www.adamj.org

    Sunday, July 9, 2017 4:44 AM
  • Dealing with the exact same issue. I had to completely remove WSUS/SUP from my environment for now inorder to even use SCCM... (running Configmgr 1702 with hotfix rollup)

    I even removed the WSUS and redeployed...

    Yes I run your script Adam.

    Once some Win10 clients get online my wsus worker process goes cpu insane, eats ALL resources and prevents use of SCCM completely. 

    My server has 8 cores, 32GB of RAM, pure SSD storage...

    We only have 60 online clients!

    I've been writing about it here with another gentleman with the same problem...

    https://www.reddit.com/r/SCCM/comments/6nvh8h/wsus_issues/

    Monday, July 17, 2017 10:39 PM
  • All:

    I am having the exact same problem for the last couple of weeks.  I have been banging my head against a wall trying everything from cleaning up WSUS DB, rebuilding it from scratch, modifying App Pool settings and nothing is helping.  We have been adding new Windows 10 1607 clients over the last couple of weeks which is when I noticed this, but we have added machines in the past and never experienced this issue.  My Windows 1511 machines, and Win 7 machines are scanning fine.  I've got it narrowed down to my new 1607 clients that are connecting.  Is anyone planning on contacting Microsoft about this?  I was about to call them this morning, until I found this thread.  The reddit site brought me here :)  I'm stuck and would like to get my machines patched.  

    Monday, July 24, 2017 11:22 AM
  • Same issue. In my case problems with wsuspool gone after unsuccessfull deinstall kb2919355 (changes rolled back after server reboot).
    Thursday, August 17, 2017 3:59 PM
  • Absolutely same issue here, 2016 VM WSUS (sccm) server, iis application pool eats 100% cpu 8 cores.

    All "stuck" clients in my environment is other only 2016 servers with august updates.

    At this time isn't possible to work on sccm server. No other workaround rather than disable wsus application pool at this time.

    Seems like any of latest MS updates cause the issue, may be august updates.

    https://support.microsoft.com/en-us/help/4034661

    WSUS servers will exhibit increased CPU, memory, and network utilization when Windows Update clients perform their first scan after installing KB4034658.

    Microsoft is investigating this issue and will provide an update as soon as possible.

    Update: Uninstalling KB4034658 from all 2016 servers solve this problem. 
    Tuesday, August 22, 2017 5:46 PM
  • If you're up for it, Andrey, contact me through my website and reference this thread and I'll give you Version 3.0 beta of my script that will help speed up WSUS Database operations by about 1000-1500 times (yes, you read that right). That may fix your issue - I'm not sure.

    Adam Marshall, MCSE: Security
    http://www.adamj.org


    • Edited by AJTek.caMVP Tuesday, August 22, 2017 5:58 PM
    Tuesday, August 22, 2017 5:58 PM
  • All:

    I am having the exact same problem for the last couple of weeks.  I have been banging my head against a wall trying everything from cleaning up WSUS DB, rebuilding it from scratch, modifying App Pool settings and nothing is helping.  We have been adding new Windows 10 1607 clients over the last couple of weeks which is when I noticed this, but we have added machines in the past and never experienced this issue.  My Windows 1511 machines, and Win 7 machines are scanning fine.  I've got it narrowed down to my new 1607 clients that are connecting.  Is anyone planning on contacting Microsoft about this?  I was about to call them this morning, until I found this thread.  The reddit site brought me here :)  I'm stuck and would like to get my machines patched.  

    Are your 1607 clients 1607 RTM? If so there is a known issue with 1607 RTM communicating with WSUS. You must install a cumulative update manually past September 2016 for it to re-communicate with WSUS again. Perhaps this is the issue for you?

    Adam Marshall, MCSE: Security
    http://www.adamj.org

    Tuesday, August 22, 2017 6:01 PM
  • We've taken the decision to disable the 'update peering' system that Microsoft introduced in Windows 10 / Server 2016.. and I found that on clients where I hadn't actually disabled the setting in Group Policy (Computer Configuration > Administrative Templates > Windows Components > Delivery Optimisation > Download Mode - Set to Bypass (100)), it would cause a near denial of service condition on the WSUS server for some reason even though currently the WSUS server is just handling a very small number of machines, and it would occur even with just one machine checking in - CPU slammed at 100% for several minutes before failing to find any updates.

    Setting the above group policy option, the clients now check in instantly and the WSUS server CPU usage barely registers again.

    Could be worth reading up more on it if you're unfamiliar as it could be related:

    https://docs.microsoft.com/en-gb/windows/deployment/update/waas-delivery-optimization

    Wednesday, August 23, 2017 10:04 AM
  • I noticed that I also couldn't get the WSUS Server Cleanup to succeed either.  I decided to reinstall WSUS (remove the WSUS DB and the downloads) and start over.  Everything is working normally now.
    • Marked as answer by Travis Mathews Thursday, September 7, 2017 10:28 PM
    Thursday, September 7, 2017 10:28 PM
  • Make sure to use my script as a method of maintenance and enjoy the speed increase with my Indexing too! If you choose not to use my script, that's fine, just make sure you do the proper maintenance for WSUS!

    Adam Marshall, MCSE: Security
    http://www.adamj.org

    Friday, September 8, 2017 12:22 AM
  • This august last KB resolve the issue:

    https://support.microsoft.com/en-us/help/4039396

    • Addressed issue where Update History and hidden updates are lost and a full scan for updates happens after installing OS Updates 14393.1532 through 14393.1613, including KB4034658. Installing this update will not restore past update history or hidden updates for users who have already installed the listed updates. However, this current update will address this issue for users who have not yet installed them.

    • Addressed issue with WSUS update metadata processing that can cause some clients to time out with a 0x8024401c error.

    Sunday, September 10, 2017 7:14 AM