locked
WSUS SP2 - lots of error after server ran out of OS partition disk space. RRS feed

  • Question

  • Hello All,

    My primary server Windows 2008 R2 running WSUS SP2 ran out of disk space in primary partition, C drive. The server had 12 GB memory hence had 16 GB page file. The tech attending the drive space issue reduced the paging to 4 GB, deleted quite a bit of IIS logs and rebooted the server. I understand the issues with WSUS started to happen right after that.

    In the application log I have these IDs generated.

    Hundreads of of 2299, IIS-W3SVC-WP every 5 min's - An application has reported as being unhealthy. The worker process will now request a recycle. Reason given: ASP.NET application initialization failed.. The data is the error.

    If I try to access WSUS using the servers WSUS Administration console, I get just the console but the local server is missing. If try to add the server, it comes up with error "Cannot connect to "Server Name". The server may be using another port or different Secure Socket Layer setting.

    I can not open Server Manager to start looking at Application logs. If I do, the moment I click Application or System under Windows Logs I get the Error window that says: Microsoft Management Console has stopped working, there are three options, 1. Check online for a solution and close the program, 2. Close the program, 3. Debug the program. I hit debug to get the following data.

    Description:
      Stopped working

    Problem signature:
      Problem Event Name: APPCRASH
      Application Name: mmc.exe
      Application Version: 6.1.7600.16385
      Application Timestamp: 4a5bc808
      Fault Module Name: mscorwks.dll
      Fault Module Version: 2.0.50727.5472
      Fault Module Timestamp: 5174ddb3
      Exception Code: c0000005
      Exception Offset: 0000000000175e64
      OS Version: 6.1.7601.2.1.0.272.7
      Locale ID: 3081

    When I get the above, I also get the following in the Application logs.

    Event ID: 1023 - .NET Runtime version 2.0.50727.5472 - Fatal Execution Engine Error (000007FEF98E5756) (80131506)

    Followed by 1000 - Faulting application name: mmc.exe, version: 6.1.7600.16385, time stamp: 0x4a5bc808
    Faulting module name: mscorwks.dll, version: 2.0.50727.5472, time stamp: 0x5174ddb3
    Exception code: 0xc0000005
    Fault offset: 0x0000000000175e64
    Faulting process id: 0x%9
    Faulting application start time: 0x%10
    Faulting application path: %11
    Faulting module path: %12
    Report Id: %13

    I can see WSUS services running, but I believe they are starting and stopping as the Event ID 2299. I also know newly built servers which are pointing to this particualar WSUS server (through GPO settings) are not getting any updates whatsoever. WSUS functionalty is basically not there. I am having to use a Windows 7 PC to connect to this server to look at its logs.

    I did however gave the server its 16 GB page file back in C drive hoping this will make some difference, but it did not. There seems to be multiple issues here, this server is the main Primary server where update approval takes place.

    I was wondering, if an uninstallation of WSUS SP2 (keeping the internal DB) and reinstallation joining the existing DB will fix the issues? I am also wondering since this is the primary server where all approval and declines are made, if I can re-join the new WSUS SP2 installation to this servers internal DB, will it get every approvals, declines back? Also, are these  MMC related issues related to WSUS? or will they need seperate troubleshooting.

    This server has 3 more layers of WSUS server under it.

    Any help to tackle the issue will be appreciated.

    • Changed type Yan Li_ Tuesday, September 17, 2013 2:29 AM
    Tuesday, September 3, 2013 1:59 AM

Answers

  • All other servers had the following WSUS patches.

    KB2720211 first, and to fix isseus caused by KB2720211, we were asked to apply KB2734608 when a call was logged with Microsoft.

    Question: Do I need to apply both?

    Yeah... actually you should have installed KB2734608 before performing the synchronization from the other downstream server.

    I think that installing the update and resynchronizing will be sufficient. Without KB2734608 the new server cannot sync the SHA-256 hashes.

    If you install KB2734608 you do not need to install KB2720211.


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2013)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    http://www.solarwinds.com/gotmicrosoft
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

    • Marked as answer by Yan Li_ Tuesday, September 17, 2013 2:29 AM
    Tuesday, September 10, 2013 7:37 PM

All replies

  • I was wondering, if an uninstallation of WSUS SP2 (keeping the internal DB) and reinstallation joining the existing DB will fix the issues?

    Not likely. Almost certainly the root cause is the corrupting of that database as a result of overflowing the SYSVOL. In such cases, reinstalling WSUS with a new database is the only known remediation (unless you have a database backup, but I suspect not since you went straight to "reinstall" without stopping at "restore").

    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2013)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    http://www.solarwinds.com/gotmicrosoft
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

    Tuesday, September 3, 2013 10:18 PM
  • Thanks Lawrence. Correct, there was no DB backup.

    I was not too sure when you mentioned Sysvol, or was it just a short cut to mentioning about System Partition Volume's Page File?

    As you mentioned, reinstalling is a likely fix. I have some questions relating to that. This server lets call it Server A, has another 3 layers of servers (I know its not MS recommended, but it has been working so far). So it goes like this, the server I am talking about is at the top called RPAWSUS01.

    RPAWSUS01 is Upstream to RPAWSUS02

    RPAWSUS02 is Upstream to LIVWSUS01 and CRGWSUS01

    LIVWSUS01 and CRGWSUS01 are Upstream to have 8 more WSUS servers

    So, for RPAWSUS01, due to the 100's of various errors relating to .NET, IIS and WSUS, do I need to re-build the whole server as in including the OS? If yes, that is not a problem but my concerns are as follows

    - How do I again make it the top level server? Concerns: With a new DB it will not actually have any of the Computer Groups (unless I create them), it will not have any of the approvals (unless I manually approve them again), basically it will not have anything from the previuos setup. Setting this up exactly could be an issue as we decline many updates dictated by various departments individual requirements. Approvals are quite an issue as well, because not everything is approved, usually security, critical and updates category, but sometimes updates rollups as well. (At this point, a DB backup would have been a sweet deal!!)

    - Instead, can I remove RPAWSUS02 (Layer 2 server) from Replica mode, as this already has all the Computer Groups, Approvals etc. Temporarily this could be used as the primary WSUS server for approval etc (Once I remove it from replica mode).

    I can then start building RPAWSUS01 (original Layer 1 server), join it under RPAWSUS02 so that it gets all the approval, computer groups etc, and starts to serve its own clients? I can then move newly build RPAWSUS01 to its original Layer 1 spot?

    Are these meaningful actions? Its already a complicated situation.

    Wednesday, September 4, 2013 1:17 AM
  • I was not too sure when you mentioned Sysvol

    For this purpose "SYStem VOLume" = the volume where the operating system is installed.

    Strictly speaking it refers to a resource defined on a Domain Controller, and the proper name I should be using is SystemDrive. :-)


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2013)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    http://www.solarwinds.com/gotmicrosoft
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

    Wednesday, September 4, 2013 5:35 PM
  • the server I am talking about is at the top called RPAWSUS01.

    So, for RPAWSUS01, due to the 100's of various errors relating to .NET, IIS and WSUS, do I need to re-build the whole server as in including the OS?

    Not likely. But that may be more a function of the efforts and success involved in getting WSUS cleanly uninstalled, than it will the process of reinstalling WSUS. If the .NET and Web Server role installations are not damaged, there should be no reason to need to reinstall them.

    How do I again make it the top level server?

    It is the top level server by virtue of the fact that the 2nd level servers are already configured to talk to it.

    Concerns: With a new DB it will not actually have any of the Computer Groups (unless I create them), it will not have any of the approvals (unless I manually approve them again), basically it will not have anything from the previous setup.

    These are all important points. One approach is to temporarily disable synchronization on your 2nd level servers until you have fully reinstalled/reconfigured this upstream server.

    Setting this up exactly could be an issue as we decline many updates dictated by various departments individual requirements. Approvals are quite an issue as well, because not everything is approved, usually security, critical and updates category, but sometimes updates rollups as well.

    Reconfiguring it exactly, as relates to approvals and declines, is not really an issue. The 2nd level servers will 'resync' whatever you configure on the reinstalled 1st level server. You could just set it up the way it should be based on current needs and go forward from that point.

    Instead, can I remove RPAWSUS02 (Layer 2 server) from Replica mode, as this already has all the Computer Groups, Approvals etc. Temporarily this could be used as the primary WSUS server for approval etc (Once I remove it from replica mode).

    Well, yes, that is one approach, but since all of the lower levels are still functional, it would be best to not disturb them at all. And actually, you're part-way onto the optimal way to recover your upstream server. Taking a page from the "How do I migrate a WSUS Server?" playbook:

    1. Turn off synchronization of the 2nd level servers.
    2. Reinstall RPAWSUS01 as a *replica* of RPAWSUS02 and replicate.
    3. When the replication is complete, reconfigure RPAWSUS01 as an upstream server and synchronize with Microsoft.
    4. When the sync is complete, reenable the synchronization of the 2nd level servers with RPAWSUS01.

    If all went well, RPAWSUS01 will be an exact copy of what existed at the last time RPAWSUS02 synchronized with it, plus all of the new stuff released by Microsoft since then.


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2013)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    http://www.solarwinds.com/gotmicrosoft
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

    Wednesday, September 4, 2013 5:46 PM
  • Thanks Lawrence. I have some questions in relation to the last options discussed, the points 1 to 4.

    Well the question is only on Point 1.

    Turn off synchronization of the 2nd level servers - Does this mean that on RPAWSUS02 (2nd level server) I do the following?

    - Set it to Synchronize Manually under Synchronization Schedule

    - Also Untick This server is a replica of the upstream server? (The reason being, since I will be rebuilding RPAWSUS01 with exact same name, new DB etc, I did not want any accidental Sync happening between RPAWSUS02 and newly built RPAWSUS01 yet until Point 2 to 4 was complete)

    Friday, September 6, 2013 1:01 AM
  • Well the question is only on Point 1.

    Turn off synchronization of the 2nd level servers - Does this mean that on RPAWSUS02 (2nd level server) I do the following?

    - Set it to Synchronize Manually under Synchronization Schedule

    Yes. The desired effect is that the downstream servers do not try to synchronize with a partially valid "upstream server" which could be catastrophic.

    - Also Untick This server is a replica of the upstream server?

    On the downstream servers.. NO. It is only necessary to temporarily stop the regular synchronization event. They will always be downstream replica servers.

    In fact, setting them as upstream servers, and then having a sync inadvertently launched (against Microsoft) could be even more catastrophic than having them sync against a partially deployed upstream server.

    Then again, worse case scenario, either way, if a replica server goes FUBAR, you simply reinstall it and re-replicate from a working upstream server. :-)

    (The reason being, since I will be rebuilding RPAWSUS01 with exact same name, new DB etc, I did not want any accidental Sync happening between RPAWSUS02 and newly built RPAWSUS01 yet until Point 2 to 4 was complete)

    Which is why we set sync to MANUAL on RPAWSUS02 -- to preclude exactly that scenario.

    Now, strictly speaking, this does not prevent a rogue (or clueless) administrator from launching a manual sync, so if you're concerned about that, then the best solution would be to STOP the Update Services service, set it to DISABLED, and not let that service run until the upstream server is ready for use.

    Stopping the database service (WID) is also another potential option.

    There are also other creative ways to preclude accesing the console... like removing memberships in Adminstrators and WSUS Administrators, or stopping the W3SVC (which would block a console connection - but also block client connections, which is not desirable in this scenario). There are also creative solutions to block access to the APIRemoting30 v-dir... (like renaming it).

    Frankly, my vote if you have such potential issues is to have a heart-to-heart ("do this and you'll get fired") conversation with anybody else who has console access. ;-)


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2013)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    http://www.solarwinds.com/gotmicrosoft
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.


    Friday, September 6, 2013 3:20 AM
  • Hi Lawrence, the server RPAWSUS01 has been rebuilt from scratch (well we had VM image with Windows 2008 R2 SP1 which speeded up the process). Points 1 and 2 are complete. The server statistics are OK in terms of Unapproved updates, Approved updates, Declined updates etc. This is good although it took 14 hours for the 10 GB DB to sync. Thats fine. Although I noticed client reporting status is very very slow, then again it hasn't been even 24 hours after the server was rebuilt and DB synced. I have seen 10 or so clients reporting from this morning.

    I will not sync new RPAWSUS01 to RPAWSUS02 again, as I think the sync was done and OK.

    I am about to start on point 3, but I just remembered that there was Automatic Approval rules set for Critical Updates to a certain group of Computers, there was also automatic approval for all other updates (except drivers) for the Trial groups in the 1st level server (when it was working). I will set them up. After I do all these, I have one issue.

    All other servers had the following WSUS patches.

    KB2720211 first, and to fix isseus caused by KB2720211, we were asked to apply KB2734608 when a call was logged with Microsoft.

    Question: Do I need to apply both? Becasue their KB article for the patches mention that these two patches are independent of each other...and then proceed to Point 3 and 4?

    For KB2720211 - http://support.microsoft.com/kb/2720211/en-au the following is mentioned.

    Replacement information

    This update does not replace a previously released update.

    Same as above for KB2734608 - http://support.microsoft.com/kb/2734608/en-au

    I guess the main issue is do I need these patches only for the sake of keeping everything alligned with the same version? Fresh installation has put my new RPAWSUS01 to 3.2.7600.226, and applying the last patch which is KB2734608 will bring it upto 3.2.7600.256. I actaully dont remember the dot change after I applied KB2720211 last year sometime.



    Tuesday, September 10, 2013 1:06 AM
  • All other servers had the following WSUS patches.

    KB2720211 first, and to fix isseus caused by KB2720211, we were asked to apply KB2734608 when a call was logged with Microsoft.

    Question: Do I need to apply both?

    Yeah... actually you should have installed KB2734608 before performing the synchronization from the other downstream server.

    I think that installing the update and resynchronizing will be sufficient. Without KB2734608 the new server cannot sync the SHA-256 hashes.

    If you install KB2734608 you do not need to install KB2720211.


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Packaging, Deployment & Servicing (2005-2013)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    http://www.solarwinds.com/gotmicrosoft
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

    • Marked as answer by Yan Li_ Tuesday, September 17, 2013 2:29 AM
    Tuesday, September 10, 2013 7:37 PM
  • I did install the patches, sync is OK with Microsoft and with other downstream servers.

    Pls. mark this thread answered. Thanks Lawrence for your inputs.

    Wednesday, September 11, 2013 1:54 AM