Sign in
 
HomeWindows Server 2012Windows Server 2008 R2Windows Server 2003LibraryForums
Windows Server TechCenter >
> 
> 
Automatic updates - Patch testing best practices / case studies?

Answered Automatic updates - Patch testing best practices / case studies?

  • Monday, May 07, 2012 2:26 PM
     
     
    Sign In to Vote
    0

    I'm at a smallish (~40 employees) organization and have been deploying automatic updates through a WSUS server for several years now.   Up to now, my testing has been pretty cursory -- two (physical) workstations and one server running in Virtual PC, assigned to special testing groups in WSUS.  The updates are approved for the test groups and if no obvious major breakage occurs immediately, the updates are approved for the production groups.

    It's clear this is going to have to change.  Our environment has gotten more complex (although hardly to the level of a major data center) and the updates have more things to interact with (possibly adversely).

    As an example, we recently had a breakage with a piece of third-party server software.   One of the initial theories was that an automatic update caused a change to a .NET framework in IIS.   This eventually turned out not to be the case, but we still have the vendor's statement that "Microsoft patches are not supported without testing." 

    I'm looking around for resources that talk about specific results of testing Microsoft patches with the vendor's software (for example, which updates pose the most risk of breakage), but they may not exist or may not be timely enough, so I may be on my own in this regard.

    Now that we have a more robust virtual environment to work with, I have a little more flexibility, in that I can take a snapshot of the test machine before installing the patch, and roll it back if there is a problem, revoking approval.

    But wow... Do I really have to test each update, and each affected application, on each configuration, one by one?   Is the order of installation going to affect the results?

    So what's out there for patch testing procedures? Is WSUS/SCE still the best tool for this when environments begin to get more complex?   I'm not sure I can get approval to purchase something like System Center Configuration Manager.

     

All Replies

  • Tuesday, May 08, 2012 5:41 AM
    Moderator
     
     
    Sign In to Vote
    0

    Hi,


    There is no specific white paper on how to test/roll-out updates with WSUS.It would help if you would identify the specific updates you believe are creating problems for your PC, as well as the exact problem(s) you believe is/are caused by the update(s).

    In general, test is a must. It is what the WSUS admins or patch management teams need to do.To your question,absolutly YES. You have to install the updates one-at-a-time,test each update, and each affected application, on each configuration,in chronological order of release and TEST after the installation of each update. If the tests show no faults, install the next update. If the tests fail, then you've identified the culprit. The next step is to identify why. Research the update and determine exactly what the update does. Then evaluate what your affected applications are doing. Almost always the conflict is casued because the application experiencing the fault has not been updated, or has an update that's not been installed, or is too dependent on a specific call in a specific DLL, or doesn't call it correctly, or test the results from the call -- and the repair that's deployed by the update is causing the application to fail.

    Even if you purchase something like System Center Configuration Manager,it is just another deployment tool that is used with additional function.Test is still needed.

    Regards,
    Clarence
    Forum Support
    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contacttnmff@microsoft.com.

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

     
  • Tuesday, May 08, 2012 12:56 PM
     
     
    Sign In to Vote
    0

    Oh, if only it were just the one PC and a couple configurations. Oh, if I only had a team, much less one that could spend all its time testing patches.  And the resources to set up test environments for all the configurations (many are unique).  Well at least it's a little better now.

    Do you think it's at least acceptable, with the rollback capabilities of VM's, to try "all the updates" up front and go into "granular testing mode" only if breakage shows up? 

    Wouldn't it be nice if vendors tested Microsoft updates on top of their applications and released their findings (breakage at a minimum)?   Does MS even let them do this?


    • Edited by Spencer Simpson Tuesday, May 08, 2012 1:00 PM complete a sentence
    •  
     
  • Tuesday, May 08, 2012 5:18 PM
     
     
    Sign In to Vote
    0

    I can relate to your dilemma Spencer, actually thats why I'm reading this thread because I'm in a similar situation where I'm the only one supporting ~75 employees and services running across 15 servers...the simple prospect of having to create a fully functional test environment and thoroughly test it to ensure patches get rolled out smoothly is almost impossible to fathom.

    For me, having been a part of the Windows update ecosystem since its inception, I think I've settled with this way of coping:

    Make sure you have good backups of the system state on regular basis knowing you can restore it as a last resort.  Preferrably to restoring from backup, and assuming a patch hasn't stopped a system from booting, there's also the option of manually removing the most recent set of updates.  Reality is that the frequency of patches breaking systems is extremely rare when you look at how many patches are released times the number of systems they're installed on times the number of apps that could be impacted and finally the fact that most aren't impacted at all.  When something does break though, it sure puts a damper on your day and no one seems to understand how you could let something like this happen! 

    Good news, if the people that count are really that worried about it, they'll let you hire someone to run tests on every patch and every piece of software you network runs, including whatever licenses and hardware may be needed to create this environment....doesn't seem very likely to me with ~40 employees. 

    I'm not sure I've provided a solution, but sometimes a settling on a method of 'coping' the best we can do with what we have....

     
  • Tuesday, May 08, 2012 7:12 PM
     
     
    Sign In to Vote
    0
    Great info, and nice to hear that someone else is in the same boat.   Do you "try all the updates first" as I suggested and roll back then go granular only when there's a problem?
     
  • Wednesday, May 09, 2012 4:16 AM
    Moderator
     
     
    Sign In to Vote
    0

    Oh, if only it were just the one PC and a couple configurations. Oh, if I only had a team, much less one that could spend all its time testing patches.  And the resources to set up test environments for all the configurations (many are unique).  Well at least it's a little better now.

    Do you think it's at least acceptable, with the rollback capabilities of VM's, to try "all the updates" up front and go into "granular testing mode" only if breakage shows up? 

    Wouldn't it be nice if vendors tested Microsoft updates on top of their applications and released their findings (breakage at a minimum)?   Does MS even let them do this?


    Yes,it's at least acceptable.  If vendor is focusing on the application compatibiliy,it is always better to test.

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

     
  • Thursday, May 10, 2012 12:30 PM
     
     Answered
    Sign In to Vote
    1
    Great info, and nice to hear that someone else is in the same boat.   Do you "try all the updates first" as I suggested and roll back then go granular only when there's a problem?

    Well, I don't install Service Packs, drivers or feature packs or tools without waiting for a while.  Basically I've prioritized updates according to what I'm comfortable with...

    1. Security updates (immediately approved)
    2. Critical Updates (immediately approved)
    3. Definition updates (immediately approved)
    4. Update Rollups (immediately approved)
    5. Updates (immediately approved)
    6. Feature packs (Approved monthly)
    7. Service packs (Wait for a few months or longer, and coincide the SP update with my test lab or a few selected test systems)
    8. Tools (Whenever I get to it...i.e. maybe never)
    9. Drivers (If I feel its needed, usually in response to a performance issue or functionality issue that is driver related)

    Sounds like a bit of a crapshoot doesn't it?  Thats because it is...I can't afford the time to test it all upfront so I just have to look back at my break-fix experiences with updates and try to make a judgement call.

    Although I've used the wording "immediately approved", we don't install updates the day they're released to production servers because reboots are done once a week.  Something that really helps, is visiting this forum and reviewing the feedback before approving updates.  If you see a post about an update thats wreaking havoc, just skip it/them until there's clarity on the situation.  Good example of that is here: http://social.technet.microsoft.com/Forums/en-US/winserverwsus/thread/5f27ee0e-d6bf-4dfa-afa0-49cec67b2ec7. 

     
  • Monday, May 14, 2012 2:56 PM
     
     
    Sign In to Vote
    0
    OK, thanks for the input. I've been installing 1-8 automatically and avoiding AU drivers entirely, installing only the manufacturer's drivers when they recommend it.
    • Edited by Spencer Simpson Monday, May 14, 2012 2:57 PM stop repeating myself
    •