none
Keeping your servers up-to-date with patches

    Question

  • Hello All,

    Please note that this is NOT a WSUS question. I have that installed in our environment and I know how to use it, or at least I like to think I do. I am a System Admin of a completely virtualized environment. As our company grows, more and more servers have been built and keeping up with the latest patches has become a challenge. So I am looking for some advice from users who have more experience here than I do.

    Up to this point, we have been updating the virtual machines as we do the ESX(i) hosts. This was okay when there were fewer machines. Now, if there are 20 or more VMs on a single host, it is just too time consuming. Also, as we add more hosts, we fall behind with the updates. So, I am thinking we can keep on top of it better if I can separate the ESX(i) updates from the Windows Updates. So perhaps I can update a different group of VMs each day. Production VMs will need to wait until the weekend of course. I am thinking I will create a new group in WSUS that has all of the latest updates approved and then move the VMs into that group as I am ready to do each machine. I manually approve the updates but I can probably auto-approve critical updates. 

    Anyway, for those of you who who work in larger environments and keep on top of Windows updates for servers, how do you do it? What are your best practices? All of our desktop computers automatically install updates in the late-night hours but we obviously cannot do that with servers. Let me know if you have any questions or need more information. Thanks for the help.

    -Adam

    Friday, April 25, 2014 7:35 PM

Answers

  • The best way I've found for handling this is spread which servers receive their updates on which days, and which servers can be allowed to automatically install them and reboot, and while need manual intervention.

    I assume you handle your WSUS settings via GPO, though if you do it manually via the registry you could still do similar, it's just more leg work.

    Rather than having a single GPO setting when and how to install updates, I've split the servers across a number of OUs in AD, and then configured a GPO for each group. Each GPO handles which day and time the updates are installed (so there are GPO's for days throughout the week), and whether to auto install them (so two variations of each day).

    Then it's just a matter of deciding which servers fit into which categories. For instance redundant servers are auto install, but are configured to do so on different days. Likewise servers which aren't customer facing, or aren't used at night can auto install. The rest are just set to download and prompt, but obviously the day they're set to do that download is spread out as well, so they're not all doing it at the same time.

    In my case I haven't bothered spreading them across multiple time periods throughout the night, but if you have  a lot of servers to configure then that could also be an option.

    Saturday, April 26, 2014 12:21 PM
  • You can use 2 or more WSUS machines if you are concerned about the load

    as for VMware, go pester them as this is a MSFT forum for Hyper-V and Server


    Corsair Carbide 300R with window
    Corsair TX850V2 70A@12V
    Asus M5A99FX PRO R2.0 CFX/SLI
    AMD Phenom II 965 C3 Black Edition @ 4.0 GHz
    G.SKILL RipjawsX DDR3-2133 8 GB
    EVGA GTX 6600 Ti FTW Signature 2(Gk104 Kepler)
    Asus PA238QR IPS LED HDMI DP 1080p
    ST2000DM001 & Windows 8.1 Enterprise x64
    Microsoft Wireless Desktop 2000
    Wacom Bamboo CHT470M
    Place your rig specifics into your signature like I have, makes it 100x easier to understand!

    Hardcore Games Legendary is the Only Way to Play!

    Saturday, April 26, 2014 12:38 PM

All replies

  • The best way I've found for handling this is spread which servers receive their updates on which days, and which servers can be allowed to automatically install them and reboot, and while need manual intervention.

    I assume you handle your WSUS settings via GPO, though if you do it manually via the registry you could still do similar, it's just more leg work.

    Rather than having a single GPO setting when and how to install updates, I've split the servers across a number of OUs in AD, and then configured a GPO for each group. Each GPO handles which day and time the updates are installed (so there are GPO's for days throughout the week), and whether to auto install them (so two variations of each day).

    Then it's just a matter of deciding which servers fit into which categories. For instance redundant servers are auto install, but are configured to do so on different days. Likewise servers which aren't customer facing, or aren't used at night can auto install. The rest are just set to download and prompt, but obviously the day they're set to do that download is spread out as well, so they're not all doing it at the same time.

    In my case I haven't bothered spreading them across multiple time periods throughout the night, but if you have  a lot of servers to configure then that could also be an option.

    Saturday, April 26, 2014 12:21 PM
  • The best way I've found for handling this is spread which servers receive their updates on which days, and which servers can be allowed to automatically install them and reboot, and while need manual intervention.

    I assume you handle your WSUS settings via GPO, though if you do it manually via the registry you could still do similar, it's just more leg work.

    Rather than having a single GPO setting when and how to install updates, I've split the servers across a number of OUs in AD, and then configured a GPO for each group. Each GPO handles which day and time the updates are installed (so there are GPO's for days throughout the week), and whether to auto install them (so two variations of each day).

    Then it's just a matter of deciding which servers fit into which categories. For instance redundant servers are auto install, but are configured to do so on different days. Likewise servers which aren't customer facing, or aren't used at night can auto install. The rest are just set to download and prompt, but obviously the day they're set to do that download is spread out as well, so they're not all doing it at the same time.

    In my case I haven't bothered spreading them across multiple time periods throughout the night, but if you have  a lot of servers to configure then that could also be an option.

    Hello,

    Thank you very much for the suggestions. I apologize for the delay in my response but things come up when working in IT which take sudden priority as you know.

    Yes, I have WSUS setup by GPO. We have various servers in the different GPOs but they all point to the WSUS server. I am not worried about the load on the WSUS server itself as it doesn't get bogged down too much as this point. Right now, the GPO policy for the vast majority of our servers is set to download the updates but prompt for installation. The workstations are set to auto install which is easy enough. 

    In our particular situation, I will be the one doing the installations. We just don't necessarily want the servers to download updates until I am ready to do the installations on those servers as this risks somebody mistakenly installing them when not wanted and causing a reboot of the server. Not good. I would like to do at least a couple of servers each night if possible in order to keep up. 

    Do you have the WSUS server auto-approve the critical updates? I am weary of using the auto-approve update. Thanks.

    -Adam

    Wednesday, May 07, 2014 5:56 PM
  • Yeah it's a tricky balance. Well you could always choose option 2 to simply notify that there are updates awaiting installation, so they don't get downloaded until you tell a machine to update. Obviously the downside to that is having to wait while a potentially large download happens (though if it's over the local network it shouldn't take too long). So you centrally control what is deployed to the servers, and get all the servers retrieving the data over the local network rather than having to download the updates over the internet, but keep the actual download and install part as a manual process.

    For desktop machines I tend to allow auto-approval and installation since the risk of a user doing something bad with an unpatched machine is higher than the risk of the update having issues. For servers though I never auto-approve, and prefer to manually go through approving the latest updates for the servers once I've had a chance to confirm there are no reports of issues with the updates. Even then I tend to stagger rolling out the updates minimise any issues, and where possible test them first on staging servers or servers which aren't as critical. I tend to look at it in terms of the two risks. The risk of the vulnerability being exploited vs the risk to the server if something went wrong. For a public facing / user facing server then the risk tends to sway towards installing earlier, where as backend servers that are never directly connected to are less at risk. Additionally, things like IE updates while critical on a desktop really aren't important on a server, so can safely wait for a while. Any sysadmin that goes surfing from a server shouldn't be in the job.

    Wednesday, May 07, 2014 6:55 PM
  • Yeah it's a tricky balance. Well you could always choose option 2 to simply notify that there are updates awaiting installation, so they don't get downloaded until you tell a machine to update. Obviously the downside to that is having to wait while a potentially large download happens (though if it's over the local network it shouldn't take too long). So you centrally control what is deployed to the servers, and get all the servers retrieving the data over the local network rather than having to download the updates over the internet, but keep the actual download and install part as a manual process.

    For desktop machines I tend to allow auto-approval and installation since the risk of a user doing something bad with an unpatched machine is higher than the risk of the update having issues. For servers though I never auto-approve, and prefer to manually go through approving the latest updates for the servers once I've had a chance to confirm there are no reports of issues with the updates. Even then I tend to stagger rolling out the updates minimise any issues, and where possible test them first on staging servers or servers which aren't as critical. I tend to look at it in terms of the two risks. The risk of the vulnerability being exploited vs the risk to the server if something went wrong. For a public facing / user facing server then the risk tends to sway towards installing earlier, where as backend servers that are never directly connected to are less at risk. Additionally, things like IE updates while critical on a desktop really aren't important on a server, so can safely wait for a while. Any sysadmin that goes surfing from a server shouldn't be in the job.

    Thank you again. But a similar problem remains with the second option of the notification without the download. A user will still be notified that updates are ready to be downloaded and then choose to do so. So at that point it's probably better to have the updates downloaded and ready to install. All of our servers report to WSUS so none of them hit the internet which makes sense considering most of the workstations are on the same LAN as the WSUS server so it's pretty quick. 

    As of run now, I don't schedule the synchronizations continually. I manually run them, update each group of computers in WSUS until all are green, and then run the synchronization again and repeat. An obvious disadvantage to this is that we won't have the latest updates available until I get through all of the groups and run the synchronization again! So that leaves us vulnerable. I also manually approve each update. People sometimes call me crazy but I have seen WSUS auto-approve updates in the past which I did not want pushed out, so I play it safe. 

    Thank you for all of the help!

    Wednesday, May 07, 2014 7:47 PM