locked
WSUS 3.0 IPv4 Issue - Download failures RRS feed

  • Question

  • We are using the latest available version of WSUS that runs on WIN2008 R2.

    We changed the IP address of the WSUS server.

    Now, our client computers are failing to download updates from the WSUS server successfully. 

    In Fiddler, the sequence of http requests looks like this on the client side:

    (1) Targets the correct host using the unchanged WSUS "hostname:port"

    HEAD /selfupdate/wuident.cab?1506041722 HTTP/1.1

    (2) Targets the correct host using the unchanged WSUS "hostname:port"

    POST /ClientWebService/client.asmx HTTP/1.1

    (3) & (4) as for (2)

    (5) Targets the OLD IP address (not the hostname) and thus fails with a 50x error

    HEAD /Content/E1/4631DA8D6454B5680C0159B5254C15D54A8184E1.exe HTTP/1.1


    What we are expecting to see is the 5th request pointing to either the unchanged hostname or at the very least, to the correct new IP address.

    Things we have checked to try to fix this:

    1/ DNS. We are absolutely confident that there is no way the client (or server) could be resolving the hostname to the old IP address. We have scavenged ruthlessly in both the forward and reverse lookup zones; scoured WINS for any stale entries; scoured the Windows HOSTS file; scoured DHCP (just in case).... we have dug as deeply as we think it is possible to go to ensure this is not a name-resolution issue

    2/ On the WSUS Server where the IP was changed, we have

    (a) triple-checked all IIS bindings and reset IIS, making sure there are no references or bindings to the old IP address

    (b) triple-checked all basic network settings and checked the hosts file

    (c) Dug into the internal SQL Server WSUS database (using SQL Server Management Studio) and looked through all tables and views to try to identify any stale references to the old IP addresses. In this respect I must admit we have cannot have covered every item of data (because there is so much) but we have dug around as much as we can and concluded that its unlikely the host IP address for these requests is stored therein. 

    (d) Checked every setting we can find in the WSUS console

    (e) Executed the clean-up wizard a number of times

    (f) run NSLOOKUP and network connectivity tests

    3/ On the affected client where we see the erroneous requests that point to the OLD ip address we have:

    (a) run NS LOOKUP and network connectivity tests; cleared the DNS cache; thoroughly reviewed all Group Policy and Local Security Policies to ensure the Window Update policy settings are all correct and use only the hostname

    (b) Cleared out the BITS .dat files and restarted both BITS and WU Services with each test 

    (c) Attempted to remove all BITS queued jobs using both the BITSAdmin tool and the Powershell cmdlets (this is on Windows 7 x64) -- however, there is one job owned by SYSTEM that we cannot remove no matter what we do 

    4/ We have also tried DECLINING the updates we know trigger the problem; letting the client run; then re-approving the updates on WSUS and then letting the client run again, but the problem persists

    5/ We have extensively tested network connectivity between the client and server at various points of the stack (e.g. TCP/IP layer tests, protocol-specific HTTP tests, SOAP tests, etc.) and cannot identify any issues.

    At this stage we are considering the possibility of having to either:

    1. Entirely trash the Windows Updates folders on the CLIENT side and see what happens

    2. Entirely re-install WSUS.

    Any suggestions that do not cover the ground we have already covered, would at this time be very gratefully received.

    thanks

    Thursday, June 4, 2015 5:57 PM

Answers

  • As we are in the fortunate position of having some clients virtualized, we were able to take a snapshot of an affected client and then go ahead and completely destroy the C:\Windows\SoftwareDistribution folder

    With this done, the problem has gone away, i.e. the CONTENT request is now targeting the hostname instead of the old redundant IP address

    So I think also, we have determined definitively that the CONTENT request hostname information is being cached on the client side, somewhere within the SoftwareDistribution folder. Since the contents of this folder look fairly cryptic and probably fairly difficult to (conceptually) "disassemble" and make any sense of, I'm not sure we are ever go to determine exactly where the cached data is stored. It may be impossible.

    So it looks like the only practical solution will be to entirely delete the SoftwareDistribution folder on all clients that are failed to operate properly once the WSUS IP address has been changed. Whether such a solution would ever be practical in an Enterprise environment is doubtful, but if you have only a limited number of client machines as we do, it looks like the only practical solution to the problem (apart from NOT changing the WSUS IP address in the first place, of course).

    HTH somebody else at some time

    Sunday, June 7, 2015 3:29 PM

All replies

  • Do you have a static DNS entry to the old IP address?

    the client PC can you ping the proper WSUS by name and IP address?

    ipconfig/flushdns on client if ip address is wrong when pinging by name.

    do you have a group policy that is configuring windows update by ip address and not name?

    check registry  I think  : hklm\software\policies\windows\windowsupdate you should see the settings for wsus server ip address? or server name? is it correct?

    If this needs to be changed you will need to restart windows update service on PC or reboot.

    if you use group policy on clients to configure wsus run gpupdate/force and reboot after fixing group policy.

    Thursday, June 4, 2015 6:04 PM
  • Do you have a static DNS entry to the old IP address?

    As mentioned above, we have ruthlessly scavenged all relevants DNS, WINS and DHCP servers, as well as triple-checking the HOSTS files on both the server and clients. At this time we are 99.999% confident that this is NOT a name resolution issue.

    the client PC can you ping the proper WSUS by name and IP address?

    Yes. We are 99.999% confident that this is not a basic networking issue. As mentioned above, we have carried out fairly extensive tests at various different layers of the stack, all successful.

    ipconfig/flushdns on client if ip address is wrong when pinging by name.

    do you have a group policy that is configuring windows update by ip address and not name?

    No. We have carefully reviewed both Group Policies and Local Security Policies on both the server and client. We have carried out RSOP tests and reviewed all settings are correct and hostname only. In any case, if the Policy settings were incorrect, I would expect to see ALL http requests targeting the wrong host, whereas in our case, as described by my original post, it is only the CONTENT requests that are targeting the old IP address.

    check registry  I think  : hklm\software\policies\windows\windowsupdate you should see the settings for wsus server ip address? or server name? is it correct?

    I had not checked this at the time of the Original Post but I have checked now, and the settings are correct in the Registry keys. They use the hostname:port format as expected. 

    If this needs to be changed you will need to restart windows update service on PC or reboot.

    if you use group policy on clients to configure wsus run gpupdate/force and reboot after fixing group policy.

    As mentioned above, this is not a group policy issue.

    Thanks for all your ideas. 

    If any one else has any more ideas, I'd be grateful. Thanks 

    Friday, June 5, 2015 11:51 AM
  • My key suspicions here are that:

    1. There is some setting or cache somewhere that WSUS is picking up and sending to the client that tells the client to use a download URL targeting the OLD IP address

    OR 

    2. The OLD IP address is being cached somewhere by the Client

    But for the life of me I cannot work out where that stale information is being cached. 

    We have reviewed IIS very carefully. 

    We have cleared all the internet caches and temporary caches we can find on the Client.

    We have tried to clear as much as we can of the BITS queue on the client

    I am just stunned by how difficult this is to resolve and find.

    Any help really gratefully appreciated.

    TNX

    Friday, June 5, 2015 12:03 PM
  • As we are in the fortunate position of having some clients virtualized, we were able to take a snapshot of an affected client and then go ahead and completely destroy the C:\Windows\SoftwareDistribution folder

    With this done, the problem has gone away, i.e. the CONTENT request is now targeting the hostname instead of the old redundant IP address

    So I think also, we have determined definitively that the CONTENT request hostname information is being cached on the client side, somewhere within the SoftwareDistribution folder. Since the contents of this folder look fairly cryptic and probably fairly difficult to (conceptually) "disassemble" and make any sense of, I'm not sure we are ever go to determine exactly where the cached data is stored. It may be impossible.

    So it looks like the only practical solution will be to entirely delete the SoftwareDistribution folder on all clients that are failed to operate properly once the WSUS IP address has been changed. Whether such a solution would ever be practical in an Enterprise environment is doubtful, but if you have only a limited number of client machines as we do, it looks like the only practical solution to the problem (apart from NOT changing the WSUS IP address in the first place, of course).

    HTH somebody else at some time

    Sunday, June 7, 2015 3:29 PM