none
WinShock (KB2992611) Patch breaks IIS

    General discussion

  • I've installed the KB2992611 patch on several Windows Server 2012 systems running IIS with PHP and all of them have stopped serving https pages altogether.  I also have a 2008 sharepoint server where the patch worked.

    On the systems effected by this broken patch I initially only installed a single patch, 2992611, which is when things broke.  I also tried installing all the other patches thinking I missed something but it was still broken.

    My system is running:
    Windows 2012 - Fully Patched
    IIS - php 5.5.18

    Problem:
    Attempt to load the page from a client on port 80 - works fine.
    Attempt to load the page from a client on port 443 - page fails to load with 'The webpage at https://test.domain.com/ might be temporarily down or it may have moved permanently to a new web address.'

    Event Log:
    Event ID 36888
    A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 20. The Windows SChannel error state is 960.

    Update: More Event Logs
    Event ID 36888
    A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 10. The Windows SChannel error state is 1203.

    I would appreciate any help in getting this patch fixed since it is an important patch and I don't want to simply uninstall it.



    • Edited by DBWYCL Wednesday, November 12, 2014 8:16 PM
    Wednesday, November 12, 2014 7:46 PM

All replies

  • Upon closer inspection this fails in chrome 38.0.2125.122 m (64-bit) but works on firefox 33.1 and IE 11.0.10
    Wednesday, November 12, 2014 9:01 PM
  • Call 1-800-Microsoft.  State you have an issue with a security patch KB2992611 andn that security patch calls are a free call and you need diagnostic assistance ASAP.

    If you have any problems opening a case, email me at susan-at-msmvps.com (change the -at- to @)

    Thursday, November 13, 2014 6:42 AM
  • We're experiencing the same issue. Windows server 2012 IIS SSL sites not loading in Google Chrome and our FTP connections using SSL also fail.
    Thursday, November 13, 2014 1:30 PM
  • Unfortunately I called, told them i was having an issue with a security patch and asked if calls for security patches were free and I was told that no, they are not.  :(
    Thursday, November 13, 2014 2:56 PM
  • grumble grumble

    Okay just emailed you to get your phone number so I can set up a case.

    Thursday, November 13, 2014 6:00 PM
  • You had the bad luck to be assigned to a call center operator who hasn't been properly trained.  I recommend you try again and/or email Susan as she suggested.
    Thursday, November 13, 2014 6:01 PM
  • I've experienced the same issue.  I'm glad I found your post.

    After KB2992611 was installed via Windows Automatic Update users of our retail site on Google Chrome were not able to establish a secure connection and could not reach our secure pages to checkout/manage accounts.

    I've since uninstalled the update until the issue can be resolved.

    Thursday, November 13, 2014 6:23 PM
  • I currently have a case open with microsoft.  In the mean time i've found that we can mitigate the issue because we have an load balancer device fronting the web server (terminating the SSL cert among other things) so the issue doesn't present itself for users besides myself who access the server directly.
    • Edited by DBWYCL Thursday, November 13, 2014 8:39 PM
    Thursday, November 13, 2014 8:38 PM
  • I currently have a case open with microsoft.  In the mean time i've found that we can mitigate the issue because we have an load balancer device fronting the web server (terminating the SSL cert among other things) so the issue doesn't present itself for users besides myself who access the server directly.
    What exactly needs to be done with a load balancer that mitigates this and is there documentation to support this?
    Thursday, November 13, 2014 10:41 PM
  • Well we know that HTTP is unaffected so one solution is to terminate the SSL at the load balancer and travel back to your web server over HTTP.  Another solution is to use HTTPS back to your web server which may also work depending on your load balancer since we know that IE and firefox work, your load balancer may not have an issue with the HTTPS page.
    Friday, November 14, 2014 12:50 PM
  • If you run a web farm, I highly recommend reading up on load balancers and how they work.

    Most are Linux based (so would have been vulnerable to Heartbleed), and they basically terminate the SSL connection to/from the client, then re-send the request to an internal server via http port 80.

    By using this method, you can mitigate the Microsoft SChannel vulnerability by making sure that ONLY the load balancer IP address can be accessed through your firewall.

    Microsoft stated there is no mitigation.  But they failed to mention that completely blocking a computer via a firewall mitigates the problem.

    I recommend and use the Kemp LoadMasters, but research and find your favorite brand.


    • Edited by Brain2000 Friday, November 14, 2014 6:51 PM
    Friday, November 14, 2014 6:50 PM
  • I experienced the same problem on Windows Server 2012 with IIS and Google Chrome. It appears that at the moment the best approach is to try and remove the 4 new ciphers that Microsoft added as part of the patch for TLS 1.2 or just disable TLS 1.2 alltogether which would make browser clients fall back to TLS 1.1.

    odetodata.com/2014/11/microsoft-patch-ms14-066-leads-to-https-problems-with-iis-and-google-chrome/

    OdeToData


    • Edited by OdeToData Sunday, November 16, 2014 12:01 AM
    Sunday, November 16, 2014 12:01 AM
  • I'm seeing this error with some of my web pages as well.

    Client in use windows 7, IE9

    Server in use windows 2012 (fully patched)

    we disabled TLS 1.2 in the browser and forced it to negotiate at TLS 1.1 and were successful.  This to me is not a long term fix.    I tried the regedits indicated in KB2992611, but the ones under Microsoft SSL Protocol Provider generates an error when I try to delete the entries indicated.

    Monday, November 17, 2014 5:49 PM
  • The fixes for https errors from this patch are hit and miss.  I made the recommended registry fixes, which didn't work by themselves, but then played with turning ssl3 back on and off as well as turning tls1.2 off and then, and got one 2012 R2 server working properly again by doing that, but couldn't replicate it on another 2008 R2 server, both of which had identical schannel registry settings before the patch was installed.  Uninstalling the patch hasn't fixed any affected servers for me even though it has for others.  My 2008 R2 server would work if I enabled ss3 while disabling tls1.2, but even then it would only work if I made IE able to use ssl3, and it won't work if just the three tls options are checked in IE. At the times the https pages are not loading, it doesn't matter what browser is being used, or if the client is local or remote.

    So I believe is some cases it is wrecking all of TLS, and registry fixes are not fixing that.  Hoping for an actual replacement patch from MS to fix these problems so I can finish patching my https web servers that have not been yet since every server I've done so far has lost all https functionality after the patch was applied, and I can't afford that to happen.


    • Edited by u2surfer Monday, November 17, 2014 11:32 PM
    Monday, November 17, 2014 11:24 PM
  • Remove the new Ciphers from the registry as a workaround to resolve this problem:

    It's detailed in the KB:  https://support2.microsoft.com/kb/2992611/en-us

    Tuesday, November 18, 2014 3:56 PM
  • I had removed the ciphers by those instructions, but it did not fix my issue.  MS just released a new version of the patch today that is separated now into two different installs, but even after applying these updated patches, I still have no https being served and the same schannel 36888 errors every time a page is called.

    They also released a new out of band patch for a whole different critical vulnerability that has been seen in the wild allowing domain accounts to be elevated to domain admin privileges.  Not a great week for MS security, and thus for me either trying to fix these issues caused by faulty patches.

    Tuesday, November 18, 2014 8:17 PM
  • Can you review your SSL cert and rebind it or see if it's older and needs to be rekeyed?
    Tuesday, November 18, 2014 10:55 PM
  • Has everyone tried the rereleased KB2992611 that was issued today for Server 2012 and 2008R2?  If you are STILL having issues with the rereleased updates, please email susan-at-msmvps.com (change the -at- to @) and I'll get a case set up.
    Wednesday, November 19, 2014 12:06 AM
  • My cert is sha1, but still is 2048 bits and is working fine on one server that had similar issues but I was finally able to fix by disabling and re-enabling ssl3 and tls1.2 in the registry. My 36888 errors are two specific ones for every page hit,

    The following fatal alert was generated: 80. The internal error state is 1250.

    The following fatal alert was generated: 80. The internal error state is 1051.

    My understanding is that this means a protocol mismatch, so it's happening before it gets to the cert and seems to be that tls although enabled in the registry, isn't actually working correctly, and with ssl3 disabled, no protocols are available to work, although on my problem server even ssl3 won't work right when enabled.  I am able to rdp though, and I made sure it was set to tls1.0, so tls seems to be working in that capacity.  I am just guessing though at this point since my experience has been worse than most of those I see online who are mostly able to roll back changes successfully, whereas I haven't.

    Wednesday, November 19, 2014 12:36 AM
  • I opened a case with Microsoft and verified with them that the KB2992611 patch is causing the LSASS.EXE process, in some scenarios, to have a high CPU usage of 100% on at least one to two cores, peaking at three cores.  We have eight core systems, so it eating about 25% of our processing power.

    

    Let me just say that it was a terrible phone experience.  The VOIP service that Microsoft uses is of the lowest quality, and to make matters worse, they use Indians that have heavy accents.  I finally realized that the person I was speaking with was clueless about how threads work.  When he saw 35% CPU usage he said "you're only using 35%, that's not maxing out the processor".  I tried to explain that the maximum that one thread could take on an eight core system is 13%.  He didn't understand.  If you've ever seen the movie "The Fifth Element", it was like the scene where the deaf guy rolls Bruce Willis the cue ball instead of throwing him the gun, and the look on Bruce Willis face...

    

    Anyways, I think Microsoft needs to wake up about the problems that the KB2992611 patch is bringing with it.  As a side note, Microsoft would be better of if they traded three of their tenderfoot techs for one developers.

    Wednesday, November 19, 2014 7:03 PM
  • Could this update have broken access to microsoft update?  My Exchange 2007 server running on Server 2003 R2 can't access microsoft update anymore (I get generic error 0x80248015).  I was able to access microsoft update just a couple days ago when I downloaded and installed KB2992611 (along with something like 11 other updates).
    Thursday, November 20, 2014 2:29 AM
  • Microsoft contacted me today and provided me with an additional patch.  It corrected the CPU issues with LSASS.EXE.  After I reported the results, they  notified me that this patch would be included in Windows Update.

    Kudos Microsoft for fixing the CPU portion of the issue.  I can't comment on whether this also fixes the TLS 1.2 issues, as my IIS servers sit behind a 3rd party proxy that handles encryption.

    Thursday, November 20, 2014 4:50 AM
  • Could this update have broken access to microsoft update?
    That appears to be a completely separate problem, a built-in expiration date for Microsoft Update.  See this thread.
    Thursday, November 20, 2014 10:32 PM
  • Could this update have broken access to microsoft update?

    That appears to be a completely separate problem, a built-in expiration date for Microsoft Update.  See this thread.
    Hey thanks for that!
    Thursday, November 20, 2014 11:23 PM
  • Yes the rereleased update fixed our issues.  Thanks susan!
    Friday, November 21, 2014 12:55 AM
  • Will the patch be rereleased for Windows 2003?

    Event Type: Error
    Event Source: .NET Runtime 2.0 Error Reporting
    Event Category: None
    Event ID: 5000
    Date:  11/19/2014
    Time:  3:00:03 PM
    User:  N/A
    Computer: xxxxxxxxxx
    Description:
    EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 opensslchannel, P5 4.1.0.0, P6 48ac3fd9, P7 7b, P8 4b, P9 system.argumentoutofrange, P10 NIL.

    Friday, November 21, 2014 2:38 PM
  • Hi,

    It seems like that update 3018238 will fix the issue caused by KB 2992611.

    Here are some references below for you guys:

    Slow SQL Server Performance after Windows Update

    https://social.technet.microsoft.com/Forums/en-US/c287fac7-32a3-4eec-91b6-249dc897f75a/slow-sql-server-performance-after-windows-update?forum=winserve

    MS14-066: Vulnerability in SChannel could allow remote code execution: November 11, 2014

    https://support.microsoft.com/kb/2992611

    Best Regards,

    Amy


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, December 10, 2014 2:49 AM
    Moderator
  • The same KB is replaced with MS14-068 on November 18, 2014, Reinstalling the KB would resolve the issue, Refer below link for more detail.

    https://support.microsoft.com/kb/2992611


    Satheesh

    Friday, January 9, 2015 3:38 PM