locked
Root Certificates Optional Windows Update December 2012 - KB931125 triggers Event ID 36885 - SCHANNEL RRS feed

  • Question

  • Log Name:      System
    Source:        Schannel
    Date:          12/12/2012 21:05:10
    Event ID:      36885
    Task Category: None
    Level:         Warning
    Keywords:     
    User:          SYSTEM
    Computer:      XXXXXX.XXXX.local
    Description:
    When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" />
        <EventID>36885</EventID>
        <Version>0</Version>
        <Level>3</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2012-12-12T21:05:10.912004100Z" />
        <EventRecordID>21210</EventRecordID>
        <Correlation />
        <Execution ProcessID="580" ThreadID="3604" />
        <Channel>System</Channel>
        <Computer>XXXXX.XXXX.local</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
      </EventData>
    </Event>

    ***************** Unfortunately this Update cannot be uninstalled once installed. What now?


    • Edited by Win7ine Wednesday, December 12, 2012 9:26 PM
    Wednesday, December 12, 2012 9:23 PM

Answers

All replies

  • This update caused havoc on our NPS server today.  Knocked out our secure wireless network.  I had to delete hundreds of certificates.  What a mess!
    Thursday, December 13, 2012 1:23 AM
  • Exact same problem here. Our 802.1x wireless went down when I installed this!
    Thursday, December 13, 2012 3:40 AM
  • Big problem for us too,

    BIG BIG MESS

    Why send out an update knowing well that you will fill the store!!

    Thursday, December 13, 2012 10:44 AM
  • WHAAAAAAH! Same thing here, big big troubles, since we use wireless & wired 802.1X auth. When about 1200 people are knocking on your door because the network is down, "hell" doesn't even come close. Took a while before I figured it out, since I updated the Intermediate Certificates just 2 days before and I figured that just *had* to have something todo with it. But no, MS decided to dump about 320 extra root certs. What the hell is this good for?
    • Edited by Gelfer Thursday, December 13, 2012 11:10 AM
    Thursday, December 13, 2012 10:56 AM
  • so what's the fix, if we can't uninstall the update? 
    Thursday, December 13, 2012 12:09 PM
  • Hi all,

    Same problem with us after installing the Decemeber 2012 Root Certificate Update kb931125.

    We were unable to authenticate to our wireless network via our NPS server.

    After getting our DC / NPS / Cert Auth server down below 100 Trusted Root Certificates we were then able to authenticate again and Event ID 36885 ceased.

    Hopefully MS will rectify this in time for their next TRC update!

    Thursday, December 13, 2012 12:11 PM
  • I ran into the same issue with NPS and Lync.  This KB article provides a registry key workaround that seemed to fix it for me: http://support.microsoft.com/kb/2464556
    • Proposed as answer by Junktemp Thursday, December 13, 2012 2:31 PM
    • Marked as answer by Win7ine Thursday, December 13, 2012 3:58 PM
    • Unmarked as answer by Win7ine Wednesday, January 16, 2013 9:48 PM
    Thursday, December 13, 2012 2:22 PM
  • You guys running Windows 2012 server?  Windows 2012 Essentials server?

    What specific OS is this?

    Thursday, December 13, 2012 4:47 PM
  • I have had huge email problems today because of this Microsoft update. Users cannot get email because their logins fail due to the huge list added by the update triggering the SCHANNEL problem.

    Like everyone else I have deleted about a hundred certificates. Running Server 2008R2, Exchange 2010

    I reckon many hundreds of thousands - maybe more - are having this issue.

    Presumably some rather red faced Microsoft employee has made a big mistake. However we are all human and we sometimes make mistakes.


    prd

    Thursday, December 13, 2012 7:37 PM
  • It seems this Windows Update has been pulled ... Just fired up my second WS2012E in Hyper-V and it does not see this malicious update anymore.
    Thursday, December 13, 2012 9:28 PM
  • Also had clients unable to connect with NPS on Server 2008 and 2008 R2 and Polycom phones unable to stay signed in with Lync 2010 on 2008 R2.  Used option 3 in the kb on the NPS servers and Lync front end servers and that fixed both immediately.  Exchange 2010 on Server 2008 R2 hasn't been affected so far...
    • Edited by PremiereMed Friday, December 14, 2012 7:47 PM
    Friday, December 14, 2012 7:47 PM
  • Hi Susan,

    My O/S is Windows Server 2012 DataCenter.

    KB931125 appears to have prevented by Lync Server 2013 Front End from starting.

    After deleting all the Trusted Root CAs (other than the ones needed for my Test Environment) and rebooting, the Lync FE started just fine.

    Stu

    Friday, December 14, 2012 8:36 PM
  • It happened on two of my Windwos 2008R2 servers. I'm not looking forward to manualy deleting hundreds of certificates on two servers. Is there any sort of script or automated way to remove them?
    Friday, December 14, 2012 11:24 PM
  • I had this problem on Windows 2008 R2, but it looks like others had it on other OS's. Disastrous update that you should not install! Kicked everyone off of wifi, and wasn't immediately apparent what caused it.

    Now I have to sift through and delete certs. Not 100% sure which ones should stay and which should be deleted. Perfect opportunity for MSFT to chime in with a patch or guideance!

    Saturday, December 15, 2012 11:14 PM
  • Also happened on our 2008 R2 NPS server, none of our students or teachers could connect via wireless. Registry fix however worked, shame Microsoft can't issue an urgent update fix for it

    Monday, December 17, 2012 12:14 PM
  • After installing update 931125 december 2012 on Windows Server 2008 x64, we had an issue with client certificates.

    We have our own internal PKI, issuing client certificates. The client certificate are in use for client authentication on IIS(7).

    First we noticed that not all client certificates were shown in the Select a Certificate in the browser on the client. Among the not shown client certificates the client certificate from our own CA. We fixed this by moving (do not copy, but move!) our own CA certificates from the Trusted Root Certification Authorities store to the Third-Party Root Certification Authorities on the IIS server.

    We also see that update 931125 december 2012 installed a lot of CA certificates on our server. Why this is I don't understand. It seems like a unwanted situation, having many CA certificates on your server. Why MS did this I don't understand.

    Monday, December 17, 2012 2:03 PM
  • It also affected my SCCM 2012 server. Now my clients are failing client check. :(

    I don't really want to mess with changing the registry. I hope microsoft come out with a patch or solution for this.


    windows 7 newbie

    Monday, December 17, 2012 7:39 PM
  • They won't.  They accidentally released this root cert to 2008 and 2008 r2 servers.  Unless you call and complain, they won't see the customer pain.

    Monday, December 17, 2012 7:45 PM
  • This happened to us too, now we have hundreds of servers with bad Trusted Root Lists.  How is the world can we fix this?  Not feasible to do it manually...

    Monday, December 17, 2012 7:59 PM
  • This happened to us too, now we have hundreds of servers with bad Trusted Root Lists.  How is the world can we fix this?  Not feasible to do it manually...


    The Root Cert Updates are not "bad" per se but the amount deployed in this update is far too large for Servers (>4K).
    • Edited by Win7ine Monday, December 17, 2012 9:54 PM
    Monday, December 17, 2012 9:53 PM
  • We got hit with this one too over the weekend.  Several hundred servers & workstations all got the updated.  It found it was approved in WSUS.

    Has anyone tried Method 3?

    Any caveats, or issues found ?

    Monday, December 17, 2012 10:19 PM
  • We had problem with Lync and I used method 3 in:

    http://support.microsoft.com/kb/2464556

    I works after that!

    Tuesday, December 18, 2012 3:21 PM
  • Add me to the mix. I'm in from vacation fixing my NPS wireless problem and also a raft of SCCM 2012 client problems.

    Orange County District Attorney

    Tuesday, December 18, 2012 3:40 PM
  • See post by Frank Carius re an alleged forthcoming KB response from MS re the KB931125 at: http://social.technet.microsoft.com/Forums/en-US/lyncdeploy/threads

    Stu

    Tuesday, December 18, 2012 9:07 PM
  • To the people with SCCM problems:

    Our client group also has problems with SCCM after the december updates.

    But the Root Cert Update is not the reason. They opened a PSS call.

    Reason is the powershell 3.0 Update. Is changing something with WMI (don't know details).

    Try remove that update. The KB931125 is not the reason in that case (but same impact).

    Thursday, December 20, 2012 3:36 PM
  • I just closed a call with MS on that issue.

    Use the provided KB article to create that reg value (I used a GPO) and it will work again.

    Still I am wondering if no one at MS tests those updates before they are being rolled out.

    For me, it installed around 300! new CAs resulting in the errors above, not to speak about my security concers regarding all those CAs...

    (still wondering why those CA`s are stored in trusted root but not third party trusted root)


    Who needs a signature?


    • Edited by GunniO Friday, December 21, 2012 9:40 AM
    Friday, December 21, 2012 9:35 AM
  • GunniO, did MS give you a way to clean your CA trusted root store back to the way it was?  I also have a case open with them, and last I heard they are "still working on a solution."
    Friday, December 21, 2012 2:00 PM
  • It seems Microsoft are blaming their customers for installing this update on their servers - disgusting behaviour. Microsoft - Admit you got it wrong!

    Taken from MS website search for KB931125

    "Some customers did not follow the guidelines published in KB 931125 before they installed the latest update from WU or WSUS, and they encountered the issue documented in KB 933430. For this reason, the KB 931125 updates for Server SKUs were expired from WSUS on 12/21/12. We recommend that you sync your WSUS server and approve the expiry. If you already applied the update without making sure that the certificate count does not exceed the limits and need assistance recovering, please contact Microsoft Support

          (http://support.microsoft.com/)   
    .

    prd

    Sunday, December 23, 2012 7:24 PM
  • I'm seeing this problem on a number of 2008 R2 and 2012 servers here.

    Interestingly, the registry key fix worked fine on the 2008R2 boxes, but *isn't* making any difference on the 2012 boxes. Is anyone else able to confirm that they're seeing the same thing?

    For the record, the registry key I'm creating is:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

    Value name: SendTrustedIssuerList 
    Value type: REG_DWORD 
    Value data: 0 (False)

    from http://support.microsoft.com/kb/2464556

    Wednesday, January 2, 2013 5:31 AM
  • Same problem.  Server 2008 R2.

    Creating the registry key did not solve the problem.

    Rick

    Monday, January 7, 2013 10:58 PM
  • Guys, another fix (if all your servers are expected to have the same list of CAs) is to find a good 2008 server with the "reduced" list, and highlight all of them, export to PFX file.

    Then, on your buggered servers, delete all the CAs from the trusted root store, and import the saved PFX into your trusted root store.   I did this with our Lync servers, and have no issues.

    Cheers,

    Jules

    Thursday, January 10, 2013 4:53 PM
  • Hi, all,

    We published a blog on this (with a link to a FixIt) on Saturday. Please see the blog for details about how to get the Fixit or to fix this manually.

    http://blogs.technet.com/b/windowsserver/archive/2013/01/12/fix-available-for-root-certificate-update-issue-on-windows-server.aspx

    Best,

    Christa


    Christa Anderson [MSFT] Want the Windows Server 2008 R2 Remote Desktop Services Resource Kit? Click here.

    Wednesday, January 16, 2013 9:22 PM
  • I am wondering what will happen to the certificates that are supposed to be in the trusted root store?

    If they will be deleted aswell, this will cause major trouble...

    Tuesday, February 5, 2013 10:35 AM
  • The blog link page has a FixIt solution and also details a manual solution.  If you run the manual solution you will be deleting all certificates in the Trusted Root store for the machine.  If you run the FixIt, then it retains those that are necessary for your servers.  I did add the Root and Intermediate certificates needed for our domain wildcard certificate back to the list.

    This resolved most of our issues, including the IAS authentication for wireless clients.  I'm still running down stray problems but the biggest issue afterwards was on the servers that host our Citrix farm.  Users were experiencing certificate errors constantly.  I ended up restoring the full list onto those servers but I now see numerous SCHANNEL errors in the event logs.  Haven't figured this one out yet.

    Rick


    • Edited by scdl Tuesday, February 5, 2013 5:31 PM
    Tuesday, February 5, 2013 5:30 PM
  • Hi, Gunni,

    Yep, that wouldn't be good! As noted in the Fixit I linked to in the blog, this is what the fix does:

    This solution removes all Third-party Root Certification Authorities. If your server has connectivity to Windows Update, it will automatically add back Third-party Root Certification Authorities as needed, as also discussed in KB 931125. If an affected server is isolated or disconnected from the Internet, you must manually add the necessary Third-party Root Certification Authorities back as you would have done in the past. (Or, you can install them by using Group Policy.)

    Best,

    Christa


    Christa Anderson [MSFT] Want the Windows Server 2008 R2 Remote Desktop Services Resource Kit? Click here.

    Tuesday, February 5, 2013 8:12 PM
  • Thank you for the feedback, Rick. I'm glad that the fix helped you. Can you tell me what version of Citrix and Windows Server you're running? I'd like to share this with the team.

    Best,

    Christa


    Christa Anderson [MSFT] Want the Windows Server 2008 R2 Remote Desktop Services Resource Kit? Click here.


    Tuesday, February 5, 2013 8:15 PM
  • Christa,

    Servers are Windows Server 2008 R2 SP1 using IE9.  Citrix version is Xenapp 6 with all current hotfixes applied.

    We kept running into certificate chains that were no longer valid.  Continuing through the warning to the site resulted in pages with elements blocked because of the certificate issue (mostly images)

    Trying to add back the individual Trusted Roots and Intermediates that were needed looked like it was going to be too daunting so I installed the large list again.  However this leaves me with SCHANNEL errors in the Event Logs of these servers.

    Rick

    Tuesday, February 5, 2013 9:16 PM
  • The Problem:

    SSL Certificate processing on WinOSes has been compromised.

    To identify if servers are effected

    Open Regedit on the server and go to:

    HKLM\Software\Microsoft\SystemCertificates\AuthRoot\Certificates 

    If there are hundreds of Certificates in there then you have a problem, it breaks Certificate processing.  If combined with a ZDE…  The problem also becomes one of too much trust (too many Certificates). IF Server Admins use IE on the Server with too much trust then we have another problem... IE should not be run on a server.

    Event IDs (EIDs) that are indicative of this issue (not complete):

    • EID #36885 - Event Source: SChannel
    • EID #36887 - Event Source: SChannel
    • EID #36855 - Event Source: SChannel
    • EID #2 -  - Event Source: IAS
    • EID #39 - Event Source: NapAgent
    • EID #20225 - Event Source: RemoteAccess
    • EID #20271 - Event Source: RemoteAccess

    Possible Exploits ?

    After exporting the SystemCertificates registry key I applied the reverse engineered KB code and Dec. 2012 KB 931125 .SSTs to it.  I again exported the SystemCertificates.  Using Beyond Compare I went to the last CERT in the list that was removed from the system and took it's registry key (the SHA1 of the CERT) and searched the Interweb.

     

    The last CERT key removed was "CE1A3553BA6155DA5160097B4B1EA1FF4CBA7195".

     

    In searching for that key it seems that several Virus'/Trojans are leveraging it:

     

    June 28th, 2013

    http://home.mcafee.com/virusinfo/virusprofile.aspx?key=3354154#none

    Sept. 1st, 2013

    http://home.mcafee.com/virusinfo/virusprofile.aspx?key=3868943#none

    Even so, if our ability to properly handle CERTs is compromised our ability to use PKI is compromised.  After the DigiNotar / Staat der Nederlanden <place w:st="on"><city w:st="on">Root</city> <state w:st="on">CA</state></place> compromise in June 2011 and VeriSign's CA compromise in 2001 we NEED to be able to properly process CAs.


    MS’s Approach to the Cleanup


    http://support.microsoft.com/kb/933430

    http://social.technet.microsoft.com/Forums/windowsserver/en-US/7b5e6cb7-a4f3-4466-afea-0c32da5d3167/updated-root-certificates-windows-update-for-windows-2008-r2-server-kb931125

    The basic issue is that MS's only solution seems to be one of four (non-viable) manual options.  Although they could have created a new KB to fix this they didn’t.  It also seems they are keeping it quiet.

    None of the MS solutions seem to completely address the issue (WinXP Root Certs being loaded into Servers and no clean, clear, or easy way to remove).

    MS Solutions:

    • Run Fix-It on every system

    NO.  This will delete the registry key including your own certificates which will lead to broken services.

    • Deleting the Registry Key manually (or via GPO or 'package')

    NO.  Again, as with the solution above, you will be deleting your own certificates which will lead to broken services.

    • Deleting expired certs, small certs, certs you know and here a cert, there a cert

    NO.  It is too hard to tell what certificates you are using for what when you are presented with hundreds of certificates.

    • Reconfigure SChannel to not send the list

    NO.  This doesn’t fix the problem.  It merely only stops the OS from telling you about it.

    Another Solution:

    Remove JUST the Certificates that were added by MS in Dec 2012.

    • Obtain a copy of KB 931125 from Dec 2012 from MS
    • Use WinRAR to export the KB contents to a directory
    • Use the UpdRoots.EXE and the .SSTs to remove the CERTs added by KB 931125


    "UpdRoots.EXE -d AuthRoots.sst" – for the Root CERTs
    "
    UpdRoots.EXE -d UpdRoots.sst" - for the updated CERTs
    "
    UpdRoots.EXE -d Roots.sst" - for the Local Machine CERTs


     

    Friday, October 4, 2013 12:21 PM