none
Anywhere Access VPN suddenly broken -- default MSCHAPv2 fails, manually enabled EAP-MSCHAPv2 works

    Question

  • Bottom line:

    After reading countless forum threads and articles that don't apply to this exact scenario and eliminating all other variables I've been able to think of, it seems that a recent Windows update may have caused network policies in NPS to no longer process authentication requests that occur over MS-CHAP-v2 properly. They always result in an Event 6273, Reason Code 16 "Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect."  However, authentication requests ARE properly processed when EAP-MSCHAP v2 is enabled in the NPS network policy and used by the client. The problem is that MS-CHAP-v2 is the only default authentication for the Anywhere Access VPN functionality built into Windows Server 2016 Essentials, so this issue has broken VPN connectivity for the entire user base until they manually reconfigure the VPN profile that was auto-created by the Essentials Connector on their systems to use EAP-MSCHAP v2 instead. And if the Anywhere Access Repair wizard in Windows Server Essentials Dashboard were ever run, it would restore the default NPS network policy for VPN, which does not include EAP-MSCHAP v2, once again breaking VPN.

    Background:

    I support a client running a Windows Server 2016 Essentials server. The Remote Access VPN capability was enabled when the server was originally deployed and has been working flawlessly for over 2 years -- until now.  I just learned that all users who try to connect to VPN receive an "Incorrect username or password" error, and the server's Event Viewer has the corresponding entry I described above. The credentials are absolutely correct, because they still work for the Remote Web Access portal and Remote Desktop Gateway, both also part of Windows Server Essentials Anywhere Access and enabled on this server.  Running the Anywhere Access Repair wizard does not resolve this.  There have been no changes to this configuration since the server was initially deployed except to update the SSL certificate (which is still current and trusted) and no recent changes to the server itself except for Windows updates.  Unfortunately I don't know precisely when the issue began because these users do not use VPN often, but I suspect a bug in a Windows update is the culprit here. It seems this behavior has arisen from a buggy update before based on this article, although that fix did not help me because the issue there involves using EAP.

    I read the numerous threads about this issue where changing the RADIUS shared secret to all caps, all lowercase, or all lowercase and numbers resolved the issue, but this setup involves no RADIUS clients. I also read the articles mentioning that using a certificate with a blank Subject field could cause this, but our certificate has a populated Subject field, and that issue seems to apply when using Windows 7 clients and PEAP authentication, neither of which we're using. I also saw suggestions that the router could be the culprit, but I can reproduce this issue even while trying to connect from an internal system on the same subnet as the server.  And that same system started working when I implemented the workaround described below, so trying to start VPN from a local system was not a problem itself.

    As a workaround, I finally modified the default NPS policy for VPN access, named "{502F03DC-1EC9-49A9-811A-99BA53619319}", to add EAP-MSCHAP v2 as an option under Constraints > Authentication Methods.  When I made the corresponding change to the VPN connection settings on the client, it connected successfully.  So again, contrary to the errors on the client and in the server's Event Viewer, credentials are not the issue.

    So why has the server suddenly stopped processing requests that arrive over regular MS-CHAP-v2 properly?  This is a major annoyance because the VPN profile that gets automatically created on workstations through Windows Server Essentials Connector specifies regular MS-CHAP-v2, and the default NPS network policy configuration also uses that. This means that a) currently, users have to manually reconfigure their VPN connection settings to get a connection, and b) if the Anywhere Access Repair wizard in Windows Server Essentials Dashboard were ever run, the NPS network policy would be returned to its default settings that do not allow EAP-MSCHAP v2, thereby breaking everything again.

    If anyone has any suggestions, I'd be happy to try them, but once again the VPN setup was originally implemented leaving everything at the default settings, worked fine for 2 years, and then suddenly broke without any changes having been made on the server except Windows updates.


    • Edited by jphughan Wednesday, April 3, 2019 8:08 PM
    Wednesday, April 3, 2019 8:06 PM

Answers

All replies

  • Hello jphughan,

    Thank you for posting in this forum.

    Since your server has not made any changes other than updates. It is recommended that you try to uninstall the update and restore your previous VPN settings to see if the issue is caused by the update.

    Best Regards,

    Leon


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, April 4, 2019 9:39 AM
  • Hello jphughan,

    Thank you for posting in this forum.

    Since your server has not made any changes other than updates. It is recommended that you try to uninstall the update and restore your previous VPN settings to see if the issue is caused by the update.

    Best Regards,

    Leon


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Hi Leon,

    Thanks for the reply, but I would appreciate it if you could step up your customer service a bit here.  I've given you a report of a serious problem described in a very clear way, and I've narrowed down the underlying cause rather than just saying, "VPN doesn't work."  All of that should make it fairly easy for someone to reproduce the issue or at least figure out what might be responsible for authentication failing over MS-CHAP-v2 even when it works over EAP MS-CHAP v2.  Telling me to simply uninstall updates is not an acceptable response for several reasons:

    - I don't know exactly when the problem started because users at this location don't use VPN often, especially right now because this client is an accounting firm at the end of tax season, so everyone has been in the office working long hours 7 days per week for the last few months.  This means that VPN might have been broken for weeks or possibly months without anybody noticing, which in turn means that I don't even know how far I'd have to roll back.  And of course even if I did, that work schedule doesn't really make downtime for rollbacks feasible, especially for a situation like this where a workaround exists.  

    - Even the above actually assumes that uninstalling updates will solve the problem, which I can't be absolutely positive about.  So I might end up incurring a bunch of downtime and inconvenience for this client without actually solving anything.

    - Even if I uninstalled one or more updates AND it resolved the issue, that would still mean that there's a live Windows update that breaks a core function of Server 2016 Essentials, so that wouldn't actually solve the underlying problem.  In addition, Server 2016 doesn't allow you to block updates, so it's not clear how I would prevent the problem from being reintroduced.  But even if I could do THAT, what do you suggest going forward?  Testing every cumulative update released each month from now on to see if it breaks VPN again, and uninstalling it if it does?  In addition to the massive inconvenience, that would force me to forego any other fixes in those cumulative updates that might be important to the system.

    If this problem was in fact introduced by an update, then Microsoft would need to develop another update to fix it -- which means they first need to become aware that there's a problem to be fixed.  That was the reason I created this thread.  I did NOT create it to simply be told to uninstall an update that might contain other important fixes.  There have been some serious security vulnerabilities fixed in the last few updates, for example.



    • Edited by jphughan Thursday, April 4, 2019 9:01 PM
    Thursday, April 4, 2019 8:47 PM
  • I'm having the same issue.  NPS configured with MS_Chapv2.   I have multiple NPS policies.  Only the policy authenticating with MS CHAPv2 is failing.  Prior to the reboot/installing the patches just a few hours ago it was working fine.  All the other policies are using other authentication methods to network devices.   Same description of the issue as jphughan.   Just looks like a password failure from NPS logs or event viewer, but users can log into other devices not using CHAPv2 without trouble.  I deleted the policy and recreated, rebooted the NPS server, stopped/started NPS, still just the one policy CHAPv2 failing while others work for the same users.  Finally changed the policy to PAP and it works fine.  Windows updates definitely foobared something.

    Last update was in Dec.  Updates applied this go round:
    2019-02 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4485447)
    2019-04 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4493470)
    Windows Malicious Software Removal Tool x64 - April 2019 (KB890830)

    Wednesday, May 1, 2019 9:12 PM
  • update, uninstalled KB4493470 and now its working fine via MSCHAPv2
    • Marked as answer by jphughan Monday, May 13, 2019 4:36 PM
    Wednesday, May 1, 2019 10:21 PM
  • update, uninstalled KB4493470 and now its working fine via MSCHAPv2

    THANK YOU MerrettW for this report!  It appears this issue was introduced even earlier though, because the update you mention wasn't available when I started this thread -- but at least this issue is now confirmed to be caused by a Windows Update that was released sometime  in the December 2018 through March 2019 range.  Our server installs every month's cumulative update, and I don't think rolling them back until I find the first one that introduces this issue will be feasible in our case.  In addition, going without recent security updates would be a particularly significant concern for this server, so for right now the workaround we have is probably the better overall solution for us.
    • Edited by jphughan Wednesday, May 15, 2019 4:49 AM
    Monday, May 13, 2019 4:36 PM
  • Hello jphughan,

    Thank you for posting in this forum.

    Since your server has not made any changes other than updates. It is recommended that you try to uninstall the update and restore your previous VPN settings to see if the issue is caused by the update.

    Best Regards,

    Leon

    Leon, based MerrettW's corroborating report of my exact issue above and his confirmation that uninstalling a Windows Update resolves it, is there any chance of you now escalating this to someone who might be able to develop an update to fix the problem that Microsoft created in a recent update, and ideally reporting back here to let us know the first Windows Update that will NOT introduce this issue if it is installed?
    • Edited by jphughan Monday, May 13, 2019 4:51 PM
    Monday, May 13, 2019 4:50 PM
  • Yea, kinda stuck on this one as I can't patch the server.  maybe related, NPS runs on an AD server in this case.
    Wednesday, May 15, 2019 2:42 AM
  • I am seeing the same issue as MerrettW - my NPS was no longer working with MSCHAPv2. Uninstalled KB4493470 and now its working fine via MSCHAPv2.  It is disturbing that this is still being pushed out over one month later by Microsoft, when it is clearly causing issues.  We had this exact same problem in January 2019 with KB4467684, and we had to uninstall that KB as well.
    Thursday, May 23, 2019 5:46 PM