locked
L2TP/IPSec Connection Fails from Windows 7 Ultimate to Windows Server 2012 R2 with error 789. RRS feed

  • Question

  • I previously posted this on the Windows Forums where I was promptly told that this is too in depth in networking for said forum.

    To preface this, I am using the server in a lab environment and am attempting to set up my own L2TP/IPSec VPN. I have opened ports 500 UDP and 1701 TCP on my router to the server's primary interface where the VPN is. This is on a consumer Comcast connection where other applications such as Arma 3 dedicated servers and IIS have worked.
    The RRAS role is running based on this tutorial: http://www.thomasmaurer.ch/2014/01/how-to-install-vpn-on-windows-server-2012-r2/ I have only deviated from this in using DHCP forwarding instead of a static IP pool as my router runs a DHCP server, and as I understand it, the router should give out IP addresses from the internal IP pool which I use for everything else. I am also using PSK authentication instead of it being certificate based. For user authentication I have MS-CHAP-V2 and CHAP enabled; I am connecting from the remote device with an account I have created on the server for the purpose of this VPN which I know RRAS connections are allowed on.

    When connecting I receive error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer. From what I have seen, this can be fixed by checking that both ends of the connection are not behind a NAT (not an option), checking PSK (already done), and checking certificates (not applicable). If there is a way to fix this issue that would be excellent, however my server will always be behind a NAT firewall as the router is one, and the modem becomes one if multiple devices are connected to it without a router in between.

    Saturday, March 14, 2015 2:36 PM

Answers

  • Hi,
     
    According to your description, my understanding is that L2TP/IPSec VPN(using a preshared key for Internet Key Exchange (IKE) authentication) prompts error 789 between client(Windows 7) and VPN Server(Windows Server 2012 R2).
     
    Confirm that if you have configured the same preshared key on both ends of the VPN connection.
    VPN server:Properties – Security – select allow IPSec… and type the preshared key. then restart the VPN service.
    VPN client: Properties – Security – set the Type of VPN as L2TP/IPSec – Advanced settings – select Use preshared key for authentication.

    Then try to reconnect the VPN again.

    If the problem still exits,  L2TP/IPSec VPN(using a preshared key) is based on PPTP VPN, reference the link below and check to see if the basic PPTP VPN connection works:
    https://technet.microsoft.com/en-us/library/cc739537%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396
    or
    https://technet.microsoft.com/en-us/library/cc757206(v=ws.10).aspx

    If it works, then reference the steps(confirm preshared key) above to change the PPTP VPN to L2TP/IPSec (with preshared key) VPN.
                                          
    Best Regards,
    Eve Wang


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Monday, March 16, 2015 8:42 AM