L2TP/IPsec VPN Problems. No ESP traffic from server


  • Hi all,

    I have a SBS2003 box at home which is set up as DC, DHCP server, DNS server, Terminal services and RAS server.

    It has two active NICs, one connected to internet through a modem router and one for the internal network at home which is connected by wire to a wireless router.

    My problem is with establishing a L2TP/IPsec VPN with the server.
    I use an XP pro SP3 with all the updates and patches, including the change in the register for the IPsec NAt-T.
    If I use a PPTP VPN or RDP it just works fine.
    If I try with L2TP/IPsec it fails with error 678 no answer from remote computer.

    I have checked all the events on the XP and on the SBS and the IKE security association is established in both main mode and quick mode and after about 30 seconds I just have events ending the IKE security association for both modes. That is all. There is no more traffic.

    No firewall. Ports 1701, 1723, 4500 and 500 are opened as well as protocol ICPM, 47, 50 and 51. I can ping the server by IP and by name.
    There are 5 free L2TP ports in the RAS.

    It seems to me that the IPsec is successful but the L2TP cannot be established.

    I am using a public network and am not sure about routers that might be between the XP machine and internet.
    I have checked the RASMAN. Log in both machines and there are no errors apart from the mentioned above.

    In the RRAS policies, the VPN is allowed.

    After much googleing, I have not found a clear coincident problem. It seems that the common problem is on establishing the IPsec, but I cannot find much about this matter. I have already spent three days doing changes and tests and am at the same point.

    Please, any light? Has someone come across this before? I don’t know where to look at anymore…

    All the best,

    Saturday, March 5, 2011 8:27 PM


All replies

  • NAT-T seems to be the problem here.  ESP can't pass through NAT/PAT natively so it's encapsulated in UDP over port 4500.  Since your IKE is working properly (it uses UDP 500) I'd look at what end is not encapsulating your ESP packets properly (or is not permitting inbound UDP 4500) which is why you aren't actually getting any data from end-to-end.

    Sunday, March 6, 2011 2:47 PM
  • Hi Matt,

    Thank you very much for your post.

    Following your advise, I have captured the traffic through the ports 500, 4500 and 1701 in both machines, client and server.

    There is no traffic at all in port 1701.

    The traffic starts at port 500 in both machines with packets showing main mode.

    After few packets, the traffic changes to 4500 in both machines and there is communication. There are some ESP packets over this port with an SPI from client to server after ISAKMP Quick mode UDENCAPin both machine's captures, but no answer as ESP from server in any of the captures. Then, some ISAKPM informational and Just some UDPENCAP Nat-Keepalive in both directions until I have the error no response from remote computer in the client.

    Also, I can port probe 4500 and 500 in the server from the client with result listening. Also 1701 once the negotiation has started.

    Many thanks,


    Monday, March 7, 2011 12:49 AM
  • Once the VPN connects and an IPSEC sa is established what are you doing to send traffic between the host and server?  Can you ping the server from the client?  Is there a route to the client computer's subnet from server via the VPN interface?  Are you using reverse route injection (or whatever MS calls it)?

    Normally when something like this happens on Cisco VPNs there's a route missing to the client VPN pool addresses or the ACL that specifies the interesting traffic to send through the VPN doesn't have the proper networks in it, i.e. the crypto ACL is wrong.

    Monday, March 7, 2011 3:45 AM
  • I am not sure if they actually connect... According to the events logs the authentication is succesful, which I believe means that the main mode and quick mode pass through.

    I can ping the server without problems and can connect to it with a PPTP VPN; is this meaning that there is a route? To be honest, I just looked at what reverse route injection is!

    I paste below the packets descriptors from wireshark of the traffic in both computers over port 4500. The public IPs have been changed.  The SPI are different because it is not the same attemp.

    Notice that the server never sends an UDP-ESP packet. Is this normal? The client sends and the server receives... never answers. Allways the same.

    This is the capture of traffic in port 4500 taken in the server where the IP is the interface conected to the modem-router and would be the public IP of the network of the client

    No.     Time        Source                Destination           Protocol Info
          1 0.000000          ISAKMP  
          2 0.001301          ISAKMP  
          3 0.739506          ISAKMP  
          4 0.741207          ISAKMP  
          5 1.389121          ISAKMP  
          6 1.462249          ISAKMP  
          7 1.462716          ISAKMP  
          8 2.183113          ESP      ESP (SPI=0x3fcf31eb)
          9 3.192558          ESP      ESP (SPI=0x3fcf31eb)
         10 5.181794          ESP      ESP (SPI=0x3fcf31eb)
         11 9.180053          ESP      ESP (SPI=0x3fcf31eb)
         12 17.179564          ESP      ESP (SPI=0x3fcf31eb)
         13 21.061898          UDPENCAP NAT-keepalive
         14 22.367351          UDPENCAP NAT-keepalive
         15 27.181213          ESP      ESP (SPI=0x3fcf31eb)
         16 37.183256          ISAKMP  
         17 37.183614          ISAKMP  
         18 37.185003          ISAKMP  
         19 37.185259          ISAKMP  
         20 41.061581          UDPENCAP NAT-keepalive
         21 42.361926          UDPENCAP NAT-keepalive
         22 61.061534          UDPENCAP NAT-keepalive
         23 62.371099          UDPENCAP NAT-keepalive
         24 81.060892          UDPENCAP NAT-keepalive
         25 82.421807          UDPENCAP NAT-keepalive
         26 101.060532          UDPENCAP NAT-keepalive
         27 102.388600          UDPENCAP NAT-keepalive
         28 121.060185          UDPENCAP NAT-keepalive
         29 122.394611          UDPENCAP NAT-keepalive
         30 141.059793          UDPENCAP NAT-keepalive
         31 142.388660          UDPENCAP NAT-keepalive
    This is a capture of traffic in port 4500 taken in the client where the IP is the local IP of the client. would be the public IP of server

    No.     Time        Source                Destination           Protocol Info
          1 0.000000        ISAKMP  
          2 0.713816         ISAKMP  
          3 0.715852        ISAKMP  
          4 1.381697        ISAKMP  
          5 1.468778         ISAKMP  
          6 1.469595        ISAKMP  
          7 2.178953         ISAKMP  
          8 2.181206        ESP      ESP (SPI=0xb51b9b11)
          9 3.180589        ESP      ESP (SPI=0xb51b9b11)
         10 5.180606        ESP      ESP (SPI=0xb51b9b11)
         11 9.180631        ESP      ESP (SPI=0xb51b9b11)
         12 17.180683        ESP      ESP (SPI=0xb51b9b11)
         13 22.382077        UDPENCAP NAT-keepalive
         14 22.569213         UDPENCAP NAT-keepalive
         15 27.180753        ESP      ESP (SPI=0xb51b9b11)
         16 37.195071        ISAKMP  
         17 37.196887        ISAKMP  
         18 37.914988         ISAKMP  
         19 37.915054         ISAKMP  
         20 42.382208        UDPENCAP NAT-keepalive
         21 43.607017         UDPENCAP NAT-keepalive
         22 62.382342        UDPENCAP NAT-keepalive
         23 62.568138         UDPENCAP NAT-keepalive
         24 82.382368        UDPENCAP NAT-keepalive
         25 82.570712         UDPENCAP NAT-keepalive


    After this, I have the connection failed prompt on the client. No response from remote computer.

    Monday, March 7, 2011 4:36 AM
  • Hi aznarepse,


         Your XP client change register for NAT-T, is it refer to KB818043? Both client and server need to support NAT-T



         The firewall need to open port 500,4500,50,1701,5500, please open port 5500


        Check certificate on RRAS and xp client setting if normal.

        Below articles maybe useful to you.


    How to configure a connection to a virtual private network (VPN) in Windows XP


    How to troubleshoot a Microsoft L2TP/IPSec virtual private network client connection


    Basic L2TP/IPSec Troubleshooting in Windows XP


    If still not work, please trace ras log and Oakley log,  then post to us to further investigate.

         "netsh ras set tracing * enabled" 

    Regards, Rick Tan
    Wednesday, March 9, 2011 9:44 AM
  • Hi Rick,

    Thank you very much for your post!

    The client runs XP Pro SP3 and I understood from that article that I didn’t have to apply the update, but only add the string HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec in the registry to allow the NAT-T that by default has value 0 and be able to change it to 2. That is what I did on the client. The initial traffic between the client and server is clearly NAT-T and runs over 4500.

    I haven’t even try with certificates, just with a shared key, which is indeed the same in both machines. The security logs in both machines show Main mode logon and Quick mode log on as successful.

    There is not firewall in the server and the Nic connected to the router has the basic firewall deactivated; there is just a NAT on the router, which is “VPN through (pptp and l2tp/ipsec)”, with the necessary ports opened for my traffic although, indeed, I didn’t open the port 5500 on it. Honestly, I didn’t find that information before.   I didn’t even try to capture that port… Thank you!

    Unfortunately, my server has become unresponsive and I won’t be back at home for the next two weeks! I rebooted it yesterday and never came back… I believe is because of this:

    I am afraid that I won’t be able to test with that port opened again until then. I am very sorry and frustrated also, for I relay on my server a lot for both, security and data. With the amount of time that I spend away from home, I will have to think in some sort of remote switch to “dirty reset” the server.

    I have those traces from rasman and  oakley which I took before though (from client and server), but I wouldn’t want you to spend time in a matter that could actually be that closed port’s fault (and of course mine as well!)

    Please, be patient and come back to this post in a couple of weeks…

    I am very sorry that I won’t be able to follow the post until then (unless some miraculous black out happens at home earlier!)

    All the best,


    Thursday, March 10, 2011 5:04 AM