none
Direct Access 2012 R2 - troubleshoot IPsec main mode negociation RRS feed

  • Question

  • Hello,

    We have a two-leg (DMZ+LAN) Direct Access infrastructure setup on Windows Server 2012 with Win 7 computers clients. Out of sudden, after the last server restart it stopped working. IP Sec Main mode negotiations are failing on both ends (client & server) with event id 4653. Can somebody help me troubleshoot this IPSec errors?

    Looking with tcpview, I can see on both sides that the connection to https port on DA server is established.

    On server side, I see only green checkmarks in Remote Access console. There are times when Network Security module is reporting that is under a DOS attack (probably caused by the high number of connections ~1000 that are failing IPSec)

    A wireshark trace is showing ipv6 traffic only in one direction, from fd00:0:0:1000::1 toward the remote client. I cannot see anything where the source is the ipv6 address of client.

    • On the client side, I also get      4563 event ids:

    The IPHTTPS interface is reporting as active, but it cannot reach the DA, DNS or any other infrastructure server.

    DirectAccess Client Troubleshooter Tool is reporting:

    [28/11/2018 10:37:30]: In worker thread, going to start the tests.
    [28/11/2018 10:37:30]: Running Network Interfaces tests.
    [28/11/2018 10:37:30]: Wireless Network Connection (Intel(R) Centrino(R) Advanced-N 6205): 10.3.77.53/255.255.252.0;
    [28/11/2018 10:37:30]: Default gateway found for Wireless Network Connection.
    [28/11/2018 10:37:30]: iphttpsinterface (iphttpsinterface): fd00::1000:4005:d6ac:8164:5a85;: fd00::1000:f45f:b394:7649:f2f9;: fe80::4005:d6ac:8164:5a85%18;
    [28/11/2018 10:37:30]: No default gateway found for iphttpsinterface.
    [28/11/2018 10:37:30]: Wireless Network Connection has configured the default gateway 10.3.79.254.
    [28/11/2018 10:37:42]: Warning - default gateway 10.3.79.254 for Wireless Network Connection does not reply on ICMP Echo requests, the request or response is maybe filtered?
    [28/11/2018 10:37:42]: Received a response from the public DNS server (8.8.8.8), RTT is 41 msec.
    [28/11/2018 10:37:42]: The public DNS Server (2001:4860:4860::8888) does not reply on ICMP Echo requests, the request or response is maybe filtered?
    [28/11/2018 10:37:42]: Running Inside/Outside location tests.
    [28/11/2018 10:37:42]: NLS is https://nls.<COMPANY>.local/.
    [28/11/2018 10:37:42]: NLS is not reachable via HTTPS, the client computer is not connected to the corporate network (external) or the NLS is offline.
    [28/11/2018 10:37:42]: NRPT contains 3 rules.
    [28/11/2018 10:37:42]:   Found (unique) DNS server: fd00::a03:ea
    [28/11/2018 10:37:42]:   Send an ICMP message to check if the server is reachable.
    [28/11/2018 10:37:54]: DNS Server fd00::a03:ea does not reply on ICMP Echo requests.
    [28/11/2018 10:37:54]: Running IP connectivity tests.
    [28/11/2018 10:37:54]: The 6to4 interface service state is default.
    [28/11/2018 10:37:54]: Teredo inferface status is offline.
    [28/11/2018 10:37:54]:  The configured Teredo server is the public Microsoft Teredo server teredo.ipv6.microsoft.com..
    [28/11/2018 10:37:54]: The IPHTTPS interface is operational.
    [28/11/2018 10:37:54]:  The IPHTTPS interface status is IPHTTPS interface active.
    [28/11/2018 10:37:54]: IPHTTPS is used as IPv6 transition technology.
    [28/11/2018 10:37:54]:  The configured IPHTTPS URL is https://da.<COMPANY>.com:443.
    [28/11/2018 10:37:54]: IPHTTPS has a single site configuration.
    [28/11/2018 10:37:54]: IPHTTPS URL endpoint is: https://da.<COMPANY>.com:443.
    [28/11/2018 10:37:55]:  Successfully connected to endpoint https://da.<COMPANY>.com:443.
    [28/11/2018 10:37:55]: No response received from <COMPANY>.local.
    [28/11/2018 10:37:55]: Running Windows Firewall tests.
    [28/11/2018 10:37:55]: The current profile of the Windows Firewall is Public.
    [28/11/2018 10:37:55]: The Windows Firewall is enabled in the current profile Public.
    [28/11/2018 10:37:55]: The outbound Windows Firewall rule Core Networking - Teredo (UDP-Out) is enabled.
    [28/11/2018 10:37:55]: The outbound Windows Firewall rule Core Networking - IPHTTPS (TCP-Out) is enabled.
    [28/11/2018 10:37:55]: Running certificate tests.
    [28/11/2018 10:37:55]: Found 1 machine certificates on this client computer.
    [28/11/2018 10:37:55]: Checking certificate [no subject] with the serial number [15CF7D9B0005000094D7].
    [28/11/2018 10:37:55]:  The certificate [15CF7D9B0005000094D7] contains the EKU Client Authentication.
    [28/11/2018 10:37:57]:  The trust chain for the certificate [15CF7D9B0005000094D7] was sucessfully verified.
    [28/11/2018 10:37:57]: Running IPsec infrastructure tunnel tests.
    [28/11/2018 10:37:57]: Failed to connect to domain sysvol share \\<COMPANY>.local\sysvol\<COMPANY>.local\Policies.
    [28/11/2018 10:37:57]: Running IPsec intranet tunnel tests.
    [28/11/2018 10:38:09]: Failed to connect to fd00:0:0:1000::1 with status TimedOut.
    [28/11/2018 10:38:21]: Failed to connect to fd00:0:0:1000::2 with status TimedOut.
    [28/11/2018 10:38:21]: Failed to connect to HTTP probe at http://directaccess-WebProbeHost.<COMPANY>.local.
    [28/11/2018 10:38:21]: Running selected post-checks script.
    [28/11/2018 10:38:21]: No post-checks script specified or the file does not exist.
    [28/11/2018 10:38:21]: Finished running post-checks script.
    [28/11/2018 10:38:21]: Finished running all tests.

    Below is the output from some common troubleshooting commands:

    <CMD>ipconfig /all

    Windows IP Configuration

       Host Name . . . . . . . . . . . . : <HOSTNAME>
       Primary Dns Suffix  . . . . . . . : <COMPANY>.local
       Node Type . . . . . . . . . . . . : Hybrid
       IP Routing Enabled. . . . . . . . : No
       WINS Proxy Enabled. . . . . . . . : No
       DNS Suffix Search List. . . . . . : <COMPANY>.local

    Ethernet adapter Bluetooth Network Connection:

       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Bluetooth Device (Personal Area Network)
       Physical Address. . . . . . . . . : C0-F8-DA-E3-1B-90
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes

    Wireless LAN adapter Wireless Network Connection:

       Connection-specific DNS Suffix  . : <COMPANY>.local
       Description . . . . . . . . . . . : Intel(R) Centrino(R) Advanced-N 6205
       Physical Address. . . . . . . . . : A0-88-B4-55-F8-F0
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 10.3.77.53(Preferred)
       Subnet Mask . . . . . . . . . . . : 255.255.252.0
       Lease Obtained. . . . . . . . . . : 28 November 2018 08:59:53
       Lease Expires . . . . . . . . . . : 02 December 2018 10:00:00
       Default Gateway . . . . . . . . . : 10.3.79.254
       DHCP Server . . . . . . . . . . . : 10.3.80.1
       DNS Servers . . . . . . . . . . . : 8.8.8.8
       NetBIOS over Tcpip. . . . . . . . : Enabled

    Tunnel adapter isatap.{4A3D349D-D1ED-4F0E-967F-D4612C286083}:

       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes

    Tunnel adapter isatap.<COMPANY>.local:

       Media State . . . . . . . . . . . : Media disconnected
       Connection-specific DNS Suffix  . : <COMPANY>.local
       Description . . . . . . . . . . . : Microsoft ISATAP Adapter #3
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes

    Tunnel adapter iphttpsinterface:

       Connection-specific DNS Suffix  . :
       Description . . . . . . . . . . . : iphttpsinterface
       Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
       DHCP Enabled. . . . . . . . . . . : No
       Autoconfiguration Enabled . . . . : Yes
       IPv6 Address. . . . . . . . . . . : fd00::1000:4005:d6ac:8164:5a85(Preferred)
       Temporary IPv6 Address. . . . . . : fd00::1000:f45f:b394:7649:f2f9(Preferred)
       Link-local IPv6 Address . . . . . : fe80::4005:d6ac:8164:5a85%18(Preferred)
       Default Gateway . . . . . . . . . :
       NetBIOS over Tcpip. . . . . . . . : Disabled

    <CMD>Netsh dnsclient show state

    Name Resolution Policy Table Options
    --------------------------------------------------------------------

    Query Failure Behavior                : Always fall back to LLMNR and NetBIOS
                                            if the name does not exist in DNS or
                                            if the DNS servers are unreachable
                                            when on a private network

    Query Resolution Behavior             : Resolve only IPv6 addresses for names

    Network Location Behavior             : Let Network ID determine when Direct
                                            Access settings are to be used

    Machine Location                      : Outside corporate network

    Direct Access Settings                : Configured and Enabled

    DNSSEC Settings                       : Not Configured


    <CMD>Netsh interface httpstunnel show interface

    Interface IPHTTPSInterface (Group Policy)  Parameters
    ------------------------------------------------------------
    Role                       : client
    URL                        : https://da.<COMPANY>.com:443/IPHTTPS
    Last Error Code            : 0x0
    Interface Status           : IPHTTPS interface active

    <CMD>Netsh namespace show effectivepolicy

    DNS Effective Name Resolution Policy Table Settings


    Settings for nls.<COMPANY>.local
    ----------------------------------------------------------------------
    Certification authority                 :
    DNSSEC (Validation)                     : disabled
    IPsec settings                          : disabled
    DirectAccess (DNS Servers)              :
    DirectAccess (Proxy Settings)           : Use default browser settings

    Settings for .<COMPANY>.local
    ----------------------------------------------------------------------
    Certification authority                 :
    DNSSEC (Validation)                     : disabled
    IPsec settings                          : disabled
    DirectAccess (DNS Servers)              : fd00::a03:ea
    DirectAccess (Proxy Settings)           : Bypass proxy

    <CMD>Netsh advfirewall monitor show mmsa

    No SAs match the specified criteria.

    <CMD>Netsh advfirewall show currentprofile

    Public Profile Settings:
    ----------------------------------------------------------------------
    State                                 ON
    Firewall Policy                       BlockInbound,AllowOutbound
    LocalFirewallRules                    N/A (GPO-store only)
    LocalConSecRules                      N/A (GPO-store only)
    InboundUserNotification               Enable
    RemoteManagement                      Disable
    UnicastResponseToMulticast            Enable

    Logging:
    LogAllowedConnections                 Enable
    LogDroppedConnections                 Enable
    FileName                              %systemroot%\system32\LogFiles\Firewall\pfirewall.log
    MaxFileSize                           24096

    Ok.

    <CMD>Certutil -store my
    my
    ================ Certificate 0 ================
    Serial Number: 61d68c3200050000946c
    Issuer: CN=<COMPANY-CAName>, DC=<COMPANY>, DC=Local
     NotBefore: 15/11/2018 13:35
     NotAfter: 02/07/2019 11:43
    Subject: EMPTY (DNS Name=<HOSTNAME>.<COMPANY>.local)
    Non-root Certificate
    Template: 1.3.6.1.4.1.311.21.8.6693252.4963786.7359385.10098729.16443910.70.7655005.1833759
    Cert Hash(sha1): fb 5d d5 b2 31 57 83 bb 9b 68 b8 91 b8 f2 b2 a4 8b a2 51 ac
      Key Container = f588ece0f8e5701064bc0b40d7c606f2_704c463e-1552-49a5-8244-f045c492456d
      Simple container name: le-SCCMClientCertificate-f86fdc58-1726-4779-9be3-aa3023c0fa21
      Provider = Microsoft RSA SChannel Cryptographic Provider
    Private key is NOT exportable
    Encryption test passed

    On the server side, I've performed a restore back in time to a point when I know for sure that DA was working. Along with this, I've also restored the DA GPOs. This has not helped, so it makes me think that the issue is not on the DA server itself.

    I don't believe that it can be on client side, as I consider that if it was this case, at least I could have seen at least one connected client. Or maybe the DOS protection of DA server is preventing all client connections.

    Does anyone have any idea what might be wrong?

    • Edited by Petru B Wednesday, November 28, 2018 7:40 PM
    Wednesday, November 28, 2018 7:35 PM

All replies

  • The reported IPSec 4563 event ids are:

    <SERVER SIDE>

    An IPsec main mode negotiation failed.

    Local Endpoint:
     Network Address: fd00:0:0:1000::1
     Keying Module Port: 500

    Remote Endpoint:
     Principal Name:  -
     Network Address: fd00::1000:486:e69e:a504:4ca
     Keying Module Port: 500

    Additional Information:
     Keying Module Name: AuthIP
     Authentication Method: Unknown authentication
     Role:   Responder
     Impersonation State: Not enabled
     Main Mode Filter ID: 4455770

    Failure Information:
     Failure Point:  Local computer
     Failure Reason:  Negotiation timed out

     State:   Sent first (SA) payload
     Initiator Cookie:  141e18021c27bf5d
     Responder Cookie: 5a40930637026985

    -----------------------------------------------------
     
     An IPsec main mode negotiation failed.

    Local Endpoint:
     Local Principal Name: -
     Network Address: fd00:0:0:1000::1
     Keying Module Port: 500

    Remote Endpoint:
     Principal Name:  -
     Network Address: fd00::1000:cb1:83a8:e7d6:1bd4
     Keying Module Port: 500

    Additional Information:
     Keying Module Name: AuthIP
     Authentication Method: Unknown authentication
     Role:   Responder
     Impersonation State: Not enabled
     Main Mode Filter ID: 0

    Failure Information:
     Failure Point:  Local computer
     Failure Reason:  Sent DoS cookie notify to initiator.

     State:   No state
     Initiator Cookie:  4391e84d89af5e82
     Responder Cookie: 0000000000000000

    Also, on server side a wireshark trace is showing ipv6 traffic only in one direction, from fd00:0:0:1000::1 toward the remote client. I cannot see anything where the source is the ipv6 address of client.

    <CLIENT SIDE>

    An IPsec main mode negotiation failed.

    Local Endpoint:
     Local Principal Name: -
     Network Address: fd00::1000:f45f:b394:7649:f2f9
     Keying Module Port: 500

    Remote Endpoint:
     Principal Name:  -
     Network Address: fd00:0:0:1000::1
     Keying Module Port: 500

    Additional Information:
     Keying Module Name: IKEv1
     Authentication Method: Unknown authentication
     Role:   Initiator
     Impersonation State: Not enabled
     Main Mode Filter ID: 0

    Failure Information:
     Failure Point:  Local computer
     Failure Reason:  No policy configured

     State:   No state
     Initiator Cookie:  7771d39d2441725a
     Responder Cookie: 0000000000000000

     -----------------------------------------
    An IPsec main mode negotiation failed.

    Local Endpoint:
     Local Principal Name: -
     Network Address: fd00::1000:f45f:b394:7649:f2f9
     Keying Module Port: 500

    Remote Endpoint:
     Principal Name:  -
     Network Address: fd00:0:0:1000::1
     Keying Module Port: 500

    Additional Information:
     Keying Module Name: AuthIP
     Authentication Method: Unknown authentication
     Role:   Initiator
     Impersonation State: Not enabled
     Main Mode Filter ID: 77992

    Failure Information:
     Failure Point:  Local computer
     Failure Reason:  Negotiation timed out

     State:   Sent first (SA) payload
     Initiator Cookie:  4a0f745933477f6a
     Responder Cookie: 0000000000000000

    Wednesday, November 28, 2018 7:42 PM
  • The first thing to check is the Windows Firewall on both the client as well as the DA server. IPsec tunnels will fail to establish if WFAS has been turned off. The ConSec rules that build the IPsec tunnels are owned by WFAS, and they are only engaged when both client and server have the Private or Public profiles turned on and active.

    So the symptoms you are setting could be caused by the DA server's Public profile being turned to OFF. Or, it could also be caused if both of your DA server's NICs have suddenly self-discovered that they are "Domain" profile. This would indicate an issue on networking, because you should never allow a DMZ NIC to be able to get to a Domain Controller, but if it could then it would cause the DMZ NIC to set itself up as Domain inside the firewall settings, and that would then also stop the IPsec tunnels from connecting.

    Wednesday, December 19, 2018 5:00 PM
  • And of course check over your machine certificates that are used as part of the IPsec tunnel authentication process. If that certificate has expired on either the client or on the DA server, it will cause problems. A machine certificate that expires on the DA server will immediately stop all DA client from being able to connect. :)
    Wednesday, December 19, 2018 5:01 PM