none
Connection Security rule doesn't work when using a computer certificate

    Question

  • Please bear with me as I am new to using AD CA as well as connection security rules.  I am trying to deploy Connection security rules as a way of verifying a remote computer that will connect to a resource in my environment.  The two resources exist in different untrusted active directory forests. 

    I have configured the Connection security rule and inbound / outbound rule for the encryption.  When using a pre shared key the rule works just fine. 

    When I switch it to a computer certificate it will not establish the IPSEC connection.  The destination (192.168.1.51) is also the AD CA.  I have requested a "server auth" certificate and approved the certificate, installed the certificate into the remote computer, and then exported and imported it into the computer's personal store.

    The only thing that I change between when it works and when it doesn't is changing the "first authentication methods" from preshared key to computer certificate.

    Additionally, I have none of the "advanced" settings configured. 

    I am literally at my wits end with this.  I have enabled Firewall verbose and connection verbose and am not getting any logs out of either.  I cleared the event logs (this is a lab) and don't see anything that pertains to IPSEC or connection security in either configuration (preshared key / computer certificate) 

    Thursday, December 12, 2013 2:17 PM

All replies

  • I have some event log information from the destination server. I cleared the logs on the servers (again this is just a lab) and found this audit failure created multiple times when I try to create the connection with the certificate configuration in place

    Log Name:      Security
    Source:        Microsoft-Windows-Security-Auditing
    Date:          12/12/2013 7:20:19 AM
    Event ID:      5061
    Task Category: System Integrity
    Level:         Information
    Keywords:      Audit Failure
    User:          N/A
    Computer:      Server2012AD.clearwaterms.local
    Description:
    Cryptographic operation.

    Subject:
     Security ID:  SYSTEM
     Account Name:  SERVER2012AD$
     Account Domain:  CLEARWATERMS
     Logon ID:  0x3E7

    Cryptographic Parameters:
     Provider Name: Microsoft Software Key Storage Provider
     Algorithm Name: RSA
     Key Name: clearwaterms-SERVER2012AD-CA
     Key Type: Machine key.

    Cryptographic Operation:
     Operation: Decrypt.
     Return Code: 0x80090010
    Event Xml:
    <Event xmlns=>
      <System>
        <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
        <EventID>5061</EventID>
        <Version>0</Version>
        <Level>0</Level>
        <Task>12290</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8010000000000000</Keywords>
        <TimeCreated SystemTime="2013-12-12T15:20:19.640481500Z" />
        <EventRecordID>2611725</EventRecordID>
        <Correlation />
        <Execution ProcessID="544" ThreadID="1736" />
        <Channel>Security</Channel>
        <Computer>Server2012AD.clearwaterms.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data Name="SubjectUserSid">S-1-5-18</Data>
        <Data Name="SubjectUserName">SERVER2012AD$</Data>
        <Data Name="SubjectDomainName">CLEARWATERMS</Data>
        <Data Name="SubjectLogonId">0x3e7</Data>
        <Data Name="ProviderName">Microsoft Software Key Storage Provider</Data>
        <Data Name="AlgorithmName">RSA</Data>
        <Data Name="KeyName">clearwaterms-SERVER2012AD-CA</Data>
        <Data Name="KeyType">%%2499</Data>
        <Data Name="Operation">%%2484</Data>
        <Data Name="ReturnCode">0x80090010</Data>
      </EventData>
    </Event>

    Thursday, December 12, 2013 3:26 PM
  • Hi,

    Thank you for your question.

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

    Thank you for your understanding and support.

    Regards,

    Yan Li

    TechNet Subscriber Support

    If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.


    Regards, Yan Li

    Friday, December 13, 2013 6:19 AM
  • Thank you for the response.  I appreciate that you are trying to find a resource to assist.  This feature set doesn't appear to be well documented / well used.  I have tried multiple internet search phrases and have found a few people that have done something similar, but nobody who is using Certificates across multiple untrusted forests.  I did find a set of architecture and design web links that appear to indicate what I am trying to do is supported, but nothing in those documents that would indicate specifically how to set it up. 

    In further research, I suspect that the certificate being given to the client is one that can not be opened / verified by the receiving computer, but I am not certain if that means I have the wrong certificate, or even how to specify exactly which certificate to use.  In the advanced section of the GUI, it has a piece for the thumbprint.  It requires a 40 character hex code with no spaces.  I have copied the thumb print from the certificate that I intended to use, removed the spaces and entered it into the GUI and it gives me an error that it isn't the correct format.

    I have specified the EKU of the certificate and verified I had a certificate with that EKU in the Local Computer/Personal store and I got the same errors. 

    Friday, December 13, 2013 3:21 PM
  • Hi, 

    As it's a cross-forest certificate usage, you need to figure out the following points:

    1. How to design the PKI hierarchy for cross-forest scenario 2. How to get the certificate issued cross-forest 3. What is the IPSec authentication method  (X.509 certificates are compatible with IKE. SSL certificates are compatible with AuthIP. If you configure a rule to use an SSL certificate, then the rule is not compatible with IKE and cannot be used to establish a connection with a computer that does not support AuthIP.)

    Regards, Brian


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Sunday, December 15, 2013 2:12 PM
  • Hi, 

    As it's a cross-forest certificate usage, you need to figure out the following points:

    1. How to design the PKI hierarchy for cross-forest scenario 2. How to get the certificate issued cross-forest 3. What is the IPSec authentication method  (X.509 certificates are compatible with IKE. SSL certificates are compatible with AuthIP. If you configure a rule to use an SSL certificate, then the rule is not compatible with IKE and cannot be used to establish a connection with a computer that does not support AuthIP.)

    Regards, Brian


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Brian,

    Thank you for the help. 

    1. Traffic between the forests is routed and I am only trying to encrypt a specific traffic type (TCP port) so HTTP is available between the two machines. As a result, I can connect to the CA's certsrv website. 

    IPSec uses X.509 but I don't know if the certificate that I can issue through a default AD CA configuration is issuing an x.509 certificate.  I am fairly certain this is the problem with my installation.  http://support.microsoft.com/kb/942957

    Here is what I am doing currently.  #1 - create the IPSEC connection security rule with a preshared key, confirm it works.  Change the authentication method to "computer certificate".  Log into the Certsrv website request an advanced certificate request, choose server authentication, for the name, I put in the FQDN of the machine.  I then log into the CA, approve the request.  Go back to the other machine, install the certificate, install the root certificate.  I then export the installed certificate from the Local User/Personal and import it into the Local Computer/Personal and do the same with the trusted root (placing it into the trusted root store)


    Monday, December 16, 2013 3:14 PM