locked
ATA Gateway terminated unexpectedly 7031 | ATA GW Log: PostAsync failed | ATA Center Log: Client certificate doesn't exist RRS feed

  • Question

  • Please Note: System worked perfectly with ATA prior to ATA 1.8 and ATA 1.8 Update 1.

    Upgraded to 1.8 & Update 1. GW Service would not upgrade and constantly restarted. Event log errors 7031.

    Uninstalled, Cleaned System (Certs, Files, etc.), Reinstalled. Same issues.  Uninstalled/Reinstalled both GW & Center. Same Issues.  Verified json files are correct and match Certs installed.

    Certs are Enterprise Root CA issued with proper CSP and 2048 bits. (Remember, system worked perfectly prior to 1.8 or 1.8 Update 1).

    ATA Version 1.8.6645.28499 (1.8 Update 1)

    ATA Center: Windows Server 2012 R2
    ATA GW: Windows Server 2012 R2 (AD Domain Controller)

    ATA GW Event logs (7031 repeated):

    The Microsoft Advanced Threat Analytics Gateway service terminated unexpectedly.  It has done this [x] time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

    ATA Center Event Logs: No error

    ATA GW file log (Microsoft.Tri.Gateway.Updater-Errors.log):

    2017-08-17 19:49:54.3592 5620 17  14689fae-b5a6-4658-81d9-1468df0bd0b6 Error [GatewayConfigurationManager] Failed to get configuration, using default configuration
    2017-08-17 19:49:55.5624 5620 16  38e075a2-44a1-4458-8892-20785b231106 Error [GatewayConfigurationManager] Failed to get configuration, using default configuration

    (This line Repeats)

    2017-08-17 19:49:55.6092 5620 15  e0d63b07-714a-4a77-b954-e698ce5949d2 Error [WebClient+<InvokeAsync>d__8`1] System.Net.Http.HttpRequestException: PostAsync failed [requestTypeName=UpdateGatewayServiceStatusRequest] ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
       at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
       at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
       --- End of inner exception stack trace ---
       at async Microsoft.Tri.Common.Communication.WebClient.PostAsync[](?)
       at async Microsoft.Tri.Common.Communication.WebClient.InvokeAsync[](?)
       --- End of inner exception stack trace ---
       at async Microsoft.Tri.Common.Communication.WebClient.InvokeAsync[](?)
       at async Microsoft.Tri.Common.Communication.WebClient.InvokeAsync[](?)
       at async Microsoft.Tri.Gateway.Updater.Communication.GatewayServiceStatusUpdaterProxy.UpdateGatewayServiceStatusAsync(?)
       at async Microsoft.Tri.Gateway.Updater.Updates.GatewayServiceController.SendGatewayServiceStatusUpdateAsync(?)
       at async Microsoft.Tri.Infrastructure.Framework.Module.<>c__DisplayClass30_0.<RegisterPeriodicTask>b__1(?)
       at async Microsoft.Tri.Infrastructure.Extensions.TaskExtension.<>c__DisplayClass33_0.<RunPeriodic>b__0(?)

    ATA GW file log (Microsoft.Tri.Gateway-Errors.log):

    (This line Repeats)

    2017-08-17 19:49:57.0937 5504 5   00000000-0000-0000-0000-000000000000 Error [WebClient+<InvokeAsync>d__8`1] System.Net.Http.HttpRequestException: PostAsync failed [requestTypeName=UpdateGatewaySystemProfileRequest] ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
       at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
       at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
       --- End of inner exception stack trace ---
       at async Microsoft.Tri.Common.Communication.WebClient.PostAsync[](?)
       at async Microsoft.Tri.Common.Communication.WebClient.InvokeAsync[](?)
       --- End of inner exception stack trace ---
       at async Microsoft.Tri.Common.Communication.WebClient.InvokeAsync[](?)
       at async Microsoft.Tri.Common.Communication.WebClient.InvokeAsync[](?)
       at async Microsoft.Tri.Gateway.Common.Service.GatewayConfigurationManager`1.GetConfigurationAsync[](?)
       at async Microsoft.Tri.Infrastructure.Framework.ConfigurationManager`2.UpdateConfigurationAsync[](?)
       at async Microsoft.Tri.Gateway.Common.Service.GatewayConfigurationManager`1.UpdateConfigurationAsync[](?)
       at async Microsoft.Tri.Infrastructure.Framework.ConfigurationManager`2.OnInitializeAsync[](?)
       at async Microsoft.Tri.Gateway.Common.Service.GatewayConfigurationManager`1.OnInitializeAsync[](?)
       at async Microsoft.Tri.Infrastructure.Framework.Module.InitializeAsync(?)
       at async Microsoft.Tri.Infrastructure.Framework.ModuleManager.OnInitializeAsync(?)
       at async Microsoft.Tri.Infrastructure.Framework.Module.InitializeAsync(?)
       at async Microsoft.Tri.Infrastructure.Framework.Service.OnStartAsync(?)
       at Microsoft.Tri.Infrastructure.Framework.Service.OnStart(String[] args)

    Thursday, August 17, 2017 8:41 PM

All replies

  • Can you look in the logs in 
    %windir%\system32\LogFiles\HTTPERR\
    on the center machine, And see if there are any errors there? 
    If there are, please share.


    Thursday, August 17, 2017 9:55 PM
  • Here is the log you requested:

    #Software: Microsoft HTTP API 2.0
    #Version: 1.0
    #Date: 2017-08-17 19:48:00
    #Fields: date time c-ip c-port s-ip s-port cs-version cs-method cs-uri sc-status s-siteid s-reason s-queuename
    2017-08-17 19:48:00 xxx.xxx.xxx.102 64304 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/compatibility/v1.0 400 - BadRequest -
    2017-08-17 19:49:54 xxx.xxx.xxx.102 64344 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:49:55 xxx.xxx.xxx.102 64346 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:49:55 xxx.xxx.xxx.102 64345 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:49:55 xxx.xxx.xxx.102 64347 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/compatibility/v1.0 400 - BadRequest -
    2017-08-17 19:49:57 xxx.xxx.xxx.102 64348 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:50:04 xxx.xxx.xxx.102 64350 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:50:10 xxx.xxx.xxx.102 64351 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:50:10 xxx.xxx.xxx.102 64352 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -
    2017-08-17 19:50:10 xxx.xxx.xxx.102 64354 xxx.xxx.xxx.106 443 HTTP/1.1 POST /api/v1.0 400 - BadRequest -

    Friday, August 18, 2017 12:53 PM
  • Interesting.

    Question: are there any other Gateways that are installed already or is it the first one?

    If it's the first, what happens if you try to install another one on another machine? same issue?

    Friday, August 18, 2017 4:37 PM
  • I actually have 3 Lightweight GW's. No Standalone GW's.  All uninstalled.  Was trying to bring the first one up.  Yes, I did install another one with the exact same results. I actually installed all of them with the same results.  None of the 1.8+ GW's have worked since the upgrade.  All GW's are uninstalled and I'm working on just getting one working with the ATA Center to figure out the issue.

    I should add that I have a completely fresh mongo db with no left over data from the 1.7 installs so this is now treated as a fresh 1.8 install (with Update 1 applied through Windows Updates).

    Friday, August 18, 2017 4:41 PM
  • Thanks for the update.

    Which Center certificate are you using? is it the default self signed the ATA deployment produces or is it your own?

    Friday, August 18, 2017 4:45 PM
  • It is a Enterprise Root CA Cert issued.  It's the same cert that was used during the 1.7 install.  It's a temple based off the Web Server template cert with CSP and RSA Public key of 2048 bits.  But the GW's are installing their own Self Signed cert and using that.

    Friday, August 18, 2017 4:50 PM
  • Yes, the GWs in 1.8 are using their own self signed certs.

    But the data between them and the center is  encrypted using the center's cert.

    I am asking because I would love to get network captures from the GW and center at the same time,

    But the SSL traffic will need to be decrypted before I can see it, and this is possible if you can export the private key of the center's cert, decrypt the recorded traffic, and then share with me the decrypted data.

    Friday, August 18, 2017 4:57 PM
  • Working on the capture now.  Weekend coming. Will update on Monday
    Friday, August 18, 2017 5:30 PM
  • I'm having a problem decrypting the messages from the GW.  

    I'm using Message Analyzer (all current updates) and the only error I get when reparsing for decryption is 'Decryption Failed The cipher suite is unsupported'

    I'm using the private key from the ATA Center server for decryption in Message Analayzer.

    These servers have the default Windows Server 2012 R2 (With all current WinUpdates applied) SSL protocols and ciphers. Nothing has been disabled or altered from default.

    forgot to add. Can't run Message Analyzer on the ATA center server because ATA is already using an ETW capture session and I get the following error message:

    Unable to start ETW session: MMA-ETW-Livecapture-a2c373e9-155b-4624-adf7-99c31be6e239

    Host Name: Localhost

    A capture session exists. It must be stopped and deleted before starting a new one, e.g., using Powershell cmdlet Get-NetEventSession, Stop-NetEventSession, Remove-NetEventSession.

    -----

    Output from Get-NetEventSession:

    Name               : Microsoft-ATA-NDISCAP-LiveCapture
    CaptureMode        : RealtimeLocal
    LocalFilePath      :
    MaxFileSize        : 250 MB
    TraceBufferSize    : 4096 KB
    MaxNumberOfBuffers : 64
    SessionStatus      : NotRunning

    ----

    Is it possible that this EWT Capture Session is a hold over from ATA 1.7?

    Before I did use the ATA Center server as a live capture and also had a GW installed.  Now, after uninstalling everything and installing 1.8 Update 1, I'm only going to run ATA Center and install GW's directly on the DC's

    • Edited by WarrenJr Tuesday, August 22, 2017 1:52 PM
    Tuesday, August 22, 2017 1:46 PM
  • Interesting.  Turns out that the server (ATA Center) is trying to use a cipher_suite of TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 according to the Server Hello message.

    According to this blog article: https://blogs.technet.microsoft.com/messageanalyzer/2014/10/21/post-decryption-of-tlsssl-traffic/

    That cipher is not support for decryption in Message Analyzer.  I'll change the ATA center cipher suites to limit to supported decryption ciphers and try again.

    Tuesday, August 22, 2017 3:33 PM
  • Got it decrypted now. Here is where it is failing:

    After a fresh install of the GW was able to capture the GW checking in with the ATA Center and passing it's configuration information:
    ----
    Operation, Status: OK (200), POST /api/management/systemProfiles/gateways, Version: HTTP/1.1
    Has Payload - XML Config Information

    Then, it will attempt to call a HTTP Post that generates the Error in the GW logfile:
    ----
    Request, POST /api/v1.0, Version: HTTP/1.1
    Header - Expect - [100-continue]
    No Payload
    Has Diagnostic Warning - HTTP: Incompelte HTTP Payload for full reassembly, due to mission continuation message
    ----
    Response, Status: Bad Request (400), Version: HTTP/1.1

    I was able to capture at the ATA Center and the ATA GW and decrypt both captures which confirmed the exact messages on both sides.

    Tuesday, August 22, 2017 4:26 PM
  • It is not supported to install Message Analyzer on the same machine where a GW or LWGW is installed.

    It will cause ATA to fail, as it shares the same parsing engine (with ATA modifications).

    Please uninstall Message Analyzer completely from the machine, it doesn't even have to run to break ATA,

    it's enough that it is just installed to break anything.

    After that, please restart all ATA services.

    For capturing, I would recommend using netmon 3.4, as it can capture while GW is running without colliding.

    (you can later open the cap file with any program you like, even WS or MA, on a different machine.


    Tuesday, August 22, 2017 6:31 PM
  • Good, this is progress.

    Can you share the captures with me?

    please email me at atashare at microsoft com, and I will send you back details for a secured workspace where you can upload the data,

    Or if you prefer , you can open a case with Premier Support Services, and have an ATA support engineer help (you can of course ask them to onboard me as well).

    The main problem is that the GW is sending a certificate for authentication as part of the payload, for some reason, this payload arrives to the center without the certificate, which break ongoing authentication.

    The errors you mentioned might support this theory, if indeed they are related to the specific authentication request.

    I am hoping the full capture from both sides will help us determine why it is happening.


    Tuesday, August 22, 2017 6:39 PM
  • Same problem here - any news reguarding this issue ?

    Tryed to install with cert from internal PKI on ATAcenter and with self-sigend cert on ATAcenter ... both setup variants are failing with the same error messages ("The request was aborted: Could not create SSL/TLS secure channel.") in Microsoft.Tri.Gateway-Errors.log

    Monday, September 25, 2017 11:26 AM
  • Can you confirm if you are getting in the logs a Warn level message that contains "Client certificate doesn't exist" ?
    Monday, September 25, 2017 8:38 PM
  • Messages like this

    2017-09-26 07:04:05.4325 2748 9   e48f5af5-e9e7-47bb-901c-21a123837cb4 Warn  [OwinContextExtension] Sending retry request [Client certificate doesn't exist]

    can be found multiple times in Microsoft.Tri.Center.log on ATAcenter 

    Tuesday, September 26, 2017 11:04 AM
  • Then yes, most likely you are hitting the same issue.

    It happened to a very limited of deployments so far, and is still under research.

    I will update here as soon as we have root cause and action plan.

    Tuesday, September 26, 2017 10:17 PM
  • Problem was solved with your posting in this thread:

    https://social.technet.microsoft.com/Forums/en-US/a7937dc5-caec-4a46-8b5f-b5f041ac9175/postasync-failed-could-not-create-ssltls-secure-channel

    by setting 

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL]
    "DisableRenegoOnServer"=dword:00000001

    to 0

    Wednesday, September 27, 2017 12:48 PM
  • I recently had this issue and I fixed uninstalling Message Analyzer first, but after reboot I still with the issue, then I uninstall Network Monitoring and ATA Gateway, and after install again the ATA Gateway and now works fine

    kldo

    Friday, July 5, 2019 5:42 PM