locked
1.8 -> 1.9 Upgrade - Gateway Crashes Microsoft.Tri.Gateway.exe and clr.dll RRS feed

  • Question

  • I upgraded my ATA environment today from 1.8 to 1.9.  After the ATA center was upgraded successfully, I pushed out the lightweight gateway and normal gateway updates.  Soon after the update, I started getting alerts from my monitoring system that the services on both the lightweight gateways and normal gateways were constantly stopping and starting.  I tried completely removing and reinstalling one of the lightweight gateways, but the same result remained.  Nothing jumped out in any of the logs in 'C:\Program Files\Microsoft Advanced Threat Analytics\Gateway\Logs'  I went through all the prerequisites and it looks like I meet them all.

    Eventviewer:

    Faulting application name: Microsoft.Tri.Gateway.exe, version: 1.9.7312.32791, time stamp: 0xa747e950
    Faulting module name: clr.dll, version: 4.7.2117.0, time stamp: 0x59cf526c
    Exception code: 0xc00000fd
    Fault offset: 0x0000000000177e27
    Faulting process id: 0x2464
    Faulting application start time: 0x01d3c51802e52184
    Faulting application path: C:\Program Files\Microsoft Advanced Threat Analytics\Gateway\Microsoft.Tri.Gateway.exe
    Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    Report Id: 4de04b77-310b-11e8-8e49-005056990dc9

    Any advice or tips other than restoring from backup?

    Monday, March 26, 2018 4:41 PM

All replies

  • The logs might help to be sure, but the error from the event logs says we most likely did not have enough memory to run.

    It usually happens in LWGW where we restrict our own memory to make sure the machine has 15% free RAM at all times.

    Note that after a fresh install/upgrade we need to recompile the parsers on first run which will take longer to start the first time, and will also consume more memory compared to normally running.

    Are you able to add some RAM to this machine and see if the problem resolves?

    Also, you mentioned standalone GWs also experience problems, are those getting the exact same error in the event log too ? (Exception code: 0xc00000fd)

    Monday, March 26, 2018 7:48 PM
  • Currently the server is utilizing only 4 GB of RAM with the total amount of 24 GB allocated to the VM.

    The exception code 0xc00000fd is the same on both the LWGW and regular GW

    Last few lines of Microsoft.Tri.Gateway.Log before it crashes:  (I can provide the whole log if you want, but it is around 627 lines long) 

    2018-03-26 21:46:48.2477 6848 11  Debug [NetworkListener] Loaded cached module 'MSRPCE_016bc621835345f616739d839aec44c1_4_0_7587_0.mdb'
    2018-03-26 21:46:48.2717 6848 11  Debug [NetworkListener] Loaded cached module 'SRVS_6c3e0de06ce6dd35c27e9dff4ba781a0_4_0_7587_0.mdb'
    2018-03-26 21:46:48.2777 6848 11  Debug [NetworkListener] Loaded cached assembly 'SPNG_dcdb65adda9e97beb2eee5701647085e_4_0_7587_0.dll'
    2018-03-26 21:46:48.2867 6848 11  Debug [NetworkListener] Loaded cached assembly 'DTYP_4199acf0eb5fb905f2a0de12c35eebb2_4_0_7587_0.dll'
    2018-03-26 21:46:48.2967 6848 11  Debug [NetworkListener] Loaded cached assembly 'NLMP_1b919357fe00e9544a56aa364da61a49_4_0_7587_0.dll'
    2018-03-26 21:46:48.3037 6848 11  Debug [NetworkListener] Loaded cached assembly 'CoreNetworkingResources_e973be36a39b8f19352aeaaf16b3fe25_4_0_7587_0.dll'
    2018-03-26 21:46:48.3097 6848 11  Debug [NetworkListener] Loaded cached assembly 'ReassembledTCP_c34f86a7dc4fce046b8a91c499d561f1_4_0_7587_0.dll'
    2018-03-26 21:46:48.3157 6848 11  Debug [NetworkListener] Loaded cached assembly 'TCP_e2e92ed303168eba132f4d2889901c0f_4_0_7587_0.dll'
    2018-03-26 21:46:48.3247 6848 11  Debug [NetworkListener] Loaded cached assembly 'UDP_4fa12d228c06fe93b66812f2d1c000fa_4_0_7587_0.dll'
    2018-03-26 21:46:48.3327 6848 11  Debug [NetworkListener] Loaded cached assembly 'X509_ef5ce6f94852dccc7a0591bb7e7a26a5_4_0_7587_0.dll'
    2018-03-26 21:46:48.3477 6848 11  Debug [NetworkListener] Loaded cached assembly 'MicrosoftCommonResources_ce18f767b75b9c811a6e7e0170e9f01e_4_0_7587_0.dll'
    2018-03-26 21:46:48.3537 6848 11  Debug [NetworkListener] Loaded cached assembly 'ERREF_0424ab69f90140a0b223d4bcd6f202ac_4_0_7587_0.dll'
    2018-03-26 21:46:48.3597 6848 11  Debug [NetworkListener] Loaded cached assembly 'KerberosV5_df67bd38df39aee184f2207c06aca59a_4_0_7587_0.dll'
    2018-03-26 21:46:48.3727 6848 11  Debug [NetworkListener] Loaded cached assembly 'GSSAPIKRB5_6754fa32d88e8595adbbb01922ecbd53_4_0_7587_0.dll'
    2018-03-26 21:46:48.3797 6848 11  Debug [NetworkListener] Loaded cached assembly 'GSSAPI_2911f93208236408d48f9d9badb8157c_4_0_7587_0.dll'
    2018-03-26 21:46:52.2937 6848 11  Debug [NetworkListener] Loaded cached assembly 'DNS_ea8aeeee348c5c63fc4ee4d2570c6ac9_4_0_7587_0.dll'
    2018-03-26 21:46:54.9297 6848 11  Debug [NetworkListener] Recompiling module assembly 'ABNF'
    2018-03-26 21:46:55.3347 6848 11  Debug [NetworkListener] Module assembly saved 'C:\Program Files\Microsoft Advanced Threat Analytics\Gateway\Pef\Cache\ABNF_892875283a4bdb8915030a4bfb11b857_4_0_7587_0.dll'
    2018-03-26 21:46:55.3357 6848 11  Debug [NetworkListener] Loaded cached assembly 'ABNF_892875283a4bdb8915030a4bfb11b857_4_0_7587_0.dll'
    2018-03-26 21:46:55.3437 6848 11  Debug [NetworkListener] Recompiling module assembly 'HTTP'

    Monday, March 26, 2018 9:58 PM
  • On the standalone GW (not the LW one) look under the Logs folder to see if a CrashDumps folder was created,

    and let me know if you can find any dmp files in it.

    If there are , let me know their rough sizes.

    Monday, March 26, 2018 10:13 PM
  • See Below:

    Name                                   Length
    ----                                   ------
    Microsoft.Tri.Gateway.exe.2876.dmp 1741010480
    Microsoft.Tri.Gateway.exe.2912.dmp 1420103760
    Microsoft.Tri.Gateway.exe.96.dmp   1369328693

    (Thank you for the quick replies)

    Tuesday, March 27, 2018 3:20 AM
  • Can you email me at atashare at microsoft com ?

    mentioned this post in the email,

    I will get back to you with instructions.

    Thanks,

    Eli

    Tuesday, March 27, 2018 5:51 AM
  • Hi Eli,

    I am seeing the exact same issue. Ill drop you an email

    Thanks

    Martyn

    Friday, April 13, 2018 3:03 PM