locked
Memory Leak in ATA version 1.9.7312.32791 RRS feed

  • Question

  • The client I am working with is beginning a large Enterprise ATA deployment.

    Upon the initial roll out of the ATA Gateways configured with a mirror port to inspect packets from a domain controller on the same VCenter host, the gateway application "Microsoft.Tri.Gateway.exe" appears to have a memory leak and eventually crashes the host server.

    Both the DC and the Gateway host server are hosted on a Windows 2012 R2 VM

    The Gateway host VM has 64 GB of memory available and 12 2.7 Ghz CPUs

    Is there any way to tune ATA so that it does not use up all the memory?

    We have already tried configuring the priority in task manager to "Low" and even reducing the processor affinity.

    Wednesday, September 5, 2018 11:44 AM

All replies

  • When working on a standalone machine the GW is OK if using 90-95% of RAM. any outside tweaking like priority or affinity will only make it worse.

    What is the total traffic this machine is getting via mirroring?

    is it getting any events forwarded as well?

    Did you use the ATA sizing tool when specing this machine?

    Also, we did improve some memory issues in Update 1, you should install it:

    https://www.microsoft.com/en-us/download/details.aspx?id=56725

    Wednesday, September 5, 2018 11:53 AM
  • The machine(s) eventually crash due to the resource over utilization.

    Yes the hosts have 12 CPU cores @ 2.7 Ghz and 64 GB of RAM.

    ATA is attempting to commit over 160 GB of RAM.

    Yes some of the machines are utilizing Windows Event Collection/Forwarding.

    Prior to installation of ATA the machines are fine.

    Wednesday, September 5, 2018 1:48 PM
  • What was the sizing tool recommendation as to total busy packets for this machine ?

    Note that a single GW can handle up to 50,000 packets per second, and adding more memory or cores to go beyond that won't help. it is limited because of other factor at this point.

    Also, follow the docs regarding making sure those cores aren't loaded on the host which makes them shared, dynamic memory off (if it's virtual), power management set to "High performance"  etc. 

     https://docs.microsoft.com/en-us/advanced-threat-analytics/ata-capacity-planning#ata-gateway-sizing

    Wednesday, September 5, 2018 2:09 PM
  • approx. 19 K packet per second and dynamic memory is not turned on
    Wednesday, September 5, 2018 3:51 PM
  • These are Virtual machines.

    I did install update 1 and deployed across the platform. It is still causing the hosts to crash.

    Are you stating that the VM resources CPU/RAM must be hard set / dedicated ?

    Also are you stating that ATA will always use the 90-95% of RAM available?
    • Edited by Securitt Wednesday, September 5, 2018 6:11 PM update
    Wednesday, September 5, 2018 4:56 PM
  • Yes, the resources needs to be dedicated, although you should be good with a bit more than 6 cores and 24 GB of RAM.

    (unless the amount of events forwarded is abnormal)

    And yes, The gateway can use 90-95% of RAM at times of peaks as it tries to reduce latency.  

    Wednesday, September 5, 2018 6:52 PM
  • Hello,

    Please view the following table, and make sure you have assigned sufficient CPU and Memory for ATA Gateway. Especially, please note that the Hyper-threading must be disabled. You can refer here for more details.

    Best regards,

    Andy Liu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.


    Thursday, September 6, 2018 6:15 AM
  • The ATA gateways are on an ESX host, so CPU's and Memory are going to be shared with other hosts on the same ESX host VM.

    The DCs are also on the same ESX host as the ATA gateways in order for the port mirror to function.

    It looks like the references used apply only to physical hosts and not virtual hosts, especailly in reagrds to the High Performance power options.

    Can you explain if the hyperthreading and dynamic memory only applies to physical hosts OR does this apply to VM hosts as well?

    It appears as though ATA gateways running on an ESX host in a virtual environment are not truly supported and can affect performance as we are experiencing in this situation.

    Monday, September 10, 2018 4:50 PM
  • Running as an ESX host is supported and working.

    hyper threading will need to be disabled in the host.

    Not sure where dynamic memory settings are set in ESX, if it's per host or per guest.

    High Perf power options shoudl be true for VMs as well, and also for the host.

    Monday, September 10, 2018 8:37 PM
  • The server systems engineers are asking why it must be disabled, it is not clear whether hyper threading is the culprit that is crashing our ATA Gateway VM servers, or if this is just a performance preference.

    Tuesday, September 11, 2018 10:46 AM
  • All the suggested changes are documented as performance requirements.

    The specs documented are assuming those settings are in place when calculating needed hardware.

    so yes, while those settings are not set correctly, it can leas to poor performance, which in turn can crash the service due to overload.

    Tuesday, September 11, 2018 10:44 PM
  • The VM system admins state that the memory and CPUs are not shared and are not being utilized by other VMs on the systems.

    Are the specs and tested results for ESX server or another type of enterprise VM ?

    Monday, September 17, 2018 2:15 PM
  • Yes, we have many customers running both on ESX and HyperV without issues.

    (Giving that the settings I mentioned earlier were applied correctly).

    Monday, September 17, 2018 6:57 PM