none
Memory leak?

    Question

  • Hi,
        As we want to process huge cap data,so I use a big loop (For example : process 10 files 10000 times) to simulate the real environment.  I just do very basic operation:  

    for( 10000 times)
    {
         (1)Open a cap file.
         (2) get raw frame count;
         (3) get every raw frame.
    }

     We are very sure we close every handle( rawFrame's and file's ) after we use it.

     But the memory usage  begin to increase until the program throw a OutOfMemory exception : (
     By the way,we've tried both in C# and C++,the result is same.
     Anyone knows why?  Thanks so much.
    Tuesday, November 03, 2009 8:05 AM

All replies

  • One way to verify you are closing all handles is that they should return the same handle ID.  But I doubt this is the problem as the limits for hanldes is 1000 and you should deplete this before running out of memory.

    If you are parsing the data, it is possilbe that your are depleting memory.  Every time you parse a frame the parsers involved might save state data which consumes memory.  Enabling Conversations will certaily make this higher.

    Are you parsing each frame? 
    Tuesday, November 03, 2009 7:18 PM
    Owner
  • Hi,paul,thanks a lot for your reply.
    Now I show you my test .There are 22 files in G:\TestData\SmallData directory,total size: 53M  (Some file is same:for example :there is a 2.cap and a Copy_of_2.cap),everyFile size between :1M ~ 3M .In this simple test,the memory keeps increase util it stops.
    By the way,there is no error occurs(Both the C# and NetMon API's return value--always zero) until OutOfMemoryException occurs. Is there something wrong with my test ? Thanks very much : )


    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Microsoft.NetworkMonitor;
    using System.IO;
    using System.Diagnostics;
    
    namespace Test.NetMon.MemoryLeak
    {
        public class Program
        {
            public static string GetApiVersion()
            {
                ushort Maj, Min, Build, Revision;
                NetmonAPI.NmGetApiVersion(out Maj, out Min, out Build, out Revision);
                return String.Format("Maj: {0}, Min: {1}, Build: {2}, Revision: {3}", Maj.ToString(), Min.ToString(),
                    Build.ToString(), Revision.ToString());
            }
    
            [STAThread]
            static void Main(string[] args)
            {
                Console.WriteLine(GetApiVersion());//The result is :Maj: 3, Min: 4, Build: 1972, Revision: 1
                //while (true)
                {
                    for (int i = 0; i < 10000; i++)
                    {
                        foreach (string strFile in Directory.GetFiles(@"G:\TestData\SmallData", "*.cap"))
                        {
                            ReadFile(strFile);
                        }
                        PrintProcessInformation(i);
                    }
                }
                Console.WriteLine("Finished!");
                Console.ReadLine();
            }
    
            public static void PrintProcessInformation(int times)
            {
                Process prc = Process.GetCurrentProcess();
                Console.WriteLine("Memory usage:{0}M  .Aready run {1} times.", ((int)(prc.PagedMemorySize64 / (1024 * 1024))),times++);
            }
    
            public static void ReadFile(string strCaptureFilePath)
            {
                IntPtr iptrCaptureFile = IntPtr.Zero;
                try
                {
                    ulong errno = NetmonAPI.NmOpenCaptureFile(strCaptureFilePath, out iptrCaptureFile);
                    WriteErrorMessageIfErrorOccurs(errno, "NmOpenCaptureFile");
                    uint uintFrameCount;
                    errno = NetmonAPI.NmGetFrameCount(iptrCaptureFile, out uintFrameCount);
                    WriteErrorMessageIfErrorOccurs(errno, "NmGetFrameCount");
                    IntPtr iptrFrame = IntPtr.Zero;
                    for (uint uintCurrentFrameIndex = 0; uintCurrentFrameIndex < uintFrameCount; uintCurrentFrameIndex++)
                    {
                        try
                        {
                            errno = NetmonAPI.NmGetFrame(iptrCaptureFile, uintCurrentFrameIndex, out iptrFrame);
                            WriteErrorMessageIfErrorOccurs(errno, "NmGetFrame");
                        }
                        finally
                        {
                            NetmonAPI.NmCloseHandle(iptrFrame);
                        }
                    }
                }
                finally
                {
                    NetmonAPI.NmCloseHandle(iptrCaptureFile);
                }
            }
    
            public static void WriteErrorMessageIfErrorOccurs(ulong errorNumber, string message)
            {
                if (errorNumber != 0)
                {
                    Console.WriteLine(string.Format("{0}  :{1}", errorNumber, message));
                }
            }
    
        }
    }
    
    
    Wednesday, November 04, 2009 1:44 AM
  • I'm able to repro a growth in memory using your test application.  I'm now checking with our dev group to see see what they say regarding this reporting of memory.  I know is some instance memory can be freed, but seeing you get an out of error message after a while, I 'd have to agree this might be a memory leak. 

    I will get back to you soon.

    Paul
    Thursday, November 05, 2009 2:35 PM
    Owner
  • Hi,paul,thanks for you reply.
    I've do another test.When I open a cap file,then keep a 10000 times to get its frame count and get frame,the memory won't increase.So I think maybe the CloseHandle has some problems.I give you another test code,Just Open and Close files,in my test ,the memory also increase.

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Microsoft.NetworkMonitor;
    using System.IO;
    using System.Diagnostics;
    
    namespace Test.NetMon.MemoryLeak
    {
        public class Program
        {
            public static string GetApiVersion()
            {
                ushort Maj, Min, Build, Revision;
                NetmonAPI.NmGetApiVersion(out Maj, out Min, out Build, out Revision);
                return String.Format("Maj: {0}, Min: {1}, Build: {2}, Revision: {3}", Maj.ToString(), Min.ToString(),
                    Build.ToString(), Revision.ToString());
            }
    
            [STAThread]
            static void Main(string[] args)
            {
                Console.WriteLine(GetApiVersion());//The result is :Maj: 3, Min: 4, Build: 1972, Revision: 1
                Console.WriteLine("Build at 20091105 1731");
                //while (true)
                {
                    for (int i = 0; i < 10000; i++)
                    {
                        foreach (string strFile in Directory.GetFiles(@"G:\TestData\SmallData", "*.cap"))
                        {
                            ReadFile(strFile);
                        }
                        PrintProcessInformation(i);
                    }
                }
                Console.WriteLine("Finished!");
                Console.ReadLine();
            }
    
            public static void PrintProcessInformation(int times)
            {
                Process prc = Process.GetCurrentProcess();
                Console.WriteLine("Memory usage:{0}M  .Aready run {1} times.", ((int)(prc.PagedMemorySize64 / (1024 * 1024))), times++);
            }
    
            public static void ReadFile(string strCaptureFilePath)
            {
                IntPtr iptrCaptureFile = IntPtr.Zero;
                try
                {
                    ulong errno = NetmonAPI.NmOpenCaptureFile(strCaptureFilePath, out iptrCaptureFile);
                    WriteErrorMessageIfErrorOccurs(errno, "NmOpenCaptureFile");
                }
                finally
                {
                    NetmonAPI.NmCloseHandle(iptrCaptureFile);
                }
            }
    
            public static void WriteErrorMessageIfErrorOccurs(ulong errorNumber, string message)
            {
                if (errorNumber != 0)
                {
                    Console.WriteLine(string.Format("{0}  :{1}", errorNumber, message));
                }
            }
    
        }
    }
    
    Friday, November 06, 2009 9:39 AM
  • Thanks for the extra info.  We've repro this now with unmanaged app as well and have filed a bug.  We'll have to take a look at it, but it probably won't be fixed till we release the next version, Nm3.4.

    Paul
    Friday, November 06, 2009 5:01 PM
    Owner
  • Hi,thanks for your reply.
    As our project use netmon to do some basic analysis works,Can I know when will netmon 3.4 release please?
    Thanks.

    Monday, November 09, 2009 7:28 AM