locked
Memory leak in FTP client RRS feed

  • Question

  • I am using the built-in FTP client (Windows Explorer) a lot. In general it works OK, but i think it must have a memory leak. Here are a couple of observations (it is repeatable):

    Observed using Windows 7 Ultimate x64, 8 GB memory installed.

    Reboot.
    When running idle Task manager reports about 1.4 GB of memory used (around 45 processes). I then connect to an FTP server and starts copying files from the remote server to my local machine. After about 25 minutes I have copied 245 MB (7670 files). Memory consumption has gone up to 3.28 GB. The only other application running is Internet Exporer (consumes about 50 MB of memory).

    Reboot
    When running idle Task manager reports about 1.4 GB of memory used (around 45 processes). I then connect to an FTP server and starts copying files from the remote server to my local machine. After about 30 minutes I have copied 284 MB (8962 files). Memory consumption has gone up to 3.9 GB. The only other application running is Internet Exporer (consumes about 50 MB of memory).
    I quit both Explorer windows (disconnecting the FTP). Memory consumption stays the same (3.9 GB).

    I reconnect to the same FTP server and starts copying again. After another 35 minutes Task Manager reports around 7 GB of memory used. FTP fails with this message:

    "An error occurred while copying the file. No more Internet handles can be allocated"

    When looking at the Process list in Task Manager and in Resource Monitor, I am not able to see where the 5.5 GB of extra memory has been used (none of the displayed processes show any significant memory consumption).

    The only way to get memory consumption back to normal in this case is a reboot.

    I guess this must be a bug?
    Any ideas?

    Wednesday, May 5, 2010 4:09 PM

Answers

All replies

  • When you reproduce the behavior, what does Process Explorer's System Information window look like (can you post a screenshot)?
    Wednesday, May 5, 2010 4:30 PM
  • Hi,

     

    I would like to suggest you update the BIOS, chipset drivers and network card driver first.

     

    If the issue persists, I would like to suggest you test the issue in Safe Mode with Networking and Clean Boot.

     

    What are the results?

     

    Regards,


    Arthur Li - MSFT
    Thursday, May 6, 2010 6:31 AM
  • Hello! Here is a screen dump showing Process Explorer (and Task Manager) after having transferred 16375 files (606 MB). It took about 50 minutes. The memory usage went from 1.4 GB when starting to 6.2 when the error mentioned above occurred:

    http://www.jjohanson.com/FTPa.png

    Process Explorer's System Information window:

    http://www.jjohanson.com/FTPb.png 

    Thursday, May 6, 2010 8:03 AM
  • Hello! When I built the machine a couple of months ago I updated the BIOS before installing Windows (updating the BIOS after having installed and activated Windows got me into activation trouble on my previous machine). The chipset drivers where the very latest from Intel when I built the PC. The Realtek network driver was updated by Windows Update 3 weeks ago.

    When running in Safe Mode with Networking or after a Clean Boot I can observe the same behavior running an FTP transfer of many files: the memory usage keeps on growing at the same rate as described above.

    It is only when using the built in (Explorer) FTP client I can observe this.

    Thursday, May 6, 2010 8:58 AM
  • Your system has a paged pool leak (5.1 GB).  Check with poolmon from the WDK which tag is the largest consumer of the pool and research that tag to try to find out what it's used for and what driver is making the allocations.  Post back if you would like more explicit instructions.

    It may be AV or similar software that is responsible or impacting this.

    Thursday, May 6, 2010 10:50 AM
  • OK. I have run the transfer again. This time with poolmon.exe active. After about 35-40 minutes (480 MB, 11940 files, 5.04 GB memory used, this time I did not let the transfer run until it fails), the poolmon.exe output looks like this:

    http://www.jjohanson.com/FTPc.png 

    The pool with the tag "MmSt" seems to be the one allocating all the memory (around 10-20 MB between poolmon refreshes, a total of 3.65 GB so far).

    Looking in pooltag.txt from the WDK:

      MmSt - nt!mm        - Mm section object prototype ptes

    Not sure what this is, but I guess that there must be something wrong going on here?

    (BTW: I do not have any antivirus running, "AV")

    Thursday, May 6, 2010 2:05 PM
  • MmSt is the tag used by the cache manager for file caching and file mappings.

    This could be a bit tedious, but consider checking in Process Explorer each process for large numbers of or large ranges of mappings.  Choose View->Show unnamed handles and mappings and sort by the Size column descending.  Any large values stand out?

     

    Thursday, May 6, 2010 3:16 PM
  • OK, but I am afraid that I am not able to follow you here. I have been able to get Process Explorer to show handles in the lower pane (View -> Lower Pane View -> Handles (ctrl+H)). And I have enabled View->Show unnamed handles. However, I am not able to see or turn on any "Size" column in the lower pane?

    In View -> Select Columns... -> Handle, all I can see is Type (checked), Name (checked), Handle Value, Access Mask, File Share Flags and Object Address.

    Also, I can't find "mappings" anywhere.

    I guess I have missed something...

    Thursday, May 6, 2010 4:01 PM
  • Sorry - you want DLL view in the lower pane, not handle view. (Size will be there for DLL view, too, in the column selection screen.)
    Thursday, May 6, 2010 4:20 PM
  • Aha! I have enabled the "Mapped Size" and "Mapping Type" columns in the DLLs view in the lower pane. After having reached about 5 GB memory usage I have gone through all the processes shown by Process Explorer. Only the processes marked with yellow have anything at all in the lower pane:

    http://www.jjohanson.com/FTPe.png 

    Of these the largest numer I found in the Size column was 0x2D6A000 (data). The results shown for iexplore.exe in the screen dump was rather typical: <Pagefile Blocked>: 0x1272000 (data). Not very large numbers. I guess this can not explain where the large amount of allocated memory has gone.

    Thursday, May 6, 2010 6:05 PM
  • I just discovered the "Show Details for All Precesses" setting. Turning this on I can see DLL information for almost all processes (the only exceptions are the System Idle Process, Interrupts and DPCs of course). Going through all the processes I can see a maximum (data) size of 0x2D6A000 for Skill.dll (process VDeck.exe). All the other processes are lower. When looking at the number of mappings I can see that for most processes the number of DLL mappings fills about a screen size. For a couple of the processes there are about 2-5 screen sizes (but the sizes shown are rather small, in the 4KB to 32KB range typically).

    So I can not see anything that stands out, no.

    Thursday, May 6, 2010 7:25 PM
  • Again, sorry for not mentioning the "Show Processes for All Users" setting - I run Process Explorer elevated (procexp.exe /e) via scheduled task when I log in, so I don't check that option much.

    Those results don't seem to point the finger at anything.

    The System Information window lists that there are ~13000+ handles, which is quite a fine number of handles for the system as a whole.  Does any one process have a large number of handles, when the memory is consumed?  (Add the Handles column and sort it in descending order to check.)

    The behavior happens in Safe Mode with Networking and with a clean boot, which makes this interesting as that would seem to rule out a number of pieces of third-party software/drivers that may otherwise be running.

    Are you able to identify the third-party drivers loaded in a safe mode with networking repro scenario?  (Process Explorer, details for all processes, select SYSTEM process (PID 4), press CTRL+D, add Company Name to DLL view in lower pane, sort by Company Name, save report)

    Can you check the output of Sysinternals' handle.exe /s, when you have reproduced the behavior?  (I've seen unreconciled situations where the output of handle.exe didn't match the number of handles reported by other utils including Task Manager and Process Explorer, and there was a pool leak.)

     

    Are you capable of performing kernel debugging on the system (access to another system and a supported debugger connection e.g. null modem)?

    Thursday, May 6, 2010 8:56 PM
  • Hello again! I have checked the number of handles for each process (the number of handles even before starting the transfer is 13000 - 15000).

    Here is a screen dump showing the output of handle.exe, Task Manager and Process Explorer after 5GB of memory consumed:

    http://www.jjohanson.com/FTPf.png

    The third-party drivers loaded in safe mode are these:

    Process: System Pid: 4

    Name   Description     Company Name   Version  Size  Mapping
    ----   -----------     ------------   -------  ----  -------
    dump_dumpata.sys             0xC000  Image
    dump_atapi.sys              0x9000  Image
    dump_dumpfve.sys             0x13000  Image
    ASACPI.sys  ATK0110 ACPI Utility         1043.6.0.0 0x8000  Image
    ATMFD.DLL  Windows NT OpenType/Type 1 Font Driver  Adobe Systems Incorporated 5.1.2.226 0x61000  Image
    amdxata.sys  Storage Filter Driver    Advanced Micro Devices  1.1.2.4  0xB000  Image
    jraid.sys  JMicron JMB36X RAID Driver   JMicron Technology Corp. 1.17.53.0 0x20000  Image
    Rt64win7.sys  Realtek 8136/8168/8169 NDIS 6.20 64-bit Driver Realtek                        7.17.304.2010 0x57000  Image

    Full list here:

    http://www.jjohanson.com/System.txt

    Have not done any kernel debugging. Think I am currently missing a few items (software (no debugger) and hardware (no RS-323 null modem cable), and there is no RS-232 interface on the Win 7 box, internal connector only).

    Just to check: I did a similar transfer (using the same FTP server) on another machine (Vista Ultimate 32-bit). No memory problems there.

    Just to check: I did the same transfer (on the Win 7 x64 machine again) with a different FTP client (WebDrive version 9.12). No memory problems here.

    Friday, May 7, 2010 8:09 AM
  • With the investigation you've done, I'm left wondering if you've encountered a bug in Windows.  I don't have the ability to attempt to repro.

    The only other thought would be if for some reason the Realtek NIC driver were involved.  Any chance you're able to try a different NIC (and thus a different driver)?

    Saturday, May 8, 2010 4:26 PM
  • A few weeks back I had a gigabit PCI Express network adapter, but I gave it to a friend (used it to connect an iSCSI DroboPro to his PC). It would interresting to check with another NIC (btw, I saw that Realtek has released an ever newer driver, dated 2010/4/21, but using an adapter with a different chip would be best I guess). I found a couple of adapters using Intel chips, but they are a bit expensive.

    If only there was a way to identify the application/service/process/driver that allocated the memory (running some software locally)...

    Saturday, May 8, 2010 8:34 PM
  • Yes - that was the thought, rather than suggesting a new driver.  Still, if the problem may be fixed with a new driver, it would be good to test.  I'm sure you have more interesting things to do than update the driver and perform the exercise in safe mode yet again, though.

    Verifier's pool tracking may be of interest, but it relies on the driver being unloaded - " At the time that the driver is unloaded, Driver Verifier ensures that all allocations made by the driver have been freed." - and in this case I'm not sure that any responsible driver would be unloaded, unless perhaps on reboot.  Still, something else to consider trying, at your leisure.

    Of course, an alternative FTP client software would seem to be an obvious workaround, but that doesn't really seem to fit in with what you're concerned about at this point...

    Sunday, May 9, 2010 2:49 PM
  • I have tried the new Realtek driver, but the results are the same. I think I will try to file this as a bug (seems to be possible at connect.microsoft.com).

    Yes, I can always use WebDrive as an FTP client (works great, I needed it as a WebDAV client, the built in WebDAV client in Windows is a bit difficult to get to work).

    Anyway, thank you so much for helping!!

    Sunday, May 9, 2010 7:41 PM
  • Yes - connect seems to be a reasonable place to go for this.  Please indicate any further information (such as a report id and link or the like) that may come about as a result of the report.

    The only additional suggestions I have would be different networking hardware (and thus a different, non-Realtek, driver), pool tracking, and kernel debugging which, with additional instructions and some potentially tedious effort, may indicate when allocations involving the MmSt tag are made; the stack of such allocations may provide some insight into why the allocations are being made.  Perhaps, kernel memory pool inspection with a manual dump or LiveKD would yield additional details.

    @VF: Holy answer trolling, Batman!

    Sunday, May 9, 2010 8:11 PM
  • Hello again,

    I first reported the problem to Microsoft Support through normal channels (ID: 1131269952). They then called my on the phone and suggested that I should report this to https://connect.microsoft.com/IE. I just did (ID: 558570).

     

    Wednesday, May 12, 2010 8:56 AM
  • Interesting - thanks for the update.  Will be interesting to see how this tracks.
    Wednesday, May 12, 2010 11:22 AM