locked
Why is my non-paged pool usage so high? RRS feed

  • Question

  • I have a couple of windows server 2012R2 file servers in cluster configuration. I have trouble figuring out why they have a very high non-page pool memory usage.

    Server configuration: DELL R620 with 16GB ram, Windows Server 2012R2 Enterprise, - fully patched.

    Symptoms:  The Server uptime right now is about 12 days, and the non-paged pool size is 5,3GB. When running poolmon, I get the following result:

    Memory:16711136K Avail: 8976988K  PageFlts: 35367   InRam Krnl:19108K P:306944K
    Commit:7727436K Limit:18349536K Peak:7933316K            Pool N:5520296K P:314028K
    System pool information
    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc
    
    Cont Nonp       5091 (   0)       441 (   0)     4650 243622448 (     0)  52391
    DdCa Nonp       1143 (   0)        14 (   0)     1129 113642640 (     0) 100657
    BCM0 Nonp        958 (   0)         0 (   0)      958 59695680 (     0)  62312
    BCM8 Nonp      80210 (   0)         0 (   0)    80210 46418688 (     0)    578
    Mdl  Nonp  246652914 ( 180) 246562964 ( 188)    89950 42161184 (-17904)    468
    SpBu Nonp  369873478 (2013) 369870040 (2013)     3438 30990000 (     0)   9013
    brcm Nonp       3209 (   0)        12 (   0)     3197 22757536 (     0)   7118
    DpmB Nonp      69890 (   0)     69700 (   0)      190 12451840 (     0)  65536
    Irp  Nonp  286795692 ( 635) 286782221 ( 634)    13471 11596256 (  -976)    860
    Sp   Nonp    2299482 (   9)   2294957 (   9)     4525 9361680 (     0)   2068
    File Nonp   91383757 ( 768)  91357626 ( 762)    26131 8697520 (  2016)    332
    LS00 Nonp    2679126 (   4)   2678771 (   3)      355 8181408 ( 86976)  23046
    EtwB Nonp       1890 (   0)      1606 (   0)      284 7053344 (     0)  24835
    ConT Nonp      58829 (   0)     58036 (   0)      793 6258688 (     0)   7892
    Ntfx Nonp     126465 (   0)    108561 (   0)    17904 6115632 (     0)    341
    FMsl Nonp     144234 (   0)    126198 (   0)    18036 5194368 (     0)    288
    Even Nonp   26907348 ( 546)  26872275 ( 543)    35073 4492832 (   384)    128

    The thing that wonders me, is that when I add the bytes for all the nonp tags (poolmon logfil > import in excel and sum the bytes column.) I only get the number to be around ~720MB. So where did the rest of the 5,3GB go? Shouldn’t poolmon show all of the non-paged memory?

    As the days go the Non-paged pool keeps growing, but poolmon shows about the same picture as now. Could that be a memory leak deep down in the system somewhere?


    Monday, February 23, 2015 8:56 AM

Answers

  • I finally found the culprit. The problem was a older driver from DELL - md3utm.sys. A driver for DELL MD3200. That driver was actually shown in the "findstr /m /l MPP *.sys" command, it was just hidden in the clutter.

    Updating to the new version solved the memory leak.

    So in conclusion: There is no reason to believe there should be a memory leak in memory pressure protection.


    Thomas Iversen

    Thursday, March 5, 2015 2:27 PM

All replies

  • Hi,

    Please first refer to this article. See if it is the tool poolman shows non-paged memory different as Task Manager. 

    Large initial Non-Paged Pool Memory is reserved after operating system boot-up

    http://support.microsoft.com/kb/2762246

    Note that the Non-paged pool area seen under RAMMAP is not the same as the non-paged pool usage observed from task manager.

    The Memory Manager keeps track of each page of memory in an array called the PFN database and, for performance, it maps the entire PFN database into virtual memory, this memory will always be resident on the RAM. The PFN database itself shows up as pool in the data that the memory manager returns to RAMMAP, so it’s not just purely non-poaged pool, but other non-pageable memory manager structures as well

    This becomes the main reason that we see a significantly high number for Non-Paged memory in Rammap.


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Tuesday, February 24, 2015 3:51 PM
  • Hi Shaon,

    Thank you for the article. I had not seen it before, so it was good information. However I don’t think it explains what I am seeing on my server.

    I just had another look at den non-paged memory usages today:

    Now the uptime of the server is 13 days and 21 hours.

    And the non-paged pool usage has risen to:

    Task Manager says: 6,3GB
    RAMMAP: 6.944.492K
    POOLMON: 6.619.844K
    POOLMON sum from pooltags: 715.059K

    Compared to the last time I looked (uptime 12 days) that’s around a gigabyte in growth.

    The mystery from my point of view, is not why there is a difference between what Task Manger, RAMMAP and POOLMON says. It’s more, why is there such a huge difference between the sum of the pooltags and the non-pages pool usage? And of cause, why does it keep growing?

    Wednesday, February 25, 2015 8:48 AM
  • Hi Thomas,

    Thank you for the reply.

    As you said we should at least saw all of these entries in Poolmon. 

    Whether the table you provided in first post lists all Non-paged entries? Sort by size to see the result like mentioned in this article:

    http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx

    Also please go to Task Manager - Detailed tab, add Paged Pool and NP pool as columns and check the result.


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Thursday, February 26, 2015 9:08 AM
  • Hi Shaon,

    The table in my first post was no a complet list, only the top of a sortede list of nonp tags.

    As I still cannot put images in my postes, I have uploaded some logfils and captures here:

    Capture of POOLMON, only nonp tags and sortede by bytes:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/poolmon.png

    Logfile POOLMON, ALL tags, sortede by bytes:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/poolmonALL.txt

    Logfile POOLMON, only nonp tags, sortede by bytes:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/poolmonNP.txt

    Caputure of Taskmanager:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/TaskManager1.png

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/TaskManager2a.png

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/TaskManager2b.png

    Capture of RAMMAP:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/RAMMAP.png

     

    One thing in the poolmonNP.txt made me wonder, on the very last line of the file:

      MPP Nonp   29480419    578083  28902336      -1        148   

    -1 what does that mean? And what does the TAG MPP cover?

    I therefore tried to search for MPP in the sys files of the driver folder, with a lot of luck, but not much wiser :-).

    C:\Windows\System32\drivers>findstr /m /l MPP *.sys
    afd.sys
    ClusDisk.sys
    CsvFlt.sys
    CsvFs.sys
    dxgkrnl.sys
    FWPKCLNT.SYS
    http.sys
    md3utm.sys
    mup.sys
    ndis.sys
    ndiswan.sys
    pacer.sys
    rassstp.sys
    sdstor.sys
    SerCx.sys
    smbdirect.sys
    SpbCx.sys
    srv2.sys
    storport.sys
    svhdxflt.sys
    tcpip.sys
    TsUsbFlt.sys
    tsusbhub.sys
    tunnel.sys
    UCX01000.SYS
    usbhub.sys
    USBHUB3.SYS

    Thursday, February 26, 2015 12:36 PM
  • Hi,

    Thank you for sharing all these files and pictures and sorry for the delay in reply.

    I can see the same result as you that most non-paged memory just disappeared in poolman.

    As it is a cluster, is there any change to do a reboot to see whether the issue still exists?

    Also the current situation leads me consider for memory leak. There is an article for a memory leak issue, which related to shared VHDX (and CSV, which seems meet your current situation):

    Memory leak when you use shared VHDX in Windows Server 2012 R2

    http://support.microsoft.com/kb/2920198/en-us

    When you use shared virtual hard disk (.vhdx) files on a Windows Server 2012 R2 computer, Windows OS may experience memory leak.

    Generally this update could be installed via Windows Update so compare the file version to see if it is applied.

    The update could be found here. It is an update package for multiple issues:

    http://support.microsoft.com/kb/2919355/en-us


    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.


    Tuesday, March 3, 2015 12:26 PM
  • Hi Shaon,

    Thanks for the links to the updates. It is true it is a cluster setup, consting of two servers (Node A and Node B) running in cluster. The cluster has two roles, a fileserver and SOFS. Normally running the fileserver role on one node, and the SOFS + CSV discs on the other.

    However I already have installed the updates them back in April 2014 as part of the cumulative update.

     

    In the meantime I have searched for what “MPP” could be, and my best guess is “Memory Pressure Protection”.

    http://support.microsoft.com/kb/974288/en-us

    http://windowsitpro.com/windows/understand-memory-pressure-protection

    Now I am saying it is a guess, because I have not seen anywhere that memory pressure protection is using a pooltag called “ MPP”.

    However I tried to disable MPP on NODE A using this command:

    netsh int tcp set security mpp=disabled

    Now take a look what happened to the poolmon logfiles:

    POOLMON – nonp tags sorted by bytes for NODE A:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-03032015/pool-13.40-03-03-2015-A.txt

    POOLMON – nonp tags sorted by bytes for NODE B:

    https://webtek.sdu.dk/user/thiv/temp/non-paged-03032015/pool-13.50-03-03-2015-B.txt

    On NODE A the “ MPP” pooltag is consuming: 642288 Bytes or 628KB

    On NODE B the “ MPP” pooltag is consuming: 981297408 Bytes or 936MB

    At this time the uptime of both servers is around 2 days and 1 hour, so it is too early to make any conclusions, but I defiantly think that I am on to something here. From what I have seen so far, this could very well be a memory leak in memory pressure protection.

    I will let you know if things change in a couple of days.


    Thomas Iversen

    Tuesday, March 3, 2015 1:51 PM
  • I finally found the culprit. The problem was a older driver from DELL - md3utm.sys. A driver for DELL MD3200. That driver was actually shown in the "findstr /m /l MPP *.sys" command, it was just hidden in the clutter.

    Updating to the new version solved the memory leak.

    So in conclusion: There is no reason to believe there should be a memory leak in memory pressure protection.


    Thomas Iversen

    Thursday, March 5, 2015 2:27 PM