none
Server 2019 spaceport.sys non-paged-pool memoryleak? RRS feed

  • Question

  • I use storage pools since server 2012 R2 and think its a good solution.
    But there are some limitations on server 2012 R2 (re-balance) where no longer exists on server 2016 and 2019.

    Now I have build a new server based on server 2019 hyper-v.

    The migration was quite easy:

    • Take the storage pool offline on server 2012 R2
    • Install server 2019
    • Attach the storage pool and take it online
    • migrate it to version server 2019

    works!

    But now I have start to copy some files to the storage pool and notice that the memory consume in the non-paged-pools grows. After copy of 1TB the pool is 1GB larger. After on or two days the server crashes with out of non-paged-pool memory.
    The analyse with poolmon identify the tag "Sp**".

    I have found in different forums that there was a problem on server 2016 with the storport.sys using ReFS.

    What I have try:

    • delete virtual drive (ReFS), recreate virtual drive NTFS
    • delete alls virtual drives, delete storage pool, recreate virtual drive

    But I always end after copy of 5TB files in a non-paged-pool whit a size of more than 10GB.

    Not sure, but I think there must be a memory leak in the storport.sys on server 2019.

    On the hardware side I use an SuperMicro board with an intel C236 chipset.
    Attached 4 SATA harddisk with same size attached to the pool.

    Have someone else this issue or can confirm Microsoft there is an issue with server 2019


    • Edited by heuby.ch Tuesday, April 23, 2019 9:10 AM
    Saturday, April 6, 2019 8:31 AM

Answers

  • After the reorg of the storage-pool and write about 4TB data to a new disk, I can no longer see a growing of the non-paged pool.

    Issue fixed! Thank you Microsoft!

    • Marked as answer by heuby.ch Thursday, June 20, 2019 1:21 PM
    Thursday, June 20, 2019 1:21 PM

All replies

  • Hi,

    Thanks for your question.

    I’d confirm that you have updated the server 2019 with the latest rollup.

    https://support.microsoft.com/en-us/help/4464619

    Please check your hardware meet server 2019 requirements.

    https://docs.microsoft.com/en-us/windows-server/get-started-19/sys-reqs-19

    Please also check your hardware would be certified for windows server 2019. We can refer to the following link,

    https://www.windowsservercatalog.com/results.aspx?bCatID=1283&cpID=0&avc=126&ava=0&avq=0&OR=1&PGS=25

    Furthermore, here’re articles talked about troubleshooting for storport.sys problems, we can refer to these to try to resolve your current situation.

    https://www.solvusoft.com/en/files/bsod-blue-screen-error/sys/windows/microsoft/microsoft-office-system-beta-2-kit-2003/storport-sys/

    https://ugetfix.com/ask/how-to-fix-storport-sys-gsod-on-windows-10/

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    Hope above information can help you. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, April 8, 2019 3:07 AM
    Moderator
  • Hi,

    Thank you for your answer.

    I have checked that the server is up to date and also meets the hardware requirements.

    I use a SuperMicro board (X11 SAE). But it doesn't seems to be certified for Server 2019. It's not special certified for any operation system, but works perfect with Server 2012 R2.

    I have also checked your links, but not found a solution. I haven't a BSOD, I "only" notice a memory leak.

    Can it be, that a Intel Chipset C236 doesn't works with Server 2019 and storage space? If yes it would be nice to find a hint about it. 

    Best regards

    Thomas

    Monday, April 8, 2019 7:11 PM
  • Hi,

    According to some research, Intel Chipset C236 is applicable with windows 10, and windows 2019 uses the same OS core version, so I think C236 also is applicable with windows 2019 and storage space. 

    https://support.lenovo.com/sg/zh/downloads/ds112447

    We'd update the driver for C236 to see if it helps.

    Furthermore, here's an article discussed how to find and fix Memory leaks in windows for your reference, hope it can help.

    https://www.thewindowsclub.com/memory-leaks-in-windows

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Tuesday, April 9, 2019 2:34 AM
    Moderator
  • Hi,

    Just checking in to see if the information provided was helpful. Please let us know if you would like further assistance.

    Best Regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Wednesday, April 10, 2019 8:18 AM
    Moderator
  • Hi,

    I have checked the drivers. All ok, the newest drivers are installed.

    After this I have make additional tests:

    • Hyper-V Server 2019 with Microsoft standard AHCI drivers
    • Hyper-V Server 2019 installed on a different hardware
    • Hyper-V Server 2019 installed on VMWare-Workstation
    • Server 2019 Standard (GUI) installed on VMWare-Workstation

    In all cases I have build an new StoragePool based on 3 disk and create a virtual harddisk of type mirror.

    Then I have copied some files (114 GB) to the virtual harddisk on the StoragePool and repeat this 3 times.

    I can notice that after each copy process the non-paged-pool grows by 20MB.

    Conclusion:

    on my point of view, I think there is a memory leak inside the storage space feature
    Can you give my findings to another person to reproduce this?

    Thursday, April 11, 2019 8:20 AM
  • Hi,

    Thanks for your detailed update.

    Did you use the Refs in your storage space? 

    If yes, you can set the tunable parameters as the following article:

    Please check this article:

    FIX: Heavy memory usage in ReFS on Windows Server 2016 and Windows 10

    https://support.microsoft.com/en-us/help/4016173/fix-heavy-memory-usage-in-refs-on-windows-server-2016-and-windows-10

    Note: You need to confirm that you patch the latest update for your OS.

    Meanwhile, please refer to the following similar thread who someone has similar problem as you:

    https://www.reddit.com/r/Windows10/comments/65hxf7/windows_10_creators_update_memory_leak/

    Highly appreciate your effort and time. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, April 12, 2019 8:12 AM
    Moderator
  • Hi,

    I don't use ReFs (no more...).

    I know about the the two links and cannot find a solution in it.

    Yes, I can confirm that I have installed the latest patches (until 2019-04)

    Best regards
    Thomas

    Friday, April 12, 2019 9:30 AM
  • Hi,

    Thanks for your update.

    For the current issue, it should be a memory leak. For memory leak issue, in general, we need two ways:

    ----------------------------

    1. We need collect performance log by internal tool to find the tag, which use the most of virtual memory. Then we collect poolmon to find the drive, which corresponds tag.
    2. We need collect dump file and analyze dump file.

    However, we cannot troubleshoot the issue because of limitation of forum. I would like to suggest you open case to Microsoft Support.

    Highly appreciate your effort and time. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, April 16, 2019 10:33 AM
    Moderator
  • Hi,

    How are things going on? 

    Please feel free to let me know if you need further assistance.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Thursday, April 18, 2019 8:11 AM
    Moderator
  • Hi Michael,

    I wasn't in the office this week.

    It is how it is. I hope someone from Microsoft read this an made some tests. I think at the moment, no many companies use windows server 2019 in a productive environment.

    I have move our Hyper-V to an independent disk located on a raid controller, an use the storage pool only for files "write once read many". Not the solution I want, but at the moment the only that work.

    I will retest it in half a year again. Hope Microsoft has fix this .....

    Best regards

    Thomas

    Sunday, April 21, 2019 7:20 PM
  • I can confirm this issue on WS2019, rollup 2019-03: A NonPagedPool memory leak occurs/grows when copying data into a single parity storage space. It can grow up to 32GB and cause a server reboot. (VMs on dynamic memory run into RAM starvation earlier.) The tag in poolmon is Sp**, as the OP reported, but it belongs to the spaceport.sys driver. (The current title states storport.sys, which is misleading; maybe, it can be updated?) Diagnostic details are in the screenshot below. For reference and reproducibility, this is the Powershell snippet that created the virtual disk getting the data:

    #the pool has 2 SSDs for journaling and the WBC. The parity space was created on 6 8TB HDDs usin these commands, essentially:
        $parityVDisk = Get-StoragePool $sPool | New-VirtualDisk -FriendlyName $sVDiskFriendlyName `
            -MediaType HDD `
            -FaultDomainAwareness PhysicalDisk `
            -ResiliencySettingName Parity -PhysicalDiskRedundancy 1 `
            -NumberOfColumns (3+1) `
            -ProvisioningType Thin `
            -Size 25TB `
            -WriteCacheSize 80GB `
            -Interleave 4MB `
        ;
        $sVolumeLabel = "sParity.4c6d.ReFS";
        $parityReFSVol = New-Volume -FriendlyName $sVolumeLabel `
            -DiskUniqueId $parityVDisk.UniqueId `
            -FileSystem ReFS `
            -AccessPath $driveLetter `
        ;
    spaceport.sys memory leak in WS2019-03

    I hope, MS finds this info helpful and that this thread is forwarded to the right storage spaces driver/QC people...
    Best, Michael.


    • Edited by 'Michael G.' Sunday, April 21, 2019 9:57 PM inserted values for powershell size vars
    Sunday, April 21, 2019 9:53 PM
  • Title corrected, sorry about mistake. But great that you can confirm my findings.

    Best regards.

    Thomas

    Tuesday, April 23, 2019 9:15 AM
  • By the way it does not matter which type of storage layout you use and what for a file system (NTFS, ReFS) you use.

    The memory leak occurs in any case.....
    Wednesday, April 24, 2019 2:51 PM
  • Hi,

    Thanks for your update.

    If it still couldn't find any clue, I'm afraid you might need to contact  Microsoft Customer Support Services (CSS) so that a dedicated Support Professional can help you on this issue.

    To obtain the phone numbers for specific technology request, please refer to the website listed below:

    https://www.microsoft.com/en-us/worldwide.aspx

    Appreciate your support and understanding.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, April 26, 2019 6:17 AM
    Moderator
  • I think, Microsoft have enough information to reproduce and fix the issue (if they want).
    Sunday, April 28, 2019 10:35 AM
  • I'm also having this issue on our new S2D cluster, we have 2 nodes, 200TB each and are experiencing the memory leak of non paged pool which during rebalance operations, can consume up to 95% (120GB) or RAM and eventually will cause the server to experience major issues. 

    Is there any fix for this? Not really acceptable for an 'Enterprise' solution Microsoft. 


    Friday, May 17, 2019 2:12 AM
  • Microsoft plans to re-release the Hyper-V Server 2019 in June.

    when I read between the lines, it's possible the issue is fixed in this release.

    At the moment I repeat my tests after each "patchday". I will post my results next week, after install the May patches.

    Saturday, May 18, 2019 12:55 PM
  • Re-test it after install updates 2019-05.

    Result: Issue not solved.

    Friday, May 24, 2019 2:15 PM
  • I just installed Server 19 on a system and set up a Storage Spaces pool. Transferring files to it initially and also noticing severe memory leak issue. For me system, non-paged pool memory seems to grow at a rate of nearly 4GB memory per 1TB files transferred.
    • Edited by m.n1 Saturday, May 25, 2019 2:37 PM
    Saturday, May 25, 2019 2:33 PM
  • I have also found that as I fill up the pool with more data, the amount of RAM that gets taken up per TB transferred increases. I have 30TB transferred in now, and as I transfer more data the non-paged pool memory is increasing at a rate of around 8GB per every 1.25TB I transfer now.
    • Edited by m.n1 Sunday, June 9, 2019 2:34 AM
    Saturday, June 1, 2019 3:58 PM
  • Can confirm, after latest updates (Server 2019 17763.437) this still happens. NTFS storage spaces simple volume.

    After 48 hours on Sp** uses 3 GB of memory on a 8 GB RAM test system, simulating database access with random reads and writes (4kb-8GB). There should be 4 GB RAM left to use for caching, but instead cache is less than 512 MB and applications are paging.

    Performance and throughput has crawled to 10-30% of frehly booted system and from experience can tell that within 24 hours a reboot or forced hard boot will be mandatory, as it won't even be possible to log on via RDP anymore.

    This a very serious bug and it irks me to no end that it has received all the quality and urgency of Mickeysoft Mousesupport. This is not ready for production for even a 5 days/week office organization, and absolutely prohibited for a 24/7 internet business model.

    *Crippling bug on storage driver* and support pastes web links. Unreal.

    Edit: reference test server, identical in all aspects but with a normal sata disk volume doesn't suffer from this at all, so it most definitely is an outright bug in Storage Spaces Direct.

    Edit2: The 3GB figure from Sp** alone doesn't tell the whole dimension of the issue, because after all user programs are closed, the commit charge remains at 7.3GB out of 8GB physical RAM. Just terrible. 


    • Edited by Subrama Sunday, June 2, 2019 5:32 PM more data
    Sunday, June 2, 2019 2:27 PM
  • Using a second pool now, also with parity config. This second pool uses 1 parity disk, and its memory issue is much slower than the first pool which uses 2 parity disks. Otherwise they are identical configs. The 2 parity pool started at a rate of 4GB RAM used per 1TB transferred and has grown to 10GB per 1TB transferred now that I have around 40TB written to the pool. The 1 parity pool has 12TB transferred into it and that pool is having RAM grow at a rate of around 1GB per 1TB transferred still. So how much parity data is configured for the pool seems to have a large impact on how much memory gets "stuck" until server reboot.
    Sunday, June 9, 2019 2:39 AM
  • I think the issues is fix with the cumulative update 2019-06.

    I have install it and copy some data. After copy 1TB I see a growing of 10MB on the non-paged pool SP**. Not nothing, but much better than 1GB per 1TB data.

    Can some other confirm this?

    Friday, June 14, 2019 12:46 PM
  • After the reorg of the storage-pool and write about 4TB data to a new disk, I can no longer see a growing of the non-paged pool.

    Issue fixed! Thank you Microsoft!

    • Marked as answer by heuby.ch Thursday, June 20, 2019 1:21 PM
    Thursday, June 20, 2019 1:21 PM
  • After the June '19 updates I no longer have memory issues with Storage Spaces either.

    Thanks Microsoft.

    Friday, August 2, 2019 10:43 PM