none
Powershell Memory Usage? RRS feed

  • Question

  • Hi,

    I am having an issue with a long running powershell script with a the memory increasing over time, like a leak. I am creating collection variables but after doing a connection to sql. Later on I am writing the output to a CSV file.  However in my functions I am setting the collection variables to $null. But the memory keeps going up and up.

    Is there something special I am missing on how powershell frees it memory. So for example if I call function a and I have a variable called $test should the memory be free for what was allocated to that function.

    So is there any general advice somebody can offer how powershell allocates and frees it's memory, or any other advice that you can offer?

    I am even calling the GC::Collect() and it is still not helping

    thanks,

    Ward

    Saturday, December 7, 2013 1:42 PM

Answers

  • The most important thing is to identify why you think it is a memory leak.

    Memory in Windows is allocated in large chunks.  Once allocate it stays with an application until Windows decides to do "workingset trimming".  In systems with large memories this will not happen very often and perhaps at all.  This will cause the workingset to never get trimmed so it will always reflect the largest allocation  required.  The GC:Collect will release memory back to the pool held by the workingset allocation.  The workingset will not be trimmed.  The system allocates from  the free memory in the workingset until the workingset is low at which time the memory manager will add to the workingset causing it to grow until the system needs to trim it.

    Allocating memory from main memory is an expensive process.  Some memory is "physical" and some is "virtual".  A programs memory is considered all virtual but room in physical memory must be maintained to keep track of pages owned by the program.  The program must also have pages reserved in the page file for swapping.  All of this is done only when necessary since it is more expensive to deallocate and reallocate then it is to just leave memory and resources assigned.

    So we need to know what makes you think you are leaking memory?


    ¯\_(ツ)_/¯

    • Marked as answer by WardH Wednesday, December 11, 2013 7:39 AM
    Saturday, December 7, 2013 5:54 PM

All replies

  • Hard to judge.

    What kind of variables are you creating?

    If you create variables and aggregate them then release the collection the individual variables will no be automatically released.

    If you have open conenctions and datasets they will no be released. 

    GC is lazy in PowerShell even if you explicitly call GC.

    COM objects must be explicitly released.

    How do you know that this is a memory leak?


    ¯\_(ツ)_/¯

    Saturday, December 7, 2013 1:56 PM
  • We'd need to see the code to know for sure.  For the most part, though, there's no reason to call GC.Collect() yourself.  .NET will take care of that on its own within a few seconds.

    You didn't mention Active Directory, but just to double check: are you using the Quest AD cmdlets in this script?  They seem to hold onto references to objects even when the PowerShell script itself is not.  I've seen a few different posts here complaining about OutOfMemoryException errors using that module.

    Saturday, December 7, 2013 2:10 PM
  • Hi,

    I am not doing any active diretory calls, nor am I doing any COM calls.

    so I am builing up collections with code like below.

    Thanks,

    Ward

    function Test1($machine_list)
    {
    $data_list = @()
    
    foreach ($pc in $machine_list)
    {
    	$obj = New-Object System.Object
    	$obj | Add-Member -MemberType NoteProperty Data1 $pc
    	$obj | Add-Member -MemberType NoteProperty Data2 $pc
    	$obj | Add-Member -MemberType NoteProperty Data3 $pc
    	$data_list += $obj
    }
    
    $data_list | Export-Csv c:\scripts\test.txt -notype
    
    $data_list = $null
    }
    
    $machine_list = @()
    
    #select Name from Table where ProductID = 'XX'
    
    # Then put each Name into a machine_list variable.
    
    Test1 $machine_list
    
    

    Saturday, December 7, 2013 2:39 PM
  • I can't troubleshoot a memory "leak" without seeing the classes and functions you're using for the SQL query, but there are definitely opportunities to improve this from a PowerShell perspective by using the pipeline instead of building arrays to store your entire result set.  For example, this:

    $data_list = @()
    
    foreach ($pc in $machine_list)
    {
    	$obj = New-Object System.Object
    	$obj | Add-Member -MemberType NoteProperty Data1 $pc
    	$obj | Add-Member -MemberType NoteProperty Data2 $pc
    	$obj | Add-Member -MemberType NoteProperty Data3 $pc
    	$data_list += $obj
    }
    
    $data_list | Export-Csv c:\scripts\test.txt -notype

    Could be written as this:

    $machine_list |
    ForEach-Object {
    	$obj = New-Object System.Object
    	$obj | Add-Member -MemberType NoteProperty Data1 $pc
    	$obj | Add-Member -MemberType NoteProperty Data2 $pc
    	$obj | Add-Member -MemberType NoteProperty Data3 $pc
    	$obj
    } |
    Export-Csv c:\scripts\test.txt -notype

    There are other ways to adjust that code; I just demonstrated one way, replacing your $data_list array and foreach loop with a single pipeline involving ForEach-Object instead.  You may be able to make a similar change with the SQL code, not storing all the records in a $machine_list array, but I can't tell for sure since you only posted pseudo-code for that part.

    • Proposed as answer by David Wyatt Wednesday, December 11, 2013 12:50 PM
    Saturday, December 7, 2013 3:48 PM
  • The most important thing is to identify why you think it is a memory leak.

    Memory in Windows is allocated in large chunks.  Once allocate it stays with an application until Windows decides to do "workingset trimming".  In systems with large memories this will not happen very often and perhaps at all.  This will cause the workingset to never get trimmed so it will always reflect the largest allocation  required.  The GC:Collect will release memory back to the pool held by the workingset allocation.  The workingset will not be trimmed.  The system allocates from  the free memory in the workingset until the workingset is low at which time the memory manager will add to the workingset causing it to grow until the system needs to trim it.

    Allocating memory from main memory is an expensive process.  Some memory is "physical" and some is "virtual".  A programs memory is considered all virtual but room in physical memory must be maintained to keep track of pages owned by the program.  The program must also have pages reserved in the page file for swapping.  All of this is done only when necessary since it is more expensive to deallocate and reallocate then it is to just leave memory and resources assigned.

    So we need to know what makes you think you are leaking memory?


    ¯\_(ツ)_/¯

    • Marked as answer by WardH Wednesday, December 11, 2013 7:39 AM
    Saturday, December 7, 2013 5:54 PM
  • Hi,

    Thanks for your response that what great. Once I converted things to use code like above using the pipeline - the memory usage dropped by a large amount. So I think it was the collection variables building up.

    Thanks again.

    Ward

    Wednesday, December 11, 2013 7:48 AM
  • Hi,

    Thanks for your response that what great. Once I converted things to use code like above using the pipeline - the memory usage dropped by a large amount. So I think it was the collection variables building up.

    Thanks again.

    Ward

    This is because you are creating a new variable in you loop and overwriting the old name.  The variable is not being reused.  The memory will eventually be released although the working set will indicate that the memory is still assigned.  By using a pipeline you are not throwing away variables as the pipeline passes a single object and does not create temporaries.

    The following method would create evenfewer temporary variables because it reuses all temporary structures.

    $props=@{
       Data1=$null
       Data2=$null
       Data3=$null
    }
    Get-Content server_list.txt|
         ForEach-Object {
            $pc=Gwmi win32_BIOS -computer $_
            $props.Data1=$pc.PSComputerName
            $props.Data2=$pc.Manufacturer
            $props.Data3=$pc.Version
    	New-Object PsObject -Property $props
    } |
    Export-Csv c:\scripts\test.txt -notype
    

    Note that the pipeline suggestion was mode by David but you marked my answer whch was not about the pipeline.


    ¯\_(ツ)_/¯

    Wednesday, December 11, 2013 1:06 PM