none
How to collect returns from recursive function calls without a global var? RRS feed

  • Question

  • Usually global variables shouldnt be used, I was told. Ok. But:

    How can I avoid using a global var when recursively using a function, when the function returns an object and when I want to have the collection of all these objects as the result?

    For example, I think of determine the users of a group including group nesting. I would write a function that adds the direct group members to a collection as a global var and call itself recusively if a member is another group. This recursively called function would do as well: Update the global var and if needed call itself.

    I'm afraid this is no good programming style. What algorithm would be better, prettier? Please dont focus on the example, it is a more general question of how to do.

    Thanks

    Walter

    Friday, November 7, 2014 4:21 PM

Answers

  • Pass collection by reference

    function Recurse-Folders([string]$path,[ref]$FoldersList) 
    {
      Set-Location -Path $Path
      $Location = Get-Item .
      [string[]]$FoldersList.Value += $Location.FullName  
    
      [String[]]$Folders  = Get-ChildItem "." -Directory
      ForEach ($F in $Folders)
      {
        Recurse-Folders $F $FoldersList
        Set-Location -Path ".." 
      }
    }
    
    $ListOfFolders = @()
    
    Recurse-Folders -Path C:\test -Folders ([ref]$ListOfFolders)
    
    $ListOfFolders

    Gives this output

    C:\test
    C:\test\Folder1
    C:\test\Folder1\SubFolder1
    C:\test\Folder2
    C:\test\Folder3


    You don't really need to build up the collection inside the function; if the caller assigns the result to a variable, PowerShell will do that for you, and it executes quite a bit faster.  I would tend to write that code like this:

    function Get-Folders([string] $Path)
    {
        $directory = Get-Item -LiteralPath $Path -ErrorAction Stop
        if ($directory -isnot [System.IO.DirectoryInfo])
        {
            throw "Path '$Path' does not refer to a valid FileSystem directory."
        }
        
        GetFoldersR -DirectoryInfo $directory
    }
    
    function GetFoldersR([System.IO.DirectoryInfo] $DirectoryInfo)
    {
        $DirectoryInfo.FullName
    
        foreach ($subdirectory in $DirectoryInfo.GetDirectories())
        {
            GetFoldersR -DirectoryInfo $subdirectory
        }
    }
    
    $ListOfFolders = Get-Folders -Path C:\test
    
    $ListOfFolders

    This way there's some validation code that gets performed once (not for each recursive call), and then the recursive helper function is about as fast as you can make it without resorting to C#.  (Technically, I'd write that code as Get-ChildItem -Directory -Recurse C:\test , but presumably the OP needs to do something other than just walk the file system.  :) )

    • Marked as answer by WalterFMB Monday, November 10, 2014 2:43 PM
    Saturday, November 8, 2014 4:23 AM

All replies

  • In general, you shouldn't need to collect the objects; just output them.

    Please write a minimal, self-contained example that doesn't depend on your system that illustrates the problem you want to solve.


    -- Bill Stewart [Bill_Stewart]

    Friday, November 7, 2014 4:44 PM
    Moderator
  • I would just use an array of objects to collect the resulting object from a function. 

    Take this sample function:

    function Get-MyProcess ($ProcessName) {
        $Proc = Get-Process -Name $ProcessName
        $Props = [ordered]@{
            Name = $Proc.Name
            ID = $Proc.Id
            Handle = $Proc.Handle
            Handles = $Proc.HandleCount
            StartTime = $Proc.StartTime
        }
        New-Object -TypeName psobject -Property $Props
    }

    To use it to collect several resulting objects:

    $MyProcs = @()
    
    "VSSVC","conhost","explorer" | % { $MyProcs += Get-MyProcess $_ }
    
    $MyProcs | FT -AutoSize


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

    Friday, November 7, 2014 6:26 PM
  • Just collect them as the aggregate.

    $MyProcs="VSSVC","conhost","explorer" | % { Get-MyProcess $_ }

    No other issues needed.  That is how the pipeline works.


    ¯\_(ツ)_/¯

    Friday, November 7, 2014 6:32 PM
  • This would be how to do it in PowerShell:

    function Get-MyProcess{
         Param(
               [Parameter(ValueFromPipeline=$true)]
              [string]$ProcessName
        )
        Process{
            Get-Process -Name $ProcessName | Select Name,ID,Handle,Handles,StartTime
        }
    }
    
    'svchost','conhost','explorer' | Get-MyProcess |ft -auto
    


    ¯\_(ツ)_/¯

    Friday, November 7, 2014 6:40 PM
  • Or you could just do this:

    $MyProcs='svchost','conhost','explorer' |   %{Get-Process $_} | Select Name,ID,Handle,Handles,StartTime |ft -auto
    
    #And this is even easier:
    
    $MyProcs=Get-Process 'svchost','conhost','explorer' | Select Name,ID,Handle,Handles,StartTime | ft -auto
    
    #And even easier:
    
    $MyProcs=Get-Process 'svchost','conhost','explorer' | ft  Name,ID,Handle,Handles,StartTime -auto
    

    The point is that recursion and looping are built into PS.

    You example is not about recursion. It is about the pipeline


    ¯\_(ツ)_/¯


    • Edited by jrv Friday, November 7, 2014 6:45 PM
    Friday, November 7, 2014 6:45 PM
  • Tiny little codes:

    ps 'svchost','conhost','explorer' | ft  Name,ID,Handle,Handles,StartTime -a


    ¯\_(ツ)_/¯

    Friday, November 7, 2014 6:47 PM
  • Pass collection by reference

    function Recurse-Folders([string]$path,[ref]$FoldersList) 
    {
      Set-Location -Path $Path
      $Location = Get-Item .
      [string[]]$FoldersList.Value += $Location.FullName  
    
      [String[]]$Folders  = Get-ChildItem "." -Directory
      ForEach ($F in $Folders)
      {
        Recurse-Folders $F $FoldersList
        Set-Location -Path ".." 
      }
    }
    
    $ListOfFolders = @()
    
    Recurse-Folders -Path C:\test -Folders ([ref]$ListOfFolders)
    
    $ListOfFolders

    Gives this output

    C:\test
    C:\test\Folder1
    C:\test\Folder1\SubFolder1
    C:\test\Folder2
    C:\test\Folder3


    Friday, November 7, 2014 7:23 PM
  • Well Brian beat me.  That is a good example of recursion.  Recursion is not about a pipeline it is about structure walking.  It is very good at sorting binary trees.  Not so good with maple trees though.


    ¯\_(ツ)_/¯

    Friday, November 7, 2014 7:37 PM
  • Well Brian beat me.  That is a good example of recursion.  Recursion is not about a pipeline it is about structure walking.  It is very good at sorting binary trees.  Not so good with maple trees though.


    ¯\_(ツ)_/¯

    ROFLOL

    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

    Friday, November 7, 2014 10:31 PM
  • I rarely needed to create/use recursive functions. Here's one example I did:

    function New-SBSeed {
    <# 
     .Synopsis
      Function to create files for disk performance testing. Files will have random numbers as content. 
      File name will have 'Seed' prefix and .txt extension. 
    
     .Description
      Function will start with the smallest seed file of 10KB, and end with the Seed file specified in the -SeedSize parameter. 
      Function will create seed files in order of magnitude starting with 10KB and ending with 'SeedSize'. 
      Files will be created in the current folder.
    
     .Parameter SeedSize
      Size of the largest seed file generated. Accepted values are:
          10KB
          100KB
          1MB
          10MB
          100MB
          1GB
          10GB
          100GB
          1TB
    
     .Example
      New-SBSeed -SeedSize 10MB -Verbose
      
      This example creates seed files starting from the smallest seed 10KB to the seed size specified in the -SeedSize parameter 10MB.
      To see the output you can type in:
    
        Get-ChildItem -Path .\ -Filter *Seed*
    
      Sample output:
     
        Mode                LastWriteTime     Length Name                                                                                                                                      
        ----                -------------     ------ ----                                                                                                                                      
        -a---          8/6/2014   8:26 AM     102544 Seed100KB.txt
        -a---          8/6/2014   8:26 AM      10254 Seed10KB.txt
        -a---          8/6/2014   8:39 AM   10254444 Seed10MB.txt
        -a---          8/6/2014   8:26 AM    1025444 Seed1MB.txt
    
     .Link
      https://superwidgets.wordpress.com/category/powershell/
    
     .Notes
      Function by Sam Boutros
      v1.0 - 08/01/2014
    
    #>
        [CmdletBinding(SupportsShouldProcess=$true,ConfirmImpact='Low')] 
        Param(
            [Parameter(Mandatory=$true,
                       ValueFromPipeLine=$true,
                       ValueFromPipeLineByPropertyName=$true,
                       Position=0)]
                [Alias('Seed')]
                [ValidateSet(10KB,100KB,1MB,10MB,100MB,1GB,10GB,100GB,1TB)] 
                [Int64]$SeedSize
        )
    
        $Acceptable = @(10KB,100KB,1MB,10MB,100MB,1GB,10GB,100GB,1TB)
        $Strings = @("10KB","100KB","1MB","10MB","100MB","1GB","10GB","100GB","1TB")
        for ($i=0; $i -lt $Acceptable.Count; $i++) {
            if ($SeedSize -eq $Acceptable[$i]) { $Seed = $i } 
        }
        $SeedName = "Seed" + $Strings[$Seed] + ".txt"
        if ($Acceptable[$Seed] -eq 10KB) { # Smallest seed starts from scratch
            $Duration = Measure-Command {
                do {Get-Random -Minimum 100000000 -Maximum 999999999 | 
                    out-file -Filepath $SeedName -append} while ((Get-Item $SeedName).length -lt $Acceptable[$Seed])
            }
        } else { # Each subsequent seed depends on the prior one
            $PriorSeed = "Seed" + $Strings[$Seed-1] + ".txt"
            if ( -not (Test-Path $PriorSeed)) { New-SBSeed $Acceptable[$Seed-1] } # Recursive function :)
            $Duration = Measure-Command {
                $command = @'
                cmd.exe /C copy $PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed+$PriorSeed $SeedName /y
    '@
                Invoke-Expression -Command:$command
                Get-Random -Minimum 100000000 -Maximum 999999999 | out-file -Filepath $SeedName -append
            }
        }
        Write-Verbose ("Created " + $Strings[$Seed] + " seed $SeedName file in " + $Duration.TotalSeconds + " seconds")
    }

    This is part of the SBTools module and is used by the Test-SBDisk function

    Example use:

    New-SBSeed 10GB -Verbose

    Test-SBDisk is a multi-threaded function that puts IO load on target disk subsystem and can be used to simulate workloads from multiple machines hitting the same SAN at the same time, and measure disk IO and performance.


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx


    • Edited by Sam Boutros Friday, November 7, 2014 10:52 PM
    Friday, November 7, 2014 10:50 PM
  • Very nice Sam.  YOu cheated a bit with the "ConfirmImpact" though.  I see no tests.  Of course all reports are low impact by nature except when they recurse through very large datasets.

    I will go through it later as it looks quite cool.


    ¯\_(ツ)_/¯

    Friday, November 7, 2014 11:25 PM
  • Yep, this and other few things can be enhanced, like adding a common logging variable throughout the entire module. May be later.. :)

    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

    Saturday, November 8, 2014 12:03 AM
  • This recursive function splits the dots of a class name into branches of a tree and the full class name as a leaf of the tree. Depending on the system it is ran on, it does about 8000+ classes

    Function Add-NetClasses
    {
        [CmdletBinding()] 
        param(
            [Object]$ClassList,
            [String]$ClassName,
            [Object]$Class
        )
        Try
        {
           $Position = $ClassName.IndexOf(".")
           IF ($Position -GT 0)
           {  
              $ClassFirst = $ClassName.Substring(0,$Position)
              If ($ClassList.Count -GT 0)
              {
                  $NewClassList = $ClassList[$ClassFirst]
                  If ($NewClassList -EQ $Null)
                  {
                     $NewClassList = New-Object System.Collections.SortedList
                     $ClassList.Add($ClassFirst,$NewClassList)
                  }
              }
              Else
              {
                  $NewClassList = New-Object System.Collections.SortedList
                  IF ($ClassList -EQ $Null)
                  {
                  }
                  Else
                  {
                    $ClassList.Add($ClassFirst,$NewClassList)
                  }
              }
              $ClassRest  = $ClassName.Substring($Position+1)
              Add-NetClasses $NewClassList $ClassRest $Class
           }     
           Else
           {
              If ($ClassList[$ClassName] -EQ $Null)
              {
                 $ClassList.Add($Class.FullName,$Class)
              }
           }
        }
        Catch [System.Exception]
        {
            Write-Verbose "Add-NetClasses Catch"
            Write-Verbose $_.Exception.Message
        }
    }
    

    Saturday, November 8, 2014 3:08 AM
  • Pass collection by reference

    function Recurse-Folders([string]$path,[ref]$FoldersList) 
    {
      Set-Location -Path $Path
      $Location = Get-Item .
      [string[]]$FoldersList.Value += $Location.FullName  
    
      [String[]]$Folders  = Get-ChildItem "." -Directory
      ForEach ($F in $Folders)
      {
        Recurse-Folders $F $FoldersList
        Set-Location -Path ".." 
      }
    }
    
    $ListOfFolders = @()
    
    Recurse-Folders -Path C:\test -Folders ([ref]$ListOfFolders)
    
    $ListOfFolders

    Gives this output

    C:\test
    C:\test\Folder1
    C:\test\Folder1\SubFolder1
    C:\test\Folder2
    C:\test\Folder3


    You don't really need to build up the collection inside the function; if the caller assigns the result to a variable, PowerShell will do that for you, and it executes quite a bit faster.  I would tend to write that code like this:

    function Get-Folders([string] $Path)
    {
        $directory = Get-Item -LiteralPath $Path -ErrorAction Stop
        if ($directory -isnot [System.IO.DirectoryInfo])
        {
            throw "Path '$Path' does not refer to a valid FileSystem directory."
        }
        
        GetFoldersR -DirectoryInfo $directory
    }
    
    function GetFoldersR([System.IO.DirectoryInfo] $DirectoryInfo)
    {
        $DirectoryInfo.FullName
    
        foreach ($subdirectory in $DirectoryInfo.GetDirectories())
        {
            GetFoldersR -DirectoryInfo $subdirectory
        }
    }
    
    $ListOfFolders = Get-Folders -Path C:\test
    
    $ListOfFolders

    This way there's some validation code that gets performed once (not for each recursive call), and then the recursive helper function is about as fast as you can make it without resorting to C#.  (Technically, I'd write that code as Get-ChildItem -Directory -Recurse C:\test , but presumably the OP needs to do something other than just walk the file system.  :) )

    • Marked as answer by WalterFMB Monday, November 10, 2014 2:43 PM
    Saturday, November 8, 2014 4:23 AM
  • I have used that function to answer at least 3 other threads. The first time, someone asked to be able to do something at each level but did not know how to write it. So I wrote it, and presented it as a suggestion the other times.

    I remember SWEEP.COM from the DOS days. (Sweep DIR *.ASM)

    PC Mag 12 Nov 1985


    Saturday, November 8, 2014 5:44 AM
  • Recurse folders and return path:

    get-childitem c:\temp -Directory -Recurse |%{$_.Fullname}


    ¯\_(ツ)_/¯

    Saturday, November 8, 2014 5:52 AM
  • David's version is much faster but throws errors which slow it down.


    ¯\_(ツ)_/¯

    Saturday, November 8, 2014 6:00 AM
  • You're missing the point on two fronts.

    The purpose of the sample is to demonstrate of how to avoid using $Script: variables. I don't know what the return type of the recursive function is but if the problem is that script can not use $Script variables then the return type is irrelevant. 

    The purpose of the function is not to duplicate the functionality of some PowerShell cmdlet, but execute custom code at each level in the directory structure. 

    Saturday, November 8, 2014 6:25 AM
  • You're missing the point on two fronts.

    The purpose of the sample is to demonstrate of how to avoid using $Script: variables. I don't know what the return type of the recursive function is but if the problem is that script can not use $Script variables then the return type is irrelevant. 

    The purpose of the function is not to duplicate the functionality of some PowerShell cmdlet, but execute custom code at each level in the directory structure. 

    Yes and David's function demonstrates true recursion that returns a collection and is not dependent on any global variables.  Output can be caught in a variable or piped to another CmdLet or file. Your function demonstrates recursion but would be slower because it constantly modifies an object collection. PowerShell is optimized to collect objects in a pipeline and can be much faster although that depends on the exact implementation.

    I have built "recursors" that take a code block as an argument.  The code block is executed on each element of the search.  Returning items or not as needed.  It is a common design pattern.  Yours is the same except it uses a fixed and embedded code block.

    David's code walks the folder tree extremely fast. I am still trying to see why that method would be faster than the CmdLet.

    Anyway.  It is all good info.


    ¯\_(ツ)_/¯

    Saturday, November 8, 2014 6:39 AM
  • Hmm, ok, but processes have no recursive character as e. g. folders or groups have.
    Monday, November 10, 2014 2:18 PM
  • Hmm, but this looks similar to a global var:
    $ListOfFolders  is defined outside the function as "a kind of" a global or script variable.

    What would be the advantage of working with [ref] ?

    Walter

    Monday, November 10, 2014 2:20 PM
  • Hmm, but this looks similar to a global var:
    $ListOfFolders  is defined outside the function as "a kind of" a global or script variable.

    What would be the advantage of working with [ref] ?

    Walter

    Are you referring to the code I posted?  In that, $ListOfFolders is defined by the caller of the function as a place to store its return value.  While this technically 'could' be a variable in the global scope (if that's where the call to the function is coming from), that's not the same thing.

    When you refer to using "global variables" in a function, that means you're accessing variables outside of the function's own scope from inside the function body.  If you only access parameters that are passed into the function, or new variables that you declare inside the function, then you're not using global variables (regardless of what the code that's calling your function decides to do with the results.)

    Monday, November 10, 2014 2:31 PM
  • WOW!
    Thanks a lot, David, that's it.

    And it works not only with just returning a string (per function call: $DirectoryInfo.FullName), but as well when returning an object:

    function Get-Folders([string] $Path) {
        $directory = Get-Item -LiteralPath $Path -ErrorAction Stop
        if ($directory -isnot [System.IO.DirectoryInfo]) {
            throw "Path '$Path' does not refer to a valid FileSystem directory."
        }
        
        GetFoldersR -DirectoryInfo $directory
    }
    
    function GetFoldersR([System.IO.DirectoryInfo] $DirectoryInfo) {
        New-Object -TypeName PSObject -Property @{
            FullName = $DirectoryInfo.FullName;
            CreationTime = $DirectoryInfo.CreationTime
        }
    
        foreach ($subdirectory in $DirectoryInfo.GetDirectories()) {
            GetFoldersR -DirectoryInfo $subdirectory
        }
    }
    
    $ListOfFolders = Get-Folders -Path C:\Path
    $ListOfFolders

    Returning an object was exactly the thing that I didnt know how to deal with.
    I was afraid that I wouldnt get a "flat" array of objects but a nested array (you know what I mean?). But Powershell is smarter than I thought! Really COOL.

    And thanks for showing me some other useful things.

    Walter

    Monday, November 10, 2014 2:42 PM