none
RunSpace Double Ups RRS feed

  • Question

  • Hey I'm just getting into really using runspaces to gather information from the network to speed along the process but I'm noticing something intriguing with doubling up of information so I have a few questions about the script I'm using, in hopes you lovely people can help clarify things I'm clearly not understanding.

    Apologies the script has gotten a little dirty as I'm trying to break it apart and test sections.

    Question 1: the "$RunspacePool = [runspacefactory]::CreateRunspacePool(1,$breakdown)" would work out to say (1,4) variable my understanding is that should mean maximum threads of 4? however I notice the chunk is actually 5 threads, if I up the count its always 1 more and I'm not sure why.

    Question 2: I noticed the last thread double ups on me if I were to run the below script against 20 PCs and expect results like:
    Machine           Username            Disabled
    Computer1       Administrator       Active
    Computer1       Guest                  Disabled
    Computer2       Administrator       Active
    Computer2       Guest                  Disabled
    Computer3       Administrator       Active
    Computer3       Guest                  Disabled
    Computer4       Administrator       Active
    Computer4       Guest                  Disabled
    Computer5       Administrator       Active
    Computer5       Guest                  Disabled

    but instead I'll get the results of:

    Machine           Username            Disabled
    Computer1       Administrator       Active
    Computer1       Guest                  Disabled
    Computer2       Administrator       Active
    Computer2       Guest                  Disabled
    Computer3       Administrator       Active
    Computer3       Guest                  Disabled
    Computer4       Administrator       Active
    Computer4       Guest                  Disabled

    Computer2       Administrator       Active
    Computer2       Guest                  Disabled
    Computer5       Administrator       Active
    Computer5       Guest                  Disabled

    noting that the last thread runs again on a previously used thread and has the previously collated data still within in it to return.  also thing to note when the script runs the next chunk despite the:

    $jobs[$jobstart].PowerShell.Dispose()
    $jobs[$jobstart].powershell = $null
    $jobs[$jobstart].handle = $null

    and
    $jobs.clear()
    $jobs = $null

    lines let alone the $job variable being re declared I notice each thread is still retaining the previously returned information from prior jobs.  how can I clear this so the threads do not retain or return the previous information again without having to "unique" the results table after the fact.

    Example Script:

    $Computers = Get-ADComputer -Filter * -Properties lastLogonTimestamp, operatingsystem, servicePrincipalName | ?{$_.enabled -eq $true} | ?{($_.servicePrincipalName -join "-") -notlike "*clustervirtual*"}
    $breakdown = [int]$env:NUMBER_OF_PROCESSORS
    $runner = $Computers.Count
    $num1 = 0
    $num2 = $breakdown
    $check = 0
    $Results =$null
    #region Runspace Pool
    [runspacefactory]::CreateRunspacePool()
    $SessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
    $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$breakdown)
    $PowerShell = [powershell]::Create()
    $PowerShell.RunspacePool = $RunspacePool
    $RunspacePool.Open()
    #endregion

    do{
        write-host "running block $num1..$num2 out of $runner" -ForegroundColor green
        $jobs = New-Object System.Collections.ArrayList
        $count = 1

        ForEach($Computer in $Computers[$num1..$num2]){
            $Parameters = @{
                Computer = $computer
                }
            $PowerShell = [powershell]::Create() 
            $PowerShell.RunspacePool = $RunspacePool
            [void]$PowerShell.AddScript({
                Param ($computer)
                run custom script
                return $Report
            })
            [void]$PowerShell.AddParameters($Parameters)
            $Handle = $PowerShell.BeginInvoke()
            $temp = '' | Select PowerShell,Handle
            $temp.PowerShell = $PowerShell
            $temp.handle = $Handle
            [void]$jobs.Add($Temp)   
        }
        $waittimer = 0
        do{
        if($jobs.handle.IsCompleted -like $false){
                                                        Write-Host "." -NoNewline
                                                        Start-Sleep -Seconds 1
                                                        $waittimer++}ELSE{$waittimer = 100}
        }until($waittimer -eq 100)
        $jobs = $jobs| ?{$_.handle.IsCompleted -like $true}
        $jobcount = $jobs.Count
        $jobstart = 0
        do{
            if($jobstart -lt $jobcount-1){
            $Results += $jobs[$jobstart].powershell.EndInvoke($jobs[$jobstart].handle)
            $jobs[$jobstart].PowerShell.Dispose()
            $jobs[$jobstart].powershell = $null
            $jobs[$jobstart].handle = $null
            }ELSE{
            $Results += $jobs[$jobstart].powershell.EndInvoke($jobs[$jobstart].handle) | ?{$_.machinename -like (($jobs[$jobstart].powershell.EndInvoke($jobs[$jobstart].handle) | select Machinename -Unique)[1]).machinename}
            $jobs[$jobstart].PowerShell.Dispose()
            $jobs[$jobstart].powershell = $null
            $jobs[$jobstart].handle = $null
            }
            $jobstart++
        }until($jobstart -eq $jobcount)
        $jobs.clear()
        $jobs = $null
        $num1 = $num1+$breakdown+1
        $num2 = $num2+$breakdown+1
        if($num2 -ge $runner){$num2 = $runner
        $check++}
        }until($check -gt 1)

    $RunspacePool.Close()
    New-TimeSpan -Start $start -End (get-date)

    Friday, July 19, 2019 4:28 AM

All replies

  • A runspacepool can have a maximum number of runspaces and a minimum number of runspaces.  The maximum is never exceeded.  The minimum is used to prevent stalling when releasing large numbers of tasks.  This ensures that there will always be a runspace available for a new task.  It also allows tasks to be queued.  The minimum should not exceed the number of cores available or overhead will be increased.  RUnspaces are most useful on multi-core/processor systems.

    I recommend that you use either jobs or a workflow. It is easier to manage and does not require systems programming experience.

    Workflows are better managed for most PS tasks and are more efficient than jobs.

    Note that a runspace is not a thread. There is a thread factory and thread creation API for threads. Runspaces are completely different.  They are objects that are part of the PowerShell "Automation" namespace.


    \_(ツ)_/

    Friday, July 19, 2019 5:06 AM
    Moderator
  • I get a lot of that and I've answered my question on question 1 with my coding its more on the seond part I'm stumped and I'd really like to get my head around it.  Lets say I move the "$jobs = New-Object System.Collections.ArrayList" line to the top of the script so it is created only once the out put for $jobs.PowerShell | ft Streams, InstanceID reads as

    Streams                                                                InstanceId
    -------                                                                   ---------- 
    System.Management.Automation.PSDataStreams     7908e81f-5bf0-42ff-8df2-bf2954d49609
    System.Management.Automation.PSDataStreams     98099023-5f63-4c97-b2d2-f96e37b1a431
    System.Management.Automation.PSDataStreams     f4c60c09-a678-4290-8aa4-8215cc84fd95
    System.Management.Automation.PSDataStreams     5abcd4f4-9437-4d25-9051-74e0b17cadd7
    System.Management.Automation.PSDataStreams     f7c5e008-9f77-4ba3-bb62-0589b3ef2022
    System.Management.Automation.PSDataStreams     2dac7fed-8c1d-4c79-a512-2390d2bda51d
    System.Management.Automation.PSDataStreams     94e65d7f-4ee4-489d-9d8e-60e4e9846986
    System.Management.Automation.PSDataStreams     113bbf7e-9f99-482f-a720-b8e799187587
    System.Management.Automation.PSDataStreams     f922dbc3-11ea-4ac1-a336-c70eafbaa37b 
    System.Management.Automation.PSDataStreams     1dd985cb-4ee3-4c56-a925-2897f6b15b3a
    System.Management.Automation.PSDataStreams     225cffe4-87c2-47c9-a55b-8c41a9fdd0d4 
    System.Management.Automation.PSDataStreams     22a4e00c-b58d-4145-a586-7e63227e269a
    System.Management.Automation.PSDataStreams     2f1ef13f-6a5d-45ac-8cf8-efcba987c702
    System.Management.Automation.PSDataStreams     f4abe787-6b84-4af1-9a14-99614af45163
    System.Management.Automation.PSDataStreams     a67cd3b0-9301-48ea-83de-987bf4382d9c
    System.Management.Automation.PSDataStreams     49c9341f-b05d-4e94-9d3a-cb540ff158d7
    System.Management.Automation.PSDataStreams     fb13b982-fcdd-49dc-bb8f-167784a83ff4

    So each object within jobs as created a unique powershell instance then when you cycle through each job the first 4 (had a maximum of 4 against the createrunspacepool) look good they each have 1 computer against them and are returning a table of data of things my custom script ran to gather. but then we hit the 5th object in the jobs list and I can see that the computer from the fourth job shows in the fifth jobs table as well as the new expected computer i check the 6th job and same deal except it has the 3rd jobs PC plus the new expected one. and if I go to the 9th it as the 3rd, 4th and its new expected computer in the jobs outputs.

    As I type this I realised I can shortcut the output issue by moving the $return value to being completely out of the do loop at the end (do loop was there while I was trying to break it all down and understand the code) and enter this code at the very end of the script.

    $Returns = $jobs[(($jobs.Count)-$breakdown-1)..$jobs.Count] | ForEach {
            $_.powershell.EndInvoke($_.handle)
            $_.PowerShell.Dispose()
        }

    Which is all fine and gets me the results I wish for but what is the point in all the other jobs and is there a more efficient way so when I run this against 10,000 devices I'm not creating so many instances.

    Once Cleaned up the new script working looks like this for anyone else in a similar issue:

    $start = Get-Date
    $Computers = Get-ADComputer -Filter * -Properties lastLogonTimestamp, operatingsystem, servicePrincipalName | ?{$_.enabled -eq $true} | ?{($_.servicePrincipalName -join "-") -notlike "*clustervirtual*"}
    $breakdown = [int]$env:NUMBER_OF_PROCESSORS
    $Results =$null
    #region Runspace Pool
    [runspacefactory]::CreateRunspacePool()
    $SessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
    $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$breakdown)
    $RunspacePool.Open()
    #endregion

    $jobs = New-Object System.Collections.ArrayList
        ForEach($Computer in $Computers){
            $Parameters = @{
                Computer = $computer
                }
            $PowerShell = [powershell]::Create() 
            $PowerShell.RunspacePool = $RunspacePool
            [void]$PowerShell.AddScript({
                Param ($computer)
                Custom Script
                return $Report
            })
            [void]$PowerShell.AddParameters($Parameters)
            $Handle = $PowerShell.BeginInvoke()
            $temp = '' | Select PowerShell,Handle
            $temp.PowerShell = $PowerShell
            $temp.handle = $Handle
            [void]$jobs.Add($Temp)   
        }
        $waittimer = 0
        do{
        if($jobs.handle.IsCompleted -like $false){
                                                        $jobs.handle | ft
                                                        Write-Host "." -NoNewline
                                                        Start-Sleep -Seconds 1
                                                        $waittimer++}ELSE{$waittimer = 100}
        }until($waittimer -eq 100)



    $Returns = $jobs[(($jobs.Count)-$breakdown-1)..$jobs.Count] | ForEach {
            $_.powershell.EndInvoke($_.handle)
        }

    $jobs| %{$_.PowerShell.Dispose()}
    $jobs.clear()
    $RunspacePool.Close()
    New-TimeSpan -Start $start -End (get-date)

    Friday, July 19, 2019 7:16 AM
  • Perhaps if you would post your code correctly we might be able to understand your issue.

    How to post code in Technet Forums

    How to ask questions in a technical forum


    \_(ツ)_/

    Friday, July 19, 2019 8:16 AM
    Moderator