locked
Workflow Start-Job Wait-Job Unexpected Behavior RRS feed

  • Question

  • I was doing some tests using Powershell Workflows and "foreach -parallel" but I am getting inconsistent results.

    I call a workflow with a foreach -parallel loop to start a bunch of jobs (separate powershell scripts). When I try to wait or get those jobs, sometimes it works but sometimes it would error:

    Microsoft.PowerShell.Core\Wait-Job : The command cannot find a job with the job ID 289. Verify the value of the Id parameter and then try the command again.
    At async:15 char:15
    + 
        + CategoryInfo          : ObjectNotFound: (289:Int32) [Wait-Job], PSArgumentException
        + FullyQualifiedErrorId : JobWithSpecifiedSessionNotFound,Microsoft.PowerShell.Commands.WaitJobCommand
        + PSComputerName        : [localhost]

    And I cannot figure out why it can't find the job despite the job ID coming from the Start-Job command.

    Below is the code I am testing

    $list = @(1,3,4,7,8,9)
    workflow async{
    Param($list)
        "Start"
        foreach -parallel ($n in $list){
            "-1: $n"
                $j = Start-Job -FilePath .\wait.ps1 -ArgumentList $n
                $j
            "--2:- $n"
                Wait-Job -Id $j.Id
            "---3: $n"
        }
    }
    async -list $list
    "Done"


    -----.\wait.ps1

    Param($val = 0)
    "Start Inside " + $val
    Start-Sleep -s 4
    "End Inside " + $val




    • Edited by David..g Monday, August 7, 2017 11:24 PM Add code tags
    Monday, August 7, 2017 11:23 PM

Answers

  • I looked into alternatives to Start-Job and found Invoke-Expression gave me the desired behavior. I am curious to the cause of the -parallel and job inconsistency, but Invoke-Expression will work. For reference I was looking for a means to duplicate of the behavior of https://caolan.github.io/async/docs.html#parallel

    Thanks for feedback.

    • Marked as answer by David..g Tuesday, August 8, 2017 5:52 PM
    Tuesday, August 8, 2017 5:51 PM

All replies

  • It doesn't make much sense to use a workflow to start a job.  Either use jobs or use a workflow.  Don't mix them if you don't want in consistent results. In a sense a workflow is a job collection managed by the workflow engine.  It will work in the background and return results as each thread completes.


    \_(ツ)_/

    • Marked as answer by David..g Tuesday, August 8, 2017 5:52 PM
    • Unmarked as answer by David..g Tuesday, August 8, 2017 6:19 PM
    Monday, August 7, 2017 11:32 PM
  • If you don't want t wait for the workflow to completer then run the workflow with the -AsJob parameter.

    Try this to see how it works.

    PS D:\scripts> workflow x{}
    PS D:\scripts> x
    PS D:\scripts> x -AsJob


    \_(ツ)_/

    Monday, August 7, 2017 11:36 PM
  • What I want to do is start a number of jobs in parallel, then wait for all jobs to finish before continuing to the next part.

    This is why I start the jobs in a foreach -parallel loop. But I've tried putting the "Wait-Job" in multiple spots with the same results:

    Wait-Job after start job in the foreach -parallel loop

    Wait job in a foreach -parallel after all start jobs ran in normal foreach loop

    Wait job in a separate workflow run after the start-job workflow

    All had the same results where some "wait jobs" would fail while others worked fine. I've tried referencing jobs with the ID and the Name.

    However, I found that when I put the wait-jobs in a normal foreach loop it appeared to work (even when start-job was parallel). This is only a partial solution since it means the loop will wait for whichever job its stuck on and not respond to when each job actually finishes.

    The design is based on node.js async parallel where you give a list of async functions ran in parallel and return when all functions are complete.

    Tuesday, August 8, 2017 1:22 AM
  • Just start the jobs in a loop.

    $jobs = @()
    foreach($x in $y){
         $jobs += Start-Job x -Arg $x
    }
    $jobs | Wait-Job
    

    There is no point to using a workflow.


    \_(ツ)_/

    Tuesday, August 8, 2017 1:51 AM
  • It seems that the problem is with wait-job and not start-job.

    Doing wait-job in a pipe has the same issue as a normal foreach loop. It will get stuck on whichever job its currently waiting for and not resume immediately when each job completes. For example if the first job took 10 minutes and every other job took 30 seconds, it would wait 10 minutes on the first one, then wait-job on the finished ones, but those actually finished almost 10 minutes ago, not right after the 10 minute job.

    When the "wait-job" is put in parallel, some fail and some succeed (no indication as to why). I am curious what causes this behavior. Below fails when wait-job foreach is -parallel but works if not parallel:

    $list = @(1,3,4,7,8,9)
    workflow async{
    Param($list)
        "Start"
        foreach -parallel ($n in $list){
            Start-Job -Name "test$n" -FilePath .\wait.ps1 -ArgumentList $n
        }
        foreach -parallel ($n in $list){
            Wait-Job -Name "test$n"
        }
    }
    async -list $list
    "Done"


    Tuesday, August 8, 2017 2:14 AM
  • Last time.  You cannot use a workflow and you cannot put the wait-job on the job.. Place it on the job collection.

    This is al you need to do.

    $list = 1, 3, 4, 7, 8, 9
    $jobs = @()
    foreach ($n in $list) {
    	$jobs += Start-Job -Name "test$n" -FilePath .\wait.ps1 -ArgumentList $n
    }
    $jobs | Wait-Job
    

    The script that you are running in the job must be deterministic.  It cannot just run a forever loop or the job will never complete.


    \_(ツ)_/

    Tuesday, August 8, 2017 2:41 AM
  • I looked into alternatives to Start-Job and found Invoke-Expression gave me the desired behavior. I am curious to the cause of the -parallel and job inconsistency, but Invoke-Expression will work. For reference I was looking for a means to duplicate of the behavior of https://caolan.github.io/async/docs.html#parallel

    Thanks for feedback.

    • Marked as answer by David..g Tuesday, August 8, 2017 5:52 PM
    Tuesday, August 8, 2017 5:51 PM
  • I looked into alternatives to Start-Job and found Invoke-Expression gave me the desired behavior. I am curious to the cause of the -parallel and job inconsistency, but Invoke-Expression will work. For reference I was looking for a means to duplicate of the behavior of https://caolan.github.io/async/docs.html#parallel

    Thanks for feedback.

    That link is for a non-PowerShell async call. It has nothing to do with what you were asking.  "Invoke-Expression" is not related to your question so it appears that you were not asking the question you thought you were asking.

    I recommend learning PowerShell  before attempting any advanced usage and definitely before trying to ask a complex question.


    \_(ツ)_/

    Tuesday, August 8, 2017 6:08 PM