locked
Power Shell Foreach loop with imported file pausing / stopping for long periods RRS feed

  • Question

  • Ok i have this simple code here. The purpose is to load a CSV and generate web traffic for some lab environments i have. There seems to be no reason / rhyme as to why it just stops, sometimes forever and others for small gaps 5-10 mins and just picks back up. I have tried smaller and larger files, Running it as administrator with similar results.  Im pulling my hair out here.

    if there is a better way im also open to that this is all i have to go on.

     

    Sites = Import-csv C:\TrafficScipt\BatchExport.csv
    foreach ($i in $sites)
    {
    $url = $i.URL
        Invoke-WebRequest -uri $url -Method head | out-null
        Start-Sleep -s 1

    }

    Monday, February 17, 2020 7:57 PM

Answers

  • Your line of questions lead me to add an output field to the loop to see what site was stopping during the process. During this posing i would get a crap ton of names flying by on output then a pause. Sometimes up to a min -5mins. That lead me to add a timeout to the string / code you helped with above. Since i dont really care if the page loads or the output i added a 1 second timeout now this thing is really cooking! Adding the timeout to the web request allows it to bypass those long pauses. Although there is no reason for them i mean the site may not be there but the connection i think should have been closing on its own. 

    Adopted this script:

    import-csv C:\TrafficScipt\BatchExport.csv |
        ForEach-Object{
            Write-Output "Loading:  " $_.url
            Invoke-WebRequest -TimeoutSec 2 -uri $_.url -Method head | out-null
            
        }

    w55c.net
    Invoke-WebRequest : The operation has timed out.
    At line:4 char:9
    +         Invoke-WebRequest -TimeoutSec 1 -uri $_.url -Method head | ou ...
    +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
        + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
     

    • Marked as answer by Rich369 Monday, February 17, 2020 10:50 PM
    Monday, February 17, 2020 10:50 PM

All replies

  • Use a workflow to  parallelize the calls.


    \_(ツ)_/

    Monday, February 17, 2020 8:10 PM
  • The following would also be better:

    Import-csv C:\TrafficScipt\BatchExport.csv |
        ForEach-Object{
            Invoke-WebRequest -uri $_.url -Method head | out-null
        }
    

    With PowerShell 7 you can do this:

    Import-csv C:\TrafficScipt\BatchExport.csv |
        ForEach-Object -Parallel {
            Invoke-WebRequest -uri $_.url -Method head | out-null
        }

    See: https://docs.microsoft.com/ja-jp/PowerShell/module/microsoft.powershell.core/foreach-object?view=powershell-7



    \_(ツ)_/

    Monday, February 17, 2020 8:13 PM
  • For workflow:

    workflow GetUril{
        $csv = Import-csv C:\TrafficScipt\BatchExport.csv
        foreach -Parallel($line in $csv){
            Invoke-WebRequest -uri $line.URL -Method head | out-null
        }
    }


    \_(ツ)_/

    Monday, February 17, 2020 8:17 PM
  • JRV makes a good point.

    Are those web requests actually doing anything? Do you know if the contents of the CSV contain valid URLs? If a URL points to a dead web site (but the URL can be resolved in DNS) the connection has to time out before the web request ends.

    Try something like this so you'll at least be able to see what's happening:

    Import-csv C:\TrafficScipt\BatchExport.csv |
        foreach {
            $uri = $_.uri
            Try{
                Invoke-WebRequest -uri $uri -Method head -ErrorAction STOP  | out-null
            }
            Catch{
                Write-host "Failed to connect to $uri"
                Write-Host $_
            }
        Start-Sleep -s 1
    }


    --- Rich Matheisen MCSE&I, Exchange Ex-MVP (16 years)

    Monday, February 17, 2020 8:17 PM
  • The long stops are likely due to unresponsive server.  You cannot alter that condition.  Adding error handling will not prevent the delays.  How you handle this depends on he purpose for this code.


    \_(ツ)_/

    Monday, February 17, 2020 8:29 PM
  • I wasn't intending to eliminate the delays, just to point out that timeouts might be the problem, and to at least indicate which URLs are the problem (not all errors will be time outs, though).

    --- Rich Matheisen MCSE&I, Exchange Ex-MVP (16 years)

    Monday, February 17, 2020 8:33 PM
  • I wasn't intending to eliminate the delays, just to point out that timeouts might be the problem, and to at least indicate which URLs are the problem (not all errors will be time outs, though).

    --- Rich Matheisen MCSE&I, Exchange Ex-MVP (16 years)

    Understood and clarified.


    \_(ツ)_/

    Monday, February 17, 2020 9:07 PM
  • Ill see if can catch anything with that. The really odd thing is i do get output even with the suppression enabled. I see mostly things like cannot reach server not allowed and forbidden. So i know its working and doesnt seem to hang on those types. 

    The real thing that bothers me is i can see this hitting the firewall im behind. Its not triggering anything other than the url blocker and that also causes an error output during the loops but all of them keep going. I have tried various delays in sleep. What bugs me is it can get through the files, they are only 200 or so records. Just sometimes it will do it others it wont. Drives me crazy same list each time same order just stops.

    Monday, February 17, 2020 10:17 PM
  • Thanks for this it does seem a bit more efficient. I will look at updating to PS7 i dont do a whole lot with this.

    Monday, February 17, 2020 10:44 PM
  • Your line of questions lead me to add an output field to the loop to see what site was stopping during the process. During this posing i would get a crap ton of names flying by on output then a pause. Sometimes up to a min -5mins. That lead me to add a timeout to the string / code you helped with above. Since i dont really care if the page loads or the output i added a 1 second timeout now this thing is really cooking! Adding the timeout to the web request allows it to bypass those long pauses. Although there is no reason for them i mean the site may not be there but the connection i think should have been closing on its own. 

    Adopted this script:

    import-csv C:\TrafficScipt\BatchExport.csv |
        ForEach-Object{
            Write-Output "Loading:  " $_.url
            Invoke-WebRequest -TimeoutSec 2 -uri $_.url -Method head | out-null
            
        }

    w55c.net
    Invoke-WebRequest : The operation has timed out.
    At line:4 char:9
    +         Invoke-WebRequest -TimeoutSec 1 -uri $_.url -Method head | ou ...
    +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
        + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
     

    • Marked as answer by Rich369 Monday, February 17, 2020 10:50 PM
    Monday, February 17, 2020 10:50 PM