none
Invoke-WebRequest opening Browser, even with -UseBasicParsing RRS feed

  • General discussion

  • Hello all,

    We have a couple SharePoint farms (Win2012R2, PowerShell v4.0), for which we have implemented a warmup script.  Anyone familiar with SharePoint (or ASP.NET in general) will understand what this is.  We use SPBestWarmUp (https://github.com/spjeff/spbestwarmup).

    I have made a few minor tweaks to the script to accommodate our round-robin load balancing proxies and make sure that all servers in the farm are fully "warmed up".  It works great.

    However, recently (~3-4 weeks) we started having issues with Scheduled Tasks starting; not just the warmup script, but any Scheduled Task on the server.  The result code led me to determine that the DesktopHeap memory was filling up (https://blogs.msdn.microsoft.com/ntdebugging/2007/01/04/desktop-heap-overview/). Digging further, I discovered that the warmup script was leaving Internet Explorer processes running, eventually filling the DesktopHeap memory.

    To troubleshoot, I ran it manually.  It turns out that, for whatever reason, about 10 (of ~200) URLs/pages are spawning IE windows, which never happened when I initially tested / configured the script (4-5 months ago).  It seems to be since a recent set of Windows Updates, which also included SharePoint updates.  So, it is difficult to tell whether it's a change in how SharePoint sends the Response, or how PowerShell handles the Response.  Either way, in a Scheduled Task, having orphaned IE session running is a problem.

    Here are the relevant functions that handle the "warmup":

    Function NavigateTo([string] $url) {
        if ($url.ToUpper().StartsWith("HTTP") -and !$url.EndsWith("/ProfileService.svc", "CurrentCultureIgnoreCase")) {
            WriteLog "  $url" -NoNewLine
            # WebRequest command line
            try {
                $wr = Invoke-WebRequest -Uri $url -UseBasicParsing -UseDefaultCredentials -TimeoutSec 120
                FetchResources $url $wr.Images
                FetchResources $url $wr.Scripts
                Write-Host "."
            }
            catch {
                $httpCode = $_.Exception.Response.StatusCode.Value__
                if ($httpCode) {
                    WriteLog "   [$httpCode]" Yellow
                }
                else {
                    Write-Host " "
                }
            }
        }
    }
    
    Function FetchResources($baseUrl, $resources) {
        # Download additional HTTP files
        [uri]$uri = $baseUrl
        $rootUrl = $uri.Scheme + "://" + $uri.Authority
    	
        # Loop
        $counter = 0
        foreach ($res in $resources) {
            # Support both abosolute and relative URLs
            $resUrl = $res.src
            if ($resUrl.ToUpper().Contains("HTTP")) {
                $fetchUrl = $res.src
            }
            else {
                if (!$resUrl.StartsWith("/")) {
                    $resUrl = "/" + $resUrl
                }
                $fetchUrl = $rootUrl + $resUrl
            }
    
            # Progress
            if (!$skipprogress) {
                Write-Progress -Activity "Opening " -Status $fetchUrl -PercentComplete (($counter / $resources.Count) * 100)
                $counter++
            }
    		
            # Execute
            Invoke-WebRequest -UseDefaultCredentials -UseBasicParsing -Uri $fetchUrl -TimeoutSec 120 | Out-Null
            Write-Host "." -NoNewLine
        }
        if (!$skipprogress) {
            Write-Progress -Activity "Completed" -Completed
        }
    }
    

    The Invoke-WebRequest lines are fairly typical for pretty much all warmup scripts out there.  The -UseDefaultCredentials option combined with the script running with our SP_Admin account give it access to all sites/collections/pages.  The -UseBasicParsing, as I understand it, should read in all of the HTML page content and not process the DOM, thus preventing IE from spawning.  In fact the Docs (https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-4.0) indicate it should be used for core installation where IE isn't installed; and in PowerShell 6.0, the option is deprecated and all requests are handled thusly.

    However, that isn't the case.  At least not entirely.  Strangely, when I manually execute the Invoke-WebRequest for the URLs that are spawning IE (when run manually, the script prints out URLs as they are processed), it doesn't seem to spawn the window.  However, I have not tried manually processing the resources yet (see Function FetchResources), so it may be something in there.  In any case, it shouldn't be spawning IE at all.

    As a stop-gap measure, I have inserted some code to kill the IE windows:

    Function NavigateTo([string] $url) {
        $ts = get-date
        if ($url.ToUpper().StartsWith("HTTP") -and !$url.EndsWith("/ProfileService.svc", "CurrentCultureIgnoreCase")) {
            WriteLog "  $url" -NoNewLine
            # WebRequest command line
            try {
                $wr = Invoke-WebRequest -Uri $url -UseBasicParsing -UseDefaultCredentials -TimeoutSec 2
                FetchResources $url $wr.Images
                FetchResources $url $wr.Scripts
                Write-Host "."
            }
            catch {
                $httpCode = $_.Exception.Response.StatusCode.Value__
                if ($httpCode) {
                    WriteLog "   [$httpCode]" Yellow
                }
                else {
                    Write-Host " "
                }
            }
        }
        
        Get-Process -IncludeUserName | where {$_.ProcessName -eq "iexplore" -and $_.UserName -match $env:USERNAME -and $_.StartTime -ge $ts} | Stop-Process -ErrorAction Continue
    }
    
    Function FetchResources($baseUrl, $resources) {
        # Download additional HTTP files
        [uri]$uri = $baseUrl
        $rootUrl = $uri.Scheme + "://" + $uri.Authority
    	
        # Loop
        $counter = 0
        foreach ($res in $resources) {
            # Support both abosolute and relative URLs
            $resUrl = $res.src
            if ($resUrl.ToUpper().Contains("HTTP")) {
                $fetchUrl = $res.src
            }
            else {
                if (!$resUrl.StartsWith("/")) {
                    $resUrl = "/" + $resUrl
                }
                $fetchUrl = $rootUrl + $resUrl
            }
    
            # Progress
            if (!$skipprogress) {
                Write-Progress -Activity "Opening " -Status $fetchUrl -PercentComplete (($counter / $resources.Count) * 100)
                $counter++
            }
    		
            # Execute
            Invoke-WebRequest -UseDefaultCredentials -UseBasicParsing -Uri $fetchUrl -TimeoutSec 2 | Out-Null
            Write-Host "." -NoNewLine
        }
        if (!$skipprogress) {
            Write-Progress -Activity "Completed" -Completed
        }
    }
    

    By grabbing the date/time when entering Function NavigateTo, I can kill any IE windows that were opened just before exiting the function.  It is not elegant, and it could indeed kill IE processes in the interactive desktop, but I couldn't come up with anything better.  Since Invoke-WebRequest is not intended to open an IE window, it doesn't have access to/knowledge of PIDs of any that do open, so I cannot kill the process based its ID.

    So, finally, to my question: 

         Is there a way to fully prevent Invoke-WebRequest from spawning IE under any condition?

    Thanks,

    Eric

    Monday, October 1, 2018 8:32 AM

All replies

  • It's a good and specific question.

    However, it sounds to me like a break/fix issue. If the code is working as designed but has side-effects not before seen after some update, then it seems the issue isn't the script.

    I'm not sure how we would be able to help. You may need to open a ticket with Microsoft.


    -- Bill Stewart [Bill_Stewart]

    Monday, October 1, 2018 2:17 PM
    Moderator