Run parallel Invoke-WebRequest jobs in PowerShell v3

Question

Running simultaneous background jobs in PowerShell in pretty straightforward, but I cannot seem to get it working with the new (in v3) cmdlet Invoke-WebRequest.

I have several thousand files I am downloading scriptomagically via PowerShell. Works great, but doing them in series is taking days:

for($f=0;$f -lt $urlList.Count;$f++)
{
    $remote = $urlList[$f] + $fileList[$f]
    $local = 'C:\folder\' + $fileList[$f]
    Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox
}

I've made several attempts at leveraging the the 'AsJob' method, but either they error-out or complete fine yet no local files get saved. Here's an example of the latter:

for($f=0;$f -lt $urlList.Count;$f++)
{
    $remote = $urlList[$f] + $fileList[$f]
    $local = 'C:\folder\' + $fileList[$f]
    $command = "Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox"
    Start-Job {Invoke-Expression -Command $command}
}
Get-Job|Wait-Job

Output examples:

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command                  
--     ----            -------------   -----         -----------     --------             -------                  
339    Job339          BackgroundJob   Running       True            localhost            Invoke-Expression -Com...
341    Job341          BackgroundJob   Running       True            localhost            Invoke-Expression -Com...
343    Job343          BackgroundJob   Running       True            localhost            Invoke-Expression -Com...
339    Job355          BackgroundJob   Completed     True            localhost            Invoke-Expression -Com...
341    Job357          BackgroundJob   Completed     True            localhost            Invoke-Expression -Com...
343    Job359          BackgroundJob   Completed     True            localhost            Invoke-Expression -Com...

The weird thing is, the jobs above complete roughly as fast as it would take to download the associated file...so it seems that data is going somewhere -- just not to my save location. I assume it's going to memory but being dumped without being saved. Tried adding and removing the 'PassThru' option, but get the same results either way. Also tried piping it to Out-File, still no joy. Not sure why this particular cmdlet is being such a bugger.

Solution

You need to pass in the parameters in such as way that they actually make it into the job. In V3, you can use the $using: syntax to do this:

for($f=0;$f -lt $urlList.Count;$f++)
{
    $remote = $urlList[$f] + $fileList[$f]
    $local = 'C:\folder\' + $fileList[$f]
    Start-Job {Invoke-WebRequest $using:remote -Method Get -OutFile $using:local -UserAgent FireFox}
}
Get-Job|Wait-Job

BTW you don't need to use Invoke-Expression. Just put the code you want to run in a job inside a scriptblock i.e. a set of curly braces {...}.

Comments

[Local Needs]

That works great. Is this documented somewhere? I've read through both the about_jobs and about_jobs_details man pages and can't see any reference to the $using: syntax

[jbockle]

I found this information on $using : link