Download Multiple Files from http using Powershell with proper names

Question

I have searched for something similar and I keep running across the FTP download answers. This is helpful information, but ultimately proving to be difficult to translate. I have found a powershell script and it works, but I am wondering if it can be tweaked for my needs. I don't have much experience with powershell scripting, but I'm trying to learn.

The need is this. I need to download and install a series of files to a remote machine, unattended. The files are distributed via email via tinyurls. I currently throw those into a .txt file, then have a powershell script read the list and download each file.

Requirements of the project and why I have turned to powershell (and not other utilities), is that these are very specialized machines. The only tools available are ones that are baked into Windows 7 embedded.

The difficulties I run into are: The files download one at the time. I would like to grab as many downloads at the same time that the web server will allow. (usually 6)

The current script creates file names based off the tinyurl. I need the actual file name from the webserver.

Thanks in advance for any suggestions.

Below is the script I’m currently using.

#  Copyright (C) 2011 by David Wright ([email protected])
#  All Rights Reserved.

#  Redistribution and use in source and binary forms, with or without
#  modification or permission, are permitted.

#  Additional information available at http://www.digitalwindfire.com.

$folder = "d:\downloads\"
$userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1"
$web = New-Object System.Net.WebClient
$web.Headers.Add("user-agent", $userAgent)


Get-Content "d:\downloads\files.txt" |
    Foreach-Object { 
        "Downloading " + $_
        try {
            $target = join-path $folder ([io.path]::getfilename($_))
            $web.DownloadFile($_, $target)
        } catch {
            $_.Exception.Message
        }
}

Solution

If you do the web request before you decide on file name you should be able to get the expanded path (otherwise you would have to make two web requests, one to get the extended path and one to download the file).

When I tried this, I found that the BaseResponse property of the Microsoft.PowerShell.Commands.HtmlWebResponseObject returned by the Invoke-WebRequest cmdlet had a ResponseUri property which was the extended path we are looking for.

If you get the correct response, just save the file using the name from the extended path, something like the following (this sample code does not look at HTTP response codes or similar, but expects everything to go well):

function Save-TinyUrlFile
{
    PARAM (
        $TinyUrl,
        $DestinationFolder
    )

    $response = Invoke-WebRequest -Uri $TinyUrl
    $filename = [System.IO.Path]::GetFileName($response.BaseResponse.ResponseUri.OriginalString)
    $filepath = [System.IO.Path]::Combine($DestinationFolder, $filename)
    try
    {
        $filestream = [System.IO.File]::Create($filepath)
        $response.RawContentStream.WriteTo($filestream)
        $filestream.Close()
    }
    finally
    {
        if ($filestream)
        {
            $filestream.Dispose();
        }
    }
}

This method could be called using something like the following, given that the $HOME\Documents\Temp folder exists:

Save-TinyUrlFile -TinyUrl http://tinyurl.com/ojt3lgz -DestinationFolder $HOME\Documents\Temp

On my computer, that saves a file called robots.txt, taken from a github repository, to my computer.

If you want to download many files at the same time, you could let PowerShell make this happen for you. Either use PowerShell workflows parallel functionality or simply start a Job for each url. Here's a sample on how you could do it using PowerShell Jobs:

Get-Content files.txt | Foreach {
    Start-Job {
        function Save-TinyUrlFile
        {
            PARAM (
                $TinyUrl,
                $DestinationFolder
            )

            $response = Invoke-WebRequest -Uri $TinyUrl
            $filename = [System.IO.Path]::GetFileName($response.BaseResponse.ResponseUri.OriginalString)
            $filepath = [System.IO.Path]::Combine($DestinationFolder, $filename)
            try
            {
                $filestream = [System.IO.File]::Create($filepath)
                $response.RawContentStream.WriteTo($filestream)
                $filestream.Close()
            }
            finally
            {
                if ($filestream)
                {
                    $filestream.Dispose();
                }
            }
        }

        Save-TinyUrlFile -TinyUrl $args[0] -DestinationFolder $args[1]
    } -ArgumentList $_, "$HOME\documents\temp"
}

Comments

[user3590927]

Robert, thank you very much. This is exactly what I was looking for. One more question to add to this. Is there a way to add a progress indicator? These are fairly large files and I don't want someone to think that the program died. I have tried adding $buffer and $count arguments and the script works, but does not show progress '

[Robert Westerlund]

When you run each download as a separate Job, you run them in a different Runspaces and will have to do some work to get it to show progress. I suppose you could write progress information to output in the jobs and have the script which started the jobs read from the output streams continously and updating a progress bar. Perhaps there are simpler ways? How to do it, however, is a completely separate question, so I suggest you write a new question regarding that, if you run into problems in your own attempts at it.

[Joshua Nurczyk]

You can possibly use write-Progress to do this. See: blogs.technet.com/b/heyscriptingguy/archive/2011/01/29/…

[Robert Westerlund]

@JoshuaNurczyk You can't use Write-Progress from a Job since it runs in a separate Runspace. If you periodically Write-Output the current state in the Job the script which started the Job should be able to read this output and turn that into Write-Progress.

[Robert Westerlund]

@JoshuaNurczyk @user3590927 Here's a quickly written gist sample of writing output from a job and collecting it to Write-Progress.