Downloading Docker Images from Docker Hub without using Docker

docker dockerhub skopeo p2iwd

103,712

Solution 1

It turned out that the Moby Project has a shell script on the Moby Github which can download images from Docker Hub in a format that can be imported into Docker:

download-frozen-image-v2.sh

The usage syntax for the script is given by the following:

download-frozen-image-v2.sh target_dir image[:tag][@digest] ...

The image can then be imported with tar and docker load:

tar -cC 'target_dir' . | docker load

To verify that the script works as expected, I downloaded an Ubuntu image from Docker Hub and loaded it into Docker:

user@host:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@host:~$ tar -cC 'ubuntu' . | docker load
user@host:~$ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

In practice I would have to first copy the data from the internet client (which does not have Docker installed) to the target/destination machine (which does have Docker installed):

user@nodocker:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@nodocker:~$ tar -C 'ubuntu' -cf 'ubuntu.tar' .
user@nodocker:~$ scp ubuntu.tar user@hasdocker:~

and then load and use the image on the target host:

user@hasdocker:~ docker load ubuntu.tar
user@hasdocker:~ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

Solution 2

There is a tool called Skopeo which can retrieve Docker images from a repository and save them in several formats.

For example:

Download the image and save the layers as a tarball: skopeo copy docker://ubuntu docker-archive:/tmp/ubuntu.tar:ubuntu
Transfer /tmp/ubuntu.tar to another machine if desired.
Load the image on a Docker instance which does not have internet connection: docker load --input /tmp/ubuntu.tar

It is available in CentOS 7 repo with the package name skopeo. There are no Debian or Ubuntu packages at this time (but it is easy to compile).

Solution 3

thanks for motivation. I made a powershell version of it. Check it out... With it you can move in dockerhub containers to a restricted docker networks with a windows desktop and an ssh-scp tool to docker machine without root or administrator rights

https://gitlab.com/Jancsoj78/dockerless_docker_downloader a new hacker tool :)

$image = "ubuntu"
$tag = "latest"
$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/"+$image+":pull"
$taguri = "https://registry-1.docker.io/v2/library/"+$image+"/manifests/"+$tag
$bloburi = "https://registry-1.docker.io/v2/library/"+$image+"/blobs/sha256:"

#token request
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token

#pull image manifest
$blobs = $($(Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $taguri | ConvertFrom-Json | Select -expand fsLayers ) -replace "sha256:" -replace "@{blobSum=" -replace "}")

#download blobs
for ($i=0; $i -lt $blobs.length; $i++) {
    $blobelement =$blobs[$i]

    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile blobtmp

    $source = "blobtmp"
    $newfile = "$blobelement.gz"

#overwrite
Copy-Item $source $newfile -Force -Recurse
#source blobs
ls *.gz
}
#postprocess
echo "copy these .gz to your docker machine"
echo "docker import .gz backward one by one"
echo "lastone with ubuntu:latest"
echo "after docker export and reimport to make a simple layer image"

Solution 4

Here is an adapted python script, thus having an OS independant solution: docker-drag

Use it like that, and it will create a TAR archive that you will be able to import using docker load :

python docker_pull.py hello-world
python docker_pull.py alpine:3.9
python docker_pull.py kalilinux/kali-linux-docker

Solution 5

I didn't really understand Jancsó's way of postprocessing, so I've spent some time to modify his script and here's what I came with:

https://github.com/meetyourturik/dockerless-docker-downloader

upd: apparently, 'a link only answer...' is something bad, so here's a whole script:

# Workaround for SelfSigned Cert an force TLS 1.2
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
        ServicePoint srvPoint, X509Certificate certificate,
        WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# use 'library/' prefix for 'official' images like postgres 
$image = "atlassian/jira-software" 
$tag = "8.13.2" 

$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${image}:pull" 
$taguri = "https://registry-1.docker.io/v2/${image}/manifests/${tag}"
$bloburi = "https://registry-1.docker.io/v2/${image}/blobs/" 

# generating folder to save image files 
$path = "$image$tag" -replace '[\\/":*?<>|]'
if (!(test-path $path)) { 
    New-Item -ItemType Directory -Force -Path $path 
} 

# token request 
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token 

# getting image manifest 
$headers = @{} 
$headers.add("Authorization", "Bearer $token") 
# this header is needed to get manifest in correct format: https://docs.docker.com/registry/spec/manifest-v2-2/ 
$headers.add("Accept", "application/vnd.docker.distribution.manifest.v2+json") 
$manifest = Invoke-Webrequest -Headers $headers -Method GET -Uri $taguri | ConvertFrom-Json 

# downloading config json 
$configSha = $manifest | Select -expand config | Select -expand digest 
$config = ".\$path\config.json" 
Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$configSha -OutFile $config 

# generating manifest.json 
$manifestJson = @{} 
$manifestJson.add("Config", "config.json") 
$manifestJson.add("RepoTags",@("${image}:${tag}")) 

# downloading layers 
$layers = $manifest | Select -expand layers | Select -expand digest 
$blobtmp = ".\$path\blobtmp" 

#downloading blobs 
$layersJson = @() 
foreach ($blobelement in $layers) { 
    # making so name doesnt start with 'sha256:' 
    $fileName = "$blobelement.gz" -replace 'sha256:' 
    $newfile = ".\$path\$fileName" 
    $layersJson += @($fileName) 

    # token expired after 5 minutes, so requesting new one for every blob just in case 
    $token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token 
    
    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile $blobtmp 
    
    Copy-Item $blobtmp $newfile -Force -Recurse 
} 

# removing temporary blob 
Remove-Item $blobtmp 

# saving manifest.json 
$manifestJson.add("Layers", $layersJson) 
ConvertTo-Json -Depth 5 -InputObject @($manifestJson) | Out-File -Encoding ascii ".\$path\manifest.json" 

# postprocessing
echo "copy generated folder to your docker machine" 
echo "tar -cvf imagename.tar *" 
echo "docker load < imagename.tar"

after sctipt downloads blobs and generates config and manifest jsons download it to docker machine and execute two following commands:

tar -cvf imagename.tar *
docker load < imagename.tar

first creates an archive, 2nd uploads image archive to docker

View more solutions

103,712

sea_1987

Updated on September 18, 2022

Comments

sea_1987 almost 2 years
I want to manually download a Docker Image from Docker Hub. More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed. I would have thought that this would be possible using the official API, but this does not appear to be the case - see the following discussion:
- Fetch docker images without docker command. e.g. with wget
Is it really the case that the API doesn't support downloading images? Is there a way to work around this?

UPDATE 1:

I came across the following ServerFault post:
- Downloading docker image for transfer to non-internet-connected machine
The accepted solution uses the docker save command, which doesn't help in my situation. But another solution posted there cites the following StackOverflow post:
- Pulling docker images
One of the solutions there refers to a command-line tool called docker-registry-debug which, among other things, can generate a curl command for downloading an image. Here is what I got:
```
user@host:~$ docker-registry-debug curlme docker ubuntu

# Reading user/passwd from env var "USER_CREDS"
# No password provided, disabling auth
# Getting token from https://index.docker.io
# Got registry endpoint from the server: https://registry-1.docker.io
# Got token: signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read
curl -i --location-trusted -I -X GET -H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" https://registry-1.docker.io/v1/images/ubuntu/layer

user@host:~$ curl \
-i --location-trusted -I -X GET \
-H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" 

https://registry-1.docker.io/v1/images/ubuntu/layer
HTTP/1.1 404 NOT FOUND
Server: gunicorn/18.0
Date: Wed, 29 Nov 2017 01:00:00 GMT
Expires: -1
Content-Type: application/json
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 29
X-Docker-Registry-Version: 0.8.15
X-Docker-Registry-Config: common
Strict-Transport-Security: max-age=31536000
```
So unfortunately it looks like the curl command generated does not work.

UPDATE 2:

It looks like I'm able to download layer blobs from Docker Hub. Here is how I'm currently going about it.

Get an authorization token:
```
user@host:~$ export TOKEN=\
"$(curl \
--silent \
--header 'GET' \
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/ubuntu:pull" \
| jq -r '.token' \
)"
```
Pull an image manifest:
```
user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq '.'
```
Pull an image manifest and extract the blob sums:
```
user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum'

sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e
sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3
sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375
sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f
sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb
```
Download a single layer blob and write it to a file:
```
user@host:~$ BLOBSUM=\
"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"

user@host:~$ curl \
--silent \
--location \
--request GET \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"
```
Write all of the blob sums to a file:
```
user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum' > ubuntu-blobsums.txt
```
Download all of the layer blobs from the manifest:
```
user@host:~$ while read BLOBSUM; do
curl \
--silent \
--location \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"; \
done < blobsums.txt
```
Now I have a bunch of layer blobs and I need to recombine them into an image - I think.

Related Links:
- Tensibai over 6 years
  
  "More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed." => What's the point to have the images on this machine then ? (easier workaround is using a pivot host, one where you acn docker pull from dockerhub and docker save/ docker push to an internal registry after)
- Tensibai over 6 years
  
  Did you had a look at the docker pull code? It sounds the way to go to build something like this from basic http calls
- sea_1987 over 6 years
  
  @Tensibai I think I figured it out. I also think I got a solution from the Docker community. I'll come back and post the solution later today.
- sea_1987 over 6 years
  
  @Tensibai I posted a solution with a shell script that solves the problem.
- Marged over 5 years
  
  Perhaps it would be easier to give the docker host Internet access by using the other as a proxy
- sea_1987 over 5 years
  
  @Marged That wasn't an option. The server was configured in such a way that it could not easily be used as a proxy.
sea_1987 over 6 years

The machine with internet connectivity does not and cannot have Docker installed. The question is asking for a way to download an image without using the Docker client. See my solution.
030 over 6 years

The machine with internet connectivity does not and cannot have Docker installed. but you apply docker load
sea_1987 over 6 years

@030 Just to test/demonstrate that the script works and that the download image data can be imported into Docker. In practice I would first have to copy the data to a machine with Docker installed.
030 over 6 years

Perhaps you could add that part for clarification
sea_1987 over 6 years

@030 I added an example session illustrating what the workflow would look like in practice.
Tensibai about 5 years

If you close your github account there's nothing left, if you're ok with sharing it, keep the link but paste the script here also please. You can edit your answer, paste your code and then select it and type ctrl+K or the {} (code) button in the editor top bar to format it.
ThePH about 5 years

I would love to paste the code here but it is 100 lines long and I don't think it will be readable. Nevertheless, you can fork the code to save your own copy of the script.
Tensibai about 5 years

That's not for me, that to have an answer which is self sustaining, if the link break, do you really think this may help someone reading this answer in a few months ? (btw the maximum size for an answer is 30k characters)
Chris M almost 4 years

FWIW there are deb packages now: github.com/containers/skopeo/blob/master/install.md
Todd Thelin about 3 years

You could put this in pip. I find it very helpful.
sea_1987 over 2 years

There doesn’t appear to be any code in that repository.
030 over 2 years

@igal version 0.1.0 has been released. It is capable to download docker images from a Nexus3 server and to upload them to another one.