gradle - download and unzip file from url

Question

What would be the proper gradle way of downloading and unzipping the file from url (http)?

If possible, I'd like to prevent re-downloading each time I run the task (in ant.get can be achieved by skipexisting: 'true').

My current solution would be:

task foo {
  ant.get(src: 'http://.../file.zip', dest: 'somedir', skipexisting: 'true')
  ant.unzip(src: 'somedir' + '/file.zip', dest: 'unpackdir')
}

still, I'd expect ant-free solution. Any chance to achieve that?

Solution 1

There isn't currently a Gradle API for downloading from a URL. You can implement this using Ant, Groovy, or, if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL. The unzipping can be done in the usual Gradle way (copy method or Copy task).

Solution 2

Let's say you want to download this zip file as a dependency:

https://github.com/jmeter-gradle-plugin/jmeter-gradle-plugin/archive/1.0.3.zip

You define your ivy repo as:

repositories {
    ivy {
        url 'https://github.com/'

        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }

        // This is required in Gradle 6.0+ as metadata file (ivy.xml) 
        // is mandatory. Docs linked below this code section
        metadataSources { artifact() } 
    }
}

reference for required metadata here

The dependency can then be used as:

dependencies {
    compile 'jmeter-gradle-plugin:jmeter-gradle-plugin:1.0.3@zip'
    //This maps to the pattern: [organisation]:[module]:[revision]:[classifier]@[ext]         
}

To unzip:

task unzip(type: Copy) {

  def zipPath = project.configurations.compile.find {it.name.startsWith("jmeter") }
  println zipPath
  def zipFile = file(zipPath)
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir

}

---

optional:

If you have more than one repository in your project, it may also help (for build time and somewhat security) to restrict dependency search with relevant repositories.

Gradle 6.2+:

repositories {
    mavenCentral()
    def github = ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
    }
    exclusiveContent {
        forRepositories(github)
        filter { includeGroup("jmeter-gradle-plugin") }
    }
}

Earlier gradle versions:

repositories {
    mavenCentral {
        content { excludeGroup("jmeter-gradle-plugin") }
    }
    ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
        content { includeGroup("jmeter-gradle-plugin") }
    }
}

Solution 3

    plugins {
        id 'de.undercouch.download' version '4.0.0'
    }

    /**
     * The following two tasks download a ZIP file and extract its
     * contents to the build directory
     */
    task downloadZipFile(type: Download) {
        src 'https://github.com/gradle-download-task/archive/1.0.zip'
        dest new File(buildDir, '1.0.zip')
    }

    task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
        from zipTree(downloadZipFile.dest)
        into buildDir
    }

https://github.com/michel-kraemer/gradle-download-task

Solution 4

Unzipping using the copy task works like this:

task unzip(type: Copy) {
  def zipFile = file('src/dists/dist.zip')
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir
}

http://mrhaki.blogspot.de/2012/06/gradle-goodness-unpacking-archive.html

Solution 5

This works with Gradle 5 (tested with 5.5.1):

task download {
    doLast {
        def f = new File('file_path')
        new URL('url').withInputStream{ i -> f.withOutputStream{ it << i }}
    }
}

Calling gradle download downloads the file from url to file_path.

You can use the other methods from other answers to unzip the file if necessary.

View more solutions

Comments

[Peter Niederwieser]

Don't forget to wrap the execution part of a task with doLast { ... } (same mistake as in your previous question).

[CMPS]

~~BOUNTY~~ Can anyone provide an example for the answer below: "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ?

[koppor]

netflix has released some plugin: github.com/nebula-plugins/nebula-core - not sure, why this is not maintained anymore.

[xpmatteo]

I would appreciate an example... I'm a total gradle newbie!

[CMPS]

~~BOUNTY~~ Can anyone provide an example for this answer "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ?

[RaGe]

@CMPS why you no groovy?

[CMPS]

@RaGe my current solution is using groovy and it works fine, but I wanted one using repository to benefit from the caching system.

[RaGe]

@CMPS Ah, caching! Posted an answer below.

[addy]

@RaGe how does 1.0.3:class@zip get matched to [revision].[ext] ?

[RaGe]

@addy class in that string is a classifier. It is optional, and here it isn't being used in the ivy url at all. 1.0.3 is revision and zip is ext.

[peuhse]

Is this for android or java? I'm trying to use it but it seems like "project.configuration.compile" does not exist? I've tried to add apply java before in the script, but that does not make a difference. Can't find it in the documentation either. Using com.android.tools.build:gradle:2.1.0 and version 2.10 of gradle.

[RaGe]

@peuhse it should be project.configurations, mind the s in configurations. The android plugin implicitly adds the java plugin to a project. It shouldn't matter if you add it explicitly again.

[peuhse]

It was a spelling mistake by me. When I write project.configurations and let the IDE autofill the possibilites the compile isn't one of them. And then I get a build error. Commenting out the line makes it build again.

[RaGe]

please raise a new question, and refer to this one.

[Akom]

Unfortunately URLs with query strings cannot be accessed this way, '?' is encoded by github.com/gradle/gradle/blob/master/subprojects/resources/s‌​rc/…

[Sourav Bhattacharjee]

When I am trying this rather than looking for .zip file it is looking for a .xml file. Any way around for this? I followed exactly the same instruction.

[Victor Lyuboslavsky]

Doesn't work for me. Gradle is looking for an ivy.xml file in addition to the artifact.

[dexter2305]

I tried moving the from zipTree() & into outputDir to the execution phase by moving them to doLast { }. What I did not understand was that it never worked. Even the println code in doLast did not work. However, this code works in the gradle configuration phase. Why?

[barfuin]

Unfortunately, this solution also has Gradle looking for an Ivy XML file.

[Michael S.]

This is not really a good solution compared to Ant as it is really slow. 26 minutes download instead of 2 minutes with Chrome.

[Vampire]

Because it configures the task. The execution still is done during execution phase. Changing the configuration during the execution phase should not be done actually and in your case the execution phase would not be executed at all, as the task thinks there is nothing to copy, so the task execution is skipped. But actually this answer is not related to the question anyway, as this does not work with an HTTP URL.

[Abhijit Sarkar]

Does work, as others have stated, Gradle looks for .xml file.

[Andy Brown]

This is a great answer. I would only add that the dependency should probably be compileOnly to prevent the raw zip ending up in the runtime artifacts.

[camikiller]

Also works in gradle 4.7. In case someone needs it.

[Greg Domjan]

gradle 7 now changing configurations - 'compile' removed. adding a configuration in replacement or updating to the new names still worked.

[rios0rios0]

Doesn't work. Gradle 7.2 is looking for a normal maven dependency.

[rios0rios0]

Doesn't work. Gradle 7.2 is looking for a normal maven dependency.

[RaGe]

@rios0rios0 I'd expect gradle to look in all available repositories for all dependencies. If you want to restrict certain deps to certain repos, see the optional section in the answer

[elect]

I can confirm this works with 7.2, @rios0rios0 have you resolved?

[rios0rios0]

@elect I solved it in another way. Because I spent a lot of time trying to work with this approach. Community on Gradle's Slack said a better way to handle my original issue. But thanks for confirming that's working. Here Gradle wasn't recognizing the pattern "/[organisation]/[module]/archive/[revision].[ext]" on dependencies.