Go Modules: finding out right pseudo-version (vX.Y.Z-<timestamp>-<commit>) of required package

Question

I am trying out Go modules. My project requires the libarary golang.org/x/net/html, so I defined this go.mod file:

module github.com/patrickbucher/prettyprint

require golang.org/x/net/html

And wrote this demo program to check if the dependency gets loaded upon compilation:

package main

import (
        "fmt"
        "log"
        "os"

        "golang.org/x/net/html"
)

func main() {
        doc, err := html.Parse(os.Stdin)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(doc)
}

When I run go build, I get this error message:

go: errors parsing go.mod:
~/prettyprint/go.mod:3: usage: require module/path v1.2.3

Obviously, I missed the version number. But which one to take? I stumbled an article called Takig Go Modules for a Spin, where I found an example of a go.mod file containing references to golang.org/x packages:

module github.com/davecheney/httpstat

require (
        github.com/fatih/color v1.5.0
        github.com/mattn/go-colorable v0.0.9
        github.com/mattn/go-isatty v0.0.3
        golang.org/x/net v0.0.0-20170922011244-0744d001aa84
        golang.org/x/sys v0.0.0-20170922123423-429f518978ab
        golang.org/x/text v0.0.0-20170915090833-1cbadb444a80
)

The author is using version strings like v0.0.0-20170922011244-0744d001aa84, consisting of the semver indication v0.0.0, a timestamp and something that looks like a git commit ID.

How do I figure out those version strings? I guess those golang.org/x packages will be versioned according to semantic versioning at some point, but to really trying out go mod, I need to figure out those now.

Solution 1

A version of the form v0.0.0-20180906233101-161cd47e91fd means that there are no tagged versions on the git repository. So go mod generates one based on the latest commit time and the prefix of the commit hash.

To get a correct go.mod file start out using the following command (assuming go 1.11):

go mod init yourmodulename

Or create an empty go.mod file that just contains the following:

module yourmodulename

then run go mod tidy, this will find all dependencies, add the missing and remove the unused dependencies.

Solution 2

Update

This command is the better solution to adding the replace command to go.mod rather than doing it manually using git that I initially posted:

go mod edit -replace github.com/docker/docker=github.com/docker/engine@ea84732a7725

produces a similar result but instead of using a pseudo-version, it finds the tagged engine version.

replace github.com/docker/docker => github.com/docker/engine v17.12.0-ce-rc1.0.20191113042239-ea84732a7725+incompatible

Alternatively, include a tagged docker version.

go mod edit -replace github.com/docker/[email protected]=github.com/docker/engine@ea84732a7725

for

replace github.com/docker/docker v1.13.1 => github.com/docker/engine v17.12.0-ce-rc1.0.20191113042239-ea84732a7725+incompatible

Thanks to @Shivam010 on Medium

---

Original, deprecated answer

Here's how I did it.

Checkout the repository on the desired branch/tag. e.g.

git clone -b v19.03.5 [email protected]:docker/engine.git

Then

cd engine
TZ=UTC git --no-pager show \
  --quiet \
  --abbrev=12 \
  --date='format-local:%Y%m%d%H%M%S' \
  --format="%cd-%h"

And i get

20191113042239-ea84732a7725

For use in go.mod as

replace github.com/docker/docker v1.13.1 => github.com/docker/engine v0.0.0-20191113042239-ea84732a7725

Solution 3

The author is using version strings like v0.0.0-20170922011244-0744d001aa84, consisting of the semver indication v0.0.0, a timestamp and something that looks like a git commit ID.

How do I figure out those version strings?

You never need to manually figure out those complex version strings, which are called pseudo-versions.

Daily workflow

Your typical daily workflow can be:

• Add import statements to your .go code as needed.
• Standard commands like go build, go test, or go mod tidy will automatically add new dependencies as needed to satisfy imports (updating go.mod and downloading the new dependencies). By default, the @latest version of a new direct dependency will be used.
• When needed, more specific versions of dependencies can be chosen with commands such as:
  • go get [email protected]
  • go get foo@e3702bed2
  • go get foo@latest
  • go get foo@branch
  • or by editing go.mod directly.

Note that you did not need to come up with a pseudo-version on your own in any of those examples, even when requesting a specific commit (e.g., @e3702bed2), or the latest commit on a branch (e.g., @master).

When do I see pseudo-versions in my go.mod?

If you end up with a version that resolves to a valid semver tag with a leading v such as v1.2.3 or v1.2.4-beta-1, then that semver tag will be recorded in your go.mod file. If the version does not have a valid semver tag, then it will be instead recorded with as a pseudo-version in your go.mod file, such as v0.0.0-20171006230638-a6e239ea1c69, which includes a version section, a commit timestamp, and a commit hash.

In your particular case, golang.org/x/net/html does not have any semver tags, which means if you do go get golang.org/x/net/html@latest, or go get golang.org/x/net/html@0744d001aa84, or just do go build after first including import "golang.org/x/net/html" in your .go file, then golang.org/x/net/html will be recorded in your go.mod as a pseudo-version, but note that you did not need to figure out the complex string yourself (because the go command translates modules queries such as go get golang.org/x/net/html@0744d001aa84 into the corresponding pseudo-version when needed, and records the result in your go.mod).

Why was the pseudo-version format chosen?

The pseudo-version format helps provide a simple total ordering across all versions based on standard semver ordering, which makes it easier to reason about what commit will be considered "later" than another commit, or whether an actual semver tag is considered "later" than a separate commit.

Controlling dependency versions

You can read more about all of the above in the "How to Upgrade and Downgrade Dependencies" section of the Go Modules wiki, which also contains additional links into the official documentation.

Solution 4

Now I read a bit further in the documentation (go help modules) and stumbled upon go mod tidy:

The 'go mod tidy' command builds that view and then adds any missing module requirements and removes unnecessary ones.

So when I leave away the requirement on golang.org/x/net/html and prune my go.mod file to this:

module github.com/patrickbucher/prettyprint

And then run go mod tidy, then the requirement with the version number is correctly figured out based on the import path in my source code, and thus go.mod becoming:

module github.com/patrickbucher/prettyprint

require golang.org/x/net v0.0.0-20180906233101-161cd47e91fd

Now both go list and go build work.

Solution 5

If you want to use a specific commit which has not yet been tagged, you could do the following -

module github.com/patrickbucher/prettyprint

require golang.org/x/net 73496e0df0ba4284f460d1955ddf6bb096957c9f

then run go mod tidy and you will automatically see the pseudo-version in the go.mod file, becoming

module github.com/patrickbucher/prettyprint

require golang.org/x/net v0.0.0-20180906233101-161cd47e91fd

(I borrowed the code snippet from Patrick's response above)

View more solutions

Comments

[thepudds]

Minor side note: you don't necessarily need to run go mod tidy to have the go command automatically determine the corresponding pseudo-version in this case. Other commands like go build, go test, etc. would also have recorded the version. Some more details in this answer. go mod tidy has its uses, but sometimes people think they need to run it more often than strictly needed. In any event, still valid to do so.

[thepudds]

FWIW, it is fairly dangerous to try to format this yourself, and you can get it subtly wrong in a way that bites you later. Much better to let the 'go' command do it for you, as described in stackoverflow.com/a/57315225/11210494

[SteveCoffman]

This is very useful! If you want to Inject build-time variables with Golang into a binary then this is as good a way to figure it out as I've found!

[Jacob]

@typical182 Can you please test github.com/docker/[email protected] with go get? I'd serious love to know how to use go get to create the psuedo version for that package. I recommend verifying that it works. Just running what you'd expect, go get github.com/docker/docker/[email protected], does not work since but does not contain package github.com/docker/docker/engine. If any of the go commands I tried had worked before posting this, i would not have shared this, i absolutely would have preferred using go. If you have a solution using go get, please share!

[thepudds]

Docker is a complex case. It probably makes sense for you to open a new question? If you comment here with a link, I will try to answer there.

[nckturner]

What if those pseudo versions are in replace directives, which is required in a project importing Kubernetes dependencies (I think)?

[Dave C]

Note that as of Go1.16 build commands such as go build and go test behave as if -mod=readonly was used and will no longer automatically update go.mod (as mentioned in your second bullet point of Daily Workflow). You can use -mod=mod to allow the command to modify go.mod as before. (The GOFLAGS environment variable can be used to set default flags to Go commands if desired).

[Dave C]

The canonical way to add such a require line to go.mod is via go get, e.g. go get golang.org/x/net@161cd47e91fd (or just go get golang.org/net for @latest).