Or perhaps that should be ‘by Dummies’…

In February (2018) Russ Cox, one of the people behind Go, published a series of blogs where he attempts to tackle one of the biggest problems in the Golang space right now. This is once again an attempt by me to understand what the bloody hell is going on in the world.

This blog is going to mostly rehash what he’s done, peppered with bits that I’ve picked up on my travels around the internet. The purpose isn’t only for your entertainment, it’s to help fix things in my mind.

Source material:

What’s the problem

Let’s say you import a package github.com/me/pkg in your project. All is well until you decide to build your project on another machine and it no longer works. Your code didn’t change but running go get githhub.com/me/pkg got an entirely different and incompatible version of the package. There’s no way of specifying what version of the package you want. If the API changes or a bug is introduced, you have no protection. A change to github.com/sartori/go.uuid is a good example of this. This issue was raised following a commit that changed the return signature of some of the methods, breaking projects that imported the package. For a while there was no way of telling which version you had without looking at the code.

It gets worse when you consider that packages that your project imports, also import their own dependencies. Again you don’t control the version that go downloads for you.

And what happens if your dependencies import the same packages you do? Let’s take go.uuid as an example again. What if my project imports go.uuid and another import does too. If I found the API change and updated my code, what about other imported packages? I have to wait for the maintainers to discover the problem and fix it when they get around to it - if they ever do!

Quick note on versioning

Believe it or not, there is a standard for what a version means. It’s called semantic versioning. In a nutshell, your version is defined as vX.Y.Z. Let’s work through this from lowest to highest importance:

Z -> Patch Version – change this to indicate a small changes such as bug fixes

Y -> Minor Version – change this for making backwards compatible changes such as adding new features

X -> Major Version – change this when you make incompatible API changes

What was done to solve it

Gopkg.in

Available via at http://gopkg.in, this service allows package developers to manage versions of their APIs. This service acts as a façade in front of Github, allowing developers to go get specific major versions of the package that you’d like to import. The Gopkg service will work out what the latest release by parsing git tags, making sure that go get -u will update your dependency.

This was one of the earliest solutions to the package versioning problem, but it never seemed to catch on. My feeling is that it should have been more popular than it is. As good as it is, there are a few problems with this approach:

You can only use Github to host your code, so no private Bitbucket or corporate repos
You can only get the latest release of a major version - no rolling back if you find a problem
Your dependency relies on 2 external services and not just Github

Lots of tools

Tools started to be developed to help the situation. There’s a fairly comprehensive list here. By and large these tools allowed you to specify which version of a dependency you would like to download, even down to the specific commit. Some work by cloning repositories in to your GOPATH, others acutally provided full build solutions.

Some of the interesting ones were:

glide was for a long time the gold standard in Go package management, using a file to specify the versions of each dependency
Godep saves the dependencies that you’ve already downloaded to allow you to commit the vendor directory
Govendor can fetch individual using the command line parameters and save them in vendor again

There is also an honourable mention for gb which created a seperate vendor directory alongside src in GOPATH but also performed the build.

All of these tools attempted to solve the problem of package versioning in slightly different ways, with varying degrees of success. Ultimately though none of them were officially supported by the Go teama and suffered because of it.

Vendor directory

Concensus had started to form around the need of the vendor directory. So in version 1.5 of Go brought official experimental support for the the vendor directory by setting the environment variable GO15VENDOREXPERIMENT to 1. This gives a good background but the essential bit is go would now prefer the vendor directory when resolving dependencies. Dependencies would be looked up as though ./vendor were the same as $GOPATH/src. Packages that had their own vendor directory was still a problem that you had to rely on tools to solve at this point.

At version 1.7, the experiment was over and the vendor directory no longer sat behind an environment variable. Subsequent versions improved support by doing things like making sure that go test ./... didn’t navigate vendor.

The intention of the Go team was that making this directory available would allow the community to come to a concensus around the right solution but this didn’t appear.

Dep

It became clear that just expecting a major change in functionality to spontaneously arrive from the community wasn’t going to work. They decided to put together a crack team to tackle the problem. The people that they chose had made important contributions, some being the authors of the packages managers above. The intention was that whatever tooling they came up with would, eventually, become a first class supported part of Go tooling.

They quickly arrived at the decision to create a new package management tool from scratch, learning all of the lessons of the tool that when before them. This tool was called dep. It allowed you to specify dependencies in the Gopkg.toml and Gopkg.lock which could be committed along with the contents of vendor if you so wish. It’s a great tool, it works out what your dependencies are and which versions are compatible.

But it can be slow. dep ensure -v will show you what it’s doing, sometimes that means it will clone a repository, build the latest tag, watch it fail and then work down the tags until something works. Sometimes not even this works.

Why won’t these work?

There’s one problem with all of these approaches so far haven’t dealt with. What happens when you have 2 dependencies that rely on different versions of the same sub-dependency? Ordinarily this isn’t a problem, but what if this makes part of the API that you’re consuming?

RSC’s solution

Russ essentially went back to basics and perhaps a little bit back in time and remembered an early proposition by the Go team that has been called the import compatibility rule. Russ has a great description on the Go Blog.

“If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.”

Is there anything more to say than this? Perhaps there is…

What it means is that if you want to make breaking API changes, you need to contrive a new import path. This becomes what he calls semantic import versioning. What this means is that if you have version 1 of your package, which in turn means putting the version of the API in the import path. github.com/my/pkg becomes github.com/my/pkg/v1. Following the semantic versioning standard means that API compatibility is indicated using the Major version, which we use in the path.

When he did a little thinking about this Russ discovered a couple of interesting implications:

If package developers keep to the semver standard, you can be confident that when you import something it will be compatible with EVERYTHING in your GOPATH
If you decide to upgrade a package, it won’t surprise you by breaking your build, or business logic

Russ’s experience with Dep showed the algorithm used to resolve package versions could be very slow. He has developed a new algorithm, minimal version selection that used the concepts from above. It proved MUCH faster to resolve versions in the early prototype. It has the added benefit that it can use a single configuration file that can be edited both by humans and tools.

An fuller prototype of a version of the Go tool implementing all of this has been written, vgo.

Blog

The New Go Package Versioning for Dummies

Pascal Dennerly