Evaluating Go's Package Management and Module Systems

This article was originally written by Ayooluwa Isaiah on the Honeybadger Developer Blog.

A package manager is a tool for automating the process of building your code, as well as downloading, updating, and removing project dependencies in a consistent manner. It can determine whether a specific version of a package is installed on a project, and then install or upgrade the package, typically from a remote host.

Package managers have been around for a long time. They were first used in operating systems and later used in programming environments. While there are differences between system-level and language-level package managers, there is also significant overlap in how they work.

In a language context, a package manager makes it easier to work with first-party and third-party libraries by helping you define and download project dependencies and pin down versions or version ranges so that you can, in theory, upgrade your dependencies without fear of breaking things. Most package managers also support a lock file through which they guarantee the reproducibility of builds in any environment.

Today, it is one of the most important things to consider when deciding whether to adopt a new programming language. This is evidenced by the fact that most mainstream languages have some sort of standard package management solution. For example, Ruby has RubyGems, Node.js has npm, and Rust has Cargo.

$ gem install dotenv

For a long time, the package management strategy in Go was inadequate, as there was no official way to version dependencies. Although third-party solutions did exist, integration with the language wasn't seamless, and adoption varied from project to project.

In 2018, the Go team finally introduced Go modules with the aim of plugging this gaping hole in the ecosystem. The feature initially landed in Go 1.11 but wasn't enabled by default. To turn it on, you had to set the $GO111MODULE environmental variable to on. As of Go 1.14, the latest version at the time of writing, modules are now enabled by default and continue to see increasing adoption in the community.

This post will walk you through how Go modules work. We'll consider all the basic use cases where the knowledge of modules will help you get things done faster and help save hours of trial and error.

The GOPATH

Before Go modules were introduced, projects had to be created inside the $GOPATH, which is an environmental variable that points to the directory where your Go workspace exists. This workspace is where Go manages your project files, dependencies, and installed binaries. The GOPATH was assumed to be $HOME/go on Unix systems and %USERPROFILE%\go on Windows by default.

Usage of $GOPATH mechanics proved to be inflexible and introduced a bit of a learning curve for Go beginners when it comes to setting up a development environment and understanding how the compiler manages dependencies for a project. In addition, versioning packages was not officially supported in the language (the way it is uses a Gemfile for a Ruby project, for example).

As of Go v1.13, $GOPATH mechanics is now largely irrelevant. While third-party dependencies are still placed in the GOPATH by default, you are now free to create a project anywhere in your filesystem, and vendoring and package versioning are now fully supported with the go tool.

In the next sections, we'll discuss how you can start using modules in your project and all the common use cases that you are likely to encounter. Make sure you have the latest version of Go installed before proceeding with the rest of this article.

Getting started with Modules

To initialize a project using Modules, enter the command below at the project root:

$ go mod init <module name>

The module name doubles as the import path, which allows internal imports to be resolved inside the module. It's also how other projects will import your package (if you're developing a library, for example). Idiomatically, this will be the URL of the repository hosting the code. Note that you don't need to have your project checked into version control or pushed to a remote repository before specifying a module name.

Suppose your project is called example; you can utilize modules in your project using the command below:

$ go mod init github.com/ayoisaiah/example
go: creating new go.mod: module github.com/ayoisaiah/example

The above command will create a go.mod file in the root of your project directory. This will contain the import path for the project and the Go version information. It's the Gemfile equivalent for Go.

$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

Installing dependencies

One of the main reasons Go modules were introduced was to make dependency management a lot easier. Adding a dependency to your project can be done using the go get command just as before:

$ go get github.com/joho/godotenv

You can target a specific branch:

$ go get github.com/joho/godotenv@master

Or a specific version:

$ go get github.com/joho/godotenv@v1.2.0

Or even a specific commit:

$ go get github.com/joho/godotenv@d6ee687

Your go.mod file should look like this now:

$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d // indirect

The // indirect comment indicates that this package is not currently being used in the project. You may also see this comment when a package is an indirect dependency (that is a dependency of another dependency).

You can import and use the newly installed godotenv package by specifying its import path and using one of its exported methods.

// main.go
package main

import (
    "github.com/joho/godotenv"
)

func main() {
    godotenv.Load()
}

At this point, you can run the go mod tidy command in your terminal to update the go.mod file. This command will remove unused dependencies in your project and add any missing ones (for example, if you import a third-party package to your project without fetching it first with go get).

Before releasing a new version of your project, and before each commit, you should run the go mod tidy command to ensure your module file is clean and accurate. This file possesses all the required information necessary for reproducible builds.

At this point, there also should be a go.sum file in your project root. It's not a lock file (like Gemfile.lock) but is maintained for the purpose of containing the expected cryptographic hashes of the content of specific module versions. You can think of it as additional verification to ensure that the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons.

$ cat go.sum
github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d h1:LRaxUhLYBFLUpSZk7X173VtzdRwPtu7HSs6avaT7lbU=
github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=

The recorded checksums are retained even if you stop using the module so that you can pick up right where you left off if you start using it again later. For this reason, make sure both your go.mod and go.sum files are checked into version control.

All downloaded modules are cached locally in your $GOPATH/pkg/mod directory by default. If you import a package to your project without downloading it first using go get, the latest tagged version of the module providing that package will be installed automatically and added to your go.mod file when you run go build or go test before your project is compiled.

You can see this in action by adding a new dependency to your project, such as this color package, and using it as shown below:

package main

import (
    "github.com/joho/godotenv"
    "gopkg.in/gookit/color.v1"
)

func main() {
    godotenv.Load()

    color.Red.Println("This is color red!")
}

Running go build will fetch the package and add it to your go.mod file:

$ go build
go: finding module for package gopkg.in/gookit/color.v1
go: found gopkg.in/gookit/color.v1 in gopkg.in/gookit/color.v1 v1.1.6
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require (
    github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d
    gopkg.in/gookit/color.v1 v1.1.6
)

Updating dependencies

Go modules use the Semantic Versioning (Semver) system for versioning, which has three parts: major, minor, and patch. A package in version 1.2.3 has 1 as its major version, 2 as its minor version, and 3 as its patch version.

Minor or patch versions

You can use go get -u or go get -u=patch to upgrade a package to its latest minor or patch version, respectively, but you can't do this for major version upgrades. This is because major version updates have different semantics for how they are published and maintained.

$ go get -u gopkg.in/gookit/color.v1
go: gopkg.in/gookit/color.v1 upgrade => v1.1.6

Major versions

The convention for code opting into Go modules is to use a different module path for each new major version. Starting at v2, the path must end in the major version. For example, if the developer of gotdotenv makes a major version release, the module path will change to [github.com/joho/godotenv/v2](http://github.com/joho/godotenv/v2), which is how you'll be able to upgrade to it. The original module path (github.com/joho/godotenv) will continue to refer to v1 of the package.

For example, let's say we're building a CLI app using v1 of this cli package in our project:

package main

import (
    "os"

    "github.com/urfave/cli"
)

func main() {
    (&cli.App{}).Run(os.Args)
}

After building the project and running go mod tidy, our go.mod file should look like this:

$ go build
go: finding module for package github.com/urfave/cli
go: found github.com/urfave/cli in github.com/urfave/cli v1.22.4
$ go mod tidy
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require github.com/urfave/cli v1.22.4

Let's say we want to upgrade to v2 of the package. All you need to do is replace the v1 import path with the v2 import path, as shown below. You should obviously read the documentation for the package you're upgrading, so any necessary changes can be made to the code. In this example, the code stays the same.

package main

import (
    "os"

    "github.com/urfave/cli/v2"
)

func main() {
    (&cli.App{}).Run(os.Args)
}

Running go build again will add the v2 package to our import path, alongside v1:

$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require (
    github.com/urfave/cli v1.22.4
    github.com/urfave/cli/v2 v2.2.0
)

If you've completed the migration successfully (and all tests pass), you can run go mod tidy to clean up the now unused v1 dependency from your go.mod file. This convention of using different module paths for major versions is known as semantic import versions. Due to this convention, it's possible to use multiple versions of a package simultaneously, such as when performing an incremental migration in a large codebase.

Removing dependencies

To delete a dependency from your project, all you need to do is remove all references to the package from your project, and then run go mod tidy on the command line to clean up your go.mod file. Remember that the cryptographic hash of a package's content will be retained in your go.sum file even after the package is removed.

Vendoring dependencies

As mentioned earlier, all downloaded dependencies for a project are placed in the $GOPATH/pkg/mod directory by default. Vendoring is the act of making a copy of the third-party packages your project depends on and placing them in a vendor directory within your project. This is one way to ensure the stability of your production builds without having to rely on external services.

Here are some other benefits of vendoring:

You'll be able to use git diff to see the changes when you update a dependency, and this history will be maintained in your git repo.
If a package suddenly disappears from the internet, you are covered.

If you want to vendor your dependencies in Ruby, you may use the following command:

$ bundle package

Here's how vendoring is achieved in Go:

$ go mod tidy
$ go mod vendor

Running go mod tidy is essential to keep the list of dependencies listed in your go.mod file accurate before vendoring. The second command will create a vendor directory in your project root, and all the third-party dependencies (direct and indirect) required to build your project are copied over to the directory. This folder should be checked into version control.

As of Go 1.14, the go command will also verify that your project’s vendor/modules.txt file is consistent with its go.mod file. If not, you may get an error, such as the one shown below, when building your project:

$ go build
go: inconsistent vendoring in /home/ayo/Developer/demo/example:
        github.com/urfave/cli@v1.22.4: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/urfave/cli/v2@v2.2.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory

To fix this error, run go mod tidy, and then run go mod vendor so that everything is consistent again.

Wrapping Things Up

In this article, we covered the most important concepts you need to know regarding Go modules before adopting them in your project. To recap,

go mod init will initialize modules in your project.
go get adds a new dependency, and you can use go get -u or go get -u=patch to upgrade a dependency to a new minor or patch version.
go mod tidy cleans up unused dependencies or adds missing dependencies.
go mod vendor copies all third-party dependencies to a vendor folder in your project root.

If you have any questions or opinions, I'd love to hear about it on Twitter. Thanks for reading, and happy coding!

Blog