Go Package Management

Go takes a semi-decentralized approach to package management, allowing any git repo to be used as a module package system. Modules can be downloaded directly from their source control servers or alternatively through a module proxy which performs a simple forwarding of requests to the appropriate source. The following diagram is a high-level take on what happens when go get attempts to resolve a package or module.

Module Proxy

The module proxy is an HTTP server which implements the module proxy protocol. It responds to GET requests matching a particular pattern, See examples below. The module proxy ensures changes or downtime to a module origin server will not bubble up and impact your build process. The module proxy can also act as a gatekeeper, restricting access to modules which have unsuitable licences or contain security vulnerabilities. Module proxies can also provide faster dependency resolution by caching previously fetched modules, which can then be used to serve future requests. A module's dependencies are defined in the following files:

Module Proxy Protocol

Since version 1.13, proxy.golang.org is the default module proxy for fetching modules.

It's possible to build your own module proxy, which consists of an HTTP server and REST API that satisfies the GOPROXY protocol.

Go Get Command

The go get command follows this protocol when attempting to resolve a package path:

• Search modules in the current project build list (go.mod) including all build lists of transitive dependencies, looking for modules with paths which are prefixes of the requested package path.
• If one module in the build list contains the required package, then that module will be used.
• If none of the modules in the searched build lists provide the package or if two or more modules provide the requested package, then go get will report an error.

When go get is resolving a new module for a package path , it uses the GOPROXY environment variable to determine whether modules will be downloaded directly, through a module proxy or some other private mirror. GOPROXY accepts a list of proxy URLs separated by either commas or pipes. The separator used in this list determines how the go get command will fallback in different scenarios:

The go get command visits each proxy in the list sequentially until it receives a successful response or error.

GoProxy Environment Variable Options

The default configuration for the GOPROXY environment variable is GOPROXY=proxy.golang.org,direct which effectively tells the go get command to first attempt module retrieval using the module mirror and if that approach is unsuccessful, fallback to the approach of connecting directly to the module origin repository. A number of other environment variables can also impact how go get will attempt to resolve dependencies. These variables are listed here.

Go Module Proxy Request Patterns

A GET request for a particular module will follow the pattern: https://$base/$module/@v/$version.zip.

Where:

The resulting GET request will download the module and dependencies from: https://proxy.golang.org/go.uber.org/zap/@v/v1.21.0.zip. The 'go get' and 'go mod tidy' commands abstract the details of this from the developer.

Go Module Proxy Response Codes

The type of separator used in the GOPROXY environment variable will determine if the go get command will fall back to the next option in the list or fail completely. See section on 'go get'.

Go Environment Variables Which Impact Module Retrieval

If set with specific values, GOPRIVATE and GONOPROXY can impact the behaviour of GOPROXY, by preventing specific modules from being downloaded through the proxy.

Verifying Modules

After downloading a .mod or .zip file, go get computes a cryptographic hash and checks that it matches a hash in the main modules go.sum. If the hash is not present in go.sum then the go command retrieves it from the checksum database. The checksum database, located at sum.golang.org, allows for global consistency and reliability for all publicly available modules. It also ensures bits associated with specific module versions do not change from one day to the next.

Module Caching

The 'go get' command will cache modules downloaded from both module proxies and directly from a module's source repository in $GOPATH/pkg/mod/cache/download. This local cache layout is identical to the proxy URL space.

Module Graph

Module graph pruning was introduced in version 1.17. See go.dev for more information. Module pruning uses Minimal version selection to select a set of module versions to use when building packages. It operates on a directed graph of modules, specified in go.mod. Each vertex in the graph represents a module version. Each edge represents a minimum required version of a dependency, specified using a require directive. The graph may be modified by exclude and replace directives in go.mod. MVS produces the build list as an output.

Full details can be found at understanding-go-get