Do you enjoy weird and strange build issues? Or do you think something we do in this blog post is fishy and you want to fix it? Want us to use bazel instead? Good news - we’ve got a role for you!
We’re on the lookout for more engineers on our Developer Infrastructure team! We’re looking to expand our sprawling development infrastructure as we grow to more people and a bigger codebase with new and exciting functionality and cloud management. If you want to help the engineers write the exciting next generation of databases by empowering their work environment, don’t delay - apply today!
-–
For a significant part of our history, CockroachDB used the dep vendoring tool for managing package dependencies. Go modules have been widely available since Go 1.11, and with Go 1.14 heavily recommending the migration as well as dependent libraries moving towards the Go module world, it was time CockroachDB followed suit.
However, migrating to go.mod was no straightforward feat for CockroachDB. Everything that could possibly complicate the upgrade seemed to come our way. We faced issues such as being unable to import vendored protobuf files, recognizing implicit upgrades and downgrades, the infamous error writing go.mod errors and a whole lot more. And in a Software Engineer’s worst nightmare, it was very difficult to find and apply StackOverflow answers, Github issues and documentation from Go that were applicable in our scenario!
Curious? Follow along our blog post as we explore migrating to Go modules, sroughly following the guidance from the “Migrating to Go Modules” page.
The Initial Migration from dep to Go Modules
Our first step was to run the migrate tool:
go mod init github.com/cockroachdb/cockroach
which migrated our Gopkg.lock file to their go.mod and go.sum equivalents. However, the go.mod file failed the goimports tool because of lines like these:
replace github.com/grpc-ecosystem/grpc-gateway 52697fc4a24978380c5ad7b80adc795336d4dfd4 => github.com/cockroachdb/grpc-gateway v1.14.6-0.20200519165156-52697fc4a249
Which caused Go’s linter to alert us with the following failure:
replace github.com/grpc-ecosystem/grpc-gateway: version "52697fc4a24978380c5ad7b80adc795336d4dfd4" invalid: must be of the form v1.2.3
Interesting. In our Gopkg.toml we had defined grpc-ecosystem/grpc-gateway to be attached to a specific branch in our fork:
[[constraint]]
name = "github.com/grpc-ecosystem/grpc-gateway"
branch = "v1.14.5-nowarning"
source = "https://github.com/cockroachdb/grpc-gateway"
But the error message gives us our clue – the 52697fc4a24978380c5ad7b80adc795336d4dfd4 component was in fact extraneous! Removing all of the third arguments in lines like these fixed that problem:
replace github.com/grpc-ecosystem/grpc-gateway => github.com/cockroachdb/grpc-gateway v1.14.6-0.20200519165156-52697fc4a249
The go.mod file still had problems - namely that the module import was defined twice: once in the require go.mod stanza and once in the replace directive. Below is an example of the duplicate definition of “github.com/cockroachdb/grpc-gateway” in the require stanza:
require (
...
github.com/cockroachdb/grpc-gateway v1.14.6-0.20200519165156-52697fc4a249
...
)
Which would error whenever we tried to build anything with Go:
go: github.com/cockroachdb/grpc-gateway@v1.14.6-0.20200519165156-52697fc4a249: parsing go.mod:
module declares its path as: github.com/grpc-ecosystem/grpc-gateway
but was required as: github.com/cockroachdb/grpc-gateway
Deleting the above line in the require stanza (and our other packages that has replace) fixed that issue.
Huzzah! Now we get some more interesting output when attempting to go build. However, from this point on, there were no more migration guide steps we could follow to resolve our issues…
Encountering error writing go.mod
After that, we should be able to build CockroachDB packages using Go modules. However, we run into the following error:
error writing go.mod: open /Users/otan/go/pkg/mod/github.com/lib/pq@v1.3.1-0.20200116171513-9eb3fc897d6f/go.mod298498081.tmp: permission denied
These seem like an error commonly encountered (judging by the many different results from a quick Google search) and something that seems tempting to blindly use chmod to fix. But hold that temptation - it will not work on other people’s systems!
We discovered these errors seem to be a result of code attempting to modify our $GOPATH/pkg/mod directory where all the go.mod modules are downloaded to. This is notably permissioned as read only to prevent corruption. This makes sense - you do not want another Go module modifying a cached shared module on your system.
In our case, we found that go/loader was deprecated and should have been upgraded to go/packages - otherwise, it will load and change the files on the module. This involved changing a number of our linters, such as our returncheck linter.
Using protobuf and C files defined in Go modules
CockroachDB imports certain libraries that require protobufs, and also relies on protobuf definitions inside certain vendored directories. Our new build errors all seemed to point to the fact that we could not import protobuf files that do not appear to be importable from our previously defined vendor directory. We realized this was because the file generator for protobufs (protoc) in the current directory, or directories specified with the -I flag. This is more verbose with Go modules as the imports have a @sha/version suffix in the directory name. As an example, we would need to do -I$GOPATH/pkg/mod/github.com to import prometheus/client_model@v0.2.0/metrics.proto in the .proto file. When using dep, we used -I./vendor/github.com so we could simply import prometheus/client_model/metrics.proto, with no notion of versioning required. Needing to include the Go module suffix in protobuf paths can be a tedious change - especially during upgrades where it could be an easily forgotten update.
This is one of the use cases for go mod vendor, which copies the module files from $GOPATH/pkg/mod into a “vendor” directory in the current Go repo. In theory, this should basically make Go modules very similar to where dep stored the vendored modules. In Go 1.13 or before, Go will look for the modules inside the vendor directory if any of the Go commands have -mod=vendor as a flag (e.g. go build -mod=vendor . or go test -mod=vendor .). It’s painful to remember to include the flag everywhere, but this was resolved in Go 1.14 as the flag -mod=vendor is implicitly added to any Go command if a vendor directory is detected.
However, another problem emerged - protobuf files did not seem to be copied into the vendor directory if there are no Go files in the same directory. In particular, we had problems with protobuf files missing in github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/api, github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/rpc and github.com/prometheus/client_model. This is because Go does not detect any Go dependencies on these packages, and as such does not attempt to copy them over when running go mod vendor. As such we have adapted and then adopted the modvendor tool to copy these additional directories from the $GOPATH/pkg/mod directory to the vendor directory.
Another use case for go mod vendor
To support cross compiling CockroachDB, we add extra .zcgo_flags.go files in directories that contain C files to tell the Go compiler know where to link certain libraries based on the OS that is being compiled. For example, in knz/go-libedit, we generate the following (.gitignore’d) file in vendor/github.com/knz/go-libedit/unix/zcgo_flags.go when compiling on OSX:
// GENERATED FILE DO NOT EDIT
// +build !make
package libedit_unix
// #cgo CPPFLAGS: -I/Users/otan/go/native/x86_64-apple-darwin19.5.0/jemalloc/include
// #cgo LDFLAGS: -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/cryptopp -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/protobuf -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/jemalloc/lib -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/snappy -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/libedit/src/.libs -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/rocksdb -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/libroach -L/Users/otan/go/native/x86_64-apple-darwin19.5.0/proj/lib
import "C"
Which tells the clang compiler to link libedit/src/.libs when compiling libedit. Since the $GOPATH/pkg/mod directory containing these modules is read only, the generated files had to be put into the vendor directory using go mod vendor. When doing this, we needed to ensure we compile the cockroach binary with -mod=vendor (for Go 1.13 or before) or else we get a link error.
Reappearing and disappearing tool related imports
At this point, we have everything building! It was time for the next step in the go.mod migration checklist: running go mod tidy. This cleans up any unused imports that are lingering in go.mod and go.sum.
Unfortunately, go mod tidy made some packages disappear! In particular, the packages that disappeared seemed to be related to tools we run that we vendor and pin at a specific revision in our Gopkg.toml and Gopkg.lock files. Instead of importing these tools, we install and run these directly from our Makefile. Since we do not explicitly import them, go mod tidy happily cleans these entries up. Any subsequent go install after the modules are cleaned up would not be pinned to the version we have previously specified.
Thus, in order to use a pinned, vendored tool, we had to make sure it does not disappear from go.mod. We created a file that imports the required tools with blank imports to ensure that these dependencies are not removed. This looks like the following (see source):
// +build tools
package main
import (
"fmt"
"github.com/client9/misspell/cmd/misspell"
"github.com/cockroachdb/crlfmt"
"github.com/cockroachdb/go-test-teamcity"
"github.com/cockroachdb/gostdlib/cmd/gofmt"
"github.com/cockroachdb/gostdlib/x/tools/cmd/goimports"
"github.com/cockroachdb/stress"
"github.com/goware/modvendor"
"github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway"
"github.com/kevinburke/go-bindata/go-bindata"
"github.com/kisielk/errcheck"
"github.com/mattn/goveralls"
"github.com/mibk/dupl"
"github.com/mmatczuk/go_generics/cmd/go_generics"
"github.com/wadey/gocovmerge"
"golang.org/x/lint/golint"
"golang.org/x/perf/cmd/benchstat"
"golang.org/x/tools/cmd/goyacc"
"golang.org/x/tools/cmd/stringer"
"golang.org/x/tools/go/analysis/passes/shadow/cmd/shadow"
"honnef.co/go/tools/cmd/staticcheck"
)
Note that +build tools directive, which tells Go to only build the package if the tools tag is provided in a Go command (e.g. go build -tags 'tools'). In practice, we never set the tools tag and thus the file is never built or included in any package.
Implicit upgrades and downgrades
When using go mod init to migrate from dep to Go modules, we found that some modules were implicitly upgraded and some modules were implicitly downgraded after running go build and go mod tidy.
As an example, take github.com/linkedin/goavro from Gopkg.lock but not present in Gopkg.toml:
[[projects]]
digest = "1:6ff6c3cb744b42df69cb874c3f19d387ece7ba7998e7d4de811c9bf61a7b1a09"
name = "github.com/linkedin/goavro"
packages = ["."]
pruneopts = "UT"
revision = "af12b3c46392134a7db8c1a8b6c6a33419fab0ea"
version = "v2.7.2"
After running go mod init, this import disappeared completely from go.mod after our migration. When running go build, it detected that github.com/linkedin/goavro did not exist, and instead reappeared as an earlier version, coincidentally the one right before the repo migrated to Go modules:
require (
...
github.com/linkedin/goavro v2.1.0+incompatible
...
)
This was an undesired downgrade in the library. This seemed to be a trend with a few Go modules packages.
[[projects]]
digest = "1:0981502f9816113c9c8c4ac301583841855c8cf4da8c72f696b3ebedf6d0e4e5"
name = "github.com/mattn/go-isatty"
packages = ["."]
pruneopts = "UT"
revision = "6ca4dbf54d38eea1a992b3c722a76a5d1c4cb25c"
version = "v0.0.4"
which gets respected in go.mod initially:
require (
…
github.com/mattn/go-isatty v0.0.4
…
)
But somewhere whilst running a go build and go mod tidy, something changes the version of the module to v0.0.9:
require (
…
github.com/mattn/go-isatty v0.0.9
)
This is seemingly because this import is used by several dependent packages, which preferred the latest version.
We could have gone down the route of trying to pin everything down to avoid vendor changes on the old Gopkg.toml. However, we decided it was easier to wave through all the upgrades and downgrades, as even if we pinned everything down, module dependencies may still end up changing after moving from dep to Go modules as some packages still had to be upgraded to properly support Go modules (we found some packages at some versions caused issues with Go modules, in which a version upgrade was required).
To ensure waving through these implicit upgrades and downgrades was safe, we wanted to ensure that all these upgrades and downgrades were audited in a systematic fashion. To do this, we wrote a script that compares the contents of SHAs and versions outlined in vendor/modules.txt against the old Gopkg.lock.
We used vendor/modules.txt to detect package changes over go.sum and go.mod, since vendor/modules.txt seems to contain the exact version of what module we were using for each dependency. go.sum had too much data and go.mod had too little. Comparing the old vendor directory and new vendor directory was untenable as thousands of lines of code and files had changed.
To illustrate the differences between go.mod, go.sum and vendor/modules.txt, let’s compare them. Here is the vendor/modules.txt entry for gogo/protobuf:
# github.com/gogo/protobuf v1.3.1 => github.com/cockroachdb/gogoproto v1.2.1-0.20190102194534-ca10b809dba0
Compare this with the go.sum file, which contains a lot of entries for the same package. This is because our dependencies all have a dependency on a different version of the protobuf library:
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
Under go.mod, it may be unclear that some implicit import could be included by surprise into vendor/modules.txt. As an example, the import of github.com/russross/blackfriday is included in vendor/modules.txt despite not being present in go.mod and is explicitly used by a main package. It is worth noting that in Go 1.14, explicit imports are marked in vendor/modules.txt with a ## explicit line right after the import to help with reasoning about implicit vs explicit imports as defined by being mentioned in go.mod:
# github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563
## explicit
github.com/rcrowley/go-metrics
github.com/rcrowley/go-metrics/exp
# github.com/russross/blackfriday v1.5.2
github.com/russross/blackfriday
The version changes dumped from the script were transformed into a checklist of issues on GitHub which engineers audited. With good integration tests and manual auditing, we found 4 unwanted behavior changes out of the 80 or so import changes caused by the implicit upgrades or downgrades which warranted further action. We are not sure how to reproduce the same list without doing go mod vendor and peering into the vendor/modules.txt file.
Some of these involved updating the import definitions to the versioned Go module equivalent. In particular, we had to change all imports of “github.com/cockroachdb/apd” to “github.com/cockroachdb/apd/v2” and “github.com/linkedin/goavro” to “github.com/linkedin/goavro/v2”. This resulted in quite a large number of file changes in the PR which were mostly just import changes leading to a larger and less well-focused code review. Unfortunately, specifying the version number in the package import does not work pre-Go modules and such we could do this as a separate commit/PR.
Why are we still seeing certain packages in go.sum despite never importing them?
Despite upgrading “github.com/cockroachdb/apd” to “github.com/cockroachdb/apd/v2”, we still found “github.com/cockroachdb/apd” in go.sum despite not being in go.mod. This was confusing to us.
However, go mod why tool told us why:
$ go mod why github.com/cockroachdb/apd
...
# github.com/cockroachdb/apd
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest
github.com/jackc/pgx
github.com/jackc/pgx.test
github.com/cockroachdb/apd
It seems as though the unit tests of github.com/jackc/pgx rely on the deprecated package which pulls it into go.sum. However, this package does not show up in vendor/modules.txt or in the vendor directory at all as we do not require pgx tests to be compiled in any part of CockroachDB.
Reproducible Builds and the Go module cache
For dep, we stored our vendor directory in a repo called “vendored”, using a git submodule on the CockroachDB repo to import the directory. This allows for reproducible builds without needing to check all the vendor directory contents into the CockroachDB repo (and as a side effect making those import git contribution stats more meaningful ;)). Further rationale and instructions for maintaining this is available here.
In the Go module world, this is achieved using the Go module cache, which copies module source files into $GOPATH/pkg/mod. In particular, all the versions stored in go.sum would download the exact same version from the Go module cache from any different source. However, we had concerns with this, such as:
What happens if a Go module becomes unavailable or deleted? Does it stay in the cache or get deleted? How does one inspect cache history?
What is the behavior of a module using the same version tag but changing SHAs. We don’t think this is possible with Github, but is it elsewhere?
These questions were blockers to us as it was possible we would not get a completely reproducible build at any SHA at some point in the future. We could alleviate this concern by hosting and owning our own Go module cache such as goproxy/goproxy but that comes with its own operational complexity and extra work for our infrastructure.
As such, we have opted to maintain the vendored repo, using the present day solution of using git / Github as our “go module cache”. This preserves reproducible builds in a way we were comfortable with. This has an added benefit of not having to run go mod vendor every time vendored packages change, which can take a while compared to simply pulling in an update to the submodule.
However, maintaining a git repo with go mod vendor is tricky as it wipes the vendor directory before recreating it. As such, we moved the git objects in .git to a separate directory before moving it back after go mod vendor is complete. Thus, our vendor directory rebuild script looks like the following:
#!/usr/bin/env bash
set -Eeoux pipefail
TMP_VENDOR_DIR=.vendor.tmp.$(date +"%Y-%m-%d.%H%M%S")
# restore the vendor directory if any of the below steps fail
trap restore 0
function restore() {
if [ -d $TMP_VENDOR_DIR ]; then
rm -rf vendor
mv $TMP_VENDOR_DIR vendor
fi
}
mv vendor $TMP_VENDOR_DIR
go mod vendor
modvendor -copy="**/*.c **/*.h **/*.proto" -include 'github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/api,github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/rpc,github.com/prometheus/client_model'
mv $TMP_VENDOR_DIR/.git vendor/.git
rm -rf $TMP_VENDOR_DIR
This script only needs to be run after a module change. More detailed instructions for our workflow when changing Go modules is available here.
Summary of our dep to Go Module Migration Steps
To migrate from dep to Go modules, we needed to perform the following:
Run
go mod init github.com/cockroachdb/cockroach.Remove the third argument for all replace directives.
Remove the libraries we needed to replace from the
requirestanza.Change all usages of go/loader to go/packages.
Run
go mod vendor.Use modvendor to include protobuf files that are not copied over in the vendor directory. At this point, we could start building compiling binaries with
go build -mod=vendor ...Add a file that imports all tools we need to pin at a specific version.
Run
go mod tidy.Audit all implicit downgrades and upgrades that occurred when running
go mod init, re-upgrading and re-downgrading packages as necessary.Update packages which required import definition changes in the codebase to use the Go module’d version.
(Go 1.13 only) Update all scripts to use
-mod=vendorto make sure everything imports from the correct directory.
Results
As of cockroachdb/cockroach#49447, we’ve moved to Go modules, with great fanfare from the Twitter community:
Our general “weirdness” with handling vendored packages is mostly alleviated using go mod vendor. It is able to handle protobufs, we are able to inject extra files into it and we’re able to commit the vendor directory as a git submodule. Furthermore, we were also successfully able to fast follow and upgrade to Go 1.14 to alleviate the need to remember to add the -mod=vendor flag to compile and test our packages (but may need to downgrade due to a Go 1.14 timer issue).
Overall, Go modules usage is a lot smoother in terms of speed compared to using dep ensure, and the management of modules feels more intuitive. Being able to use different versions of the same module at the same time also makes migration of certain packages to newer modules easier.
However, we found for those migrating that Go modules could be complicated and overwhelming - especially for large and complex code bases - as evident by our wordier-than-expected migration process. There were a few steps here which were not immediately obvious from the “Migrating to Go Modules” docs page. However, reading the Go mod reference page and the entire blog post on Go modules on golang.org was useful as a base step. Some of the build errors and required package upgrades were difficult to resolve without a deep understanding of the Go module system, which these texts offered insight on.
We’re hiring for our Developer Infrastructure team!
Do you enjoy weird and strange build issues? Or do you think something we do in this blog post is fishy and you want to fix it? Want us to use bazel instead? Good news - we’ve got a role for you!
We’re on the lookout for more engineers on our Developer Infrastructure team! We’re looking to expand our sprawling development infrastructure as we grow to more people and a bigger codebase with new and exciting functionality and cloud management. If you want to help the engineers write the exciting next generation of databases by empowering their work environment, don’t delay - apply today!
