GoLink failing to link golang.org/x/sys/unix after upgrading Go toolchain from 1.19.13 to >=1.22.0

[brianchesko]

What version of rules_go are you using?

v0.44.2

What version of gazelle are you using?

v0.35.0

What version of Bazel are you using?

7.1.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Ubuntu amd64

Any other potentially useful information about your toolchain?

Using WORKSPACE-based Gazelle deps due to incompatibilities with other dependencies not yet on bzlmod (namely https://github.com/stackb/rules_proto)

What did you do?

Updated go_register_toolchains(version = "1.20.7") to go_register_toolchains(version = "1.22.0"), cleared out go.mod and ran

bazel run @io_bazel_rules_go//go -- mod tidy && bazel run //:gazelle-update-repos && bazel run //:gazelle --verbose_failures

What did you expect to see?

Linking and build succeeds

What did you see instead?

brian$ bazel run //:gazelle --verbose_failures
INFO: Analyzed target //:gazelle (0 packages loaded, 0 targets configured).
ERROR: /home/brian/repos/agent/BUILD.bazel:15:15: GoLink gazelle-protobuf_/gazelle-protobuf [for tool] failed: (Exit 1): builder failed: error executing GoLink command (from target //:gazelle-protobuf) 
  (cd /home/brian/.cache/bazel/_bazel_brian/f3602e9cba0a194428e46041ff38163b/sandbox/linux-sandbox/1880/execroot/_main && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOEXPERIMENT=nocoverageredesign \
    GOOS=linux \
    GOPATH='' \
    GOROOT=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/io_bazel_rules_go/stdlib_ \
    GOROOT_FINAL=GOROOT \
    GOTOOLCHAIN=local \
    PATH=/usr/bin:/bin \
    ZERO_AR_DATE=1 \
  bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/go_sdk/builder_reset/builder link -sdk external/go_sdk -installsuffix linux_amd64 -arc '@@bazel_gazelle//internal/wspace:wspace=github.com/bazelbuild/bazel-gazelle/internal/wspace=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/internal/wspace/wspace.a' -arc '@@bazel_gazelle//pathtools:pathtools=github.com/bazelbuild/bazel-gazelle/pathtools=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/pathtools/pathtools.a' -arc '@@bazel_gazelle//label:label=github.com/bazelbuild/bazel-gazelle/label=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/label/label.a' -arc '@@com_github_bazelbuild_buildtools//tables:tables=github.com/bazelbuild/buildtools/tables=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_github_bazelbuild_buildtools/tables/tables.a' -arc '@@com_github_bazelbuild_buildtools//build:build=github.com/bazelbuild/buildtools/build=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_github_bazelbuild_buildtools/build/build.a' -arc '@@bazel_gazelle//rule:rule=github.com/bazelbuild/bazel-gazelle/rule=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/rule/rule.a' -arc '@@bazel_gazelle//config:config=github.com/bazelbuild/bazel-gazelle/config=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/config/config.a' -arc '@@bazel_gazelle//flag:flag=github.com/bazelbuild/bazel-gazelle/flag=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/flag/flag.a' -arc '@@bazel_gazelle//internal/module:module=github.com/bazelbuild/bazel-gazelle/internal/module=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/internal/module/module.a' -arc '@@bazel_gazelle//internal/version:version=github.com/bazelbuild/bazel-gazelle/internal/version=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/internal/version/version.a' -arc '@@org_golang_x_mod//internal/lazyregexp:lazyregexp=golang.org/x/mod/internal/lazyregexp=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_mod/internal/lazyregexp/lazyregexp.a' -arc '@@org_golang_x_mod//semver:semver=golang.org/x/mod/semver=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_mod/semver/semver.a' -arc '@@org_golang_x_mod//module:module=golang.org/x/mod/module=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_mod/module/module.a' -arc '@@org_golang_x_mod//modfile:modfile=golang.org/x/mod/modfile=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_mod/modfile/modfile.a' -arc '@@org_golang_x_sys//execabs:execabs=golang.org/x/sys/execabs=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_sys/execabs/execabs.a' -arc '@@org_golang_x_tools_go_vcs//:vcs=golang.org/x/tools/go/vcs=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_tools_go_vcs/vcs.a' -arc '@@bazel_gazelle//repo:repo=github.com/bazelbuild/bazel-gazelle/repo=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/repo/repo.a' -arc '@@bazel_gazelle//resolve:resolve=github.com/bazelbuild/bazel-gazelle/resolve=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/resolve/resolve.a' -arc '@@bazel_gazelle//language:language=github.com/bazelbuild/bazel-gazelle/language=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/language/language.a' -arc '@@bazel_gazelle//language/proto:proto=github.com/bazelbuild/bazel-gazelle/language/proto=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/language/proto/proto.a' -arc '@@org_golang_x_sync//errgroup:errgroup=golang.org/x/sync/errgroup=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_sync/errgroup/errgroup.a' -arc '@@bazel_gazelle//language/go:go=github.com/bazelbuild/bazel-gazelle/language/go=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/language/go/go.a' -arc '@@com_github_emicklei_proto//:proto=github.com/emicklei/proto=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_github_emicklei_proto/proto.a' -arc '@@in_gopkg_yaml_v3//:yaml_v3=gopkg.in/yaml.v3=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/in_gopkg_yaml_v3/yaml_v3.a' -arc '@@net_starlark_go//internal/spell:spell=go.starlark.net/internal/spell=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/net_starlark_go/internal/spell/spell.a' -arc '@@net_starlark_go//syntax:syntax=go.starlark.net/syntax=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/net_starlark_go/syntax/syntax.a' -arc '@@net_starlark_go//resolve:resolve=go.starlark.net/resolve=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/net_starlark_go/resolve/resolve.a' -arc '@@net_starlark_go//internal/compile:compile=go.starlark.net/internal/compile=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/net_starlark_go/internal/compile/compile.a' -arc '@@org_golang_x_sys//unix:unix=golang.org/x/sys/unix=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/org_golang_x_sys/unix/unix.a' -arc '@@net_starlark_go//starlark:starlark=go.starlark.net/starlark=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/net_starlark_go/starlark/starlark.a' -arc '@@net_starlark_go//starlarkstruct:starlarkstruct=go.starlark.net/starlarkstruct=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/net_starlark_go/starlarkstruct/starlarkstruct.a' -arc '@@build_stack_rules_proto//pkg/protoc:protoc=github.com/stackb/rules_proto/pkg/protoc=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/protoc/protoc.a' -arc '@@build_stack_rules_proto//pkg/language/protobuf:protobuf=github.com/stackb/rules_proto/pkg/language/protobuf=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/language/protobuf/protobuf.a' -arc '@@build_stack_rules_proto//pkg/plugin/bufbuild:bufbuild=github.com/stackb/rules_proto/pkg/plugin/bufbuild=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/bufbuild/bufbuild.a' -arc '@@build_stack_rules_proto//pkg/plugin/builtin:builtin=github.com/stackb/rules_proto/pkg/plugin/builtin=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/builtin/builtin.a' -arc '@@build_stack_rules_proto//pkg/plugin/gogo/protobuf:protobuf=github.com/stackb/rules_proto/pkg/plugin/gogo/protobuf=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/gogo/protobuf/protobuf.a' -arc '@@build_stack_rules_proto//pkg/plugin/golang/protobuf:protobuf=github.com/stackb/rules_proto/pkg/plugin/golang/protobuf=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/golang/protobuf/protobuf.a' -arc '@@build_stack_rules_proto//pkg/plugin/grpc/grpc:grpc=github.com/stackb/rules_proto/pkg/plugin/grpc/grpc=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/grpc/grpc/grpc.a' -arc '@@build_stack_rules_proto//pkg/plugin/grpc/grpcgo:grpcgo=github.com/stackb/rules_proto/pkg/plugin/grpc/grpcgo=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/grpc/grpcgo/grpcgo.a' -arc '@@build_stack_rules_proto//pkg/plugin/grpc/grpcjava:grpcjava=github.com/stackb/rules_proto/pkg/plugin/grpc/grpcjava=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/grpc/grpcjava/grpcjava.a' -arc '@@build_stack_rules_proto//pkg/plugin/grpc/grpcnode:grpcnode=github.com/stackb/rules_proto/pkg/plugin/grpc/grpcnode=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/grpc/grpcnode/grpcnode.a' -arc '@@build_stack_rules_proto//pkg/plugin/grpc/grpcweb:grpcweb=github.com/stackb/rules_proto/pkg/plugin/grpc/grpcweb=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/grpc/grpcweb/grpcweb.a' -arc '@@build_stack_rules_proto//pkg/plugin/grpcecosystem/grpcgateway:grpcgateway=github.com/stackb/rules_proto/pkg/plugin/grpcecosystem/grpcgateway=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/grpcecosystem/grpcgateway/grpcgateway.a' -arc '@@build_stack_rules_proto//pkg/plugin/scalapb/scalapb:scalapb=github.com/stackb/rules_proto/pkg/plugin/scalapb/scalapb=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/scalapb/scalapb/scalapb.a' -arc '@@build_stack_rules_proto//pkg/plugin/stackb/grpc_js:grpc_js=github.com/stackb/rules_proto/pkg/plugin/stackb/grpc_js=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/stackb/grpc_js/grpc_js.a' -arc '@@build_stack_rules_proto//pkg/plugin/stephenh/ts-proto:ts-proto=github.com/stackb/rules_proto/pkg/plugin/stephenh/ts-proto=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/stephenh/ts-proto/ts-proto.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_cc:rules_cc=github.com/stackb/rules_proto/pkg/rule/rules_cc=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_cc/rules_cc.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_closure:rules_closure=github.com/stackb/rules_proto/pkg/rule/rules_closure=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_closure/rules_closure.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_go:rules_go=github.com/stackb/rules_proto/pkg/rule/rules_go=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_go/rules_go.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_java:rules_java=github.com/stackb/rules_proto/pkg/rule/rules_java=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_java/rules_java.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_nodejs:rules_nodejs=github.com/stackb/rules_proto/pkg/rule/rules_nodejs=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_nodejs/rules_nodejs.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_python:rules_python=github.com/stackb/rules_proto/pkg/rule/rules_python=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_python/rules_python.a' -arc '@@build_stack_rules_proto//pkg/plugin/akka/akka_grpc:akka_grpc=github.com/stackb/rules_proto/pkg/plugin/akka/akka_grpc=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/plugin/akka/akka_grpc/akka_grpc.a' -arc '@@com_github_bmatcuk_doublestar//:doublestar=github.com/bmatcuk/doublestar=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_github_bmatcuk_doublestar/doublestar.a' -arc '@@build_stack_rules_proto//pkg/rule/rules_scala:rules_scala=github.com/stackb/rules_proto/pkg/rule/rules_scala=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/pkg/rule/rules_scala/rules_scala.a' -arc '@@build_stack_rules_proto//language/protobuf:protobuf=github.com/stackb/rules_proto/language/protobuf=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/build_stack_rules_proto/language/protobuf/protobuf.a' -arc '@@bazel_gazelle//merger:merger=github.com/bazelbuild/bazel-gazelle/merger=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/merger/merger.a' -arc '@@com_github_bmatcuk_doublestar_v4//:doublestar=github.com/bmatcuk/doublestar/v4=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_github_bmatcuk_doublestar_v4/doublestar.a' -arc '@@bazel_gazelle//walk:walk=github.com/bazelbuild/bazel-gazelle/walk=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/bazel_gazelle/walk/walk.a' -arc '@@com_github_pmezard_go_difflib//difflib:difflib=github.com/pmezard/go-difflib/difflib=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_github_pmezard_go_difflib/difflib/difflib.a' -package_list bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/go_sdk/packages.txt -o bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/gazelle-protobuf_/gazelle-protobuf -main bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/gazelle-protobuf.a -p gazelle-protobuf -- -extld /usr/bin/gcc '-buildid=redacted' -s -w -extldflags '-fuse-ld=gold -B/usr/bin -Wl,-no-as-needed -Wl,-z,relro,-z,now -pass-exit-codes -Wl,--push-state,-as-needed -Wl,--pop-state -Wl,--push-state,-as-needed -lm -Wl,--pop-state')
# Configuration: 483ff52d1cc3bec7c615c61cedc125113bf2164229673de4b29a62d65dfe72ae
# Execution platform: @@local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/go_sdk/pkg/tool/linux_amd64/link: running /usr/bin/gcc failed: exit status 1
/tmp/go-link-1852342043/go.o:go.go:function golang.org/x/sys/unix.munmap: error: undefined reference to 'golang.org/x/sys/unix.Syscall'
/tmp/go-link-1852342043/go.o:go.go:function golang.org/x/sys/unix.mremap: error: undefined reference to 'golang.org/x/sys/unix.Syscall6'
/tmp/go-link-1852342043/go.o:go.go:function golang.org/x/sys/unix.mmap: error: undefined reference to 'golang.org/x/sys/unix.Syscall6'
collect2: error: ld returned 1 exit status

link: error running subcommand GOROOT_FINAL=GOROOT \
GOROOT=bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/io_bazel_rules_go/stdlib_ \
PATH=/usr/bin:/bin \
GOPATH= \
GOEXPERIMENT=nocoverageredesign \
GOTOOLCHAIN=local \
TMPDIR=/tmp \
CGO_ENABLED=1 \
GOOS=linux \
ZERO_AR_DATE=1 \
GOARCH=amd64 \
external/go_sdk/pkg/tool/linux_amd64/link -importcfg /tmp/bazel-working-directory/_main/bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/gazelle-protobuf_/importcfg408211853 -o /tmp/bazel-working-directory/_main/bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/gazelle-protobuf_/gazelle-protobuf -extld /usr/bin/gcc -buildid=redacted -s -w -extldflags "-fuse-ld=gold -B/usr/bin -Wl,-no-as-needed -Wl,-z,relro,-z,now -pass-exit-codes -Wl,--push-state,-as-needed -Wl,--pop-state -Wl,--push-state,-as-needed -lm -Wl,--pop-state" /tmp/bazel-working-directory/_main/bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/gazelle-protobuf.a: exit status 2
Target //:gazelle failed to build
INFO: Elapsed time: 0.397s, Critical Path: 0.27s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

The same is true with or without clearing the Go cache.

Upgrading to versions below 1.22.0 e.g. go_register_toolchains(version = "1.21.13") does not fail to build, but due to a transitive requirement still updates go.mod to incorrectly claim go 1.22.7

brian:~/repos/agent$ bazel run @io_bazel_rules_go//go -- mod tidy && bazel run //:gazelle-update-repos && bazel run //:gazelle --verbose_failures
INFO: Analyzed target @@io_bazel_rules_go//go:go (9 packages loaded, 9615 targets configured).
INFO: Found 1 target...
Target @@io_bazel_rules_go//go/tools/go_bin_runner:go_bin_runner up-to-date:
  bazel-bin/external/io_bazel_rules_go/go/tools/go_bin_runner/bin/go
INFO: Elapsed time: 36.223s, Critical Path: 16.54s
INFO: 11 processes: 6 internal, 5 linux-sandbox.
INFO: Build completed successfully, 11 total actions
INFO: Running command line: bazel-bin/external/io_bazel_rules_go/go/tools/go_bin_runner/bin/go mod tidy
go: finding module for package github.com/apenella/go-ansible/v2/pkg/playbook
...
go: finding module for package github.com/apenella/go-ansible/v2/pkg/execute
go: toolchain upgrade needed to resolve github.com/apenella/go-ansible/v2/pkg/execute
go: github.com/apenella/go-ansible/[email protected] requires go >= 1.22; switching to go1.22.7
go: finding module for package github.com/apenella/go-ansible/v2/pkg/playbook
go: finding module for package google.golang.org/grpc
...
go: found github.com/apenella/go-ansible/v2/pkg/playbook in github.com/apenella/go-ansible/v2 v2.0.1
INFO: Analyzed target //:gazelle-update-repos (94 packages loaded, 11102 targets configured).
INFO: Found 1 target...
Target //:gazelle-update-repos up-to-date:
  bazel-bin/gazelle-update-repos-runner.bash
  bazel-bin/gazelle-update-repos
INFO: Elapsed time: 10.162s, Critical Path: 9.21s
INFO: 41 processes: 1 internal, 40 linux-sandbox.
INFO: Build completed successfully, 41 total actions
INFO: Running command line: bazel-bin/gazelle-update-repos
INFO: Analyzed target //:gazelle (34 packages loaded, 449 targets configured).
INFO: Found 1 target...
Target //:gazelle up-to-date:
  bazel-bin/gazelle-runner.bash
  bazel-bin/gazelle
INFO: Elapsed time: 1.899s, Critical Path: 1.67s
INFO: 37 processes: 1 internal, 36 linux-sandbox.
INFO: Build completed successfully, 37 total actions
INFO: Running command line: bazel-bin/gazelle