[hermes] metis #297: go_module_download_network_unreachable #7

New Issue

hermes-automation · 2026-08-17T13:57:00Z

hermes-automation commented

2026-08-17 13:57:00 +00:00

Hermes auto-triage classified incident metis/297 as go_module_download_network_unreachable (confidence 0.85); first failed gate: Build & push images.

Why a human is needed

Build 297 failed while building and pushing container images, not because of a code or test problem. All 309 tests passed and the SonarQube scan completed normally. The failure happened inside the Dockerfile's Go build stage: go build needed to fetch the oras.land/oras v1.2.2 module from the Go module proxy (proxy.golang.org), and the connection failed with 'network is unreachable' trying to reach an IPv6 address for that host. That's an environment/networking problem on the build node, not a defect in metis's Go code or its dependency versions. A fix here doesn't involve changing application code -- it means confirming the build node has working IPv6 (or a working IPv4 fallback/route) egress to proxy.golang.org and then re-running the build; if this keeps happening, it's worth asking platform/infra whether that build node's network path to the Go module proxy is flaky, or setting up a local GOPROXY mirror/vendoring so builds don't depend on live egress to proxy.golang.org. Separately, and unrelated to this specific failure, the log also shows the build node failing to register qemu-aarch64 for cross-arch emulation ('cannot register "/usr/bin/qemu-aarch64" ... no such file or directory'), which means arm64 image builds on this node may not actually work even though arm64 is listed as a supported platform -- that looks like a real gap worth investigating on its own, since it will surface again the next time an arm64 build is attempted.

Ariadne did not authorize automated remediation: classification_not_supported: 'go_module_download_network_unreachable'.

Facts

jenkins — Jenkins build metis #297 (https://ci.bstein.dev/job/metis/297/) finished with result FAILURE; Ariadne's bundle marks 'Build & push images' as the first failed pipeline stage. (jenkins.result=FAILURE, jenkins.first_failed_stage='Build & push images')
jenkins — The build's own JUnit results show all 309 tests passed (the 'Tests / Declarative: Post Actions' Gitea check reported SUCCESS, passed: 309), and jenkins.failed_tests is empty, so no test assertions failed in this build. (console_tail; jenkins.failed_tests=[])
jenkins — At console line 2052, the Docker build step 'RUN ... go build -o /out/metis ... -o /out/oras oras.land/oras/cmd/oras' failed with: 'oras.land/oras@v1.2.2: Get "https://proxy.golang.org/oras.land/oras/@v/v1.2.2.zip": dial tcp [2607:f8b0:4023:100b::8d]:443: connect: network is unreachable', followe... (console_failures[4] (marker 'ERROR:', line_number 2052))
jenkins — Immediately before the 'Build & push images' stage begins (console line ~1572), a buildx diagnostic call to the builder node ('Get http://localhost:2375/v1.47/containers/buildx_buildkit_metis-builder-2970/json') failed with 'context deadline exceeded', but the script explicitly returned 0 afterwa... (console_failures[3] (marker 'ERROR:', line_number 1572))
jenkins — At console line 1524, binfmt_misc registration for qemu-aarch64 failed ('cannot register "/usr/bin/qemu-aarch64" ... no such file or directory'), and the resulting emulator list only shows python3.12 and qemu-x86_64 (no qemu-aarch64), even though linux/arm64 is listed as a supported target platform. (console_failures[2] (marker 'No such file', line_number 1524))
jenkins — At console line 1355, SonarQube's Go analyzer logged parser errors for five files (pkg/service/remote.go, cluster.go, remote_helpers.go, app_helpers.go, node_recovery.go: 'Go parser external process returned non-zero exit value: 2'), but the Sonar sensor still completed ('46/46 source files have ... (console_failures[1] (marker 'non-zero exit', line_number 1355))
opensearch — The log query for the build's time window (2026-08-17T13:14:00Z to 13:54:47Z) returned zero records, so no supplementary application/service logs are available for this incident. (log_evidence.records=[], query_window 2026-08-17T13:14:00Z-13:54:47Z)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "metis-297-gz5mh-ttkrj-lk3l7"

Inferences

The failing go build step was compiling for GOARCH=amd64, so the earlier qemu-aarch64/binfmt registration failure a bit earlier in the log does not look like the direct cause of this failure, though it separately suggests arm64 emulation isn't fully set up on this build node and is worth checki...
The SonarQube parser errors on specific Go files look like a pre-existing scanner limitation rather than the cause of this failure, since the Sonar analysis finished normally (46/46 files analyzed) and the pipeline moved on to the image-build stage afterward.
Because the failure is a network-reachability problem reaching proxy.golang.org from the build container rather than anything in metis's own source, no code change is likely needed to fix it.

Links

Failed build: https://ci.bstein.dev/job/metis/297/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_4306dc82f0e84633a002dced911b7161). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `metis/297` as **go_module_download_network_unreachable** (confidence 0.85); first failed gate: `Build & push images`. ## Why a human is needed Build 297 failed while building and pushing container images, not because of a code or test problem. All 309 tests passed and the SonarQube scan completed normally. The failure happened inside the Dockerfile's Go build stage: `go build` needed to fetch the `oras.land/oras` v1.2.2 module from the Go module proxy (proxy.golang.org), and the connection failed with 'network is unreachable' trying to reach an IPv6 address for that host. That's an environment/networking problem on the build node, not a defect in metis's Go code or its dependency versions. A fix here doesn't involve changing application code -- it means confirming the build node has working IPv6 (or a working IPv4 fallback/route) egress to proxy.golang.org and then re-running the build; if this keeps happening, it's worth asking platform/infra whether that build node's network path to the Go module proxy is flaky, or setting up a local GOPROXY mirror/vendoring so builds don't depend on live egress to proxy.golang.org. Separately, and unrelated to this specific failure, the log also shows the build node failing to register qemu-aarch64 for cross-arch emulation ('cannot register "/usr/bin/qemu-aarch64" ... no such file or directory'), which means arm64 image builds on this node may not actually work even though arm64 is listed as a supported platform -- that looks like a real gap worth investigating on its own, since it will surface again the next time an arm64 build is attempted. Ariadne did not authorize automated remediation: `classification_not_supported: 'go_module_download_network_unreachable'`. ## Facts - **jenkins** — Jenkins build metis #297 (https://ci.bstein.dev/job/metis/297/) finished with result FAILURE; Ariadne's bundle marks 'Build & push images' as the first failed pipeline stage. (`jenkins.result=FAILURE, jenkins.first_failed_stage='Build & push images'`) - **jenkins** — The build's own JUnit results show all 309 tests passed (the 'Tests / Declarative: Post Actions' Gitea check reported SUCCESS, passed: 309), and jenkins.failed_tests is empty, so no test assertions failed in this build. (`console_tail; jenkins.failed_tests=[]`) - **jenkins** — At console line 2052, the Docker build step 'RUN ... go build -o /out/metis ... -o /out/oras oras.land/oras/cmd/oras' failed with: 'oras.land/oras@v1.2.2: Get "https://proxy.golang.org/oras.land/oras/@v/v1.2.2.zip": dial tcp [2607:f8b0:4023:100b::8d]:443: connect: network is unreachable', followe... (`console_failures[4] (marker 'ERROR:', line_number 2052)`) - **jenkins** — Immediately before the 'Build & push images' stage begins (console line ~1572), a buildx diagnostic call to the builder node ('Get http://localhost:2375/v1.47/containers/buildx_buildkit_metis-builder-2970/json') failed with 'context deadline exceeded', but the script explicitly returned 0 afterwa... (`console_failures[3] (marker 'ERROR:', line_number 1572)`) - **jenkins** — At console line 1524, binfmt_misc registration for qemu-aarch64 failed ('cannot register "/usr/bin/qemu-aarch64" ... no such file or directory'), and the resulting emulator list only shows python3.12 and qemu-x86_64 (no qemu-aarch64), even though linux/arm64 is listed as a supported target platform. (`console_failures[2] (marker 'No such file', line_number 1524)`) - **jenkins** — At console line 1355, SonarQube's Go analyzer logged parser errors for five files (pkg/service/remote.go, cluster.go, remote_helpers.go, app_helpers.go, node_recovery.go: 'Go parser external process returned non-zero exit value: 2'), but the Sonar sensor still completed ('46/46 source files have ... (`console_failures[1] (marker 'non-zero exit', line_number 1355)`) - **opensearch** — The log query for the build's time window (2026-08-17T13:14:00Z to 13:54:47Z) returned zero records, so no supplementary application/service logs are available for this incident. (`log_evidence.records=[], query_window 2026-08-17T13:14:00Z-13:54:47Z`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "metis-297-gz5mh-ttkrj-lk3l7" ``` ## Inferences - The failing `go build` step was compiling for GOARCH=amd64, so the earlier qemu-aarch64/binfmt registration failure a bit earlier in the log does not look like the direct cause of this failure, though it separately suggests arm64 emulation isn't fully set up on this build node and is worth checki... - The SonarQube parser errors on specific Go files look like a pre-existing scanner limitation rather than the cause of this failure, since the Sonar analysis finished normally (46/46 files analyzed) and the pipeline moved on to the image-build stage afterward. - Because the failure is a network-reachability problem reaching proxy.golang.org from the build container rather than anything in metis's own source, no code change is likely needed to fix it. ## Links - Failed build: https://ci.bstein.dev/job/metis/297/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_4306dc82f0e84633a002dced911b7161](https://triage.hermes.bstein.dev/chat?resume=run_4306dc82f0e84633a002dced911b7161)). Hermes has no write access to this repository; no files or infrastructure were changed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/metis#7