[hermes] metis #390: registry_push_unauthorized #288

New Issue

hermes-automation · 2026-09-29T14:05:16Z

hermes-automation commented

2026-09-29 14:05:16 +00:00

Hermes auto-triage classified incident metis/390 as registry_push_unauthorized (confidence 0.85); first failed gate: Build & push images.

Why a human is needed

Build 390 compiled, passed all 309 tests, and completed SonarQube analysis, but failed while pushing images: buildx authenticated to registry.bstein.dev, pushed the 0.1.0-390-amd64 manifest successfully, then after about four more minutes of layer pushing received 401 Unauthorized on a blob HEAD request for the latest-amd64 tag and aborted. Credentials that work at the start of a push and fail minutes later point to the registry's bearer token expiring before the multi-tag push completed, not to a bad secret or a problem in the metis code. A one-off re-run will likely pass because the already-pushed layers are cached, but any build with a cold cache or a larger image will hit this again. The durable fix is on the registry/CI side: confirm the token TTL registry.bstein.dev issues for CI push scopes and raise it above worst-case push duration, or rework the push so each tag starts with a fresh token. Confirming the actual configured TTL and choosing between those options is a call for a maintainer, which is why this needs a human rather than an automatic action.

Ariadne did not authorize automated remediation: human_required.

Facts

jenkins — Jenkins build metis #390 finished FAILURE after 2062 seconds; Jenkins reports the first failed stage as 'Build & push images'. (https://ci.bstein.dev/job/metis/390/ (jenkins.result, jenkins.first_failed_stage))
jenkins — The test gate passed: jenkins.failed_tests is empty and the Gitea check 'Tests / Declarative: Post Actions' was published as SUCCESS with 'passed: 309'. (jenkins.failed_tests; jenkins.console_tail)
jenkins — In the 'Build & push images' stage, buildx obtained a pull,push token for bstein/metis ('#25 [auth] bstein/metis:pull,push token for registry.bstein.dev') and successfully pushed the manifest for registry.bstein.dev/bstein/metis:0.1.0-390-amd64. (jenkins.console_failures, line 2080 region)
jenkins — After roughly four further minutes of push activity in the same step (97.6s + 17.8s + 133.2s), the push of registry.bstein.dev/bstein/metis:latest-amd64 failed with 'unexpected status from HEAD request to https://registry.bstein.dev/v2/bstein/metis/blobs/sha256:79522c73...: 401 Unauthorized', and... (jenkins.console_failures, line 2080 region; jenkins.console_tail)
jenkins — Earlier ERROR lines were non-fatal: SonarQube's Go parser failed on 6 files (pkg/image/rootfs.go, pkg/service/remote.go, cluster.go, remote_helpers.go, app_helpers.go, node_recovery.go) yet the sensor completed 46/46 files and that stage passed; the buildx inspect 'context deadline exceeded' erro... (jenkins.console_failures, lines 1355, 1521, 1569)
jenkins — The build agent metis-390-70lsl-7ng2q-cgvbv dropped offline twice during the push (java.lang.InterruptedException) and reconnected both times; the push continued afterwards, so this did not terminate the build. (jenkins.console_failures, line 2080 region)
opensearch — No OpenSearch log records are available for the incident window: 'opensearch request failed: [Errno 111] Connection refused'. (log_evidence.error)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "metis-390-70lsl-7ng2q-cgvbv"

Inferences

The registry credential itself is valid: the very same push step authenticated and pushed the versioned tag before failing. A credential that works at the start of a push and returns 401 several minutes later most plausibly means the registry bearer token expired before the multi-tag push finishe...
Total push time in this step was around four minutes on what looks like a cold layer cache; any registry token lifetime shorter than that will reproduce this on large or uncached builds, so the exposure is systemic to slow pushes rather than specific to this commit.
A fix would involve raising the token lifetime registry.bstein.dev issues for CI push scopes, or restructuring the push so each tag re-authenticates or both tags are exported in one shorter operation; smaller images or better layer caching would also shrink the exposure window. Nothing in the met...
An immediate re-run of the same commit would probably succeed because the already-pushed layers are cached and the retry push would fit inside one token lifetime, but that masks the underlying lifetime problem rather than fixing it.
The SonarQube Go parser failures mean those 6 files were skipped from static analysis; this is unrelated to the build failure but worth a separate look, most likely an analyzer/Go language-version mismatch.

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

Proposed action: retry_push_after_registry_token_expiry
What it would do: Re-run a build once when its image push fails with 401 Unauthorized partway through a push step that had already authenticated and pushed at least one manifest or layer set, since the warm layer cache makes the retried push short enough to complete within a single registry token lifetime.
Evidence that should be required first: Console output from the same build showing a successful pull,push token grant and at least one completed manifest or layer push to the same registry earlier in the same step, with the 401 appearing only later in that step. A 401 on the first authenticated request must not qualify, because that indicates bad or revoked credentials rather than mid-push expiry.

Links

Failed build: https://ci.bstein.dev/job/metis/390/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_08de97794ed54d79941c136a48445108). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `metis/390` as **registry_push_unauthorized** (confidence 0.85); first failed gate: `Build & push images`. ## Why a human is needed Build 390 compiled, passed all 309 tests, and completed SonarQube analysis, but failed while pushing images: buildx authenticated to registry.bstein.dev, pushed the 0.1.0-390-amd64 manifest successfully, then after about four more minutes of layer pushing received 401 Unauthorized on a blob HEAD request for the latest-amd64 tag and aborted. Credentials that work at the start of a push and fail minutes later point to the registry's bearer token expiring before the multi-tag push completed, not to a bad secret or a problem in the metis code. A one-off re-run will likely pass because the already-pushed layers are cached, but any build with a cold cache or a larger image will hit this again. The durable fix is on the registry/CI side: confirm the token TTL registry.bstein.dev issues for CI push scopes and raise it above worst-case push duration, or rework the push so each tag starts with a fresh token. Confirming the actual configured TTL and choosing between those options is a call for a maintainer, which is why this needs a human rather than an automatic action. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Jenkins build metis #390 finished FAILURE after 2062 seconds; Jenkins reports the first failed stage as 'Build & push images'. (`https://ci.bstein.dev/job/metis/390/ (jenkins.result, jenkins.first_failed_stage)`) - **jenkins** — The test gate passed: jenkins.failed_tests is empty and the Gitea check 'Tests / Declarative: Post Actions' was published as SUCCESS with 'passed: 309'. (`jenkins.failed_tests; jenkins.console_tail`) - **jenkins** — In the 'Build & push images' stage, buildx obtained a pull,push token for bstein/metis ('#25 [auth] bstein/metis:pull,push token for registry.bstein.dev') and successfully pushed the manifest for registry.bstein.dev/bstein/metis:0.1.0-390-amd64. (`jenkins.console_failures, line 2080 region`) - **jenkins** — After roughly four further minutes of push activity in the same step (97.6s + 17.8s + 133.2s), the push of registry.bstein.dev/bstein/metis:latest-amd64 failed with 'unexpected status from HEAD request to https://registry.bstein.dev/v2/bstein/metis/blobs/sha256:79522c73...: 401 Unauthorized', and... (`jenkins.console_failures, line 2080 region; jenkins.console_tail`) - **jenkins** — Earlier ERROR lines were non-fatal: SonarQube's Go parser failed on 6 files (pkg/image/rootfs.go, pkg/service/remote.go, cluster.go, remote_helpers.go, app_helpers.go, node_recovery.go) yet the sensor completed 46/46 files and that stage passed; the buildx inspect 'context deadline exceeded' erro... (`jenkins.console_failures, lines 1355, 1521, 1569`) - **jenkins** — The build agent metis-390-70lsl-7ng2q-cgvbv dropped offline twice during the push (java.lang.InterruptedException) and reconnected both times; the push continued afterwards, so this did not terminate the build. (`jenkins.console_failures, line 2080 region`) - **opensearch** — No OpenSearch log records are available for the incident window: 'opensearch request failed: [Errno 111] Connection refused'. (`log_evidence.error`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "metis-390-70lsl-7ng2q-cgvbv" ``` ## Inferences - The registry credential itself is valid: the very same push step authenticated and pushed the versioned tag before failing. A credential that works at the start of a push and returns 401 several minutes later most plausibly means the registry bearer token expired before the multi-tag push finishe... - Total push time in this step was around four minutes on what looks like a cold layer cache; any registry token lifetime shorter than that will reproduce this on large or uncached builds, so the exposure is systemic to slow pushes rather than specific to this commit. - A fix would involve raising the token lifetime registry.bstein.dev issues for CI push scopes, or restructuring the push so each tag re-authenticates or both tags are exported in one shorter operation; smaller images or better layer caching would also shrink the exposure window. Nothing in the met... - An immediate re-run of the same commit would probably succeed because the already-pushed layers are cached and the retry push would fit inside one token lifetime, but that masks the underlying lifetime problem rather than fixing it. - The SonarQube Go parser failures mean those 6 files were skipped from static analysis; this is unrelated to the build failure but worth a separate look, most likely an analyzer/Go language-version mismatch. ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `retry_push_after_registry_token_expiry` - What it would do: Re-run a build once when its image push fails with 401 Unauthorized partway through a push step that had already authenticated and pushed at least one manifest or layer set, since the warm layer cache makes the retried push short enough to complete within a single registry token lifetime. - Evidence that should be required first: Console output from the same build showing a successful pull,push token grant and at least one completed manifest or layer push to the same registry earlier in the same step, with the 401 appearing only later in that step. A 401 on the first authenticated request must not qualify, because that indicates bad or revoked credentials rather than mid-push expiry. ## Links - Failed build: https://ci.bstein.dev/job/metis/390/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_08de97794ed54d79941c136a48445108](https://triage.bstein.dev/chat?resume=run_08de97794ed54d79941c136a48445108)). Hermes has no write access to this repository; no files or infrastructure were changed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/metis#288