[hermes] soteria #954: transient_infra_failure #53

New Issue

hermes-automation · 2026-09-27T07:17:00Z

hermes-automation commented

2026-09-27 07:17:00 +00:00

Hermes auto-triage classified incident soteria/954 as transient_infra_failure (confidence 0.93); first failed gate: Declarative: Checkout SCM (git fetch from scm.bstein.dev returned HTTP 500).

Why a human is needed

Build 954 failed during the initial SCM checkout: the Gitea host at scm.bstein.dev answered the git fetch with an HTTP 500 and truncated the ref advertisement, so Jenkins exhausted its checkout retries and aborted before any code was compiled or tested. This is a server-side failure of the git host, not a defect in the soteria repository; the fix is to re-run the build once the SCM host is healthy, and if the 500s recur, to investigate the Gitea service and its logs for the 07:10-07:14 UTC window.

Ariadne did not authorize automated remediation: evidence_signature_missing.

Facts

jenkins — Build soteria #954 finished FAILURE after 240 seconds (2026-09-27T07:10:15Z to 07:14:15Z). (https://ci.bstein.dev/job/soteria/954/)
jenkins — The first failed pipeline stage was 'Declarative: Checkout SCM'; the build failed before any project stage ran. (jenkins.first_failed_stage)
jenkins — git fetch against https://scm.bstein.dev/titan/soteria.git exited with status 128; stderr was 'error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500' followed by 'fatal: expected flush after ref listing'. (jenkins.console_failures[1] (marker 'ERROR:', line 187))
jenkins — Jenkins retried the checkout until 'Maximum checkout retry attempts reached, aborting', then the post actions could not run because the workspace context was gone (MissingContextVariableException). (jenkins.console_tail)
jenkins — No structured test results were published (jenkins.failed_tests is empty), consistent with the build never reaching a test stage. (jenkins.failed_tests)
opensearch — Correlated application logs are unavailable for the incident window: 'opensearch request failed: [Errno 111] Connection refused'. (log_evidence.error)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "soteria-954-0ng47-pzgj7-whhwn"

Inferences

The build never obtained a copy of the repository, so no soteria code or tests executed; nothing in this failure implicates the repository's own source or test suite.
The HTTP 500 was returned by the git host scm.bstein.dev during the ref-listing phase of the fetch ('expected flush after ref listing' means the server's smart-HTTP response was cut off mid-protocol), which points at a server-side error on the Gitea SCM service at 07:10-07:14 UTC rather than at a...
Re-running the same build against the same commit should succeed once the SCM host is responding normally; if subsequent builds keep failing the same way, the Gitea service itself (or the ingress in front of it) needs investigation, and its server logs for this window are the next evidence to pul...

Links

Failed build: https://ci.bstein.dev/job/soteria/954/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_d5f3b35e2dd442e88337f2975f518958). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `soteria/954` as **transient_infra_failure** (confidence 0.93); first failed gate: `Declarative: Checkout SCM (git fetch from scm.bstein.dev returned HTTP 500)`. ## Why a human is needed Build 954 failed during the initial SCM checkout: the Gitea host at scm.bstein.dev answered the git fetch with an HTTP 500 and truncated the ref advertisement, so Jenkins exhausted its checkout retries and aborted before any code was compiled or tested. This is a server-side failure of the git host, not a defect in the soteria repository; the fix is to re-run the build once the SCM host is healthy, and if the 500s recur, to investigate the Gitea service and its logs for the 07:10-07:14 UTC window. Ariadne did not authorize automated remediation: `evidence_signature_missing`. ## Facts - **jenkins** — Build soteria #954 finished FAILURE after 240 seconds (2026-09-27T07:10:15Z to 07:14:15Z). (`https://ci.bstein.dev/job/soteria/954/`) - **jenkins** — The first failed pipeline stage was 'Declarative: Checkout SCM'; the build failed before any project stage ran. (`jenkins.first_failed_stage`) - **jenkins** — git fetch against https://scm.bstein.dev/titan/soteria.git exited with status 128; stderr was 'error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500' followed by 'fatal: expected flush after ref listing'. (`jenkins.console_failures[1] (marker 'ERROR:', line 187)`) - **jenkins** — Jenkins retried the checkout until 'Maximum checkout retry attempts reached, aborting', then the post actions could not run because the workspace context was gone (MissingContextVariableException). (`jenkins.console_tail`) - **jenkins** — No structured test results were published (jenkins.failed_tests is empty), consistent with the build never reaching a test stage. (`jenkins.failed_tests`) - **opensearch** — Correlated application logs are unavailable for the incident window: 'opensearch request failed: [Errno 111] Connection refused'. (`log_evidence.error`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "soteria-954-0ng47-pzgj7-whhwn" ``` ## Inferences - The build never obtained a copy of the repository, so no soteria code or tests executed; nothing in this failure implicates the repository's own source or test suite. - The HTTP 500 was returned by the git host scm.bstein.dev during the ref-listing phase of the fetch ('expected flush after ref listing' means the server's smart-HTTP response was cut off mid-protocol), which points at a server-side error on the Gitea SCM service at 07:10-07:14 UTC rather than at a... - Re-running the same build against the same commit should succeed once the SCM host is responding normally; if subsequent builds keep failing the same way, the Gitea service itself (or the ingress in front of it) needs investigation, and its server logs for this window are the next evidence to pul... ## Links - Failed build: https://ci.bstein.dev/job/soteria/954/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_d5f3b35e2dd442e88337f2975f518958](https://triage.bstein.dev/chat?resume=run_d5f3b35e2dd442e88337f2975f518958)). Hermes has no write access to this repository; no files or infrastructure were changed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/soteria#53