[hermes] soteria #305: quality_gate_enforcement_failure #14

New Issue

hermes-automation · 2026-08-19T21:42:41Z

hermes-automation commented

2026-08-19 21:42:41 +00:00

Hermes auto-triage classified incident soteria/305 as quality_gate_enforcement_failure (confidence 0.6); first failed gate: Enforce quality gate.

Why a human is needed

Build 305 of soteria failed in the "Enforce quality gate" stage after the SonarQube and supply-chain evidence collection stage had run; the image build and push stage was skipped as a result, and the pipeline exited with code 1. No failing tests were published with the build, so nothing points at a test regression. The retained console is truncated and does not include the gate stage's own output, so the specific condition that tripped it is not visible in the evidence available here, and deciding whether it is legitimate or an artifact of incomplete analysis needs a maintainer to look at the full log or the archived gate report. The same run does show two concrete analysis defects worth fixing regardless: the Go analyzer could not parse internal/server/policy_runtime.go (external parser exit code 2, while all 25 other Go files parsed), so that file was excluded from the results; and the TypeScript/JavaScript sensor aborted because the analysis image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64 has no node binary, so any JS/TS sources were not analyzed at all. Both errors were non-fatal to the scan, which is exactly why they can silently skew whatever the gate then measured. A fix means adding Node.js to the quality-tools image (or excluding JS/TS sources if the repository has none worth scanning), and either upgrading the sonar-go analyzer in that image or correcting the syntax in policy_runtime.go if it genuinely fails to build.

Ariadne did not authorize automated remediation: human_required.

Facts

jenkins — Build 305 of the soteria job started 2026-08-19T20:59:00.722Z, ran 2397.6s, and finished with result FAILURE at 2026-08-19T21:38:58.277Z. (https://ci.bstein.dev/job/soteria/305/ (jenkins.result, jenkins.timestamps, jenkins.duration_seconds))
jenkins — The pipeline stage reported as the first to fail is "Enforce quality gate". (jenkins.first_failed_stage)
jenkins — The next stage, "Build & push image", did not run: the console records 'Stage "Build & push image" skipped due to earlier failure(s)', and the pipeline ended with 'ERROR: script returned exit code 1'. (jenkins.console_tail; jenkins.console_failures[5] (line 3326))
jenkins — No failing structured test results were published by this build; jenkins.failed_tests is an empty array. (jenkins.failed_tests)
jenkins — The stage "Collect SonarQube and supply-chain evidence" ran SonarScanner CLI 8.0.1.6346 against SonarQube Server 9.9.8.100196 starting 21:00:46 and proceeded through its sensors without the stage itself being reported as failed. (jenkins.console_failures[2] (line 221))
jenkins — At 21:06:16 the Go analyzer failed on one file: "Unable to parse file: file:///home/jenkins/agent/workspace/Soteria/internal/server/policy_runtime.go" / "Cannot parse 'internal/server/policy_runtime.go': Go parser external process returned non-zero exit value: 2". The other 25 Go source files wer... (jenkins.console_failures[3] (line 293))
jenkins — At 21:07:27 the TypeScript/JavaScript sensor aborted because the Node.js runtime is absent from the analysis container: "Error when running: 'node -v'. Is Node.js available during analysis?" caused by 'java.io.IOException: Cannot run program "node": error=2, No such file or directory'. (jenkins.console_failures[3] (line 293) and jenkins.console_failures[4] (line 347))
jenkins — The analysis container for that stage is the image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, mounted on the agent workspace volume at /home/jenkins/agent. (jenkins.console_failures[0] (line 97, pod template))
jenkins — The retained console is truncated (jenkins.console_truncated = true) and none of the captured excerpts contain the output produced inside the "Enforce quality gate" stage itself; the latest excerpt is only the end-of-pipeline exit-code line. (jenkins.console_truncated; jenkins.console_failures[5] (line 3326))
opensearch — No log records were returned for the correlation window 2026-08-19T20:54:00.722Z to 2026-08-19T21:43:58.277Z, with no query error reported. (log_evidence.records (empty), log_evidence.error = null)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "soteria-305-4j2xj-0724c-8l6kq"

Inferences

The build completed checkout and the SonarQube/supply-chain evidence collection, then failed at the step that enforces the quality gate; image build and publish never ran, which is why the Gitea integration reports no assets published. The exit code 1 at the end of the pipeline is the propagated ...
This is not a test regression. No failing test results were published with the build, and no assertion or test class is named anywhere in the retained evidence, so the failure is a threshold/report check rather than broken test code.
Two analyzer defects visible in the same run degraded the data the gate was evaluated against, and either could be enough to push a threshold-based gate below its limit. First, internal/server/policy_runtime.go was not analyzed at all, so any issues, duplication and coverage attributable to that ...
The Go parse failure is more likely a toolchain mismatch than invalid source. The other 25 Go files parsed cleanly, and the sonar-go parser exiting with status 2 on a single file is the usual signature of language syntax the bundled parser does not understand (for example newer generics or langua...
A fix has three parts, in the order a maintainer would tackle them. (1) Read the untruncated console for the "Enforce quality gate" stage or the archived quality-gate artifact from build 305 to see which specific condition failed, since that determines whether the two analyzer errors are actually...
Timing is not implicated. The 40-minute duration is dominated by legitimate work, including a 193-second plugin download at 21:04:48, and nothing in the evidence shows a timeout, a cancelled step, or a resource limit being hit.

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

Proposed action: fetch_full_failed_stage_console
What it would do: For a build whose retained console excerpts do not cover the stage that failed, pull the untruncated console region for that stage plus any archived quality-gate report artifacts and attach them to the incident record.
Evidence that should be required first: The build reported a terminal failure with a named failed stage, the retained console is marked truncated, no captured excerpt overlaps that stage's output, and Jenkins still retains the full console log and archived artifacts for that build number.

Links

Failed build: https://ci.bstein.dev/job/soteria/305/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and `hermes_autotriage_diagno

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `soteria/305` as **quality_gate_enforcement_failure** (confidence 0.6); first failed gate: `Enforce quality gate`. ## Why a human is needed Build 305 of soteria failed in the "Enforce quality gate" stage after the SonarQube and supply-chain evidence collection stage had run; the image build and push stage was skipped as a result, and the pipeline exited with code 1. No failing tests were published with the build, so nothing points at a test regression. The retained console is truncated and does not include the gate stage's own output, so the specific condition that tripped it is not visible in the evidence available here, and deciding whether it is legitimate or an artifact of incomplete analysis needs a maintainer to look at the full log or the archived gate report. The same run does show two concrete analysis defects worth fixing regardless: the Go analyzer could not parse internal/server/policy_runtime.go (external parser exit code 2, while all 25 other Go files parsed), so that file was excluded from the results; and the TypeScript/JavaScript sensor aborted because the analysis image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64 has no node binary, so any JS/TS sources were not analyzed at all. Both errors were non-fatal to the scan, which is exactly why they can silently skew whatever the gate then measured. A fix means adding Node.js to the quality-tools image (or excluding JS/TS sources if the repository has none worth scanning), and either upgrading the sonar-go analyzer in that image or correcting the syntax in policy_runtime.go if it genuinely fails to build. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build 305 of the soteria job started 2026-08-19T20:59:00.722Z, ran 2397.6s, and finished with result FAILURE at 2026-08-19T21:38:58.277Z. (`https://ci.bstein.dev/job/soteria/305/ (jenkins.result, jenkins.timestamps, jenkins.duration_seconds)`) - **jenkins** — The pipeline stage reported as the first to fail is "Enforce quality gate". (`jenkins.first_failed_stage`) - **jenkins** — The next stage, "Build & push image", did not run: the console records 'Stage "Build & push image" skipped due to earlier failure(s)', and the pipeline ended with 'ERROR: script returned exit code 1'. (`jenkins.console_tail; jenkins.console_failures[5] (line 3326)`) - **jenkins** — No failing structured test results were published by this build; jenkins.failed_tests is an empty array. (`jenkins.failed_tests`) - **jenkins** — The stage "Collect SonarQube and supply-chain evidence" ran SonarScanner CLI 8.0.1.6346 against SonarQube Server 9.9.8.100196 starting 21:00:46 and proceeded through its sensors without the stage itself being reported as failed. (`jenkins.console_failures[2] (line 221)`) - **jenkins** — At 21:06:16 the Go analyzer failed on one file: "Unable to parse file: file:///home/jenkins/agent/workspace/Soteria/internal/server/policy_runtime.go" / "Cannot parse 'internal/server/policy_runtime.go': Go parser external process returned non-zero exit value: 2". The other 25 Go source files wer... (`jenkins.console_failures[3] (line 293)`) - **jenkins** — At 21:07:27 the TypeScript/JavaScript sensor aborted because the Node.js runtime is absent from the analysis container: "Error when running: 'node -v'. Is Node.js available during analysis?" caused by 'java.io.IOException: Cannot run program "node": error=2, No such file or directory'. (`jenkins.console_failures[3] (line 293) and jenkins.console_failures[4] (line 347)`) - **jenkins** — The analysis container for that stage is the image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, mounted on the agent workspace volume at /home/jenkins/agent. (`jenkins.console_failures[0] (line 97, pod template)`) - **jenkins** — The retained console is truncated (jenkins.console_truncated = true) and none of the captured excerpts contain the output produced inside the "Enforce quality gate" stage itself; the latest excerpt is only the end-of-pipeline exit-code line. (`jenkins.console_truncated; jenkins.console_failures[5] (line 3326)`) - **opensearch** — No log records were returned for the correlation window 2026-08-19T20:54:00.722Z to 2026-08-19T21:43:58.277Z, with no query error reported. (`log_evidence.records (empty), log_evidence.error = null`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "soteria-305-4j2xj-0724c-8l6kq" ``` ## Inferences - The build completed checkout and the SonarQube/supply-chain evidence collection, then failed at the step that enforces the quality gate; image build and publish never ran, which is why the Gitea integration reports no assets published. The exit code 1 at the end of the pipeline is the propagated ... - This is not a test regression. No failing test results were published with the build, and no assertion or test class is named anywhere in the retained evidence, so the failure is a threshold/report check rather than broken test code. - Two analyzer defects visible in the same run degraded the data the gate was evaluated against, and either could be enough to push a threshold-based gate below its limit. First, internal/server/policy_runtime.go was not analyzed at all, so any issues, duplication and coverage attributable to that ... - The Go parse failure is more likely a toolchain mismatch than invalid source. The other 25 Go files parsed cleanly, and the sonar-go parser exiting with status 2 on a single file is the usual signature of language syntax the bundled parser does not understand (for example newer generics or langua... - A fix has three parts, in the order a maintainer would tackle them. (1) Read the untruncated console for the "Enforce quality gate" stage or the archived quality-gate artifact from build 305 to see which specific condition failed, since that determines whether the two analyzer errors are actually... - Timing is not implicated. The 40-minute duration is dominated by legitimate work, including a 193-second plugin download at 21:04:48, and nothing in the evidence shows a timeout, a cancelled step, or a resource limit being hit. ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `fetch_full_failed_stage_console` - What it would do: For a build whose retained console excerpts do not cover the stage that failed, pull the untruncated console region for that stage plus any archived quality-gate report artifacts and attach them to the incident record. - Evidence that should be required first: The build reported a terminal failure with a named failed stage, the retained console is marked truncated, no captured excerpt overlaps that stage's output, and Jenkins still retains the full console log and archived artifacts for that build number. ## Links - Failed build: https://ci.bstein.dev/job/soteria/305/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagno _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/soteria#14