titan/ananke
3
0
Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[hermes] ananke #321: unattributed_quality_gate_failure #17

New Issue
Open
opened 2026-09-11 18:28:11 +00:00 by hermes-automation · 0 comments
hermes-automation commented 2026-09-11 18:28:11 +00:00
Member
Copy Link

Hermes auto-triage classified incident ananke/321 as unattributed_quality_gate_failure (confidence 0.5); first failed gate: unknown — the 'Enforce quality gate' stage exited 1, but sonarqube and supply_chain both recorded ok, so the failing check is an earlier one (style, loc, coverage, tests, or gate_glue) whose output lies in the truncated portion of the console.

Why a human is needed

Build 321 of ananke failed in the 'Enforce quality gate' stage after about 29 minutes. No test results were published with the build, so nothing points at a specific failing test. The end of the console shows the gate script recording sonar_status=ok and supply_status=ok and then exiting 1, which means the check that failed the build ran earlier in that same script, and its output sits in the part of the console log that was cut off before it reached this report. Identifying it requires opening the full console for build 321 and finding the first gate check that recorded a non-ok status (style, lines-of-code, coverage, unit tests, or the gate glue step), which is a judgement a maintainer has to make against the actual log. Separately, the one failure excerpt that was captured shows a real problem worth fixing: during the SonarQube scan at 18:12:57 the Go analyzer subprocess died on the first file with a 'Broken pipe' and then returned exit status 2 for every remaining Go file, so none of the roughly 74 Go source files were analyzed in this run. That scan still reported ok, so it did not fail the build, but Go quality and coverage data is currently being dropped silently; the usual fixes are raising the memory available to the scanner container or aligning the Go toolchain version in the agent image with what the analyzer expects. Correlated application logs for this build window could not be read (the log store refused the connection), so there is no second evidence source to attribute the failing check.

Ariadne did not authorize automated remediation: human_required.

Facts

  • jenkins — Build 321 of job ananke ended in FAILURE, running from 2026-09-11T17:57:00.647Z to 2026-09-11T18:25:41.583Z (1720.9 seconds). (https://ci.bstein.dev/job/ananke/321/ (jenkins.result, jenkins.timestamps, jenkins.duration_seconds))
  • jenkins — The pipeline stage that failed is 'Enforce quality gate'. (jenkins.first_failed_stage)
  • jenkins — The build published no structured test results: the failed test list is empty. (jenkins.failed_tests (empty array))
  • jenkins — At the end of the build the gate script recorded sonar_status=ok and supply_status=ok, then executed 'exit 1'; the pipeline reported 'ERROR: script returned exit code 1'. (jenkins.console_tail)
  • jenkins — The only captured failure excerpt (console line 282, timestamps 18:12:57) is from the SonarQube scan: the Go sensor announced '74 source files to be analyzed', then failed on internal/cluster/orchestrator_longhorn_reconcile.go with 'Broken pipe' and on every subsequent Go file with 'Go parser ext... (jenkins.console_failures[0] (marker 'non-zero exit', line_number 282))
  • jenkins — The console text in the bundle is truncated, so the middle of the build log is not present. (jenkins.console_truncated = true)
  • opensearch — No correlated log records could be retrieved for the build window 17:52:00.647Z–18:30:41.583Z; the log query failed with 'opensearch request failed: [Errno 111] Connection refused'. (log_evidence.error, log_evidence.records (empty))

Evidence

The earliest failure region of the build console:

18:12:57.918 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/cluster/orchestrator_service_stability.go. 
18:12:57.918 ERROR Cannot parse 'internal/cluster/orchestrator_service_stability.go': Go parser external process returned non-zero exit value: 2
18:12:57.925 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/cluster/orchestrator_shutdown_mode.go. 
18:12:57.926 ERROR Cannot parse 'internal/cluster/orchestrator_shutdown_mode.go': Go parser external process returned non-zero exit value: 2
18:12:57.934 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/apply_defaults.go. 
18:12:57.934 ERROR Cannot parse 'internal/config/apply_defaults.go': Go parser external process returned non-zero exit value: 2
18:12:57.945 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/service/daemon.go. 
18:12:57.945 ERROR Cannot parse 'internal/service/daemon.go': Go parser external process returned non-zero exit value: 2
18:12:57.957 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/load.go. 
18:12:57.957 ERROR Cannot parse 'internal/confi
... (truncated; the full log is in the linked build)

Inferences

  • The check that actually failed the build ran earlier in the same gate script than the SonarQube and supply-chain checks: both of those recorded 'ok' in the console tail, and the script still exited 1 immediately afterwards. Its output falls inside the truncated region of the console, so it is not...
  • The Sonar Go parse errors are almost certainly not the reason the build failed — the scan that produced them still reported ok. They were captured only because a generic 'non-zero exit' text match hit the analyzer's own error wording.
  • The Go analyzer failure is nonetheless real and independent: the parser subprocess died on the first file ('Broken pipe') and returned exit status 2 for every file after it, which is the signature of a crashed or killed helper process rather than bad source syntax. The practical effect is that no...
  • Likely causes for the Go parser dying on every file are a memory limit on the scanner container killing the helper process, or a mismatch between the Go toolchain in the agent image and the version the analyzer expects. Neither can be confirmed from this bundle.
  • There is no evidence in this bundle of an agent provisioning problem, a disk-exhaustion error, or a network/registry error; the build ran for roughly 29 minutes and reached the final gate script.

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

  • Proposed action: fetch_full_gate_console_and_artifacts
  • What it would do: Read back the complete, untruncated Jenkins console log and the archived quality-gate artifacts for the failed build, and extract the per-gate status lines so the first non-ok check is named directly in the evidence instead of being inferred.
  • Evidence that should be required first: The build is in a terminal FAILURE state, the captured console is marked truncated, no structured test results were published, and Jenkins still retains the build's console log and archived artifacts.

Links

  • Failed build: https://ci.bstein.dev/job/ananke/321/
  • Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_4239e7a4013647958b466afea42ef32a). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `ananke/321` as **unattributed_quality_gate_failure** (confidence 0.5); first failed gate: `unknown — the 'Enforce quality gate' stage exited 1, but sonarqube and supply_chain both recorded ok, so the failing check is an earlier one (style, loc, coverage, tests, or gate_glue) whose output lies in the truncated portion of the console`. ## Why a human is needed Build 321 of ananke failed in the 'Enforce quality gate' stage after about 29 minutes. No test results were published with the build, so nothing points at a specific failing test. The end of the console shows the gate script recording sonar_status=ok and supply_status=ok and then exiting 1, which means the check that failed the build ran earlier in that same script, and its output sits in the part of the console log that was cut off before it reached this report. Identifying it requires opening the full console for build 321 and finding the first gate check that recorded a non-ok status (style, lines-of-code, coverage, unit tests, or the gate glue step), which is a judgement a maintainer has to make against the actual log. Separately, the one failure excerpt that was captured shows a real problem worth fixing: during the SonarQube scan at 18:12:57 the Go analyzer subprocess died on the first file with a 'Broken pipe' and then returned exit status 2 for every remaining Go file, so none of the roughly 74 Go source files were analyzed in this run. That scan still reported ok, so it did not fail the build, but Go quality and coverage data is currently being dropped silently; the usual fixes are raising the memory available to the scanner container or aligning the Go toolchain version in the agent image with what the analyzer expects. Correlated application logs for this build window could not be read (the log store refused the connection), so there is no second evidence source to attribute the failing check. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build 321 of job ananke ended in FAILURE, running from 2026-09-11T17:57:00.647Z to 2026-09-11T18:25:41.583Z (1720.9 seconds). (`https://ci.bstein.dev/job/ananke/321/ (jenkins.result, jenkins.timestamps, jenkins.duration_seconds)`) - **jenkins** — The pipeline stage that failed is 'Enforce quality gate'. (`jenkins.first_failed_stage`) - **jenkins** — The build published no structured test results: the failed test list is empty. (`jenkins.failed_tests (empty array)`) - **jenkins** — At the end of the build the gate script recorded sonar_status=ok and supply_status=ok, then executed 'exit 1'; the pipeline reported 'ERROR: script returned exit code 1'. (`jenkins.console_tail`) - **jenkins** — The only captured failure excerpt (console line 282, timestamps 18:12:57) is from the SonarQube scan: the Go sensor announced '74 source files to be analyzed', then failed on internal/cluster/orchestrator_longhorn_reconcile.go with 'Broken pipe' and on every subsequent Go file with 'Go parser ext... (`jenkins.console_failures[0] (marker 'non-zero exit', line_number 282)`) - **jenkins** — The console text in the bundle is truncated, so the middle of the build log is not present. (`jenkins.console_truncated = true`) - **opensearch** — No correlated log records could be retrieved for the build window 17:52:00.647Z–18:30:41.583Z; the log query failed with 'opensearch request failed: [Errno 111] Connection refused'. (`log_evidence.error, log_evidence.records (empty)`) ## Evidence The earliest failure region of the build console: ``` 18:12:57.918 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/cluster/orchestrator_service_stability.go. 18:12:57.918 ERROR Cannot parse 'internal/cluster/orchestrator_service_stability.go': Go parser external process returned non-zero exit value: 2 18:12:57.925 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/cluster/orchestrator_shutdown_mode.go. 18:12:57.926 ERROR Cannot parse 'internal/cluster/orchestrator_shutdown_mode.go': Go parser external process returned non-zero exit value: 2 18:12:57.934 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/apply_defaults.go. 18:12:57.934 ERROR Cannot parse 'internal/config/apply_defaults.go': Go parser external process returned non-zero exit value: 2 18:12:57.945 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/service/daemon.go. 18:12:57.945 ERROR Cannot parse 'internal/service/daemon.go': Go parser external process returned non-zero exit value: 2 18:12:57.957 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/load.go. 18:12:57.957 ERROR Cannot parse 'internal/confi ... (truncated; the full log is in the linked build) ``` ## Inferences - The check that actually failed the build ran earlier in the same gate script than the SonarQube and supply-chain checks: both of those recorded 'ok' in the console tail, and the script still exited 1 immediately afterwards. Its output falls inside the truncated region of the console, so it is not... - The Sonar Go parse errors are almost certainly not the reason the build failed — the scan that produced them still reported ok. They were captured only because a generic 'non-zero exit' text match hit the analyzer's own error wording. - The Go analyzer failure is nonetheless real and independent: the parser subprocess died on the first file ('Broken pipe') and returned exit status 2 for every file after it, which is the signature of a crashed or killed helper process rather than bad source syntax. The practical effect is that no... - Likely causes for the Go parser dying on every file are a memory limit on the scanner container killing the helper process, or a mismatch between the Go toolchain in the agent image and the version the analyzer expects. Neither can be confirmed from this bundle. - There is no evidence in this bundle of an agent provisioning problem, a disk-exhaustion error, or a network/registry error; the build ran for roughly 29 minutes and reached the final gate script. ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `fetch_full_gate_console_and_artifacts` - What it would do: Read back the complete, untruncated Jenkins console log and the archived quality-gate artifacts for the failed build, and extract the per-gate status lines so the first non-ok check is named directly in the evidence instead of being inferred. - Evidence that should be required first: The build is in a terminal FAILURE state, the captured console is marked truncated, no structured test results were published, and Jenkins still retains the build's console log and archived artifacts. ## Links - Failed build: https://ci.bstein.dev/job/ananke/321/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_4239e7a4013647958b466afea42ef32a](https://triage.bstein.dev/chat?resume=run_4239e7a4013647958b466afea42ef32a)). Hermes has no write access to this repository; no files or infrastructure were changed. <!-- hermes-triage job=ananke classification=unattributed_quality_gate_failure incident=ananke/321 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/ananke#17
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?