[hermes] ananke #325: sonarqube_go_analyzer_crash #19

New Issue

hermes-automation · 2026-09-13T18:29:40Z

hermes-automation commented

2026-09-13 18:29:40 +00:00

Hermes auto-triage classified incident ananke/325 as sonarqube_go_analyzer_crash (confidence 0.55); first failed gate: sonarqube.

Why a human is needed

The build compiled and its tests passed (no test failures were published), but the quality-gate stage exited 1. The only failure recorded anywhere in the build is that the SonarQube Go analyzer's external parser process died at 18:11:51 ('Stream closed' on the first file, then 'Broken pipe' / 'returned non-zero exit value: 2' for every subsequent file), leaving most of the 74 Go files unanalyzed; the gate script's final two visible checks both read ok before it exited 1, so the failing check sits in the truncated middle of the log. Because the console truncation hides exactly which check turned the exit code red, and because the fix requires a judgement call between upgrading the Go analyzer plugin, giving the scanner more memory, and excluding a file that crashes the parser, a maintainer should confirm the root cause before changing the pipeline or scanner configuration.

Ariadne did not authorize automated remediation: human_required.

Facts

jenkins — Build ananke #325 ended in FAILURE at 2026-09-13T18:25:21Z after about 28 minutes, and the first failed pipeline stage was 'Enforce quality gate'. (https://ci.bstein.dev/job/ananke/325/ (jenkins.result, jenkins.first_failed_stage))
jenkins — No structured test failures were published for this build. (jenkins.failed_tests is empty)
jenkins — The console tail shows the gate script computing sonar_status=ok and supply_status=ok immediately before 'exit 1' and 'ERROR: script returned exit code 1'. (jenkins.console_tail)
jenkins — In the earliest retained console failure region, the SonarQube scanner's Go sensor reported '74 source files to be analyzed' and then failed to parse at least 40 of them; the first error at 18:11:51 was Cannot parse 'internal/cluster/orchestrator_longhorn_reconcile.go': Stream closed, followed by... (jenkins.console_failures[0] (marker 'non-zero exit', line 290))
jenkins — The Python analysis in the same scanner run completed normally (2/2 files analyzed; Cobertura and XUnit sensors finished without error) immediately before the Go sensor errors began. (jenkins.console_failures[0])
jenkins — The console log is truncated, so the portion of the gate script between the scanner output and the final two visible status checks is not retained. (jenkins.console_truncated = true)
opensearch — Centralized log correlation was unavailable for this incident window; the OpenSearch query failed with connection refused. (log_evidence.error: 'opensearch request failed: [Errno 111] Connection refused')

Evidence

The earliest failure region of the build console:

18:11:51.508 ERROR Cannot parse 'internal/cluster/orchestrator_service_stability.go': Go parser external process returned non-zero exit value: 2
18:11:51.515 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/cluster/orchestrator_shutdown_mode.go. 
18:11:51.515 ERROR Cannot parse 'internal/cluster/orchestrator_shutdown_mode.go': Go parser external process returned non-zero exit value: 2
18:11:51.524 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/apply_defaults.go. 
18:11:51.524 ERROR Cannot parse 'internal/config/apply_defaults.go': Go parser external process returned non-zero exit value: 2
18:11:51.533 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/service/daemon.go. 
18:11:51.534 ERROR Cannot parse 'internal/service/daemon.go': Go parser external process returned non-zero exit value: 2
18:11:51.542 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/load.go. 
18:11:51.542 ERROR Cannot parse 'internal/config/load.go': Go parser external process returned non-zero exit value: 2
18:11:51.550 ERROR Unable to parse file: file:///home/jenkins/agent
... (truncated; the full log is in the linked build)

Inferences

The error pattern — one file failing with 'Stream closed', then every later Go file failing with 'exit value: 2' or 'Broken pipe' — indicates the scanner's single external Go parser subprocess died early in the analysis and never recovered, rather than dozens of files being individually unparseab...
The two status values visible at the end of the gate script (sonar_status=ok, supply_status=ok) both passed, yet the script exited 1, so the enforced failure was recorded earlier in the script in a portion of console that was not retained. The Go-analysis crash is the only failure activity record...
A fix would start by re-running sonar-scanner against this same commit to see whether the Go parser crash reproduces. If it does, upgrade or pin the SonarQube Go analyzer plugin, raise the memory available to the scanner step, or bisect for a specific file that kills the parser and temporarily ex...

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

Proposed action: rerun_analysis_on_go_parser_crash
What it would do: Re-run the failed build once when the SonarQube Go parser subprocess crash signature is present, to distinguish a transient parser-process death (for example memory pressure on the agent) from a deterministic analyzer fault that needs a plugin or configuration change.
Evidence that should be required first: Console from the failed build shows the Sonar Go sensor's external parser dying across many files within one analysis ('Stream closed', 'Broken pipe', or 'returned non-zero exit value'), with no published test failures and no other gate failure recorded in the retained console.

Links

Failed build: https://ci.bstein.dev/job/ananke/325/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_5f7497fcb242402cbf761f90068cfc80). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `ananke/325` as **sonarqube_go_analyzer_crash** (confidence 0.55); first failed gate: `sonarqube`. ## Why a human is needed The build compiled and its tests passed (no test failures were published), but the quality-gate stage exited 1. The only failure recorded anywhere in the build is that the SonarQube Go analyzer's external parser process died at 18:11:51 ('Stream closed' on the first file, then 'Broken pipe' / 'returned non-zero exit value: 2' for every subsequent file), leaving most of the 74 Go files unanalyzed; the gate script's final two visible checks both read ok before it exited 1, so the failing check sits in the truncated middle of the log. Because the console truncation hides exactly which check turned the exit code red, and because the fix requires a judgement call between upgrading the Go analyzer plugin, giving the scanner more memory, and excluding a file that crashes the parser, a maintainer should confirm the root cause before changing the pipeline or scanner configuration. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build ananke #325 ended in FAILURE at 2026-09-13T18:25:21Z after about 28 minutes, and the first failed pipeline stage was 'Enforce quality gate'. (`https://ci.bstein.dev/job/ananke/325/ (jenkins.result, jenkins.first_failed_stage)`) - **jenkins** — No structured test failures were published for this build. (`jenkins.failed_tests is empty`) - **jenkins** — The console tail shows the gate script computing sonar_status=ok and supply_status=ok immediately before 'exit 1' and 'ERROR: script returned exit code 1'. (`jenkins.console_tail`) - **jenkins** — In the earliest retained console failure region, the SonarQube scanner's Go sensor reported '74 source files to be analyzed' and then failed to parse at least 40 of them; the first error at 18:11:51 was Cannot parse 'internal/cluster/orchestrator_longhorn_reconcile.go': Stream closed, followed by... (`jenkins.console_failures[0] (marker 'non-zero exit', line 290)`) - **jenkins** — The Python analysis in the same scanner run completed normally (2/2 files analyzed; Cobertura and XUnit sensors finished without error) immediately before the Go sensor errors began. (`jenkins.console_failures[0]`) - **jenkins** — The console log is truncated, so the portion of the gate script between the scanner output and the final two visible status checks is not retained. (`jenkins.console_truncated = true`) - **opensearch** — Centralized log correlation was unavailable for this incident window; the OpenSearch query failed with connection refused. (`log_evidence.error: 'opensearch request failed: [Errno 111] Connection refused'`) ## Evidence The earliest failure region of the build console: ``` 18:11:51.508 ERROR Cannot parse 'internal/cluster/orchestrator_service_stability.go': Go parser external process returned non-zero exit value: 2 18:11:51.515 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/cluster/orchestrator_shutdown_mode.go. 18:11:51.515 ERROR Cannot parse 'internal/cluster/orchestrator_shutdown_mode.go': Go parser external process returned non-zero exit value: 2 18:11:51.524 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/apply_defaults.go. 18:11:51.524 ERROR Cannot parse 'internal/config/apply_defaults.go': Go parser external process returned non-zero exit value: 2 18:11:51.533 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/service/daemon.go. 18:11:51.534 ERROR Cannot parse 'internal/service/daemon.go': Go parser external process returned non-zero exit value: 2 18:11:51.542 ERROR Unable to parse file: file:///home/jenkins/agent/workspace/ananke/internal/config/load.go. 18:11:51.542 ERROR Cannot parse 'internal/config/load.go': Go parser external process returned non-zero exit value: 2 18:11:51.550 ERROR Unable to parse file: file:///home/jenkins/agent ... (truncated; the full log is in the linked build) ``` ## Inferences - The error pattern — one file failing with 'Stream closed', then every later Go file failing with 'exit value: 2' or 'Broken pipe' — indicates the scanner's single external Go parser subprocess died early in the analysis and never recovered, rather than dozens of files being individually unparseab... - The two status values visible at the end of the gate script (sonar_status=ok, supply_status=ok) both passed, yet the script exited 1, so the enforced failure was recorded earlier in the script in a portion of console that was not retained. The Go-analysis crash is the only failure activity record... - A fix would start by re-running sonar-scanner against this same commit to see whether the Go parser crash reproduces. If it does, upgrade or pin the SonarQube Go analyzer plugin, raise the memory available to the scanner step, or bisect for a specific file that kills the parser and temporarily ex... ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `rerun_analysis_on_go_parser_crash` - What it would do: Re-run the failed build once when the SonarQube Go parser subprocess crash signature is present, to distinguish a transient parser-process death (for example memory pressure on the agent) from a deterministic analyzer fault that needs a plugin or configuration change. - Evidence that should be required first: Console from the failed build shows the Sonar Go sensor's external parser dying across many files within one analysis ('Stream closed', 'Broken pipe', or 'returned non-zero exit value'), with no published test failures and no other gate failure recorded in the retained console. ## Links - Failed build: https://ci.bstein.dev/job/ananke/325/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_5f7497fcb242402cbf761f90068cfc80](https://triage.bstein.dev/chat?resume=run_5f7497fcb242402cbf761f90068cfc80)). Hermes has no write access to this repository; no files or infrastructure were changed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/ananke#19