titan/soteria
3
0
Fork 0
Code Issues 23 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[hermes] soteria #337: sonarqube_analyzer_toolchain_unavailable #21

New Issue
Open
opened 2026-09-04 21:27:04 +00:00 by hermes-automation · 0 comments
hermes-automation commented 2026-09-04 21:27:04 +00:00
Member
Copy Link

Hermes auto-triage classified incident soteria/337 as sonarqube_analyzer_toolchain_unavailable (confidence 0.55); first failed gate: sonarqube.

Why a human is needed

Build 337 of the soteria job failed in the "Enforce quality gate" stage, and no test failed: the build published no test results, and no compile or test error appears in the retained log. What actually broke is the static-analysis step's own environment. Between 21:04:16 and 21:04:18 the Go analyzer failed to parse all 26 Go files in the repository - every file from cmd/soteria/main.go through the internal/api, internal/config, internal/k8s, internal/longhorn and internal/server packages - each with "Go parser external process returned non-zero exit value: 2". Seconds later, at 21:04:33, the TypeScript analyzer aborted with "Cannot run program "node": error=2, No such file or directory". A uniform parser exit across every file, plus a completely absent node binary, indicates the analysis container lacks working language tooling - most plausibly an analyzer parser binary built for a different CPU architecture than the build agent, and an image with no Node.js installed. With no Go file parsed, the analysis had no view of the service at all, and the pipeline exited 1 and skipped the image build. Fixing this means changing where and how the analysis stage runs, not changing the service: use an image for that stage that ships Node.js and an analyzer whose Go parser is executable on the agent's architecture (or upgrade the analyzer plugin to a release with an arm64 parser), and drop the JavaScript/TypeScript sensor if this repository has no JS/TS to analyze. Two things keep confidence short of certain: the console is truncated between the analyzer errors and the pipeline exit, so the gate's own verdict message is not in evidence and the link from unparsed code to the non-zero exit is reasoned rather than observed; and the log store was unreachable during collection ("opensearch request failed: [Errno 111] Connection refused"), so there is no second source. Confirm by running uname -m, node -v, and the analyzer's Go parser binary directly inside the image that stage uses.

Ariadne did not authorize automated remediation: human_required.

Facts

  • jenkins — Build 337 of the Jenkins job soteria ended with result FAILURE after 1389 seconds (started 2026-09-04T20:59:00Z, ended 2026-09-04T21:22:09Z). (https://ci.bstein.dev/job/soteria/337/ (jenkins.result, jenkins.timestamps))
  • jenkins — The pipeline stage reported as the first to fail is "Enforce quality gate". (jenkins.first_failed_stage)
  • jenkins — The build published no structured test results; jenkins.failed_tests is an empty array, so no named test, class, or assertion failed in the recorded results. (jenkins.failed_tests)
  • jenkins — In the earliest detected failure region, the SonarQube Go sensor failed to parse every Go source file it was given: 26 of 26 files logged "Cannot parse '': Go parser external process returned non-zero exit value: 2", covering cmd/soteria/main.go and all internal/{api,config,k8s,longhorn,ser... (jenkins.console_failures[0], console line ~308, marker "non-zero exit")
  • jenkins — Immediately after, the SonarQube TypeScript/JavaScript sensor aborted at 21:04:33 with org.sonarsource.nodejs.NodeCommandException: "Error when running: 'node -v'. Is Node.js available during analysis?", root-caused in the stack trace by java.io.IOException: Cannot run program "node": error=2, No... (jenkins.console_failures[0] and jenkins.console_failures[1], console lines ~308 and ~411)
  • jenkins — Analysis ran against the workspace path /home/jenkins/agent/workspace/Soteria; the parse errors reference that path, and no "no space left on device", "disk quota exceeded", agent-offline, or pod-provisioning message appears anywhere in the retained excerpts. (jenkins.console_failures[0] file URIs)
  • jenkins — The build terminated with "ERROR: script returned exit code 1" at end of pipeline, and the later stage "Build & push image" was skipped due to earlier failure(s). (jenkins.console_tail and jenkins.console_failures[2], console line ~3357)
  • jenkins — The console log is truncated (console_truncated=true), so the region between the scanner errors at line ~411 and the pipeline exit at line ~3357 - which would contain the quality gate's own verdict - is not present in the evidence. (jenkins.console_truncated)
  • opensearch — No log records were retrieved for the incident window 20:54:00Z-21:27:09Z; the log query itself failed with "opensearch request failed: [Errno 111] Connection refused", so there is no corroborating log evidence either way. (log_evidence.error, log_evidence.records = [])

Evidence

The earliest failure region of the build console:

21:04:18.336 INFO  Sensor JaCoCo XML Report Importer [jacoco] (done) | time=12ms
21:04:18.336 INFO  Sensor IaC CloudFormation Sensor [iac]
21:04:18.515 INFO  0 source files to be analyzed
21:04:18.610 INFO  0/0 source files have been analyzed
21:04:18.610 INFO  Sensor IaC CloudFormation Sensor [iac] (done) | time=274ms
21:04:18.611 INFO  Sensor IaC Kubernetes Sensor [iac]
21:04:18.683 INFO  2 source files to be analyzed
21:04:19.307 INFO  2/2 source files have been analyzed
21:04:19.327 INFO  Sensor IaC Kubernetes Sensor [iac] (done) | time=717ms
21:04:19.328 INFO  Sensor TypeScript analysis [javascript]
21:04:33.305 ERROR Error when running: 'node -v'. Is Node.js available during analysis?
org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis?
	at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143)
	at org.sonar.plugins.javascrip
... (truncated; the full log is in the linked build)

Inferences

  • The static analysis step could not analyze this repository's code at all. The Go analyzer's external parser process exited with status 2 on all 26 Go files, and the JavaScript/TypeScript analyzer never started because the node executable does not exist in the container that runs the scanner. Bo...
  • The most likely cause of the uniform Go parser failure is that the bundled parser binary cannot execute in the analysis container at all - typically a CPU-architecture mismatch (the analyzer shipping only x86-64 parser binaries while the agent runs on arm64) or a missing runtime dependency. The m...
  • Because every Go file was unparsed, the analysis reported essentially no findings and no coverage for the service, and the stage that enforces the quality gate then exited 1. This causal link is inference, not observation: the truncated console does not include the gate's own verdict message, so ...
  • Nothing in the evidence indicates a regression in the service's source or tests. No test failures were published, no compilation or go test error appears, and the failure is confined to the analysis step.
  • A fix belongs in the build environment rather than the service code: make the stage run on an image that contains Node.js and a Go parser binary executable on the agent's architecture, or upgrade the scanner/Go analyzer plugin to a version that ships an arm64 parser. If JavaScript/TypeScript anal...
  • A first confirming check is to run node -v and the analyzer's Go parser binary with uname -m inside the same container image used by this stage; if the parser is a foreign-architecture ELF or node is absen

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `soteria/337` as **sonarqube_analyzer_toolchain_unavailable** (confidence 0.55); first failed gate: `sonarqube`. ## Why a human is needed Build 337 of the soteria job failed in the "Enforce quality gate" stage, and no test failed: the build published no test results, and no compile or test error appears in the retained log. What actually broke is the static-analysis step's own environment. Between 21:04:16 and 21:04:18 the Go analyzer failed to parse all 26 Go files in the repository - every file from cmd/soteria/main.go through the internal/api, internal/config, internal/k8s, internal/longhorn and internal/server packages - each with "Go parser external process returned non-zero exit value: 2". Seconds later, at 21:04:33, the TypeScript analyzer aborted with "Cannot run program \"node\": error=2, No such file or directory". A uniform parser exit across every file, plus a completely absent `node` binary, indicates the analysis container lacks working language tooling - most plausibly an analyzer parser binary built for a different CPU architecture than the build agent, and an image with no Node.js installed. With no Go file parsed, the analysis had no view of the service at all, and the pipeline exited 1 and skipped the image build. Fixing this means changing where and how the analysis stage runs, not changing the service: use an image for that stage that ships Node.js and an analyzer whose Go parser is executable on the agent's architecture (or upgrade the analyzer plugin to a release with an arm64 parser), and drop the JavaScript/TypeScript sensor if this repository has no JS/TS to analyze. Two things keep confidence short of certain: the console is truncated between the analyzer errors and the pipeline exit, so the gate's own verdict message is not in evidence and the link from unparsed code to the non-zero exit is reasoned rather than observed; and the log store was unreachable during collection ("opensearch request failed: [Errno 111] Connection refused"), so there is no second source. Confirm by running `uname -m`, `node -v`, and the analyzer's Go parser binary directly inside the image that stage uses. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build 337 of the Jenkins job soteria ended with result FAILURE after 1389 seconds (started 2026-09-04T20:59:00Z, ended 2026-09-04T21:22:09Z). (`https://ci.bstein.dev/job/soteria/337/ (jenkins.result, jenkins.timestamps)`) - **jenkins** — The pipeline stage reported as the first to fail is "Enforce quality gate". (`jenkins.first_failed_stage`) - **jenkins** — The build published no structured test results; jenkins.failed_tests is an empty array, so no named test, class, or assertion failed in the recorded results. (`jenkins.failed_tests`) - **jenkins** — In the earliest detected failure region, the SonarQube Go sensor failed to parse every Go source file it was given: 26 of 26 files logged "Cannot parse '<file>': Go parser external process returned non-zero exit value: 2", covering cmd/soteria/main.go and all internal/{api,config,k8s,longhorn,ser... (`jenkins.console_failures[0], console line ~308, marker "non-zero exit"`) - **jenkins** — Immediately after, the SonarQube TypeScript/JavaScript sensor aborted at 21:04:33 with org.sonarsource.nodejs.NodeCommandException: "Error when running: 'node -v'. Is Node.js available during analysis?", root-caused in the stack trace by java.io.IOException: Cannot run program "node": error=2, No... (`jenkins.console_failures[0] and jenkins.console_failures[1], console lines ~308 and ~411`) - **jenkins** — Analysis ran against the workspace path /home/jenkins/agent/workspace/Soteria; the parse errors reference that path, and no "no space left on device", "disk quota exceeded", agent-offline, or pod-provisioning message appears anywhere in the retained excerpts. (`jenkins.console_failures[0] file URIs`) - **jenkins** — The build terminated with "ERROR: script returned exit code 1" at end of pipeline, and the later stage "Build & push image" was skipped due to earlier failure(s). (`jenkins.console_tail and jenkins.console_failures[2], console line ~3357`) - **jenkins** — The console log is truncated (console_truncated=true), so the region between the scanner errors at line ~411 and the pipeline exit at line ~3357 - which would contain the quality gate's own verdict - is not present in the evidence. (`jenkins.console_truncated`) - **opensearch** — No log records were retrieved for the incident window 20:54:00Z-21:27:09Z; the log query itself failed with "opensearch request failed: [Errno 111] Connection refused", so there is no corroborating log evidence either way. (`log_evidence.error, log_evidence.records = []`) ## Evidence The earliest failure region of the build console: ``` 21:04:18.336 INFO Sensor JaCoCo XML Report Importer [jacoco] (done) | time=12ms 21:04:18.336 INFO Sensor IaC CloudFormation Sensor [iac] 21:04:18.515 INFO 0 source files to be analyzed 21:04:18.610 INFO 0/0 source files have been analyzed 21:04:18.610 INFO Sensor IaC CloudFormation Sensor [iac] (done) | time=274ms 21:04:18.611 INFO Sensor IaC Kubernetes Sensor [iac] 21:04:18.683 INFO 2 source files to be analyzed 21:04:19.307 INFO 2/2 source files have been analyzed 21:04:19.327 INFO Sensor IaC Kubernetes Sensor [iac] (done) | time=717ms 21:04:19.328 INFO Sensor TypeScript analysis [javascript] 21:04:33.305 ERROR Error when running: 'node -v'. Is Node.js available during analysis? org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis? at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79) at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203) at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169) at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143) at org.sonar.plugins.javascrip ... (truncated; the full log is in the linked build) ``` ## Inferences - The static analysis step could not analyze this repository's code at all. The Go analyzer's external parser process exited with status 2 on all 26 Go files, and the JavaScript/TypeScript analyzer never started because the `node` executable does not exist in the container that runs the scanner. Bo... - The most likely cause of the uniform Go parser failure is that the bundled parser binary cannot execute in the analysis container at all - typically a CPU-architecture mismatch (the analyzer shipping only x86-64 parser binaries while the agent runs on arm64) or a missing runtime dependency. The m... - Because every Go file was unparsed, the analysis reported essentially no findings and no coverage for the service, and the stage that enforces the quality gate then exited 1. This causal link is inference, not observation: the truncated console does not include the gate's own verdict message, so ... - Nothing in the evidence indicates a regression in the service's source or tests. No test failures were published, no compilation or `go test` error appears, and the failure is confined to the analysis step. - A fix belongs in the build environment rather than the service code: make the stage run on an image that contains Node.js and a Go parser binary executable on the agent's architecture, or upgrade the scanner/Go analyzer plugin to a version that ships an arm64 parser. If JavaScript/TypeScript anal... - A first confirming check is to run `node -v` and the analyzer's Go parser binary with `uname -m` inside the same container image used by this stage; if the parser is a foreign-architecture ELF or `node` is absen _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._ <!-- hermes-triage job=soteria classification=sonarqube_analyzer_toolchain_unavailable incident=soteria/337 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/soteria#21
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?