titan/soteria
3
0
Fork 0
Code Issues 23 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[hermes] soteria #328: sonarqube_scanner_toolchain_broken #17

New Issue
Open
opened 2026-08-31 09:37:08 +00:00 by hermes-automation · 0 comments
hermes-automation commented 2026-08-31 09:37:08 +00:00
Member
Copy Link

Hermes auto-triage classified incident soteria/328 as sonarqube_scanner_toolchain_broken (confidence 0.62); first failed gate: sonarqube.

Why a human is needed

Build 328 failed in the 'Enforce quality gate' stage, not in the tests: no test results were published and no assertion failed. The earliest failure in the log is the SonarQube scanner being unable to run its analysis tools. At 09:02:52 every one of the 26 Go source files failed with 'Go parser external process returned non-zero exit value: 2' (internal/k8s/state.go, internal/server/server.go, cmd/soteria/main.go and the rest), and at 09:03:14 the JS/TS sensor died with 'Cannot run program "node": error=2, No such file or directory'. The build ended with 'ERROR: script returned exit code 1' and the image build was skipped. A parser that exits non-zero on all 26 files, with no compile or test failure ahead of it, is a broken tool rather than broken Go: check first whether the analyzer's helper binary matches the agent's CPU architecture (the agents are arm64) and that it is present and executable in the scanner image; the missing 'node' in the same container points the same way - the image is missing runtimes its enabled sensors need. The fix belongs in the CI pipeline definition and the scanner container image, not in the service source: supply a Node.js runtime (or disable JS/TS analysis if this repo has no JS/TS), and supply a working Go analyzer for the agent architecture. I have flagged this for a maintainer because the console log is truncated between the scanner errors and the pipeline's final error line, so the bundle does not show which command actually returned exit 1 - the scanner itself aborting, or the gate rejecting an analysis in which zero Go code and zero coverage were recorded. Deciding between repairing the image and adjusting what the gate enforces is your call, but re-running the same commit unchanged will reproduce this exactly.

Ariadne did not authorize automated remediation: human_required.

Facts

  • jenkins — Build 328 of job soteria ended in FAILURE after 1806.464s, running 2026-08-31T08:59:00.284Z to 2026-08-31T09:29:06.748Z. (https://ci.bstein.dev/job/soteria/328/ (jenkins.result, jenkins.timestamps))
  • jenkins — The pipeline reported its first failed stage as 'Enforce quality gate', and the later stage 'Build & push image' was skipped due to earlier failure(s). (jenkins.first_failed_stage; jenkins.console_tail)
  • jenkins — The build published no structured test results: jenkins.failed_tests is empty. (jenkins.failed_tests = [])
  • jenkins — In the earliest failure region (console line 296), at 09:02:52 the SonarQube Go sensor failed on every one of the 26 Go source files with 'Go parser external process returned non-zero exit value: 2', including internal/k8s/state.go, internal/config/config.go, internal/server/server.go and cmd/sot... (jenkins.console_failures[0] (marker 'non-zero exit', line 296))
  • jenkins — In the same region at 09:03:14 the TypeScript/JavaScript sensor failed with NodeCommandException: "Error when running: 'node -v'. Is Node.js available during analysis?", and the continuation at console line 399 shows 'Caused by: java.io.IOException: Cannot run program "node": error=2, No such fil... (jenkins.console_failures[0] and jenkins.console_failures[1] (marker 'No such file', line 399))
  • jenkins — The analysed workspace path is /home/jenkins/agent/workspace/Soteria, i.e. the build had an agent and a checked-out workspace and was executing pipeline stages. (jenkins.console_failures[0], file:/// URIs in the Go parser errors)
  • jenkins — The end of the build shows only 'ERROR: script returned exit code 1' followed by 'Finished: FAILURE', with no gate condition or assertion text. (jenkins.console_tail; jenkins.console_failures[2] (line 3345))
  • jenkins — The console log is truncated and no excerpt covers the region between line 399 and line 3345, so the command that returned exit code 1 is not present in the bundle. (jenkins.console_truncated = true)
  • opensearch — No application or CI log records were retrievable for the incident window: the log query failed with 'opensearch request failed: [Errno 111] Connection refused'. (log_evidence.error, window 2026-08-31T08:54:00Z to 2026-08-31T09:34:06Z)

Evidence

The earliest failure region of the build console:

09:02:53.483 INFO  Sensor JaCoCo XML Report Importer [jacoco] (done) | time=34ms
09:02:53.484 INFO  Sensor IaC CloudFormation Sensor [iac]
09:02:53.754 INFO  0 source files to be analyzed
09:02:53.947 INFO  0/0 source files have been analyzed
09:02:53.948 INFO  Sensor IaC CloudFormation Sensor [iac] (done) | time=465ms
09:02:53.948 INFO  Sensor IaC Kubernetes Sensor [iac]
09:02:54.082 INFO  2 source files to be analyzed
09:02:54.842 INFO  2/2 source files have been analyzed
09:02:54.845 INFO  Sensor IaC Kubernetes Sensor [iac] (done) | time=896ms
09:02:54.848 INFO  Sensor TypeScript analysis [javascript]
09:03:14.414 ERROR Error when running: 'node -v'. Is Node.js available during analysis?
org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis?
	at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143)
	at org.sonar.plugins.javascrip
... (truncated; the full log is in the linked build)

Inferences

  • The quality-gate stage ran a SonarQube scanner whose analysis toolchain is incomplete inside the build container. Two independent tools it needs were unusable in the same run: the external Go parser binary exited with status 2 on every file it was handed, and the Node.js runtime the JS/TS sensor ...
  • The Go failure is almost certainly environmental rather than a syntax error in the Go sources. A real parse error would affect the specific offending file, not all 26 uniformly, and the compile and test stages earlier in the pipeline did not fail. A parser process that exits non-zero on every inp...
  • Because no Go file was actually parsed, Sonar received an empty picture of the project: zero analysable Go code, no issues, and no coverage (the JaCoCo importer also reported 'No report imported'). Any gate condition expressed over new or overall code would therefore be evaluated against empty data.
  • Whether the stage's exit code 1 came from the scanner aborting or from a gate condition computed on that empty analysis is not determinable from this bundle, since the console is truncated between the scanner errors and the pipeline's final ERROR line. Either way the same root cause precedes it.
  • Re-running the same commit will fail identically: nothing about this depends on the commit, only on the container the scanner runs in.
  • A fix means restoring the analysis toolchain in the image/pod template used by the quality-gate stage: provide a Node.js runtime on PATH for the JS/TS sensor (or explicitly exclude JS/TS analysis if this repository genuinely has none), and provide a Go analyzer whose helper binary can execute on ...

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

  • Proposed action: preflight_sonar_scanner_toolchain
  • What it would do: Before the quality-gate stage runs the scanner, probe the scanner container for the tools the enabled sensors require - that 'node -v' succeeds

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `soteria/328` as **sonarqube_scanner_toolchain_broken** (confidence 0.62); first failed gate: `sonarqube`. ## Why a human is needed Build 328 failed in the 'Enforce quality gate' stage, not in the tests: no test results were published and no assertion failed. The earliest failure in the log is the SonarQube scanner being unable to run its analysis tools. At 09:02:52 every one of the 26 Go source files failed with 'Go parser external process returned non-zero exit value: 2' (internal/k8s/state.go, internal/server/server.go, cmd/soteria/main.go and the rest), and at 09:03:14 the JS/TS sensor died with 'Cannot run program "node": error=2, No such file or directory'. The build ended with 'ERROR: script returned exit code 1' and the image build was skipped. A parser that exits non-zero on all 26 files, with no compile or test failure ahead of it, is a broken tool rather than broken Go: check first whether the analyzer's helper binary matches the agent's CPU architecture (the agents are arm64) and that it is present and executable in the scanner image; the missing 'node' in the same container points the same way - the image is missing runtimes its enabled sensors need. The fix belongs in the CI pipeline definition and the scanner container image, not in the service source: supply a Node.js runtime (or disable JS/TS analysis if this repo has no JS/TS), and supply a working Go analyzer for the agent architecture. I have flagged this for a maintainer because the console log is truncated between the scanner errors and the pipeline's final error line, so the bundle does not show which command actually returned exit 1 - the scanner itself aborting, or the gate rejecting an analysis in which zero Go code and zero coverage were recorded. Deciding between repairing the image and adjusting what the gate enforces is your call, but re-running the same commit unchanged will reproduce this exactly. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build 328 of job soteria ended in FAILURE after 1806.464s, running 2026-08-31T08:59:00.284Z to 2026-08-31T09:29:06.748Z. (`https://ci.bstein.dev/job/soteria/328/ (jenkins.result, jenkins.timestamps)`) - **jenkins** — The pipeline reported its first failed stage as 'Enforce quality gate', and the later stage 'Build & push image' was skipped due to earlier failure(s). (`jenkins.first_failed_stage; jenkins.console_tail`) - **jenkins** — The build published no structured test results: jenkins.failed_tests is empty. (`jenkins.failed_tests = []`) - **jenkins** — In the earliest failure region (console line 296), at 09:02:52 the SonarQube Go sensor failed on every one of the 26 Go source files with 'Go parser external process returned non-zero exit value: 2', including internal/k8s/state.go, internal/config/config.go, internal/server/server.go and cmd/sot... (`jenkins.console_failures[0] (marker 'non-zero exit', line 296)`) - **jenkins** — In the same region at 09:03:14 the TypeScript/JavaScript sensor failed with NodeCommandException: "Error when running: 'node -v'. Is Node.js available during analysis?", and the continuation at console line 399 shows 'Caused by: java.io.IOException: Cannot run program "node": error=2, No such fil... (`jenkins.console_failures[0] and jenkins.console_failures[1] (marker 'No such file', line 399)`) - **jenkins** — The analysed workspace path is /home/jenkins/agent/workspace/Soteria, i.e. the build had an agent and a checked-out workspace and was executing pipeline stages. (`jenkins.console_failures[0], file:/// URIs in the Go parser errors`) - **jenkins** — The end of the build shows only 'ERROR: script returned exit code 1' followed by 'Finished: FAILURE', with no gate condition or assertion text. (`jenkins.console_tail; jenkins.console_failures[2] (line 3345)`) - **jenkins** — The console log is truncated and no excerpt covers the region between line 399 and line 3345, so the command that returned exit code 1 is not present in the bundle. (`jenkins.console_truncated = true`) - **opensearch** — No application or CI log records were retrievable for the incident window: the log query failed with 'opensearch request failed: [Errno 111] Connection refused'. (`log_evidence.error, window 2026-08-31T08:54:00Z to 2026-08-31T09:34:06Z`) ## Evidence The earliest failure region of the build console: ``` 09:02:53.483 INFO Sensor JaCoCo XML Report Importer [jacoco] (done) | time=34ms 09:02:53.484 INFO Sensor IaC CloudFormation Sensor [iac] 09:02:53.754 INFO 0 source files to be analyzed 09:02:53.947 INFO 0/0 source files have been analyzed 09:02:53.948 INFO Sensor IaC CloudFormation Sensor [iac] (done) | time=465ms 09:02:53.948 INFO Sensor IaC Kubernetes Sensor [iac] 09:02:54.082 INFO 2 source files to be analyzed 09:02:54.842 INFO 2/2 source files have been analyzed 09:02:54.845 INFO Sensor IaC Kubernetes Sensor [iac] (done) | time=896ms 09:02:54.848 INFO Sensor TypeScript analysis [javascript] 09:03:14.414 ERROR Error when running: 'node -v'. Is Node.js available during analysis? org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis? at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79) at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203) at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169) at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143) at org.sonar.plugins.javascrip ... (truncated; the full log is in the linked build) ``` ## Inferences - The quality-gate stage ran a SonarQube scanner whose analysis toolchain is incomplete inside the build container. Two independent tools it needs were unusable in the same run: the external Go parser binary exited with status 2 on every file it was handed, and the Node.js runtime the JS/TS sensor ... - The Go failure is almost certainly environmental rather than a syntax error in the Go sources. A real parse error would affect the specific offending file, not all 26 uniformly, and the compile and test stages earlier in the pipeline did not fail. A parser process that exits non-zero on every inp... - Because no Go file was actually parsed, Sonar received an empty picture of the project: zero analysable Go code, no issues, and no coverage (the JaCoCo importer also reported 'No report imported'). Any gate condition expressed over new or overall code would therefore be evaluated against empty data. - Whether the stage's exit code 1 came from the scanner aborting or from a gate condition computed on that empty analysis is not determinable from this bundle, since the console is truncated between the scanner errors and the pipeline's final ERROR line. Either way the same root cause precedes it. - Re-running the same commit will fail identically: nothing about this depends on the commit, only on the container the scanner runs in. - A fix means restoring the analysis toolchain in the image/pod template used by the quality-gate stage: provide a Node.js runtime on PATH for the JS/TS sensor (or explicitly exclude JS/TS analysis if this repository genuinely has none), and provide a Go analyzer whose helper binary can execute on ... ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `preflight_sonar_scanner_toolchain` - What it would do: Before the quality-gate stage runs the scanner, probe the scanner container for the tools the enabled sensors require - that 'node -v' succeeds _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._ <!-- hermes-triage job=soteria classification=sonarqube_scanner_toolchain_broken incident=soteria/328 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/soteria#17
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?