titan/soteria
3
0
Fork 0
Code Issues 23 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[hermes] soteria #308: sonarqube_analysis_toolchain_failure #15

New Issue
Open
opened 2026-08-21 09:43:23 +00:00 by hermes-automation · 0 comments
hermes-automation commented 2026-08-21 09:43:23 +00:00
Member
Copy Link

Hermes auto-triage classified incident soteria/308 as sonarqube_analysis_toolchain_failure (confidence 0.55); first failed gate: sonarqube (Jenkins stage 'Enforce quality gate').

Why a human is needed

Build 308 failed in the 'Enforce quality gate' stage; no tests failed and no test results were published. During the SonarQube scan at 09:18 the Go analyzer could not parse a single one of the 26 Go files in the repository: 25 reported 'Go parser external process returned non-zero exit value: 2' and internal/k8s/state.go reported 'Stream closed'. Thirteen seconds later the TypeScript sensor could not start either, because 'node' does not exist in the build environment. Both are the analyzers failing to launch the external helper processes they need, not defects in Soteria's source. The result is that SonarQube got no Go code, issues, or coverage for this run, which is the likely reason the gate stage then exited 1 and the image build was skipped. Fixing this means repairing the scan environment rather than the service: use a scanner image matching the agent's CPU architecture that ships Node.js and a runnable Go parser, make sure the directory the analyzer extracts its helper binary into is not mounted noexec, and verify by running 'node -v' and scanning one Go file in that container. It is worth comparing the agent image and pod template against the last build where the Go analyzer produced results, since that will pinpoint what changed. A maintainer is needed because the decisive part of the console covering the ~21 minutes up to the failure is missing from the retained evidence, so the specific gate condition that failed is not visible, and the choice between rebuilding the scanner image, pinning a different one, or temporarily narrowing what the gate enforces is a call about this pipeline that only its owner should make.

Ariadne did not authorize automated remediation: human_required.

Facts

  • jenkins — Build 308 of the soteria job finished with result FAILURE, running from 2026-08-21T08:59:00.797Z to 2026-08-21T09:39:13.945Z (2413s). (https://ci.bstein.dev/job/soteria/308/ (jenkins.result, jenkins.timestamps))
  • jenkins — The pipeline reported 'Enforce quality gate' as the first failed stage; 'Build & push image' was then reported as 'skipped due to earlier failure(s)', and the pipeline ended with 'ERROR: script returned exit code 1'. (jenkins.first_failed_stage; jenkins.console_tail; console_failures[2] at line 3375)
  • jenkins — The build published no structured test results: jenkins.failed_tests is an empty array. (jenkins.failed_tests)
  • jenkins — At 09:18:13-09:18:14 the SonarQube Go sensor failed to parse every one of the 26 Go source files: 'internal/k8s/state.go' failed with 'Stream closed' and the remaining 25 files (including cmd/soteria/main.go, internal/server/.go, internal/k8s/.go, internal/config/config.go, internal/longhorn/cl... (console_failures[0], marker 'non-zero exit', line 328)
  • jenkins — At 09:18:31 the SonarQube TypeScript sensor failed to start its analysis bridge: 'Error when running: node -v. Is Node.js available during analysis?' caused by 'java.io.IOException: Cannot run program "node": error=2, No such file or directory'. The sensor was still logged as '(done)' and later s... (console_failures[0] tail and console_failures[1], marker 'No such file', line 429)
  • jenkins — The build workspace was /home/jenkins/agent/workspace/Soteria, i.e. an agent was allocated and the checkout was present when the scanner ran. (console_failures[0], file URIs in the Go parser errors)
  • opensearch — No log records were returned for the correlation window 2026-08-21T08:54:00.797Z to 2026-08-21T09:44:13.945Z, with no query error reported. (log_evidence.records (empty), log_evidence.error = null)
  • jenkins — The captured console is truncated: the ~21 minutes between the scanner output at 09:18:31 and the end of the build at 09:39:13 are not present in the bundle, and the last retained failure excerpt is only the generic end-of-pipeline error. (jenkins.console_truncated = true; console_failures[2] at line 3375)

Evidence

The earliest failure region of the build console:

09:18:14.263 INFO  Sensor JaCoCo XML Report Importer [jacoco] (done) | time=9ms
09:18:14.263 INFO  Sensor IaC CloudFormation Sensor [iac]
09:18:14.440 INFO  0 source files to be analyzed
09:18:14.533 INFO  0/0 source files have been analyzed
09:18:14.537 INFO  Sensor IaC CloudFormation Sensor [iac] (done) | time=274ms
09:18:14.538 INFO  Sensor IaC Kubernetes Sensor [iac]
09:18:14.648 INFO  2 source files to be analyzed
09:18:15.042 INFO  2/2 source files have been analyzed
09:18:15.044 INFO  Sensor IaC Kubernetes Sensor [iac] (done) | time=506ms
09:18:15.045 INFO  Sensor TypeScript analysis [javascript]
09:18:31.235 ERROR Error when running: 'node -v'. Is Node.js available during analysis?
org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis?
	at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169)
	at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143)
	at org.sonar.plugins.javascript
... (truncated; the full log is in the linked build)

Inferences

  • No tests were recorded as failing and the build reached static analysis, so the failure is in the quality-gate stage rather than in the Soteria unit or integration suites.
  • The Go analyzer failing on all 26 files with the same 'external process returned non-zero exit value: 2' is very unlikely to be a code problem. A real syntax error would break the file that contains it, not every file in the module, and the message comes from the helper parser binary the analyzer...
  • The 'node not found' failure in the TypeScript sensor is a second instance of the same underlying condition: the container running the scanner does not have the external runtimes the analyzers expect. It is reported as an error but the analysis continued past it, so on its own it did not end the ...
  • Because every Go file failed to parse, SonarQube received no Go code, no issues, and no coverage measures for this project on this run. A quality gate with conditions on coverage, duplication, or new-code issues will evaluate against an effectively empty analysis, which is the most plausible reas...
  • The exact gate condition that failed is not visible: the console excerpts stop at 09:18:31 and the retained tail only shows the generic 'script returned exit code 1'. That gap is why the causal link between the parser failures and the gate outcome is inference rather than fact.
  • A fix is on the CI side, not in the Soteria source. It means making the scanner environment able to execute the analyzers' helper processes: use a scanner image that matches the agent architecture and includes Node.js and a working Go parser, ensure the temporary directory the analyzer extracts i...

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

  • Proposed action: verify_sonar_scanner_toolchain
  • What it would do: Run a read-only probe inside the scanner container/agent image used by the build that executes the external runtimes the SonarQube analyzers depend on (for example 'node -v', the extracted Go parser binary, and an exec test on the analyzer temp directory) and report which ones cannot start and why.
  • Ev

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `soteria/308` as **sonarqube_analysis_toolchain_failure** (confidence 0.55); first failed gate: `sonarqube (Jenkins stage 'Enforce quality gate')`. ## Why a human is needed Build 308 failed in the 'Enforce quality gate' stage; no tests failed and no test results were published. During the SonarQube scan at 09:18 the Go analyzer could not parse a single one of the 26 Go files in the repository: 25 reported 'Go parser external process returned non-zero exit value: 2' and internal/k8s/state.go reported 'Stream closed'. Thirteen seconds later the TypeScript sensor could not start either, because 'node' does not exist in the build environment. Both are the analyzers failing to launch the external helper processes they need, not defects in Soteria's source. The result is that SonarQube got no Go code, issues, or coverage for this run, which is the likely reason the gate stage then exited 1 and the image build was skipped. Fixing this means repairing the scan environment rather than the service: use a scanner image matching the agent's CPU architecture that ships Node.js and a runnable Go parser, make sure the directory the analyzer extracts its helper binary into is not mounted noexec, and verify by running 'node -v' and scanning one Go file in that container. It is worth comparing the agent image and pod template against the last build where the Go analyzer produced results, since that will pinpoint what changed. A maintainer is needed because the decisive part of the console covering the ~21 minutes up to the failure is missing from the retained evidence, so the specific gate condition that failed is not visible, and the choice between rebuilding the scanner image, pinning a different one, or temporarily narrowing what the gate enforces is a call about this pipeline that only its owner should make. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build 308 of the soteria job finished with result FAILURE, running from 2026-08-21T08:59:00.797Z to 2026-08-21T09:39:13.945Z (2413s). (`https://ci.bstein.dev/job/soteria/308/ (jenkins.result, jenkins.timestamps)`) - **jenkins** — The pipeline reported 'Enforce quality gate' as the first failed stage; 'Build & push image' was then reported as 'skipped due to earlier failure(s)', and the pipeline ended with 'ERROR: script returned exit code 1'. (`jenkins.first_failed_stage; jenkins.console_tail; console_failures[2] at line 3375`) - **jenkins** — The build published no structured test results: jenkins.failed_tests is an empty array. (`jenkins.failed_tests`) - **jenkins** — At 09:18:13-09:18:14 the SonarQube Go sensor failed to parse every one of the 26 Go source files: 'internal/k8s/state.go' failed with 'Stream closed' and the remaining 25 files (including cmd/soteria/main.go, internal/server/*.go, internal/k8s/*.go, internal/config/config.go, internal/longhorn/cl... (`console_failures[0], marker 'non-zero exit', line 328`) - **jenkins** — At 09:18:31 the SonarQube TypeScript sensor failed to start its analysis bridge: 'Error when running: node -v. Is Node.js available during analysis?' caused by 'java.io.IOException: Cannot run program "node": error=2, No such file or directory'. The sensor was still logged as '(done)' and later s... (`console_failures[0] tail and console_failures[1], marker 'No such file', line 429`) - **jenkins** — The build workspace was /home/jenkins/agent/workspace/Soteria, i.e. an agent was allocated and the checkout was present when the scanner ran. (`console_failures[0], file URIs in the Go parser errors`) - **opensearch** — No log records were returned for the correlation window 2026-08-21T08:54:00.797Z to 2026-08-21T09:44:13.945Z, with no query error reported. (`log_evidence.records (empty), log_evidence.error = null`) - **jenkins** — The captured console is truncated: the ~21 minutes between the scanner output at 09:18:31 and the end of the build at 09:39:13 are not present in the bundle, and the last retained failure excerpt is only the generic end-of-pipeline error. (`jenkins.console_truncated = true; console_failures[2] at line 3375`) ## Evidence The earliest failure region of the build console: ``` 09:18:14.263 INFO Sensor JaCoCo XML Report Importer [jacoco] (done) | time=9ms 09:18:14.263 INFO Sensor IaC CloudFormation Sensor [iac] 09:18:14.440 INFO 0 source files to be analyzed 09:18:14.533 INFO 0/0 source files have been analyzed 09:18:14.537 INFO Sensor IaC CloudFormation Sensor [iac] (done) | time=274ms 09:18:14.538 INFO Sensor IaC Kubernetes Sensor [iac] 09:18:14.648 INFO 2 source files to be analyzed 09:18:15.042 INFO 2/2 source files have been analyzed 09:18:15.044 INFO Sensor IaC Kubernetes Sensor [iac] (done) | time=506ms 09:18:15.045 INFO Sensor TypeScript analysis [javascript] 09:18:31.235 ERROR Error when running: 'node -v'. Is Node.js available during analysis? org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis? at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79) at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203) at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169) at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143) at org.sonar.plugins.javascript ... (truncated; the full log is in the linked build) ``` ## Inferences - No tests were recorded as failing and the build reached static analysis, so the failure is in the quality-gate stage rather than in the Soteria unit or integration suites. - The Go analyzer failing on all 26 files with the same 'external process returned non-zero exit value: 2' is very unlikely to be a code problem. A real syntax error would break the file that contains it, not every file in the module, and the message comes from the helper parser binary the analyzer... - The 'node not found' failure in the TypeScript sensor is a second instance of the same underlying condition: the container running the scanner does not have the external runtimes the analyzers expect. It is reported as an error but the analysis continued past it, so on its own it did not end the ... - Because every Go file failed to parse, SonarQube received no Go code, no issues, and no coverage measures for this project on this run. A quality gate with conditions on coverage, duplication, or new-code issues will evaluate against an effectively empty analysis, which is the most plausible reas... - The exact gate condition that failed is not visible: the console excerpts stop at 09:18:31 and the retained tail only shows the generic 'script returned exit code 1'. That gap is why the causal link between the parser failures and the gate outcome is inference rather than fact. - A fix is on the CI side, not in the Soteria source. It means making the scanner environment able to execute the analyzers' helper processes: use a scanner image that matches the agent architecture and includes Node.js and a working Go parser, ensure the temporary directory the analyzer extracts i... ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `verify_sonar_scanner_toolchain` - What it would do: Run a read-only probe inside the scanner container/agent image used by the build that executes the external runtimes the SonarQube analyzers depend on (for example 'node -v', the extracted Go parser binary, and an exec test on the analyzer temp directory) and report which ones cannot start and why. - Ev _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._ <!-- hermes-triage job=soteria classification=sonarqube_analysis_toolchain_failure incident=soteria/308 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/soteria#15
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?