[hermes] soteria #321: sonarqube_analyzer_toolchain_failure #16

New Issue

hermes-automation · 2026-08-27T21:38:53Z

hermes-automation commented

2026-08-27 21:38:53 +00:00

Hermes auto-triage classified incident soteria/321 as sonarqube_analyzer_toolchain_failure (confidence 0.62); first failed gate: sonarqube (Jenkins stage: "Enforce quality gate").

Why a human is needed

Build 321 failed in the "Enforce quality gate" stage; the "Build & push image" stage was skipped and the pipeline ended with "script returned exit code 1". No failing tests were published, so nothing in the Go test suite reported an error. What is plainly broken is the static-analysis step that runs in the quality-tools container (registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64). Between 21:05:41 and 21:05:42 the SonarQube Go sensor failed on all 26 Go source files - "Go parser external process returned non-zero exit value: 2" on most, "Stream closed" or "Broken pipe" on internal/k8s/state.go, internal/k8s/client.go, internal/k8s/volumes.go and internal/server/backup_safety.go - and at 21:06:07 the TypeScript sensor died with "Cannot run program "node": error=2, No such file or directory". Both are the analyzer's helper processes failing to execute inside the container, not results about your code: a real syntax error is reported against one file at a specific position and would not knock out all 26 files uniformly, and the JVM never even reached node -v. The consequence is that the Sonar run for this build contains no Go analysis at all, which is the most likely reason the gate stage exited 1. A fix therefore belongs in the analysis image rather than in Soteria: rebuild or repin quality-tools so it contains an arm64 Node.js on PATH (or point sonar.nodejs.executable at a real binary) and a Go parser that runs on this agent's architecture, and verify inside the image with node -v plus a single-file sonar-scanner run before re-running the pipeline; if an arm64 Go parser is genuinely unavailable for this Sonar version, the alternative is to pin a Sonar/plugin version that ships one rather than to silence the sensor. Flagged for a maintainer because the evidence stops short of proof: the console is truncated between 21:06:08 and the 21:31:51 end of the build and the log store was unreachable ("opensearch request failed: [Errno 111] Connection refused"), so the exact condition that returned exit 1 is not in hand - the archived build artifacts and the SonarQube project page for build 321 will confirm it - and deciding how to repair the shared analysis image is a maintainer's call.

Ariadne did not authorize automated remediation: human_required.

Facts

jenkins — Build soteria #321 finished with result FAILURE at 2026-08-27T21:31:51Z after 1971s (started 20:59:00Z). (https://ci.bstein.dev/job/soteria/321/)
jenkins — Jenkins reports the first failed stage as "Enforce quality gate"; the later "Build & push image" stage was skipped due to earlier failure(s) and the pipeline ended with "ERROR: script returned exit code 1". (jenkins.first_failed_stage; jenkins.console_tail; console_failures[3] (line 3351))
jenkins — The build published no failing structured test results (jenkins.failed_tests is an empty array). (jenkins.failed_tests)
jenkins — Between 21:05:41 and 21:05:42 the SonarQube Go sensor failed to parse all 26 Go source files, e.g. "Cannot parse 'cmd/soteria/main.go': Go parser external process returned non-zero exit value: 2", "Cannot parse 'internal/k8s/state.go': Stream closed", and "Cannot parse 'internal/server/backup_saf... (console_failures[1] (line 304))
jenkins — At 21:06:07 the SonarQube TypeScript/JavaScript sensor aborted with org.sonarsource.nodejs.NodeCommandException: "Error when running: 'node -v'. Is Node.js available during analysis?", caused by java.io.IOException: Cannot run program "node": error=2, No such file or directory. (console_failures[1] (line 304) and console_failures[2] (line 405))
jenkins — The analysis ran inside the pod container "quality-tools" from image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, on agent soteria-321-cf90d-bmtg6-p4q5z, with the workspace mounted at /home/jenkins/agent (analysed paths are under /home/jenkins/agent/workspace/... (console_failures[0] (line 109); file URIs in console_failures[1])
jenkins — The console log is truncated (console_truncated = true) and no retained excerpt covers the period between 21:06:08 and the end of the build at 21:31:51, so the output of the quality gate evaluation itself is not present in the bundle. (jenkins.console_truncated; jenkins.console_failures)
jenkins — The retained console excerpts contain no "no space left on device" or "disk quota exceeded" message, no image pull error, no DNS/connection-refused/reset/timeout or TLS error, and no agent provisioning error; an agent was assigned and the build ran for ~33 minutes. (jenkins.console_failures; jenkins.duration_seconds)
opensearch — Centralised log evidence for the window 20:54:00Z-21:36:51Z could not be retrieved: "opensearch request failed: [Errno 111] Connection refused"; zero records were returned. (log_evidence.error)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "soteria-321-cf90d-bmtg6-p4q5z"

Inferences

The Go analyzer failures are not findings about Soteria's source. A genuine syntax error is reported against a specific file and position; here every one of the 26 Go files failed identically with the helper process exiting non-zero (value 2), and three files failed with "Stream closed" or "Broke...
Node.js is simply not present in the analysis container: the JVM could not fork "node" at all ("error=2, No such file or directory"), so the JS/TS sensor could not start. That is a gap in the quality-tools image contents (or an unset sonar.nodejs.executable), not anything in the repository.
Because the Go sensor produced no usable analysis for any file, the SonarQube run for this build effectively carries no Go issues or coverage data. That is the most plausible reason the "Enforce quality gate" stage returned exit code 1 and the image build was skipped.
This is not a test regression and not a compile break: no failing tests were published, and Sonar's parse step is independent of go build, so a broken compile would have surfaced earlier and differently.
Re-running the same commit will schedule the same container image and reproduce the same result; the change needed is to the analysis toolchain, not to Soteria's code.
The evidence is not conclusive about the gate's exact failing condition: the console is truncated across the ~25 minutes between the Sonar errors and the end of the build, and the log store was unreachable, so a different check inside the same stage (for example a coverage threshold or the Trivy/...

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

Proposed action: rebuild_quality_tools_analysis_image
What it would do: Rebuild and republish the pinned quality-tools container image so the SonarQube analyzers have working helper runtimes for the CI agent's architec

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `soteria/321` as **sonarqube_analyzer_toolchain_failure** (confidence 0.62); first failed gate: `sonarqube (Jenkins stage: "Enforce quality gate")`. ## Why a human is needed Build 321 failed in the "Enforce quality gate" stage; the "Build & push image" stage was skipped and the pipeline ended with "script returned exit code 1". No failing tests were published, so nothing in the Go test suite reported an error. What is plainly broken is the static-analysis step that runs in the quality-tools container (registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64). Between 21:05:41 and 21:05:42 the SonarQube Go sensor failed on all 26 Go source files - "Go parser external process returned non-zero exit value: 2" on most, "Stream closed" or "Broken pipe" on internal/k8s/state.go, internal/k8s/client.go, internal/k8s/volumes.go and internal/server/backup_safety.go - and at 21:06:07 the TypeScript sensor died with "Cannot run program \"node\": error=2, No such file or directory". Both are the analyzer's helper processes failing to execute inside the container, not results about your code: a real syntax error is reported against one file at a specific position and would not knock out all 26 files uniformly, and the JVM never even reached `node -v`. The consequence is that the Sonar run for this build contains no Go analysis at all, which is the most likely reason the gate stage exited 1. A fix therefore belongs in the analysis image rather than in Soteria: rebuild or repin quality-tools so it contains an arm64 Node.js on PATH (or point sonar.nodejs.executable at a real binary) and a Go parser that runs on this agent's architecture, and verify inside the image with `node -v` plus a single-file sonar-scanner run before re-running the pipeline; if an arm64 Go parser is genuinely unavailable for this Sonar version, the alternative is to pin a Sonar/plugin version that ships one rather than to silence the sensor. Flagged for a maintainer because the evidence stops short of proof: the console is truncated between 21:06:08 and the 21:31:51 end of the build and the log store was unreachable ("opensearch request failed: [Errno 111] Connection refused"), so the exact condition that returned exit 1 is not in hand - the archived build artifacts and the SonarQube project page for build 321 will confirm it - and deciding how to repair the shared analysis image is a maintainer's call. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build soteria #321 finished with result FAILURE at 2026-08-27T21:31:51Z after 1971s (started 20:59:00Z). (`https://ci.bstein.dev/job/soteria/321/`) - **jenkins** — Jenkins reports the first failed stage as "Enforce quality gate"; the later "Build & push image" stage was skipped due to earlier failure(s) and the pipeline ended with "ERROR: script returned exit code 1". (`jenkins.first_failed_stage; jenkins.console_tail; console_failures[3] (line 3351)`) - **jenkins** — The build published no failing structured test results (jenkins.failed_tests is an empty array). (`jenkins.failed_tests`) - **jenkins** — Between 21:05:41 and 21:05:42 the SonarQube Go sensor failed to parse all 26 Go source files, e.g. "Cannot parse 'cmd/soteria/main.go': Go parser external process returned non-zero exit value: 2", "Cannot parse 'internal/k8s/state.go': Stream closed", and "Cannot parse 'internal/server/backup_saf... (`console_failures[1] (line 304)`) - **jenkins** — At 21:06:07 the SonarQube TypeScript/JavaScript sensor aborted with org.sonarsource.nodejs.NodeCommandException: "Error when running: 'node -v'. Is Node.js available during analysis?", caused by java.io.IOException: Cannot run program "node": error=2, No such file or directory. (`console_failures[1] (line 304) and console_failures[2] (line 405)`) - **jenkins** — The analysis ran inside the pod container "quality-tools" from image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, on agent soteria-321-cf90d-bmtg6-p4q5z, with the workspace mounted at /home/jenkins/agent (analysed paths are under /home/jenkins/agent/workspace/... (`console_failures[0] (line 109); file URIs in console_failures[1]`) - **jenkins** — The console log is truncated (console_truncated = true) and no retained excerpt covers the period between 21:06:08 and the end of the build at 21:31:51, so the output of the quality gate evaluation itself is not present in the bundle. (`jenkins.console_truncated; jenkins.console_failures`) - **jenkins** — The retained console excerpts contain no "no space left on device" or "disk quota exceeded" message, no image pull error, no DNS/connection-refused/reset/timeout or TLS error, and no agent provisioning error; an agent was assigned and the build ran for ~33 minutes. (`jenkins.console_failures; jenkins.duration_seconds`) - **opensearch** — Centralised log evidence for the window 20:54:00Z-21:36:51Z could not be retrieved: "opensearch request failed: [Errno 111] Connection refused"; zero records were returned. (`log_evidence.error`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "soteria-321-cf90d-bmtg6-p4q5z" ``` ## Inferences - The Go analyzer failures are not findings about Soteria's source. A genuine syntax error is reported against a specific file and position; here every one of the 26 Go files failed identically with the helper process exiting non-zero (value 2), and three files failed with "Stream closed" or "Broke... - Node.js is simply not present in the analysis container: the JVM could not fork "node" at all ("error=2, No such file or directory"), so the JS/TS sensor could not start. That is a gap in the quality-tools image contents (or an unset sonar.nodejs.executable), not anything in the repository. - Because the Go sensor produced no usable analysis for any file, the SonarQube run for this build effectively carries no Go issues or coverage data. That is the most plausible reason the "Enforce quality gate" stage returned exit code 1 and the image build was skipped. - This is not a test regression and not a compile break: no failing tests were published, and Sonar's parse step is independent of `go build`, so a broken compile would have surfaced earlier and differently. - Re-running the same commit will schedule the same container image and reproduce the same result; the change needed is to the analysis toolchain, not to Soteria's code. - The evidence is not conclusive about the gate's exact failing condition: the console is truncated across the ~25 minutes between the Sonar errors and the end of the build, and the log store was unreachable, so a different check inside the same stage (for example a coverage threshold or the Trivy/... ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `rebuild_quality_tools_analysis_image` - What it would do: Rebuild and republish the pinned quality-tools container image so the SonarQube analyzers have working helper runtimes for the CI agent's architec _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/soteria#16