titan/soteria
3
0
Fork 0
Code Issues 23 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[hermes] soteria #335: sonar_analyzer_runtime_broken_in_ci_image #19

New Issue
Open
opened 2026-09-03 21:25:47 +00:00 by hermes-automation · 0 comments
hermes-automation commented 2026-09-03 21:25:47 +00:00
Member
Copy Link

Hermes auto-triage classified incident soteria/335 as sonar_analyzer_runtime_broken_in_ci_image (confidence 0.65); first failed gate: sonarqube (Jenkins stage "Enforce quality gate").

Why a human is needed

Build 335 failed in the "Enforce quality gate" stage. No tests were published and no test failed. Inside that stage the SonarQube scan could not analyze the codebase: all 26 Go source files failed to parse, 24 with "Go parser external process returned non-zero exit value: 2" and internal/k8s/state.go and internal/k8s/jobs.go with "Stream closed", and the scanner could not start Node.js at all ("Cannot run program "node": error=2, No such file or directory"). A syntax error in the repository would fail a single file, not all 26 including cmd/soteria/main.go, so the Go analyzer helper and the Node runtime are both failing to execute inside the analysis container registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64. With no file parsed, the scan produced no issues and imported no coverage for the Go code, which is the most likely reason the gate rejected the build and the image push was skipped. The fix is to the scanner image, not to this service's source: republish quality-tools for arm64 with node on PATH and a Go analyzer binary that runs on the agent architecture, then confirm inside the container with 'node -v' and a scan over internal/. A maintainer needs to make the call because the retained console jumps from the scanner output to the end of the pipeline, so the gate's own verdict line is not in the captured evidence, and because the scanner image is maintained outside this service's tree.

Ariadne did not authorize automated remediation: human_required.

Facts

  • jenkins — Build 335 of job soteria ran 2026-09-03T20:59:00Z to 2026-09-03T21:22:44Z (1423.9s) and ended with result FAILURE. (https://ci.bstein.dev/job/soteria/335/ (jenkins.result, jenkins.timestamps))
  • jenkins — The first failed pipeline stage reported by the build is "Enforce quality gate"; the later stage "Build & push image" was skipped due to earlier failure(s) and the pipeline ended with "ERROR: script returned exit code 1". (jenkins.first_failed_stage; jenkins.console_tail; jenkins.console_failures[3] line 3357)
  • jenkins — The build published no structured test results: jenkins.failed_tests is empty, and no failing test, class, or assertion appears anywhere in the retained evidence. (jenkins.failed_tests (empty array))
  • jenkins — The earliest detected failure region is SonarQube scanner output at 21:04:34-21:04:35, where the Go sensor loaded 26 source files and every one of them failed to parse: 24 files reported "Go parser external process returned non-zero exit value: 2" and internal/k8s/state.go and internal/k8s/jobs.g... (jenkins.console_failures[1] line 310 (marker "non-zero exit"))
  • jenkins — Despite every file erroring, the Go sensor logged "26/26 source files have been analyzed" and completed, and no coverage report was imported. (jenkins.console_failures[1] line 310 (21:04:35.796, 21:04:35.817))
  • jenkins — In the same scanner run the TypeScript sensor failed with NodeCommandException "Error when running: 'node -v'. Is Node.js available during analysis?", caused by java.io.IOException: Cannot run program "node": error=2, No such file or directory. (jenkins.console_failures[1] line 310 (21:04:50.730); jenkins.console_failures[2] line 411)
  • jenkins — The analysis executed inside the pod container "quality-tools" built from image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, with the workspace mounted at /home/jenkins/agent. (jenkins.console_failures[0] line 115)
  • jenkins — An agent was provisioned and the build ran on it (agent name soteria-335-j2jr2-lrqgv-c84b2, workspace path /home/jenkins/agent/workspace/Soteria); there is no disk-space, image-pull, DNS, TLS, connection, or provisioning error anywhere in the retained console excerpts. (jenkins.console_failures[0] line 115; jenkins.console_failures[1] file paths)
  • jenkins — The console is truncated: jenkins.console_truncated is true and the retained excerpts jump from line 411 to line 3357, so the output between the scanner run and the end of the pipeline is not in the evidence. (jenkins.console_truncated; jenkins.console_failures line numbers 411 -> 3357)
  • opensearch — No application or build logs are available for the incident window: the log query returned zero records with the error "opensearch request failed: [Errno 111] Connection refused". (log_evidence.error, window 2026-09-03T20:54:00Z to 2026-09-03T21:27:44Z)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "soteria-335-j2jr2-lrqgv-c84b2"

Inferences

  • The Go parse errors are not a defect in this repository's source. A real syntax error fails the one file that contains it; here all 26 Go files failed at once, including cmd/soteria/main.go and every package under internal/. "Go parser external process returned non-zero exit value: 2" and "Stream...
  • Node.js is definitively absent from the analysis container: the scanner tried to run 'node -v' and the OS returned error=2, No such file or directory. That is a property of the image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, not of the commit. It makes the ...
  • Because no Go file parsed, the scan recorded no issues, no duplication, and no coverage for the Go code, so the quality gate was evaluated against effectively empty measures for the entire service. That is the most plausible reason the "Enforce quality gate" stage exited 1. This last link is infe...
  • Re-running the same commit will land on the same container image and fail the same way; nothing in this failure depends on what the commit changed.
  • A fix belongs in the CI toolchain image rather than in the service: republish quality-tools for arm64 with a node binary on PATH and a Sonar Go analyzer that executes on the agent's architecture, and verify inside the container by running 'node -v' and a scan over internal/ before trusting the ga...

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

  • Proposed action: republish_quality_tools_image_with_working_analyzer_runtimes
  • What it would do: Rebuild and publish the quality-tools scanner image for the agent architecture with Node.js on PATH and a Sonar Go analyzer binary that executes successfully, then re-run the affected build so the quality gate scores a real analysis instead of an empty one.
  • Evidence that should be required first: Console evidence that analyzer runtimes, not the commit, failed: every source file of a language failing with "external process returned non-zero exit value" or "Stream closed", and/or "Cannot run program" / "error=2, No such file or directory" for an analyzer runtime, with no compile or test failure of the repository's own code in the same build. A verification run of the candidate image showi...

Links

  • Failed build: https:/

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `soteria/335` as **sonar_analyzer_runtime_broken_in_ci_image** (confidence 0.65); first failed gate: `sonarqube (Jenkins stage "Enforce quality gate")`. ## Why a human is needed Build 335 failed in the "Enforce quality gate" stage. No tests were published and no test failed. Inside that stage the SonarQube scan could not analyze the codebase: all 26 Go source files failed to parse, 24 with "Go parser external process returned non-zero exit value: 2" and internal/k8s/state.go and internal/k8s/jobs.go with "Stream closed", and the scanner could not start Node.js at all ("Cannot run program \"node\": error=2, No such file or directory"). A syntax error in the repository would fail a single file, not all 26 including cmd/soteria/main.go, so the Go analyzer helper and the Node runtime are both failing to execute inside the analysis container registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64. With no file parsed, the scan produced no issues and imported no coverage for the Go code, which is the most likely reason the gate rejected the build and the image push was skipped. The fix is to the scanner image, not to this service's source: republish quality-tools for arm64 with node on PATH and a Go analyzer binary that runs on the agent architecture, then confirm inside the container with 'node -v' and a scan over internal/. A maintainer needs to make the call because the retained console jumps from the scanner output to the end of the pipeline, so the gate's own verdict line is not in the captured evidence, and because the scanner image is maintained outside this service's tree. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Build 335 of job soteria ran 2026-09-03T20:59:00Z to 2026-09-03T21:22:44Z (1423.9s) and ended with result FAILURE. (`https://ci.bstein.dev/job/soteria/335/ (jenkins.result, jenkins.timestamps)`) - **jenkins** — The first failed pipeline stage reported by the build is "Enforce quality gate"; the later stage "Build & push image" was skipped due to earlier failure(s) and the pipeline ended with "ERROR: script returned exit code 1". (`jenkins.first_failed_stage; jenkins.console_tail; jenkins.console_failures[3] line 3357`) - **jenkins** — The build published no structured test results: jenkins.failed_tests is empty, and no failing test, class, or assertion appears anywhere in the retained evidence. (`jenkins.failed_tests (empty array)`) - **jenkins** — The earliest detected failure region is SonarQube scanner output at 21:04:34-21:04:35, where the Go sensor loaded 26 source files and every one of them failed to parse: 24 files reported "Go parser external process returned non-zero exit value: 2" and internal/k8s/state.go and internal/k8s/jobs.g... (`jenkins.console_failures[1] line 310 (marker "non-zero exit")`) - **jenkins** — Despite every file erroring, the Go sensor logged "26/26 source files have been analyzed" and completed, and no coverage report was imported. (`jenkins.console_failures[1] line 310 (21:04:35.796, 21:04:35.817)`) - **jenkins** — In the same scanner run the TypeScript sensor failed with NodeCommandException "Error when running: 'node -v'. Is Node.js available during analysis?", caused by java.io.IOException: Cannot run program "node": error=2, No such file or directory. (`jenkins.console_failures[1] line 310 (21:04:50.730); jenkins.console_failures[2] line 411`) - **jenkins** — The analysis executed inside the pod container "quality-tools" built from image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, with the workspace mounted at /home/jenkins/agent. (`jenkins.console_failures[0] line 115`) - **jenkins** — An agent was provisioned and the build ran on it (agent name soteria-335-j2jr2-lrqgv-c84b2, workspace path /home/jenkins/agent/workspace/Soteria); there is no disk-space, image-pull, DNS, TLS, connection, or provisioning error anywhere in the retained console excerpts. (`jenkins.console_failures[0] line 115; jenkins.console_failures[1] file paths`) - **jenkins** — The console is truncated: jenkins.console_truncated is true and the retained excerpts jump from line 411 to line 3357, so the output between the scanner run and the end of the pipeline is not in the evidence. (`jenkins.console_truncated; jenkins.console_failures line numbers 411 -> 3357`) - **opensearch** — No application or build logs are available for the incident window: the log query returned zero records with the error "opensearch request failed: [Errno 111] Connection refused". (`log_evidence.error, window 2026-09-03T20:54:00Z to 2026-09-03T21:27:44Z`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "soteria-335-j2jr2-lrqgv-c84b2" ``` ## Inferences - The Go parse errors are not a defect in this repository's source. A real syntax error fails the one file that contains it; here all 26 Go files failed at once, including cmd/soteria/main.go and every package under internal/. "Go parser external process returned non-zero exit value: 2" and "Stream... - Node.js is definitively absent from the analysis container: the scanner tried to run 'node -v' and the OS returned error=2, No such file or directory. That is a property of the image registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64, not of the commit. It makes the ... - Because no Go file parsed, the scan recorded no issues, no duplication, and no coverage for the Go code, so the quality gate was evaluated against effectively empty measures for the entire service. That is the most plausible reason the "Enforce quality gate" stage exited 1. This last link is infe... - Re-running the same commit will land on the same container image and fail the same way; nothing in this failure depends on what the commit changed. - A fix belongs in the CI toolchain image rather than in the service: republish quality-tools for arm64 with a node binary on PATH and a Sonar Go analyzer that executes on the agent's architecture, and verify inside the container by running 'node -v' and a scan over internal/ before trusting the ga... ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `republish_quality_tools_image_with_working_analyzer_runtimes` - What it would do: Rebuild and publish the quality-tools scanner image for the agent architecture with Node.js on PATH and a Sonar Go analyzer binary that executes successfully, then re-run the affected build so the quality gate scores a real analysis instead of an empty one. - Evidence that should be required first: Console evidence that analyzer runtimes, not the commit, failed: every source file of a language failing with "external process returned non-zero exit value" or "Stream closed", and/or "Cannot run program" / "error=2, No such file or directory" for an analyzer runtime, with no compile or test failure of the repository's own code in the same build. A verification run of the candidate image showi... ## Links - Failed build: https:/ _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._ <!-- hermes-triage job=soteria classification=sonar_analyzer_runtime_broken_in_ci_image incident=soteria/335 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/soteria#19
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?