[hermes] ananke #374: sonarqube_server_internal_error #228

New Issue

hermes-automation · 2026-10-03T06:43:44Z

hermes-automation commented

2026-10-03 06:43:44 +00:00

Hermes auto-triage classified incident ananke/374 as sonarqube_server_internal_error (confidence 0.75); first failed gate: sonarqube.

Why a human is needed

Build 374 failed in the 'Enforce quality gate' stage because the SonarScanner could not upload its analysis report to the in-cluster SonarQube server: the server returned a bare HTTP 500 (Tomcat 'Internal Server Error') on the report-submit call, which the pipeline script then treated as a gate failure and exited non-zero. No structured test failures were published for this build, and the other gate checks visible in the log (ironbank, supply chain) evaluated normally, so this really is isolated to the SonarQube upload step. There are also a handful of 'Go parser ... non-zero exit value: 2' errors on nine specific files earlier in the log, but the scan continued past them and analyzed all 74 files, so they look more like analyzer noise than the actual cause of the exit code; they're worth a second look only if this keeps failing after SonarQube itself is confirmed healthy. A fix here isn't a code change to ananke - it's making sure the SonarQube service the pipeline talks to is actually healthy (check its pod/logs around 06:24 UTC for the cause of the 500, e.g. DB connection issues or memory pressure) and then re-running the build once that's confirmed.

Ariadne did not authorize automated remediation: classification_not_supported: 'sonarqube_server_internal_error'.

Facts

jenkins — Jenkins build ananke #374 resulted in FAILURE with jenkins.first_failed_stage = "Enforce quality gate". (jenkins.first_failed_stage, jenkins.result)
jenkins — jenkins.failed_tests is empty, meaning Jenkins published no structured failing test results for this build. (jenkins.failed_tests)
jenkins — The SonarScanner CLI aborted with java.lang.IllegalStateException: 'Failed to upload report: Error 500 on http://sonarqube.quality.svc.cluster.local:9000/api/ce/submit?projectKey=ananke&projectName=ananke', with the response body being a generic Tomcat 'HTTP Status 500 – Internal Server Error' page. (jenkins.console_failures[2] (marker=Exception, line_number=372))
jenkins — Immediately after the scanner failure, the pipeline script captured sonar_status=error, printed 'sonarqube gate failed: error', and set fail=1 before exiting 1, which is what made the 'Enforce quality gate' stage fail. (jenkins.console_tail)
jenkins — Before the upload failure, SonarQube's Go analyzer logged 'Unable to parse file' / 'Go parser external process returned non-zero exit value: 2' for nine files (internal/config/validate.go, internal/cluster/orchestrator_access_fluxsource.go, internal/cluster/orchestrator_service_auth.go, internal/... (jenkins.console_failures[1] (marker=non-zero exit, line_number=293))
opensearch — The log_evidence query against OpenSearch for this incident's time window returned no records because the OpenSearch connection was refused. (log_evidence.error)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "ananke-374-6770k-s6hgc-hdsk2"

Inferences

The quality gate failed because the SonarScanner could not hand its analysis report to the SonarQube server, not because of a measured code-quality violation; the server's own Tomcat process returned a bare 500 with no application detail, which usually points to a problem on the SonarQube side (d...
The Go-parser 'non-zero exit value: 2' errors on nine specific files are most likely a tooling/version mismatch in the SonarQube Go analyzer rather than real syntax errors in those files, since the scan continued and finished analyzing all 74 Go files afterward; this can't be fully ruled out with...
Because failed_tests is empty and the console tail shows the ironbank and supply-chain checks evaluating normally (ironbank_required=0, supply_status=ok), the only check that failed within 'Enforce quality gate' was the SonarQube step itself.

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

Proposed action: retry_after_sonarqube_health_check
What it would do: Check that the in-cluster SonarQube service (sonarqube.quality.svc.cluster.local) is healthy and able to accept /api/ce/submit calls, and if so, re-run the failed build so the scanner can re-upload its report.
Evidence that should be required first: Evidence that the SonarQube server was not in a persistently broken state at the time of the 500 (e.g. its own logs/pod status around 06:24:27Z show the error was a one-off rather than an ongoing crash, OOM, or DB-connectivity problem), so a retry isn't just repeating against a service that's still down.

Links

Failed build: https://ci.bstein.dev/job/ananke/374/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_875e26a80dd74d46b48061823e1ab8ca). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `ananke/374` as **sonarqube_server_internal_error** (confidence 0.75); first failed gate: `sonarqube`. ## Why a human is needed Build 374 failed in the 'Enforce quality gate' stage because the SonarScanner could not upload its analysis report to the in-cluster SonarQube server: the server returned a bare HTTP 500 (Tomcat 'Internal Server Error') on the report-submit call, which the pipeline script then treated as a gate failure and exited non-zero. No structured test failures were published for this build, and the other gate checks visible in the log (ironbank, supply chain) evaluated normally, so this really is isolated to the SonarQube upload step. There are also a handful of 'Go parser ... non-zero exit value: 2' errors on nine specific files earlier in the log, but the scan continued past them and analyzed all 74 files, so they look more like analyzer noise than the actual cause of the exit code; they're worth a second look only if this keeps failing after SonarQube itself is confirmed healthy. A fix here isn't a code change to ananke - it's making sure the SonarQube service the pipeline talks to is actually healthy (check its pod/logs around 06:24 UTC for the cause of the 500, e.g. DB connection issues or memory pressure) and then re-running the build once that's confirmed. Ariadne did not authorize automated remediation: `classification_not_supported: 'sonarqube_server_internal_error'`. ## Facts - **jenkins** — Jenkins build ananke #374 resulted in FAILURE with jenkins.first_failed_stage = "Enforce quality gate". (`jenkins.first_failed_stage, jenkins.result`) - **jenkins** — jenkins.failed_tests is empty, meaning Jenkins published no structured failing test results for this build. (`jenkins.failed_tests`) - **jenkins** — The SonarScanner CLI aborted with java.lang.IllegalStateException: 'Failed to upload report: Error 500 on http://sonarqube.quality.svc.cluster.local:9000/api/ce/submit?projectKey=ananke&projectName=ananke', with the response body being a generic Tomcat 'HTTP Status 500 – Internal Server Error' page. (`jenkins.console_failures[2] (marker=Exception, line_number=372)`) - **jenkins** — Immediately after the scanner failure, the pipeline script captured sonar_status=error, printed 'sonarqube gate failed: error', and set fail=1 before exiting 1, which is what made the 'Enforce quality gate' stage fail. (`jenkins.console_tail`) - **jenkins** — Before the upload failure, SonarQube's Go analyzer logged 'Unable to parse file' / 'Go parser external process returned non-zero exit value: 2' for nine files (internal/config/validate.go, internal/cluster/orchestrator_access_fluxsource.go, internal/cluster/orchestrator_service_auth.go, internal/... (`jenkins.console_failures[1] (marker=non-zero exit, line_number=293)`) - **opensearch** — The log_evidence query against OpenSearch for this incident's time window returned no records because the OpenSearch connection was refused. (`log_evidence.error`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ananke-374-6770k-s6hgc-hdsk2" ``` ## Inferences - The quality gate failed because the SonarScanner could not hand its analysis report to the SonarQube server, not because of a measured code-quality violation; the server's own Tomcat process returned a bare 500 with no application detail, which usually points to a problem on the SonarQube side (d... - The Go-parser 'non-zero exit value: 2' errors on nine specific files are most likely a tooling/version mismatch in the SonarQube Go analyzer rather than real syntax errors in those files, since the scan continued and finished analyzing all 74 Go files afterward; this can't be fully ruled out with... - Because failed_tests is empty and the console tail shows the ironbank and supply-chain checks evaluating normally (ironbank_required=0, supply_status=ok), the only check that failed within 'Enforce quality gate' was the SonarQube step itself. ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `retry_after_sonarqube_health_check` - What it would do: Check that the in-cluster SonarQube service (sonarqube.quality.svc.cluster.local) is healthy and able to accept /api/ce/submit calls, and if so, re-run the failed build so the scanner can re-upload its report. - Evidence that should be required first: Evidence that the SonarQube server was not in a persistently broken state at the time of the 500 (e.g. its own logs/pod status around 06:24:27Z show the error was a one-off rather than an ongoing crash, OOM, or DB-connectivity problem), so a retry isn't just repeating against a service that's still down. ## Links - Failed build: https://ci.bstein.dev/job/ananke/374/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_875e26a80dd74d46b48061823e1ab8ca](https://triage.bstein.dev/chat?resume=run_875e26a80dd74d46b48061823e1ab8ca)). Hermes has no write access to this repository; no files or infrastructure were changed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/ananke#228