[hermes] bstein-dev-home #546: jenkins_agent_channel_interruption_frontend_tests #145

New Issue

hermes-automation · 2026-09-23T10:07:56Z

hermes-automation commented

2026-09-23 10:07:56 +00:00

Hermes auto-triage classified incident bstein-dev-home/546 as jenkins_agent_channel_interruption_frontend_tests (confidence 0.65); first failed gate: tests.

Why a human is needed

Build 546 failed in the Frontend tests stage, not because of a failing test: no test failures were published (jenkins.failed_tests is empty) and the backend suite had just completed cleanly (128 passed in 46.66s). Partway through Frontend tests, the Jenkins agent pod bstein-dev-home-546-6h0qg-5zgwp-dpqpl reported 'seems to be removed or offline (java.lang.InterruptedException)', waited, and came back online, but the shell step that runs the frontend tests never actually launched ('process apparently never started in .../durable-a831f6fc'). Every later stage (quality gate, SonarQube evidence refresh, test metrics publish, image builds) was skipped as a consequence, and the pipeline ended with 'script returned exit code -2'. This points to the agent's connection to Jenkins being briefly interrupted while mid-stage, which broke the durable-task launch for the frontend test shell step; nothing in the repository's frontend code or test suite appears to be at fault. There's no repo fix to make here — the practical remedy is re-running the frontend test stage, since a fresh build on a stable agent connection would normally proceed past this point. If this recurs, it's worth checking whether the CI agent pods are seeing network interruptions, node pressure, or evictions around this time window. Separately and unrelated to this specific failure: the SonarQube scan in this same build logged that the 'node' binary isn't available in the scanner container ('Cannot run program node: No such file or directory'), which silently disables the JavaScript/TypeScript analysis sensor. It didn't cause this failure, but it means Sonar currently isn't analyzing this repo's frontend code at all — worth flagging to whoever maintains the quality-tools scanner image. Note that OpenSearch application-log correlation wasn't available for this build (connection refused), so there's no pod-event evidence beyond the Jenkins console to corroborate exactly why the agent dropped.

Ariadne did not authorize automated remediation: classification_not_supported: 'jenkins_agent_channel_interruption_frontend_tests'.

Facts

jenkins — Build 546 result is FAILURE with first_failed_stage reported as 'Frontend tests'. (jenkins.result, jenkins.first_failed_stage)
jenkins — jenkins.failed_tests is empty; no structured failing test name, class, or assertion was published for this build. (jenkins.failed_tests)
jenkins — The backend test stage completed successfully immediately before Frontend tests began: '128 passed in 46.66s'. (console_failures[4] (marker 'Exception', line_number 703))
jenkins — Inside the Frontend tests stage, agent bstein-dev-home-546-6h0qg-5zgwp-dpqpl reported 'seems to be removed or offline (java.lang.InterruptedException)', then came back online, and the shell step failed to launch: 'process apparently never started in /home/jenkins/agent/workspace/bstein-dev-home@t... (console_failures[4] (marker 'Exception', line_number 703))
jenkins — Every stage after Frontend tests (Run quality gate, Refresh SonarQube evidence, Publish test metrics, Enforce quality gate, Build & push frontend, Build & push backend) was skipped due to the earlier failure. (console_tail)
jenkins — Pipeline ended with 'ERROR: script returned exit code -2' and 'Finished: FAILURE'; Gitea asset publishing was withheld because the build was non-successful. (console_failures[5] (marker 'ERROR:', line_number 775))
jenkins — Earlier in the same build, the SonarQube scanner's JavaScript analysis sensor errored with "Error when running: 'node -v'... Cannot run program 'node': No such file or directory", but the scanner continued on to the TypeScript sensor rather than aborting the build at that point. (console_failures[2]-[3] (markers 'Exception'/'No such file', line_number 341-377))
opensearch — OpenSearch log correlation for this incident window was unavailable: 'opensearch request failed: [Errno 111] Connection refused'. (log_evidence.error)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "bstein-dev-home-546-6h0qg-5zgwp-dpqpl"

Inferences

The Frontend tests stage did not fail because of a failing test or a code regression — it failed because the Jenkins agent's connection to the controller was interrupted mid-stage, which broke the durable-task launch for the shell step before any test could even start running.
The Node.js-missing error in the SonarQube stage is a separate, pre-existing gap in the quality-tools scanning image (no 'node' binary), unrelated to this build's failure; it silently disables JS/TS analysis whenever Sonar runs, rather than causing a build failure.

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it.

Proposed action: retry_after_agent_channel_interruption
What it would do: Automatically retry a build when the console log shows the Jenkins agent going offline then back online mid-stage, immediately followed by a durable-task 'process apparently never started' error, since that pattern is an agent-connectivity disruption rather than a code or test failure.
Evidence that should be required first: Console log must show the agent 'removed or offline' / 'back online' message directly followed by a 'process apparently never started' durable-task error in the same stage, with jenkins.failed_tests empty and no test assertion text in the surrounding console output.

Links

Failed build: https://ci.bstein.dev/job/bstein-dev-home/546/
Full evidence bundle and audit trail live in Ariadne at /api/admin/audit/events, event types hermes_autotriage_incident and hermes_autotriage_diagnosis.

Filed automatically by Ariadne from a Hermes Agent diagnosis (run run_c169c8b802b44e91aa33573dd14c5923). Hermes has no write access to this repository; no files or infrastructure were changed.

Hermes auto-triage classified incident `bstein-dev-home/546` as **jenkins_agent_channel_interruption_frontend_tests** (confidence 0.65); first failed gate: `tests`. ## Why a human is needed Build 546 failed in the Frontend tests stage, not because of a failing test: no test failures were published (jenkins.failed_tests is empty) and the backend suite had just completed cleanly (128 passed in 46.66s). Partway through Frontend tests, the Jenkins agent pod bstein-dev-home-546-6h0qg-5zgwp-dpqpl reported 'seems to be removed or offline (java.lang.InterruptedException)', waited, and came back online, but the shell step that runs the frontend tests never actually launched ('process apparently never started in .../durable-a831f6fc'). Every later stage (quality gate, SonarQube evidence refresh, test metrics publish, image builds) was skipped as a consequence, and the pipeline ended with 'script returned exit code -2'. This points to the agent's connection to Jenkins being briefly interrupted while mid-stage, which broke the durable-task launch for the frontend test shell step; nothing in the repository's frontend code or test suite appears to be at fault. There's no repo fix to make here — the practical remedy is re-running the frontend test stage, since a fresh build on a stable agent connection would normally proceed past this point. If this recurs, it's worth checking whether the CI agent pods are seeing network interruptions, node pressure, or evictions around this time window. Separately and unrelated to this specific failure: the SonarQube scan in this same build logged that the 'node' binary isn't available in the scanner container ('Cannot run program node: No such file or directory'), which silently disables the JavaScript/TypeScript analysis sensor. It didn't cause this failure, but it means Sonar currently isn't analyzing this repo's frontend code at all — worth flagging to whoever maintains the quality-tools scanner image. Note that OpenSearch application-log correlation wasn't available for this build (connection refused), so there's no pod-event evidence beyond the Jenkins console to corroborate exactly why the agent dropped. Ariadne did not authorize automated remediation: `classification_not_supported: 'jenkins_agent_channel_interruption_frontend_tests'`. ## Facts - **jenkins** — Build 546 result is FAILURE with first_failed_stage reported as 'Frontend tests'. (`jenkins.result, jenkins.first_failed_stage`) - **jenkins** — jenkins.failed_tests is empty; no structured failing test name, class, or assertion was published for this build. (`jenkins.failed_tests`) - **jenkins** — The backend test stage completed successfully immediately before Frontend tests began: '128 passed in 46.66s'. (`console_failures[4] (marker 'Exception', line_number 703)`) - **jenkins** — Inside the Frontend tests stage, agent bstein-dev-home-546-6h0qg-5zgwp-dpqpl reported 'seems to be removed or offline (java.lang.InterruptedException)', then came back online, and the shell step failed to launch: 'process apparently never started in /home/jenkins/agent/workspace/bstein-dev-home@t... (`console_failures[4] (marker 'Exception', line_number 703)`) - **jenkins** — Every stage after Frontend tests (Run quality gate, Refresh SonarQube evidence, Publish test metrics, Enforce quality gate, Build & push frontend, Build & push backend) was skipped due to the earlier failure. (`console_tail`) - **jenkins** — Pipeline ended with 'ERROR: script returned exit code -2' and 'Finished: FAILURE'; Gitea asset publishing was withheld because the build was non-successful. (`console_failures[5] (marker 'ERROR:', line_number 775)`) - **jenkins** — Earlier in the same build, the SonarQube scanner's JavaScript analysis sensor errored with "Error when running: 'node -v'... Cannot run program 'node': No such file or directory", but the scanner continued on to the TypeScript sensor rather than aborting the build at that point. (`console_failures[2]-[3] (markers 'Exception'/'No such file', line_number 341-377)`) - **opensearch** — OpenSearch log correlation for this incident window was unavailable: 'opensearch request failed: [Errno 111] Connection refused'. (`log_evidence.error`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "bstein-dev-home-546-6h0qg-5zgwp-dpqpl" ``` ## Inferences - The Frontend tests stage did not fail because of a failing test or a code regression — it failed because the Jenkins agent's connection to the controller was interrupted mid-stage, which broke the durable-task launch for the shell step before any test could even start running. - The Node.js-missing error in the SonarQube stage is a separate, pre-existing gap in the quality-tools scanning image (no 'node' binary), unrelated to this build's failure; it silently disables JS/TS analysis whenever Sonar runs, rather than causing a build failure. ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; it was not performed, and Ariadne cannot perform it until a person builds and deploys it. - Proposed action: `retry_after_agent_channel_interruption` - What it would do: Automatically retry a build when the console log shows the Jenkins agent going offline then back online mid-stage, immediately followed by a durable-task 'process apparently never started' error, since that pattern is an agent-connectivity disruption rather than a code or test failure. - Evidence that should be required first: Console log must show the agent 'removed or offline' / 'back online' message directly followed by a 'process apparently never started' durable-task error in the same stage, with jenkins.failed_tests empty and no test assertion text in the surrounding console output. ## Links - Failed build: https://ci.bstein.dev/job/bstein-dev-home/546/ - Full evidence bundle and audit trail live in Ariadne at `/api/admin/audit/events`, event types `hermes_autotriage_incident` and `hermes_autotriage_diagnosis`. Filed automatically by Ariadne from a Hermes Agent diagnosis (run [run_c169c8b802b44e91aa33573dd14c5923](https://triage.bstein.dev/chat?resume=run_c169c8b802b44e91aa33573dd14c5923)). Hermes has no write access to this repository; no files or infrastructure were changed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bstein/bstein-dev-home#145