[hermes] ananke #314: test_timeout_deadlock #16

New Issue

hermes-automation · 2026-09-08T06:23:35Z

hermes-automation commented

2026-09-08 06:23:35 +00:00

Hermes auto-triage classified incident ananke/314 as test_timeout_deadlock (confidence 0.88); first failed gate: tests.

Why a human is needed

Build 314 of ananke failed because the test binary for scm.bstein.dev/bstein/ananke/testing/orchestrator hung and was killed by Go's 10-minute package timeout. The console shows 'panic: test timed out after 10m0s' with TestHookVaultLifecycleBranchMatrix/critical-vault-low-branches/workload-ready-no-value-and-ensure-error stuck for 6m57s, followed by 'FAIL scm.bstein.dev/bstein/ananke/testing/orchestrator 600.063s'. Every other package passed. Because the timeout panic kills the whole binary, no per-test report was published, so the console output is the only record of which test hung. The static-analysis and supply-chain steps ran after this and both passed; the exit 1 at the end of the log is just the pipeline reporting the earlier test failure, so the tail of the build is not where the problem is. Two clues point at the cause: the goroutine dump taken at the timeout shows an HTTP/2 client read loop parked in net/http, and six seconds earlier the test logged 'http: TLS handshake error from 127.0.0.1:60938: remote error: tls: bad certificate' from a loopback listener. That is consistent with the code under test making an HTTPS call to a test server that rejected the handshake, and the call never returning because there is no context deadline or client timeout on that path - so instead of surfacing an error the subtest waits forever. A fix has two parts. First, in the product code exercised by this subtest, make every outbound call on the vault-hook lifecycle path carry a context with a deadline (or an http.Client with Timeout set) and make sure the cancel is deferred before the ensure call so an error return cannot leak a blocked goroutine; if there is a wait or retry loop for 'workload ready with no value', give it a bounded number of attempts and a select on ctx.Done(). Second, in the test itself, give the subtest its own context with a short timeout and assert the expected error, so a regression of this kind fails in seconds with a useful message instead of taking the whole package down at the 10-minute mark. It is also worth checking that the test server's TLS material matches what the client trusts, since the bad-certificate handshake may itself be the trigger. This needs a maintainer with the source in hand: the evidence identifies the hanging test precisely but not the specific blocking call, and the choice between fixing the timeout handling in the product code and fixing the test's TLS fixture is a judgement that requires reading that package. Note also that SonarQube could not parse seven Go files ('Go parser external process returned non-zero exit value: 2', including internal/config/validate.go and internal/service/daemon.go); that did not fail this build, but those files are going unanalyzed and deserve a separate look. Finally, centralized log search was unavailable for this window (connection refused), so this analysis is based solely on the retained build console.

Ariadne did not authorize automated remediation: human_required.

Facts

jenkins — Jenkins job ananke build 314 ended with result FAILURE, starting 2026-09-08T05:57:00Z and ending 2026-09-08T06:19:14Z (1333.774s). (https://ci.bstein.dev/job/ananke/314/)
jenkins — The build reported its first failed stage as 'Enforce quality gate'. (jenkins.first_failed_stage)
jenkins — No structured test results were published for this build; the failed-test list is empty. (jenkins.failed_tests (empty array))
jenkins — The earliest console failure region records 'panic: test timed out after 10m0s' with running tests TestHookVaultLifecycleBranchMatrix, subtest critical-vault-low-branches/workload-ready-no-value-and-ensure-error, each still running at 6m57s. (jenkins.console_failures[1], line 217)
jenkins — The Go test run reports 'FAIL scm.bstein.dev/bstein/ananke/testing/orchestrator 600.063s'; every other package in the same run reported ok. (jenkins.console_failures[2], line 321)
jenkins — The goroutine dump printed with the timeout panic includes net/http HTTP/2 client frames: http2ClientConn.readLoop and http2Transport.newClientConn (h2_bundle.go:9408, :8192). (jenkins.console_failures[2], line 321)
jenkins — Immediately before the panic the test output logged '2026/09/08 06:06:37 http: TLS handshake error from 127.0.0.1:60938: remote error: tls: bad certificate'. (jenkins.console_failures[1], line 217)
jenkins — The build checked out commit 76876ef895 on refs/remotes/origin/main, commit message 'support terraform-owned bootstrap inputs'. (jenkins.console_failures[1], line 217 (Checkout stage))
jenkins — SonarQube analysis ran to completion after the test failure; the gate script recorded sonar_status=ok and supply_status=ok and then exited 1. (jenkins.console_tail)
jenkins — SonarQube's Go analyzer failed to parse 7 source files with 'Go parser external process returned non-zero exit value: 2' (including internal/config/validate.go and internal/service/daemon.go) yet still analyzed 74/74 files. (jenkins.console_failures[3], line 392)

Evidence

The earliest failure region of the build console:

tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - command:
    - "cat"
    image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64"
    name: "quality-tools"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "ananke-314-l0ddg-08d4p-4dfjn"

Inferences

The build failed because the Go test suite hung, not because an assertion failed. The package scm.bstein.dev/bstein/ananke/testing/orchestrator hit Go's 10-minute per-package test timeout at 600.063s while TestHookVaultLifecycleBranchMatrix/critical-vault-low-branches/workload-ready-no-value-and-...
The test failure is the first thing that actually failed. The static-analysis and supply-chain steps ran afterwards and both reported ok, and the pipeline exit 1 at the very end of the log is the enforcement step reporting the earlier test failure. The end-of-build output is downstream noise, not...
The goroutine dump captured at the timeout shows an HTTP/2 client connection read loop parked in net/http, and six seconds of test output earlier there is a 'tls: bad certificate' handshake rejection from a loopback listener (127.0.0.1). The most likely reading is that the code under test issued ...
The hanging subtest name, workload-ready-no-value-and-ensure-error, describes an error path: the workload is ready, there is no value, and the ensure step returns an error. That error path is the prime suspect - a retry or wait loop that has no deadline, a channel receive with no matching send, o...
Nothing in the build points to infrastructure: source checkout, image pulls, agent startup and workspace writes all succeeded, and the only network error is on a loopback address inside the test process itself, not against a registry or the Git host.
The SonarQube Go parse errors on seven files are analyzer-side noise for this build; analysis still completed and the static-analysis step passed, so they did not contribute to the failure. They are worth fixing separately because those files are effectively unanalyzed.

Suggested remediation (not available)

No automated remediation exists for this failure. Hermes proposes one; i

Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail.

Hermes auto-triage classified incident `ananke/314` as **test_timeout_deadlock** (confidence 0.88); first failed gate: `tests`. ## Why a human is needed Build 314 of ananke failed because the test binary for scm.bstein.dev/bstein/ananke/testing/orchestrator hung and was killed by Go's 10-minute package timeout. The console shows 'panic: test timed out after 10m0s' with TestHookVaultLifecycleBranchMatrix/critical-vault-low-branches/workload-ready-no-value-and-ensure-error stuck for 6m57s, followed by 'FAIL scm.bstein.dev/bstein/ananke/testing/orchestrator 600.063s'. Every other package passed. Because the timeout panic kills the whole binary, no per-test report was published, so the console output is the only record of which test hung. The static-analysis and supply-chain steps ran after this and both passed; the exit 1 at the end of the log is just the pipeline reporting the earlier test failure, so the tail of the build is not where the problem is. Two clues point at the cause: the goroutine dump taken at the timeout shows an HTTP/2 client read loop parked in net/http, and six seconds earlier the test logged 'http: TLS handshake error from 127.0.0.1:60938: remote error: tls: bad certificate' from a loopback listener. That is consistent with the code under test making an HTTPS call to a test server that rejected the handshake, and the call never returning because there is no context deadline or client timeout on that path - so instead of surfacing an error the subtest waits forever. A fix has two parts. First, in the product code exercised by this subtest, make every outbound call on the vault-hook lifecycle path carry a context with a deadline (or an http.Client with Timeout set) and make sure the cancel is deferred before the ensure call so an error return cannot leak a blocked goroutine; if there is a wait or retry loop for 'workload ready with no value', give it a bounded number of attempts and a select on ctx.Done(). Second, in the test itself, give the subtest its own context with a short timeout and assert the expected error, so a regression of this kind fails in seconds with a useful message instead of taking the whole package down at the 10-minute mark. It is also worth checking that the test server's TLS material matches what the client trusts, since the bad-certificate handshake may itself be the trigger. This needs a maintainer with the source in hand: the evidence identifies the hanging test precisely but not the specific blocking call, and the choice between fixing the timeout handling in the product code and fixing the test's TLS fixture is a judgement that requires reading that package. Note also that SonarQube could not parse seven Go files ('Go parser external process returned non-zero exit value: 2', including internal/config/validate.go and internal/service/daemon.go); that did not fail this build, but those files are going unanalyzed and deserve a separate look. Finally, centralized log search was unavailable for this window (connection refused), so this analysis is based solely on the retained build console. Ariadne did not authorize automated remediation: `human_required`. ## Facts - **jenkins** — Jenkins job ananke build 314 ended with result FAILURE, starting 2026-09-08T05:57:00Z and ending 2026-09-08T06:19:14Z (1333.774s). (`https://ci.bstein.dev/job/ananke/314/`) - **jenkins** — The build reported its first failed stage as 'Enforce quality gate'. (`jenkins.first_failed_stage`) - **jenkins** — No structured test results were published for this build; the failed-test list is empty. (`jenkins.failed_tests (empty array)`) - **jenkins** — The earliest console failure region records 'panic: test timed out after 10m0s' with running tests TestHookVaultLifecycleBranchMatrix, subtest critical-vault-low-branches/workload-ready-no-value-and-ensure-error, each still running at 6m57s. (`jenkins.console_failures[1], line 217`) - **jenkins** — The Go test run reports 'FAIL scm.bstein.dev/bstein/ananke/testing/orchestrator 600.063s'; every other package in the same run reported ok. (`jenkins.console_failures[2], line 321`) - **jenkins** — The goroutine dump printed with the timeout panic includes net/http HTTP/2 client frames: http2ClientConn.readLoop and http2Transport.newClientConn (h2_bundle.go:9408, :8192). (`jenkins.console_failures[2], line 321`) - **jenkins** — Immediately before the panic the test output logged '2026/09/08 06:06:37 http: TLS handshake error from 127.0.0.1:60938: remote error: tls: bad certificate'. (`jenkins.console_failures[1], line 217`) - **jenkins** — The build checked out commit 76876ef895bf48a8bf975cead2f4f7a962056373 on refs/remotes/origin/main, commit message 'support terraform-owned bootstrap inputs'. (`jenkins.console_failures[1], line 217 (Checkout stage)`) - **jenkins** — SonarQube analysis ran to completion after the test failure; the gate script recorded sonar_status=ok and supply_status=ok and then exited 1. (`jenkins.console_tail`) - **jenkins** — SonarQube's Go analyzer failed to parse 7 source files with 'Go parser external process returned non-zero exit value: 2' (including internal/config/validate.go and internal/service/daemon.go) yet still analyzed 74/74 files. (`jenkins.console_failures[3], line 392`) ## Evidence The earliest failure region of the build console: ``` tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - command: - "cat" image: "registry.bstein.dev/bstein/quality-tools:sonar8.0.1-trivy0.70.0-db20260422-arm64" name: "quality-tools" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins.jenkins.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ananke-314-l0ddg-08d4p-4dfjn" ``` ## Inferences - The build failed because the Go test suite hung, not because an assertion failed. The package scm.bstein.dev/bstein/ananke/testing/orchestrator hit Go's 10-minute per-package test timeout at 600.063s while TestHookVaultLifecycleBranchMatrix/critical-vault-low-branches/workload-ready-no-value-and-... - The test failure is the first thing that actually failed. The static-analysis and supply-chain steps ran afterwards and both reported ok, and the pipeline exit 1 at the very end of the log is the enforcement step reporting the earlier test failure. The end-of-build output is downstream noise, not... - The goroutine dump captured at the timeout shows an HTTP/2 client connection read loop parked in net/http, and six seconds of test output earlier there is a 'tls: bad certificate' handshake rejection from a loopback listener (127.0.0.1). The most likely reading is that the code under test issued ... - The hanging subtest name, workload-ready-no-value-and-ensure-error, describes an error path: the workload is ready, there is no value, and the ensure step returns an error. That error path is the prime suspect - a retry or wait loop that has no deadline, a channel receive with no matching send, o... - Nothing in the build points to infrastructure: source checkout, image pulls, agent startup and workspace writes all succeeded, and the only network error is on a loopback address inside the test process itself, not against a registry or the Git host. - The SonarQube Go parse errors on seven files are analyzer-side noise for this build; analysis still completed and the static-analysis step passed, so they did not contribute to the failure. They are worth fixing separately because those files are effectively unanalyzed. ## Suggested remediation (not available) No automated remediation exists for this failure. Hermes proposes one; i _Truncated by Ariadne: the diagnosis exceeded the configured issue body limit. The complete evidence bundle is in the Ariadne audit trail._

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/ananke#16