titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

hermes: harden worker isolation and blocked-task semantics #13

Merged
bstein merged 2 commits from wt/t_cca008de into main 2026-08-17 08:33:42 +00:00
Conversation 0 Commits 2 Files Changed 6 +420 -9
hermes-automation commented 2026-08-16 20:54:18 +00:00
Member
Copy Link

Three narrowly scoped Hermes reliability fixes, each backed by live evidence from the Cassandra/titan-iac proof run. No merge, no Flux reconcile, no deploy, no agent restart.

1. Worker concurrency starved the dashboard and auth sidecars

Three simultaneous direct CLI workers on the 4-core hermes-agent node drove load to ~45, made the hermes and oauth2-proxy containers fail their probes and restart, and temporarily left the pod 8/10 Ready. Two simultaneous workers stayed at 10/10.

  • HERMES_CLI_LANE_CONCURRENCY: 4 → 2
  • cli-lane-runner CPU limit: 3 → 2

Confirmed from inside the running cli-lane-runner container that this is the right cgroup to cap — it reports HERMES_CLI_LANE_CONCURRENCY=4 and cpu.max = 300000 100000 (3 cores), so the container limit is what actually bounds worker CPU.

Requests remain sensible and are unchanged. The pod requests 745m CPU in total (cli-lane-runner 100m, hermes 350m). Placement does not move. Worst case the lane runner now takes 2 of 4 cores, leaving the dashboard (350m guaranteed) and oauth2-proxy real headroom — which is the property the incident showed was missing.

2. Service-link variables contaminate worker and test environments

The deployment already addresses every service by DNS name, but Kubernetes still injects a service-link variable pair per namespace service. hermes-claude-broker yields:

HERMES_CLAUDE_BROKER_PORT=tcp://10.43.31.76:9006

which claude_oauth_broker.py parses as an int. 93 such variables are present in the live worker environment right now.

Set enableServiceLinks: false on the hermes-agent pod spec, with a manifest regression asserting it.

This is not theoretical — it currently breaks the test suite:

environment result
as-is in the live pod 2 failed, 105 passed — ValueError: invalid literal for int() with base 10: 'tcp://10.43.31.76:9006'
service-link vars stripped 107 passed

3. Scheduler defect: explicitly blocked tasks auto-promote

create_task(initial_status="blocked") records a created event whose payload carries status=blocked, but never a blocked event. _has_sticky_block() only inspects blocked/unblocked events, and recompute_ready() considers blocked tasks — so an explicitly parked task with no incomplete parent silently promoted itself to ready on the next dispatcher cycle.

The fix teaches _has_sticky_block() to also recognize a created event whose payload status is blocked. Because created is always the first event, it is only the most recent of the three kinds when no blocked/unblocked event has fired, so unblock_task() still releases either kind of block. This also covers tasks created before the image patch without adding a persisted field.

Dependency-driven promotion and the circuit-breaker failure-limit guard are untouched.

RED / GREEN evidence

Proven directly against the real upstream kanban_db API — not against a rendered manifest — by applying the Dockerfile's verbatim patch script to an isolated copy of /opt/hermes/hermes_cli and running both variants against throwaway SQLite databases. No live Kanban data was read or written.

test RED (unpatched) GREEN (patched)
initial_block_without_parents_is_sticky FAIL — recompute_ready() == 1 ok
initial_block_with_complete_parents_is_sticky FAIL — recompute_ready() == 1 ok
initial_block_is_released_by_an_explicit_unblock ok ok
dependency_block_promotes_after_parent_completes ok ok
explicit_block_and_unblock_remain_sticky_and_reversible ok ok
circuit_breaker_block_is_not_auto_promoted ok ok
FAILED (failures=2) OK (6/6)

Exactly the two defect tests flip. The four guard tests pass in both directions, so the fix does not weaken dependency-driven promotion, block_task/unblock_task, or circuit-breaker stickiness.

hermes-kanban-blocked-regression.py runs during the image build, so a regression fails the build. Verified this gate is real: the exact build-step invocation exits 1 against the current pre-fix image.

Verification

check result
test_hermes_cli_lanes.py + test_hermes_chat_quality.py 107 passed (clean env)
full testing/tests/ suite 291 passed, 2 pre-existing failures
ruff check (0.8.4, pinned) All checks passed
kustomize build services/hermes exit 0
kubectl apply --dry-run=client exit 0
git diff --check clean

The 2 full-suite failures (test_provider_status_separates_observed_activity_from_plan_quota, test_cassandra_sync_repairs_origin_without_token) are pre-existing and unrelated — reproduced identically on a pristine git archive of main HEAD 0dd6ea0f.

Blocker: image cannot be built or published

This change patches Dockerfile.hermes-agent, so it needs a new image to take effect. I could not produce one:

  • No Docker daemon is reachable from the worker (/var/run/docker.sock absent).
  • The repo has no tracked build/publish pipeline for Dockerfile.hermes-agent. Jenkinsfile / ci/Jenkinsfile.titan-iac are quality-gate pipelines only; the sole Kaniko reference is services/logging/Jenkinsfile.data-prepper, and there is no ImageRepository for hermes-agent.

Per the shepherd's safety correction I did not stand up a privileged Docker-in-Docker builder or expose an unauthenticated Docker/BuildKit API. The image digest in this PR is therefore unchanged and the manifest still references the current sha256:81970563…. A maintainer must build and publish via an unprivileged bounded builder on a healthy node, then bump the digest.

Residual risks

  • The hermes Flux Kustomization is not reconciling — dependency 'flux-system/cert-manager' is not ready. This is why live runs sha256:37ebf720… while main HEAD pins sha256:81970563…. cert-manager is now healthy, so this looks transient, but merging this PR will not take effect until that Kustomization reconciles. Not touched, per instructions.
  • The scheduler fix reads the created event payload. A task whose created event predates the payload status field would fall back to non-sticky, i.e. current behavior — no new failure mode.
  • Lowering concurrency to 2 reduces worker throughput; that is the intended trade for UI/auth availability.
  • The two manifest/Dockerfile string-match assertions are text-coupled to the patch and will need updating if the patch is reworded.

🤖 Generated with Claude Code

Three narrowly scoped Hermes reliability fixes, each backed by live evidence from the Cassandra/titan-iac proof run. No merge, no Flux reconcile, no deploy, no agent restart. ## 1. Worker concurrency starved the dashboard and auth sidecars Three simultaneous direct CLI workers on the 4-core `hermes-agent` node drove load to ~45, made the `hermes` and `oauth2-proxy` containers fail their probes and restart, and temporarily left the pod 8/10 Ready. Two simultaneous workers stayed at 10/10. - `HERMES_CLI_LANE_CONCURRENCY`: `4` → `2` - `cli-lane-runner` CPU limit: `3` → `2` Confirmed from inside the running `cli-lane-runner` container that this is the right cgroup to cap — it reports `HERMES_CLI_LANE_CONCURRENCY=4` and `cpu.max = 300000 100000` (3 cores), so the container limit is what actually bounds worker CPU. **Requests remain sensible and are unchanged.** The pod requests 745m CPU in total (`cli-lane-runner` 100m, `hermes` 350m). Placement does not move. Worst case the lane runner now takes 2 of 4 cores, leaving the dashboard (350m guaranteed) and `oauth2-proxy` real headroom — which is the property the incident showed was missing. ## 2. Service-link variables contaminate worker and test environments The deployment already addresses every service by DNS name, but Kubernetes still injects a service-link variable pair per namespace service. `hermes-claude-broker` yields: ``` HERMES_CLAUDE_BROKER_PORT=tcp://10.43.31.76:9006 ``` which `claude_oauth_broker.py` parses as an int. **93 such variables are present in the live worker environment right now.** Set `enableServiceLinks: false` on the `hermes-agent` pod spec, with a manifest regression asserting it. This is not theoretical — it currently breaks the test suite: | environment | result | |---|---| | as-is in the live pod | `2 failed, 105 passed` — `ValueError: invalid literal for int() with base 10: 'tcp://10.43.31.76:9006'` | | service-link vars stripped | **`107 passed`** | ## 3. Scheduler defect: explicitly blocked tasks auto-promote `create_task(initial_status="blocked")` records a `created` event whose payload carries `status=blocked`, but never a `blocked` event. `_has_sticky_block()` only inspects `blocked`/`unblocked` events, and `recompute_ready()` considers blocked tasks — so an explicitly parked task with no incomplete parent silently promoted itself to `ready` on the next dispatcher cycle. The fix teaches `_has_sticky_block()` to also recognize a `created` event whose payload status is `blocked`. Because `created` is always the first event, it is only the most recent of the three kinds when no `blocked`/`unblocked` event has fired, so `unblock_task()` still releases either kind of block. This also covers tasks created before the image patch without adding a persisted field. Dependency-driven promotion and the circuit-breaker failure-limit guard are untouched. ### RED / GREEN evidence Proven directly against the real upstream `kanban_db` API — not against a rendered manifest — by applying the Dockerfile's verbatim patch script to an isolated copy of `/opt/hermes/hermes_cli` and running both variants against throwaway SQLite databases. **No live Kanban data was read or written.** | test | RED (unpatched) | GREEN (patched) | |---|---|---| | `initial_block_without_parents_is_sticky` | **FAIL** — `recompute_ready() == 1` | ok | | `initial_block_with_complete_parents_is_sticky` | **FAIL** — `recompute_ready() == 1` | ok | | `initial_block_is_released_by_an_explicit_unblock` | ok | ok | | `dependency_block_promotes_after_parent_completes` | ok | ok | | `explicit_block_and_unblock_remain_sticky_and_reversible` | ok | ok | | `circuit_breaker_block_is_not_auto_promoted` | ok | ok | | | `FAILED (failures=2)` | **`OK` (6/6)** | Exactly the two defect tests flip. The four guard tests pass in both directions, so the fix does not weaken dependency-driven promotion, `block_task`/`unblock_task`, or circuit-breaker stickiness. `hermes-kanban-blocked-regression.py` runs during the image build, so a regression fails the build. Verified this gate is real: the exact build-step invocation exits 1 against the current pre-fix image. ## Verification | check | result | |---|---| | `test_hermes_cli_lanes.py` + `test_hermes_chat_quality.py` | 107 passed (clean env) | | full `testing/tests/` suite | 291 passed, 2 pre-existing failures | | `ruff check` (0.8.4, pinned) | All checks passed | | `kustomize build services/hermes` | exit 0 | | `kubectl apply --dry-run=client` | exit 0 | | `git diff --check` | clean | The 2 full-suite failures (`test_provider_status_separates_observed_activity_from_plan_quota`, `test_cassandra_sync_repairs_origin_without_token`) are **pre-existing and unrelated** — reproduced identically on a pristine `git archive` of main HEAD `0dd6ea0f`. ## Blocker: image cannot be built or published This change patches `Dockerfile.hermes-agent`, so it needs a new image to take effect. I could not produce one: - No Docker daemon is reachable from the worker (`/var/run/docker.sock` absent). - The repo has **no tracked build/publish pipeline** for `Dockerfile.hermes-agent`. `Jenkinsfile` / `ci/Jenkinsfile.titan-iac` are quality-gate pipelines only; the sole Kaniko reference is `services/logging/Jenkinsfile.data-prepper`, and there is no `ImageRepository` for `hermes-agent`. Per the shepherd's safety correction I did **not** stand up a privileged Docker-in-Docker builder or expose an unauthenticated Docker/BuildKit API. **The image digest in this PR is therefore unchanged and the manifest still references the current `sha256:81970563…`.** A maintainer must build and publish via an unprivileged bounded builder on a healthy node, then bump the digest. ## Residual risks - **The `hermes` Flux Kustomization is not reconciling** — `dependency 'flux-system/cert-manager' is not ready`. This is why live runs `sha256:37ebf720…` while main HEAD pins `sha256:81970563…`. `cert-manager` is now healthy, so this looks transient, but **merging this PR will not take effect until that Kustomization reconciles.** Not touched, per instructions. - The scheduler fix reads the `created` event payload. A task whose `created` event predates the payload `status` field would fall back to non-sticky, i.e. current behavior — no new failure mode. - Lowering concurrency to 2 reduces worker throughput; that is the intended trade for UI/auth availability. - The two manifest/Dockerfile string-match assertions are text-coupled to the patch and will need updating if the patch is reworded. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
hermes-automation added 1 commit 2026-08-16 20:54:18 +00:00 
Three narrowly scoped Hermes reliability fixes backed by live evidence
from the Cassandra/titan-iac proof run.

Worker concurrency. Three simultaneous direct CLI workers on the 4-core
hermes-agent node drove load to ~45 and made the hermes and oauth2-proxy
containers fail their probes, leaving the pod 8/10 Ready; two workers
stayed at 10/10. Cap HERMES_CLI_LANE_CONCURRENCY at 2 and lower the
cli-lane-runner CPU limit from 3 to 2 so the dashboard and auth sidecars
keep a guaranteed share of the node. Requests are unchanged: the pod
still asks for 745m total, so placement does not move.

Service links. Kubernetes injects a service-link variable pair for every
service in the namespace, and hermes-claude-broker produces
HERMES_CLAUDE_BROKER_PORT=tcp://10.43.31.76:9006 — a value the broker
parses as an int. That contaminated worker and test environments even
though the deployment already addresses every service by DNS name. Set
enableServiceLinks: false on the hermes-agent pod spec.

Blocked-task scheduling. create_task(initial_status="blocked") records a
created event carrying status=blocked but never a blocked event, while
_has_sticky_block() only inspects blocked/unblocked events. recompute_ready()
considers blocked tasks, so an explicitly parked task with no incomplete
parent auto-promoted on the next dispatcher cycle. Teach _has_sticky_block()
to also recognize a created event whose payload status is blocked, which
covers tasks created before this image patch without adding a persisted
field. Dependency-driven promotion and the circuit-breaker failure-limit
guard are untouched; unblock_task() still releases either kind of block.

hermes-kanban-blocked-regression.py runs against the real upstream
kanban_db API during the image build, so the build fails if any of these
semantics regress.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hermes-automation added 1 commit 2026-08-16 22:42:28 +00:00 
The sticky-block gate added in the previous commit classifies a task from
the `created` event payload that upstream `create_task` writes. That
producer is code we do not own, so trusting it silently was the gap: if
upstream renamed the key, dropped it, or stopped deriving it from
`initial_status`, the image would still build and ship a consumer that
mis-classifies every task it reads.

Anchor the producer contract at build time, before the regression suite
runs, with three assert-only preconditions: the `initial_status="blocked"`
park resolves `task_status` to `"blocked"`, every non-park creation
resolves it to something else, and the `created` event carries that same
variable under `"status"`. None of them rewrite the producer.

Textual anchors cannot see dataflow, so add the runtime net the reviewer
asked for. The suite now drives the real API: create + claim an ordinary
task, trip the circuit breaker once at failure_limit=1 so it parks with a
`gave_up` event (leaving its own `created` event as the most recent
create/block/unblock row), then recompute at failure_limit=2 and require
promotion to ready. That case is red under an unconditional-true created
predicate and red under producer drift that labels every created event
blocked, while the explicit block/unblock, dependency-promotion and
circuit-breaker-at-current-limit cases stay green. Non-blocked and
malformed created payloads are pinned as controls, and the gate now
rejects non-dict payloads rather than trusting `.get`.

Also make the live placement correction durable: titan-04 is cordoned
after repeated kernel undervoltage and kubelet failure and titan-19 was
probe/Longhorn unstable under worker load, so both join the hard NotIn
list; titan-05 is healthy but sits at 3592m/3600m requested CPU, so the
main hermes container gives back 50m (350m -> 300m) to schedule there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bstein merged commit ab346f5550 into main 2026-08-17 08:33:42 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#13
No description provided.
Delete Branch "wt/t_cca008de"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?