APPLY ONLY AFTER the multi-arch hermes-agent image is built + validated
(both arch leaves + promoted index). Supersedes the earlier titan-22
flip on feature/hermes-agent-multiarch (dfa50b75), which required
worker=true and tolerated the old media-primary taint.
Rewrites the runtime node affinity so hermes-agent runs on titan-22:
- Adds a second, OR'd nodeSelectorTerm matching amd64 + hostname
titan-22 + node-role.kubernetes.io/accelerator=true. It does NOT
require node-role.kubernetes.io/worker (titan-22 is no longer a
generic worker).
- Keeps the arm64 pi-fleet term untouched as an OR'd fallback so the
worker is never stranded if titan-22 is unavailable.
- Makes titan-22 the STRONG/primary preference: a hostname=titan-22
preference at weight 100 (the scheduler maximum) outranks the pi-fleet
rpi5 nudge, lowered to weight 50, so hermes actually lives on titan-22.
- Tolerates node-role.kubernetes.io/accelerator=true:NoSchedule so it
can consider titan-22; this does not change jellyfin's media-core
priority or preemption.
Updates test_hermes_agent_layout.py to the two-term topology, the
[100, 50] preference weights, the titan-22 primary preference, the
absence of a worker requirement on the titan-22 term, and the toleration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Requiring rpi5 while the historical hostname exclusion list removes the very rpi5
nodes that currently have headroom (titan-04/06) left only loaded rpi5s
(titan-05/07/11), so the agent could not schedule and worker went down. Revert to
rpi5-PREFERRED (soft) so it schedules again; proper rpi5 placement needs the
exclusion list refreshed against current node health/capacity, tracked separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Un-pinning let the scheduler land worker on titan-12 (rpi4). The agent is heavy
enough that an rpi4 risks the /api/status slowness that trips its liveness probe
- the exact flap we are avoiding. Make hardware=rpi5 a hard requirement so it
runs only on rpi5 storage workers (excluding the saturated/known-flaky ones);
the scheduler places it on a roomy rpi5 (titan-05).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Worker (hermes-agent) was hard-pinned to titan-08 (a workaround after an earlier
attempt to place it on the amd64 titan-22 failed on architecture). That single-
node pin is exactly what makes it fragile: a titan-08 blip (as just happened when
the node's Longhorn CSI went down) strands worker, and the Recreate strategy then
deadlocks because the replacement can't schedule on the one tight node.
Restore the intended multi-node design: run on any arm64 storage worker except the
known-bad/weak ones (matching the repo's own affinity test, which was red). Its
Longhorn volumes have data-locality disabled with replicas on titan-15/17/19, so
there is no locality penalty to running on another node; the scheduler now places
it on a roomier Pi (e.g. titan-05) and a node blip simply reschedules it.
Also relax the gateway /api/status liveness probe (timeout 10s->15s,
failureThreshold 3->5) so a transient slowness (e.g. a brief storage hiccup) no
longer trips a kill-and-restart cascade.
Follow-up (separate): multi-arch agent image to enable the amd64 titan-22 target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
worker.bstein.dev (the hermes-agent Deployment) gains the same HUX
shape as chat, staged and inert: a foundation-only hux sidecar on the
reviewed WebUI image line (Flux setters bound, 5s probe budgets), an
init that provisions the HMAC identity as slot-100 on the durable home
subtree (create-once context key, O_EXCL subject binding, per-pod
worker key; no relay/router/evidence keys so those trusts fail closed),
and observe-only hook env in the agent container with the runtime
plugin mounted but deliberately NOT enabled - activation is a reviewed
one-line flip per docs/hux/WORKER-PLAN.md, which carries the rollout,
verification gates, canary/rollback ladder and open questions.
Cross-surface continuity remains unclaimed until the live gates pass.
7 new topology-adaptive delivery gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.
The rename is additive, matching the shape #38 restored for chat and triage.
CoreDNS, both agent Ingresses and the hermes-sites certificate now serve BOTH
names, so merging this cannot take away the endpoint the operator uses to
reach the coordinator. Retiring agent.hermes.bstein.dev is a separate,
separately scheduled change. No redirect middleware is added.
What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak agent proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default
What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
rollback only needs the oauth2-proxy --redirect-url reverted and does not
require re-running the ensure job
The agent client passes its legacy origin through the optional fourth argument
#38 added to ensure_proxy_client, so no second mechanism is introduced. The
immutable ensure Job goes -11 -> -12 because #38 already consumed -11 and that
run has completed; without a further bump this change would never be applied.
Login on the new host fails until the -12 Job completes.
Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.
#38's public-host continuity test now covers the agent proxy's dual origins
rather than asserting the agent surface was untouched by the rename.
Knowledge catalogs and diagrams regenerated with `make knowledge`.
Drive the cross-card implement->review->repair->re-review chain from inside
the pod so unattended runs no longer stall once the first implementation card
completes. Today that chain exists only as an external codex-shepherd session;
this adds a bounded in-pod poll loop that reads board state via
hermes_cli.kanban_db and creates Kanban follow-up cards (subscription lanes
only) with no provider/metered path of its own.
- kanban_supervisor.py (I/O shell) + supervisor_policy.py (pure state machine):
impl-done+PR -> review; review SHIP -> mark impl ready-for-human (never
merges/approves/clears WIP); review BLOCK -> bounded repair; repair new
commit -> re-review. Fail-closed on unparseable/ambiguous state; per-
(parent, head_commit) dedup safe beside the external shepherd; bounded review
<->repair cycle count and max concurrent chains.
- Deployed as a hardened non-root sidecar (drop ALL caps, read-only rootfs, no
runtime-access/credential mount) alongside model-steward; scripts packaged in
the coordinator configMapGenerator.
- Gated by new kanban.auto_supervise config key (default false, re-read each
tick like auto_decompose) so it is inert until the external shepherd retires.
- Fix latent goal_max_turns NULL fallback in cli_lane_execution (1 -> documented
default 20).
- 62 new behavioral tests at 100% line+branch on both modules.
Stacks on the merge train (base 5f27e50c).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add kanban routing keys (provider_quota_min_remaining_percent: 15,
provider_capacity_cooldown_seconds: 300, provider_auth_cooldown_seconds:
3600), a lane-metrics port/Service on 9011 with service-annotation
scraping, monitoring ingress for the new port, and the lane's quota
metrics URL env. Based on PR #15 (fix/hermes-result-decomposition-
reliability); stacked because this work rides on the decomposed lane.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sticky-block gate added in the previous commit classifies a task from
the `created` event payload that upstream `create_task` writes. That
producer is code we do not own, so trusting it silently was the gap: if
upstream renamed the key, dropped it, or stopped deriving it from
`initial_status`, the image would still build and ship a consumer that
mis-classifies every task it reads.
Anchor the producer contract at build time, before the regression suite
runs, with three assert-only preconditions: the `initial_status="blocked"`
park resolves `task_status` to `"blocked"`, every non-park creation
resolves it to something else, and the `created` event carries that same
variable under `"status"`. None of them rewrite the producer.
Textual anchors cannot see dataflow, so add the runtime net the reviewer
asked for. The suite now drives the real API: create + claim an ordinary
task, trip the circuit breaker once at failure_limit=1 so it parks with a
`gave_up` event (leaving its own `created` event as the most recent
create/block/unblock row), then recompute at failure_limit=2 and require
promotion to ready. That case is red under an unconditional-true created
predicate and red under producer drift that labels every created event
blocked, while the explicit block/unblock, dependency-promotion and
circuit-breaker-at-current-limit cases stay green. Non-blocked and
malformed created payloads are pinned as controls, and the gate now
rejects non-dict payloads rather than trusting `.get`.
Also make the live placement correction durable: titan-04 is cordoned
after repeated kernel undervoltage and kubelet failure and titan-19 was
probe/Longhorn unstable under worker load, so both join the hard NotIn
list; titan-05 is healthy but sits at 3592m/3600m requested CPU, so the
main hermes container gives back 50m (350m -> 300m) to schedule there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three narrowly scoped Hermes reliability fixes backed by live evidence
from the Cassandra/titan-iac proof run.
Worker concurrency. Three simultaneous direct CLI workers on the 4-core
hermes-agent node drove load to ~45 and made the hermes and oauth2-proxy
containers fail their probes, leaving the pod 8/10 Ready; two workers
stayed at 10/10. Cap HERMES_CLI_LANE_CONCURRENCY at 2 and lower the
cli-lane-runner CPU limit from 3 to 2 so the dashboard and auth sidecars
keep a guaranteed share of the node. Requests are unchanged: the pod
still asks for 745m total, so placement does not move.
Service links. Kubernetes injects a service-link variable pair for every
service in the namespace, and hermes-claude-broker produces
HERMES_CLAUDE_BROKER_PORT=tcp://10.43.31.76:9006 — a value the broker
parses as an int. That contaminated worker and test environments even
though the deployment already addresses every service by DNS name. Set
enableServiceLinks: false on the hermes-agent pod spec.
Blocked-task scheduling. create_task(initial_status="blocked") records a
created event carrying status=blocked but never a blocked event, while
_has_sticky_block() only inspects blocked/unblocked events. recompute_ready()
considers blocked tasks, so an explicitly parked task with no incomplete
parent auto-promoted on the next dispatcher cycle. Teach _has_sticky_block()
to also recognize a created event whose payload status is blocked, which
covers tasks created before this image patch without adding a persisted
field. Dependency-driven promotion and the circuit-breaker failure-limit
guard are untouched; unblock_task() still releases either kind of block.
hermes-kanban-blocked-regression.py runs against the real upstream
kanban_db API during the image build, so the build fails if any of these
semantics regress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
