Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.
The rename is additive, matching the shape #38 restored for chat and triage.
CoreDNS, both agent Ingresses and the hermes-sites certificate now serve BOTH
names, so merging this cannot take away the endpoint the operator uses to
reach the coordinator. Retiring agent.hermes.bstein.dev is a separate,
separately scheduled change. No redirect middleware is added.
What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak agent proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default
What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
rollback only needs the oauth2-proxy --redirect-url reverted and does not
require re-running the ensure job
The agent client passes its legacy origin through the optional fourth argument
#38 added to ensure_proxy_client, so no second mechanism is introduced. The
immutable ensure Job goes -11 -> -12 because #38 already consumed -11 and that
run has completed; without a further bump this change would never be applied.
Login on the new host fails until the -12 Job completes.
Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.
#38's public-host continuity test now covers the agent proxy's dual origins
rather than asserting the agent surface was untouched by the rename.
Knowledge catalogs and diagrams regenerated with `make knowledge`.
Drive the cross-card implement->review->repair->re-review chain from inside
the pod so unattended runs no longer stall once the first implementation card
completes. Today that chain exists only as an external codex-shepherd session;
this adds a bounded in-pod poll loop that reads board state via
hermes_cli.kanban_db and creates Kanban follow-up cards (subscription lanes
only) with no provider/metered path of its own.
- kanban_supervisor.py (I/O shell) + supervisor_policy.py (pure state machine):
impl-done+PR -> review; review SHIP -> mark impl ready-for-human (never
merges/approves/clears WIP); review BLOCK -> bounded repair; repair new
commit -> re-review. Fail-closed on unparseable/ambiguous state; per-
(parent, head_commit) dedup safe beside the external shepherd; bounded review
<->repair cycle count and max concurrent chains.
- Deployed as a hardened non-root sidecar (drop ALL caps, read-only rootfs, no
runtime-access/credential mount) alongside model-steward; scripts packaged in
the coordinator configMapGenerator.
- Gated by new kanban.auto_supervise config key (default false, re-read each
tick like auto_decompose) so it is inert until the external shepherd retires.
- Fix latent goal_max_turns NULL fallback in cli_lane_execution (1 -> documented
default 20).
- 62 new behavioral tests at 100% line+branch on both modules.
Stacks on the merge train (base 5f27e50c).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add kanban routing keys (provider_quota_min_remaining_percent: 15,
provider_capacity_cooldown_seconds: 300, provider_auth_cooldown_seconds:
3600), a lane-metrics port/Service on 9011 with service-annotation
scraping, monitoring ingress for the new port, and the lane's quota
metrics URL env. Based on PR #15 (fix/hermes-result-decomposition-
reliability); stacked because this work rides on the decomposed lane.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sticky-block gate added in the previous commit classifies a task from
the `created` event payload that upstream `create_task` writes. That
producer is code we do not own, so trusting it silently was the gap: if
upstream renamed the key, dropped it, or stopped deriving it from
`initial_status`, the image would still build and ship a consumer that
mis-classifies every task it reads.
Anchor the producer contract at build time, before the regression suite
runs, with three assert-only preconditions: the `initial_status="blocked"`
park resolves `task_status` to `"blocked"`, every non-park creation
resolves it to something else, and the `created` event carries that same
variable under `"status"`. None of them rewrite the producer.
Textual anchors cannot see dataflow, so add the runtime net the reviewer
asked for. The suite now drives the real API: create + claim an ordinary
task, trip the circuit breaker once at failure_limit=1 so it parks with a
`gave_up` event (leaving its own `created` event as the most recent
create/block/unblock row), then recompute at failure_limit=2 and require
promotion to ready. That case is red under an unconditional-true created
predicate and red under producer drift that labels every created event
blocked, while the explicit block/unblock, dependency-promotion and
circuit-breaker-at-current-limit cases stay green. Non-blocked and
malformed created payloads are pinned as controls, and the gate now
rejects non-dict payloads rather than trusting `.get`.
Also make the live placement correction durable: titan-04 is cordoned
after repeated kernel undervoltage and kubelet failure and titan-19 was
probe/Longhorn unstable under worker load, so both join the hard NotIn
list; titan-05 is healthy but sits at 3592m/3600m requested CPU, so the
main hermes container gives back 50m (350m -> 300m) to schedule there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three narrowly scoped Hermes reliability fixes backed by live evidence
from the Cassandra/titan-iac proof run.
Worker concurrency. Three simultaneous direct CLI workers on the 4-core
hermes-agent node drove load to ~45 and made the hermes and oauth2-proxy
containers fail their probes, leaving the pod 8/10 Ready; two workers
stayed at 10/10. Cap HERMES_CLI_LANE_CONCURRENCY at 2 and lower the
cli-lane-runner CPU limit from 3 to 2 so the dashboard and auth sidecars
keep a guaranteed share of the node. Requests are unchanged: the pod
still asks for 745m total, so placement does not move.
Service links. Kubernetes injects a service-link variable pair for every
service in the namespace, and hermes-claude-broker produces
HERMES_CLAUDE_BROKER_PORT=tcp://10.43.31.76:9006 — a value the broker
parses as an int. That contaminated worker and test environments even
though the deployment already addresses every service by DNS name. Set
enableServiceLinks: false on the hermes-agent pod spec.
Blocked-task scheduling. create_task(initial_status="blocked") records a
created event carrying status=blocked but never a blocked event, while
_has_sticky_block() only inspects blocked/unblocked events. recompute_ready()
considers blocked tasks, so an explicitly parked task with no incomplete
parent auto-promoted on the next dispatcher cycle. Teach _has_sticky_block()
to also recognize a created event whose payload status is blocked, which
covers tasks created before this image patch without adding a persisted
field. Dependency-driven promotion and the circuit-breaker failure-limit
guard are untouched; unblock_task() still releases either kind of block.
hermes-kanban-blocked-regression.py runs against the real upstream
kanban_db API during the image build, so the build fails if any of these
semantics regress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
