atlas-iac/docs/hux/WORKER-PLAN.md

Worker HUX integration plan (worker.bstein.dev)

Status: PREPARED, working tree only — reviewed and applied at a controlled moment by the coordinator. The hermes-agent pod is the live automation brain; nothing here is pushed, merged, or reconciled by the preparing agent.

Which workload is the Worker

worker.bstein.dev → Ingress hermes-agent-dashboard → Service oauth2-proxy-hermes-agent (selector app: hermes-agent) → the oauth2-proxy container inside the hermes-agent Deployment (services/hermes/agent-deployment.yaml), which proxies loopback 127.0.0.1:9119 (dashboard) and 127.0.0.1:7681 (terminal). The hermes Deployment in services/hermes/deployment.yaml is the DASHBOARD (triage) workload and is untouched. All edits land in agent-deployment.yaml + agent-configmap.yaml only.

Design summary

Mirrors the reviewed chat topology (chat-statefulset.yaml), adapted to a single-replica Recreate Deployment:

• hux sidecar on the SAME reviewed WebUI image line as chat (registry.bstein.dev/bstein/hermes-webui, Flux marker hermes:hermes-webui-release, plus :tag/:digest setters for HUX_IMAGE_TAG/HUX_IMAGE_DIGEST). python -m hux.server, PYTHONPATH=/opt/hermes-hux, binds 127.0.0.1:8790 only; port 8790 appears in no Service, oauth2-proxy upstream, or NetworkPolicy. Probes are loopback /healthz exec probes with the corrected 5s timeout budget (the chat canary's 2s budget caused transient probe kills).
• Flags: HUX_FLAGS=hux.foundation only. HUX_TOOL_ENFORCEMENT=0 (observe-only). Card widening and enforcement are separate, gated commits.
• Identity: new init-hux-runtime init container provisions, exactly as chat's (same HMAC scheme usr_ + HMAC-SHA256(context-key, "hux.subject.id.v1\0<slot>")), a persistent 32-byte context key (0600), an immutable O_EXCL subject binding (0440, verified on every restart), a redaction canary, and a worker transport key in a Memory emptyDir (0400, regenerated every pod replacement). Slot is pinned to slot-100 (the service's SLOT_RE requires slot-[0-9]{1,3}; 100 is disjoint from the chat tenants' slot-0..3). Surface worker, trust worker. No relay key, no router key, no evidence key: with those env/file inputs absent, hux/identity.py fails those trusts closed — worker trust is the only live caller class on this instance.
• Storage: the store is a subtree of the existing durable hermes-agent-home PVC — hux/ at the claim root, mounted into the sidecar as /var/lib/hux via fixed subPath: hux (NOT $(POD_NAME): Deployment pod names churn, and identity must survive replacement). The hermes container receives only read-only views: /run/hermes-hux-context = hux/context, /run/hermes-hux-subject = hux/binding. Chat's hermes-chat-hux-data claim is not referenced.
• Agent hook wiring (hermes gateway container): HUX_BASE_URL, HUX_RUNTIME_ENABLED=1, HUX_TOOL_ENFORCEMENT=0, HUX_TENANT_SLOT=slot-100, HUX_WORKER_KEY_FILE, HUX_SUBJECT_FILE, HUX_CONTEXT_KEY_FILE, HUX_TIMEOUT_SECONDS=3, per dockerfiles/hermes-worker-hux/hux_hook/NOTES.md. No key or subject is ever a literal env value. The hermes-hux-runtime-plugin ConfigMap (already generated by kustomization.yaml for chat) is mounted read-only at /opt/data/plugins/hux-runtime, but hux-runtime is deliberately absent from plugins.enabled in agent-configmap.yaml, so the whole hook is inert until the activation commit flips that one line.

Storage justification (home-PVC subtree vs dedicated PVC)

Chosen: hermes-agent-home subtree. Reasons:

1. Self-contained: no new PVC resource. pvc.yaml and kustomization.yaml stay untouched, and the pod can never be Pending on an unbound claim — this pod is the live automation brain and must always schedule.
2. Single replica + Recreate strategy = single writer on the RWO claim; chat needed RWX + subPathExpr because four pods share one claim. Here one pod owns everything.
3. Durability is identical to the rest of the coordinator state (Kanban DB, workspace) — one claim to back up, one failure domain.

Honest caveat, stated for review: the hermes container runs as root and mounts the whole home claim at /opt/data, so the store subtree is technically readable at /opt/data/hux by the agent process. On chat the dedicated claim makes the store physically unreachable; here the boundary is "the agent only uses the loopback API" plus 0700 modes (which do not bind root). This is acceptable for a single-operator, single-subject instance whose pod is already one trust domain — but if harder isolation is wanted, the follow-up in "Coordinator TODOs" moves the store to a dedicated hermes-agent-hux-data claim (a two-line volume swap plus data copy).

Release rendering / Flux automation impact (coordinator, do not edit ci/)

• Flux ImageUpdateAutomation for hermes uses update: {strategy: Setters, path: services/hermes} — the new markers in agent-deployment.yaml are inside that path, so Flux keeps the sidecar image + tag/digest bindings current with zero renderer changes.
• ci/scripts/hermes_webui_flux_release.py rewrites only StatefulSet/hermes-chat-tenant (1–3 consumers) and Deployment/hermes (exactly 1); it never opens agent-deployment.yaml, so its expected_images contracts are unaffected. test_hermes_image_automation.py counts markers only in the chat and dashboard files — verified unaffected.
• OPTIONAL renderer extension (coordinator decision): add a third target (agent_manifest, "Deployment", "hermes-agent", "hermes-agent-deployment.yaml", 1) plus HUX build-metadata binding for the agent file, so the release belt covers the Worker too. Requires matching updates to test_hermes_webui_release.py expectations. Not required for correctness (Flux setters are authoritative); listed for completeness.
• Operational consequence: once merged, every WebUI release digest bump rewrites agent-deployment.yaml and therefore rolls the hermes-agent pod (Recreate ⇒ brief worker.bstein.dev outage, minutes) on every WebUI release. Chat already rolls on those releases; the Worker joining that cadence must be an accepted trade. If not accepted, the alternative is pinning the sidecar image without markers and bumping it manually — that deviates from the "same reviewed image line" requirement and is NOT what is prepared here.

Coordinator TODOs (files outside this prepare's ownership)

1. None required for kustomize/dry-run correctness. All referenced objects (hermes-hux-runtime-plugin ConfigMap, hermes-agent-home PVC) already exist in the build. testing/quality_contract.json already globs testing/**/*.py, so the new test needs no registration.
2. At activation (separate commit, after staged-state gates pass):
   • add - hux-runtime to plugins.enabled in agent-configmap.yaml;
   • update testing/tests/test_hermes_agent_layout.py (test_agent_uses_one_native_kanban_control_plane asserts config["plugins"]["enabled"] == ["auto-router"] — exact equality) to the new list. The prepared delivery test intentionally pins the staged state ("hux-runtime" not in plugins.enabled) and must be updated in the same activation commit.
3. Optional hardening: dedicated hermes-agent-hux-data PVC (RWO, astreae, 5Gi) in pvc.yaml + swap the sidecar's home/subPath: hux volume for it; copy hux/ content across during a quiet window.
4. Optional: renderer third target (above).
5. Pre-existing note: test_hermes_ai_usage_exporter.py pins ai.bstein.dev/config-rev == "20260823-dual-provider-quota-health" while the manifest carries "20260823-image-release-automation" — mismatched before this work; the config-rev annotation was deliberately left untouched here.

Rollout steps (exact)

1. Review this diff; run the full gate locally (PYTHONPATH=. .venv/bin/pytest -q testing/tests/test_hermes_worker_hux_delivery.py plus the agent/webui suites) and kustomize build services/hermes.
2. Merge to main in a quiet window (no cli-lane tasks in flight: check the Kanban board and /api/status; the Recreate roll kills live worker sessions — durable CLI lanes resume, native children do not).
3. Let Flux reconcile. The pod recreates once with the sidecar. Staged state: hux serving loopback foundation-only; hook completely inert (plugin not enabled).
4. Verify the staged gates (below). Leave the staged state soaking for at least one WebUI release cycle to observe the digest-roll behavior.
5. Activation commit (coordinator TODO 2) — the hook starts observing: events/spend posted from real worker turns, zero enforcement.
6. Card widening (activity_timeline first), each with its lifecycle gate and a delivery-test flag-pin update. hux.autonomy is BLOCKED on the human-surface question below. Enforcement stays 0 throughout this plan.

Verification gates (all must pass before any wider claim)

Staged (after step 3):

• Pod hermes-agent-* all containers Ready, zero restart loops; public https://worker.bstein.dev 200 behind SSO; terminal reachable.
• Sidecar loopback health: from the hux container, python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8790/healthz').read())" → status: ok.
• Identity provisioning: hux/binding/subject exists (0440, usr_ + 64 hex), hux/context/context-key 32 bytes 0600; record sha256(context-key) and the subject value.
• Capabilities matrix on the loopback: no headers → 401; worker key + wrong slot → 401; worker key + slot-100 + bound subject → 200 with ONLY HUX-11 foundation routes; any card route → 404 flag_off. Confirm relay trust 401s (no relay key exists).
• Identity persistence across a pod replacement: delete the pod (or ride the next WebUI release roll); after the new pod is Ready, re-hash context-key and re-read the subject — both MUST be identical. The worker-key MUST differ (rotation proof).
• Store isolation spot check: no Service/NetworkPolicy exposes 8790; store dirs 0700 uid 10000.

Activated (after step 5):

• Drive one real worker turn (dashboard chat or a Kanban dispatch) and read GET /hux/v1/conversations/…/events via worker trust on the loopback: the turn's events appear with Idempotency-Key-deduped rows and no raw arguments/output in any summary (redaction canary from hux/context/redaction-canary never appears in stored records).
• Gateway, terminal, cli-lane-runner all still Ready; a cli-lane task completes end-to-end (the hook must never block at enforcement 0).

Cross-surface continuity is NOT claimed until every gate above has passed on the live Worker — and even then it is a separate, explicit follow-up (the chat and worker instances have distinct subjects and stores by design; any continuity feature is new work, not a side effect of this rollout).

Canary / rollback

This is a single-replica Deployment: there is no partition canary. The canary IS the staged inert state (steps 3–4) — sidecar live, hook off — held for a soak period before activation. Roll it as a Flux-tracked change only.

Rollback ladder (fastest first):

1. Activation rollback: revert the activation commit (remove hux-runtime from plugins.enabled). One pod roll; sidecar keeps running; hook inert again.
2. Full rollback: revert the integration commit(s) touching agent-deployment.yaml/agent-configmap.yaml. Flux recreates the pod without sidecar/init/env. The hux/ subtree on the home claim is retained data, not referenced by anything — harmless, removable later.
3. Emergency (cluster is broken, Flux unusable): Brad applies kubectl -n hermes rollout undo deployment/hermes-agent — then reconciles Git to match, since the agent itself must never mutate workloads.

What to watch during any roll: kubectl -n hermes get pods -w until ALL containers Ready (13 app containers once rendered: the 12 in agent-deployment.yaml including hux, plus execution-pool-coordinator from execution-coordinator-patch.yaml); https://worker.bstein.dev/api/status 200; Kanban dispatch resumes (cli-lane-runner readiness); Vault agent-inject succeeded (init order unchanged); zero CrashLoopBackOff on hux.

Open questions

1. Approvals need a human surface. This instance has only worker trust; hux.autonomy decisions are rejected for worker callers by design (F1), so enabling autonomy/enforcement on the Worker requires a human-surface path first (candidates: a BFF on the worker dashboard asserting router trust like chat's, or driving decisions from the chat instance — which reopens cross-surface identity). Until decided, hux.autonomy and HUX_TOOL_ENFORCEMENT=1 are out of scope.
2. Terminal and cli-lane-runner containers run their own Hermes processes but were intentionally NOT wired (env absent → plugin HUX_RUNTIME_ENABLED gate keeps them clean even after activation). If observation should cover TUI/lane turns too, mirror the hermes container's env+mounts onto them in a follow-up.
3. HUX_PROJECT_SOURCE is profile:default for parity with chat; a worker-specific source (e.g. profile:worker) would partition worker activity into its own HUX project — decide before activation, changing it later re-keys the derived project id.
4. Whether to take Coordinator TODO 3 (dedicated PVC) before or after activation.