atlas-iac/docs/hux/WORKER-PLAN.md

# Worker HUX integration plan (worker.bstein.dev)

Status: PREPARED, working tree only — reviewed and applied at a controlled
moment by the coordinator. The hermes-agent pod is the live automation brain;
nothing here is pushed, merged, or reconciled by the preparing agent.

## Which workload is the Worker

`worker.bstein.dev` → Ingress `hermes-agent-dashboard` → Service
`oauth2-proxy-hermes-agent` (selector `app: hermes-agent`) → the
`oauth2-proxy` container inside the **`hermes-agent` Deployment**
(`services/hermes/agent-deployment.yaml`), which proxies loopback
`127.0.0.1:9119` (dashboard) and `127.0.0.1:7681` (terminal). The
`hermes` Deployment in `services/hermes/deployment.yaml` is the DASHBOARD
(triage) workload and is untouched. All edits land in
`agent-deployment.yaml` + `agent-configmap.yaml` only.

## Design summary

Mirrors the reviewed chat topology (`chat-statefulset.yaml`), adapted to a
single-replica `Recreate` Deployment:

- **`hux` sidecar** on the SAME reviewed WebUI image line as chat
  (`registry.bstein.dev/bstein/hermes-webui`, Flux marker
  `hermes:hermes-webui-release`, plus `:tag`/`:digest` setters for
  `HUX_IMAGE_TAG`/`HUX_IMAGE_DIGEST`). `python -m hux.server`,
  `PYTHONPATH=/opt/hermes-hux`, binds `127.0.0.1:8790` only; port 8790
  appears in no Service, oauth2-proxy upstream, or NetworkPolicy. Probes are
  loopback `/healthz` exec probes with the corrected 5s timeout budget (the
  chat canary's 2s budget caused transient probe kills).
- **Flags**: `HUX_FLAGS=hux.foundation` only. `HUX_TOOL_ENFORCEMENT=0`
  (observe-only). Card widening and enforcement are separate, gated commits.
- **Identity**: new `init-hux-runtime` init container provisions, exactly as
  chat's (same HMAC scheme `usr_ + HMAC-SHA256(context-key,
  "hux.subject.id.v1\0<slot>")`), a persistent 32-byte context key (0600),
  an immutable `O_EXCL` subject binding (0440, verified on every restart), a
  redaction canary, and a **worker transport key** in a `Memory` emptyDir
  (0400, regenerated every pod replacement). Slot is pinned to **`slot-100`**
  (the service's `SLOT_RE` requires `slot-[0-9]{1,3}`; 100 is disjoint from
  the chat tenants' slot-0..3). Surface `worker`, trust `worker`.
  **No relay key, no router key, no evidence key**: with those env/file
  inputs absent, `hux/identity.py` fails those trusts closed — worker trust
  is the only live caller class on this instance.
- **Storage**: the store is a subtree of the existing durable
  `hermes-agent-home` PVC — `hux/` at the claim root, mounted into the
  sidecar as `/var/lib/hux` via fixed `subPath: hux` (NOT `$(POD_NAME)`:
  Deployment pod names churn, and identity must survive replacement). The
  `hermes` container receives only read-only views:
  `/run/hermes-hux-context` = `hux/context`, `/run/hermes-hux-subject` =
  `hux/binding`. Chat's `hermes-chat-hux-data` claim is not referenced.
- **Agent hook wiring** (`hermes` gateway container): `HUX_BASE_URL`,
  `HUX_RUNTIME_ENABLED=1`, `HUX_TOOL_ENFORCEMENT=0`,
  `HUX_TENANT_SLOT=slot-100`, `HUX_WORKER_KEY_FILE`, `HUX_SUBJECT_FILE`,
  `HUX_CONTEXT_KEY_FILE`, `HUX_TIMEOUT_SECONDS=3`, per
  `dockerfiles/hermes-worker-hux/hux_hook/NOTES.md`. No key or subject is
  ever a literal env value. The `hermes-hux-runtime-plugin` ConfigMap
  (already generated by `kustomization.yaml` for chat) is mounted read-only
  at `/opt/data/plugins/hux-runtime`, but `hux-runtime` is **deliberately
  absent from `plugins.enabled`** in `agent-configmap.yaml`, so the whole
  hook is inert until the activation commit flips that one line.

### Storage justification (home-PVC subtree vs dedicated PVC)

Chosen: `hermes-agent-home` subtree. Reasons:

1. Self-contained: no new PVC resource. `pvc.yaml` and `kustomization.yaml`
   stay untouched, and the pod can never be Pending on an unbound claim —
   this pod is the live automation brain and must always schedule.
2. Single replica + `Recreate` strategy = single writer on the RWO claim;
   chat needed RWX + `subPathExpr` because four pods share one claim. Here
   one pod owns everything.
3. Durability is identical to the rest of the coordinator state (Kanban DB,
   workspace) — one claim to back up, one failure domain.

Honest caveat, stated for review: the `hermes` container runs as root and
mounts the whole home claim at `/opt/data`, so the store subtree is
*technically readable* at `/opt/data/hux` by the agent process. On chat the
dedicated claim makes the store physically unreachable; here the boundary is
"the agent only *uses* the loopback API" plus 0700 modes (which do not bind
root). This is acceptable for a single-operator, single-subject instance
whose pod is already one trust domain — but if harder isolation is wanted,
the follow-up in "Coordinator TODOs" moves the store to a dedicated
`hermes-agent-hux-data` claim (a two-line volume swap plus data copy).

## Release rendering / Flux automation impact (coordinator, do not edit ci/)

- Flux `ImageUpdateAutomation` for hermes uses `update: {strategy: Setters,
  path: services/hermes}` — the new markers in `agent-deployment.yaml` are
  inside that path, so Flux keeps the sidecar image + tag/digest bindings
  current with zero renderer changes.
- `ci/scripts/hermes_webui_flux_release.py` rewrites only
  `StatefulSet/hermes-chat-tenant` (1–3 consumers) and `Deployment/hermes`
  (exactly 1); it never opens `agent-deployment.yaml`, so its
  `expected_images` contracts are unaffected. `test_hermes_image_automation.py`
  counts markers only in the chat and dashboard files — verified unaffected.
- OPTIONAL renderer extension (coordinator decision): add a third target
  `(agent_manifest, "Deployment", "hermes-agent", "hermes-agent-deployment.yaml",
  1)` plus HUX build-metadata binding for the agent file, so the release
  belt covers the Worker too. Requires matching updates to
  `test_hermes_webui_release.py` expectations. Not required for correctness
  (Flux setters are authoritative); listed for completeness.
- **Operational consequence**: once merged, every WebUI release digest bump
  rewrites `agent-deployment.yaml` and therefore **rolls the hermes-agent pod
  (Recreate ⇒ brief worker.bstein.dev outage, minutes)** on every WebUI
  release. Chat already rolls on those releases; the Worker joining that
  cadence must be an accepted trade. If not accepted, the alternative is
  pinning the sidecar image without markers and bumping it manually — that
  deviates from the "same reviewed image line" requirement and is NOT what
  is prepared here.

## Coordinator TODOs (files outside this prepare's ownership)

1. **None required for kustomize/dry-run correctness.** All referenced
   objects (`hermes-hux-runtime-plugin` ConfigMap, `hermes-agent-home` PVC)
   already exist in the build. `testing/quality_contract.json` already globs
   `testing/**/*.py`, so the new test needs no registration.
2. **At activation** (separate commit, after staged-state gates pass):
   - add `- hux-runtime` to `plugins.enabled` in `agent-configmap.yaml`;
   - update `testing/tests/test_hermes_agent_layout.py`
     (`test_agent_uses_one_native_kanban_control_plane` asserts
     `config["plugins"]["enabled"] == ["auto-router"]` — exact equality) to
     the new list. The prepared delivery test intentionally pins the staged
     state (`"hux-runtime" not in plugins.enabled`) and must be updated in
     the same activation commit.
3. **Optional hardening**: dedicated `hermes-agent-hux-data` PVC (RWO,
   astreae, 5Gi) in `pvc.yaml` + swap the sidecar's `home/subPath: hux`
   volume for it; copy `hux/` content across during a quiet window.
4. **Optional**: renderer third target (above).
5. Pre-existing note: `test_hermes_ai_usage_exporter.py` pins
   `ai.bstein.dev/config-rev == "20260823-dual-provider-quota-health"` while
   the manifest carries `"20260823-image-release-automation"` — mismatched
   before this work; the config-rev annotation was deliberately left
   untouched here.

## Rollout steps (exact)

1. Review this diff; run the full gate locally
   (`PYTHONPATH=. .venv/bin/pytest -q testing/tests/test_hermes_worker_hux_delivery.py`
   plus the agent/webui suites) and `kustomize build services/hermes`.
2. Merge to `main` in a **quiet window** (no cli-lane tasks in flight:
   check the Kanban board and `/api/status`; the Recreate roll kills live
   worker sessions — durable CLI lanes resume, native children do not).
3. Let Flux reconcile. The pod recreates once with the sidecar. Staged
   state: hux serving loopback foundation-only; hook completely inert
   (plugin not enabled).
4. Verify the staged gates (below). Leave the staged state soaking for at
   least one WebUI release cycle to observe the digest-roll behavior.
5. Activation commit (coordinator TODO 2) — the hook starts observing:
   events/spend posted from real worker turns, zero enforcement.
6. Card widening (activity_timeline first), each with its lifecycle gate and
   a delivery-test flag-pin update. `hux.autonomy` is BLOCKED on the
   human-surface question below. Enforcement stays 0 throughout this plan.

## Verification gates (all must pass before any wider claim)

Staged (after step 3):

- Pod `hermes-agent-*` all containers Ready, zero restart loops; public
  `https://worker.bstein.dev` 200 behind SSO; terminal reachable.
- Sidecar loopback health: from the hux container,
  `python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8790/healthz').read())"`
  → `status: ok`.
- Identity provisioning: `hux/binding/subject` exists (0440, `usr_` +
  64 hex), `hux/context/context-key` 32 bytes 0600; record
  `sha256(context-key)` and the subject value.
- Capabilities matrix on the loopback: no headers → 401; worker key + wrong
  slot → 401; worker key + `slot-100` + bound subject → 200 with ONLY
  HUX-11 foundation routes; any card route → 404 `flag_off`. Confirm relay
  trust 401s (no relay key exists).
- **Identity persistence across a pod replacement**: delete the pod (or ride
  the next WebUI release roll); after the new pod is Ready, re-hash
  context-key and re-read the subject — both MUST be identical. The
  worker-key MUST differ (rotation proof).
- Store isolation spot check: no Service/NetworkPolicy exposes 8790; store
  dirs 0700 uid 10000.

Activated (after step 5):

- Drive one real worker turn (dashboard chat or a Kanban dispatch) and read
  `GET /hux/v1/conversations/…/events` via worker trust on the loopback:
  the turn's events appear with `Idempotency-Key`-deduped rows and no raw
  arguments/output in any summary (redaction canary from
  `hux/context/redaction-canary` never appears in stored records).
- Gateway, terminal, cli-lane-runner all still Ready; a cli-lane task
  completes end-to-end (the hook must never block at enforcement 0).

Cross-surface continuity is NOT claimed until every gate above has passed on
the live Worker — and even then it is a separate, explicit follow-up (the
chat and worker instances have distinct subjects and stores by design; any
continuity feature is new work, not a side effect of this rollout).

## Canary / rollback

This is a single-replica Deployment: there is no partition canary. The
canary IS the staged inert state (steps 3–4) — sidecar live, hook off — held
for a soak period before activation. Roll it as a Flux-tracked change only.

Rollback ladder (fastest first):

1. **Activation rollback**: revert the activation commit (remove
   `hux-runtime` from `plugins.enabled`). One pod roll; sidecar keeps
   running; hook inert again.
2. **Full rollback**: revert the integration commit(s) touching
   `agent-deployment.yaml`/`agent-configmap.yaml`. Flux recreates the pod
   without sidecar/init/env. The `hux/` subtree on the home claim is
   retained data, not referenced by anything — harmless, removable later.
3. **Emergency (cluster is broken, Flux unusable)**: Brad applies
   `kubectl -n hermes rollout undo deployment/hermes-agent` — then
   reconciles Git to match, since the agent itself must never mutate
   workloads.

What to watch during any roll: `kubectl -n hermes get pods -w` until ALL
containers Ready (13 app containers once rendered: the 12 in
`agent-deployment.yaml` including `hux`, plus `execution-pool-coordinator`
from `execution-coordinator-patch.yaml`); `https://worker.bstein.dev/api/status` 200; Kanban
dispatch resumes (`cli-lane-runner` readiness); Vault agent-inject succeeded
(init order unchanged); zero CrashLoopBackOff on `hux`.

## Open questions

1. **Approvals need a human surface.** This instance has only worker trust;
   `hux.autonomy` decisions are rejected for worker callers by design (F1),
   so enabling autonomy/enforcement on the Worker requires a human-surface
   path first (candidates: a BFF on the worker dashboard asserting router
   trust like chat's, or driving decisions from the chat instance — which
   reopens cross-surface identity). Until decided, `hux.autonomy` and
   `HUX_TOOL_ENFORCEMENT=1` are out of scope.
2. **Terminal and cli-lane-runner containers** run their own Hermes
   processes but were intentionally NOT wired (env absent → plugin
   `HUX_RUNTIME_ENABLED` gate keeps them clean even after activation). If
   observation should cover TUI/lane turns too, mirror the `hermes`
   container's env+mounts onto them in a follow-up.
3. **`HUX_PROJECT_SOURCE`** is `profile:default` for parity with chat;
   a worker-specific source (e.g. `profile:worker`) would partition worker
   activity into its own HUX project — decide before activation, changing it
   later re-keys the derived project id.
4. Whether to take Coordinator TODO 3 (dedicated PVC) before or after
   activation.