diff --git a/docs/hux/HANDOFF.md b/docs/hux/HANDOFF.md index 0d09cd6e..05e57d64 100644 --- a/docs/hux/HANDOFF.md +++ b/docs/hux/HANDOFF.md @@ -14,6 +14,11 @@ Run everything with the CI interpreter: ## Codex integration requirements common to every card +(Superseded 2026-08-24 where it conflicts with "Integrated topology" at the +end of this file: the manifests, plugin, router headers and Flux wiring are +now in-repo, the data root moved off the tenant PVC, and the Worker has no +HUX yet.) + - Package `dockerfiles/hermes-hux-foundation/hux/` runs as `python -m hux.server` inside each `hermes-chat-tenant` pod (and the Worker) on `127.0.0.1:8790`. Image, manifest, NetworkPolicy and Flux wiring are Codex's (SO-01, SO-02, @@ -100,7 +105,195 @@ Suite after repairs: 377 tests, 99% line / 99% branch over `hermes-hux-foundatio ## Open items (not blockers for Codex integration) -- SO-46 hash-chained audit, SO-48 single-writer lock, SO-53 rate limits. +- SO-46 hash-chained audit, SO-48 single-writer lock. (SO-53 rate limits + shipped since: `hux/http.py` `RateLimiter`, per-subject read/write + buckets, `429` + `Retry-After`, tuned by `HUX_READS_PER_MINUTE` / + `HUX_WRITES_PER_MINUTE` — see the integrated-topology section.) - Retention scheduler: `privacy.run_retention` is on-demand; Codex decides cron vs sidecar. - Search over `message_text`; artifact sharing (`shared_readonly`). - `testing/quality_contract.json` line-limit globs / managed modules for the two new package paths (Codex's file). + +## Integrated topology (2026-08-24, local chain, staged — not deployed) + +Everything in this section exists only on local `main`. `origin/main` is +still `5558c24f`; Flux reconciles origin, so none of this is running +anywhere. Chat is staged; the Worker is untouched. The chain since origin +(oldest first): `964103f0` merge of the hardened foundation, `438180a9` +tool runtime gated through HUX, `a1070449` vendored runtime hooks, +`53d7c2c5` backend families complete, `31c5eae5` chat router image +automation, `1eda927b` trusted HUX workspace in the WebUI, `47165274` +HUX-12 evidence trust isolation, `b66c762f` HUX card UI models, +`bd63b568` router strips inbound `X-Hux-*`, `9e99470f` one-or-two WebUI +consumers in release rendering, `0b56d04e` quality-contract gates, +`71c05cf9` staged sidecar activation topology, `659f70ee` WebUI +integration contract notes, `13359769` all-or-nothing topology gates. + +### Pod topology (`services/hermes/chat-statefulset.yaml`) + +Each `hermes-chat-tenant` pod runs a fourth container, `hux`, on the SAME +reviewed WebUI image as the `webui` container +(`registry.bstein.dev/bstein/hermes-webui`, Flux +`$imagepolicy hermes:hermes-webui-release`), started as +`python -m hux.server` with `PYTHONPATH=/opt/hermes-hux`. It binds +`127.0.0.1:8790` (`HUX_BIND`, `HUX_PORT`), runs as uid/gid 10000 with +`readOnlyRootFilesystem`, all capabilities dropped, tmpfs `/tmp`, and +`/healthz` probes. `HUX_TENANT_SLOT=slot-N` comes from the pod ordinal, +as for the agent. Port 8790 appears in no Service or NetworkPolicy; +loopback is the only path (delivery-gated). + +### Storage: shared RWX claim, kubelet-scoped subtrees + +The plan's `/opt/data/hux` on the tenant `home` PVC is gone. Storage is a +standalone `ReadWriteMany` PVC `hermes-chat-hux-data` (storageClass +`astreae`, 10Gi, `services/hermes/chat-pvcs.yaml`) shared by all tenant +pods, with per-pod isolation enforced by kubelet +`subPathExpr: $(POD_NAME)` mounts: + +- `hux` container: `/var/lib/hux` = its own `$(POD_NAME)` subtree; + `HUX_DATA_ROOT=/var/lib/hux/store`. +- `hermes` (agent) container: read-only `/run/hermes-hux-context` = + `$(POD_NAME)/context` and `/run/hermes-hux-subject` = + `$(POD_NAME)/binding` only. It has NO mount of `/var/lib/hux/store`; + records are reachable from the agent only through the loopback API. +- `webui` container: read-only `/run/hermes-hux-context` only. + +No container in any pod mounts another pod's subtree. + +### Identity provisioning (`init-hux-runtime`) + +A dedicated init container (root, capabilities dropped to +CHOWN/DAC_OVERRIDE/FOWNER) provisions, idempotently per pod: + +- 0700 roots `binding/`, `context/`, `store/`, owned 10000:10000; +- a persistent 32-byte context key `context/context-key` (0600), created + once and kept across restarts; +- an immutable subject binding `binding/subject` (0440, created `O_EXCL`): + `usr_` + HMAC-SHA256(context-key, `"hux.subject.id.v1\0slot-N"`). On + restart the file is verified (regular file, uid 10000, mode 0440, one + hardlink, expected content) or the pod refuses to start; +- a redaction canary `context/redaction-canary` (0400); +- relay and worker keys in `Memory`-medium emptyDirs + (`/run/hermes-webui-hux/relay-key`, `/run/hermes-hux-worker/worker-key`, + 0400) — regenerated on every pod restart, never touching the PVC. + +The subject is stable across restarts, derived from a per-pod random key, +and never derived from the Keycloak subject. + +### Env contracts + +`hux` sidecar: `HUX_FLAGS` lists all twelve cards (`hux.foundation, +hux.activity_timeline, hux.memory_control, hux.projects, hux.artifacts, +hux.autonomy, hux.friendly_modes, hux.multimodal, hux.research, +hux.onboarding, hux.privacy, hux.release_followthrough`); +`HUX_RELAY_KEY_FILE`, `HUX_WORKER_KEY_FILE`, `HUX_SUBJECT_BINDING_FILE`, +`HUX_CONTEXT_KEY_FILE`, `HUX_CANARY_FILE`; rate limits +`HUX_READS_PER_MINUTE=600`, `HUX_WRITES_PER_MINUTE=120`; and build +provenance `HUX_IMAGE_TAG` / `HUX_IMAGE_DIGEST` bound by Flux setters +(`$imagepolicy hermes:hermes-webui-release:tag` / `:digest` comments). + +Agent container: `HUX_BASE_URL=http://127.0.0.1:8790`, +`HUX_RUNTIME_ENABLED=1`, and `HUX_TOOL_ENFORCEMENT=0` — the first rollout +is observe-only until approval parking/resume is connected to the +upstream tool loop and proven live — plus `HUX_WORKER_KEY_FILE`, +`HUX_SUBJECT_FILE`, `HUX_CONTEXT_KEY_FILE`, `HUX_TIMEOUT_SECONDS=3`. + +### Agent runtime plugin + +`services/hermes/plugins/hux-runtime/` renders into the +`hermes-hux-runtime-plugin` ConfigMap (`services/hermes/kustomization.yaml`) +and mounts read-only into the agent at `/opt/data/plugins/hux-runtime`: +`runtime.py`, `tool_policy.py`, `context_ids.py`, `plugin.yaml` +(`on_session_end` hook + `tool_execution` middleware, fail-closed), and a +vendored `hux_hook/` package kept in parity with the reviewed worker hook +library (parity is delivery-gated). + +### Router boundary + +`services/hermes/router/main.go` deletes EVERY inbound header whose name +starts with `x-hux-` (case-insensitive) at the authenticated tenant +boundary — in the same Director that strips OAuth headers — before +asserting `X-Hermes-Tenant-Identity: slot-N`. Regression tests: +`services/hermes/router/main_test.go`. Defense in depth: the sidecar is +loopback-only anyway, and the in-pod WebUI BFF asserts its own `X-Hux-*` +from trusted context. + +### WebUI image (`dockerfiles/Dockerfile.hermes-webui`) + +The image ships the service and contracts at `/opt/hermes-hux` (`hux/` +package + `contracts/`, `HUX_CONTRACT_DIR`), the BFF proxy +(`api/hux_bff.py`) and trusted context attachment (`api/hux_context.py`, +`hux.webui_context.v1`), the bootstrap bridge +(`static/hux/bootstrap.js/.css`) and the wave A/B/C runtime bundles. A +build-time gate compiles the package and constructs the router with +`HUX_FLAGS=""`: images carry the code but no activation — flags live only +in manifests. + +### Release rendering (`ci/scripts/hermes_webui_flux_release.py`) + +The release script updates one-or-two WebUI image consumers in +`StatefulSet/hermes-chat-tenant` (the `webui` container, plus the `hux` +sidecar when staged) and exactly one in `Deployment/hermes`, and binds +`HUX_IMAGE_TAG`/`HUX_IMAGE_DIGEST` to the released tag and digest whenever +the sidecar fields are present — failing when the binding fields are +incomplete (exactly one of each, or none at all). + +### HUX-12 evidence trust (`hux/release_security.py`, `hux/releases.py`, `47165274`) + +Release follow-through has a dedicated fourth trust class, `evidence`: + +- File-only authentication: `HUX_RELEASE_EVIDENCE_KEY_FILE` must be a 0400 + regular file (symlinks refused) holding a key ≥ 32 chars, and + `HUX_RELEASE_EVIDENCE_POLICY_FILE` a strictly validated + `hux.release_evidence_policy.v1` allowlist (≤ 16 workloads, + credential-free HTTPS URLs, exact field sets, bounded evidence age). + Without both healthy the HUX-12 capability is off entirely — fail closed. +- Worker trust is read-only on releases (list/read in `WORKER_ROUTES`); it + can no longer create releases or transitions. +- Only router trust creates the single `reviewed` proposal per scoped + workload commit; only evidence trust performs transitions, and its route + allowlist in `hux/http.py` blocks everything else. +- Transition evidence binds exactly: `merged` the 40-hex merge commit; + `built` the CI URL plus image ref / image digest / Harbor digest all + equal; `deployed` a `main@sha1:<40-hex>` Flux revision; `converged` + pod digest == image digest; `live_verified` one passing health receipt; + `rolled_back` an exact target digest. +- Ledger entries are hash-chained (`previous_hash`/`entry_hash`) and + verified on every read; `Idempotency-Key` replays return the recorded + view with `HUX-Replayed: true`. + +### Worker state + +`worker.bstein.dev` (the `hermes` Deployment and the execution-worker +manifests) has NO HUX sidecar, plugin or environment. Cross-surface +continuity is not claimed; HUX is chat-staged only. + +### Test state + +Verified 2026-08-24 with the CI interpreter: +`python -m pytest testing/tests/test_hermes_hux_*.py` = 646 passed +(20.8 s). Delivery gates in `testing/tests/test_hermes_hux_delivery.py` +assert the staged topology against the manifests themselves: +all-or-nothing activation, loopback-only sidecar on the reviewed WebUI +image, least-privilege storage and key mounts, init-container identity +persistence + transport-key rotation, file-backed authentication inputs, +and plugin/vendored-hook render parity. UI card model suites: +`testing/tests/test_hermes_hux_ui_*.py`, each driving a Node harness +(`test_hermes_hux_ui_*.mjs`). Release rendering: +`test_hermes_webui_release.py` (18) and `test_hermes_image_automation.py` +(2). + +### Superseded statements + +- "`/opt/data/hux` on the tenant PVC; SO-45 separate uid, mode 0700" — + superseded. Storage is the shared RWX claim with kubelet `subPathExpr` + isolation above; every container runs uid 10000 and the boundary is + mount scoping, not a uid split. See `THREAT-MODEL.md`, "Integration + amendments (2026-08-24)". +- "`HUX_DATA_ROOT` (tenant PVC, default `/opt/data`)" — now + `/var/lib/hux/store` on the dedicated claim. +- "runs … inside each `hermes-chat-tenant` pod (and the Worker)" — the + Worker has no HUX yet. +- "Image, manifest, NetworkPolicy and Flux wiring are Codex's" — landed + in-repo with the chain above. +- Open item "SO-53 rate limits" — implemented in `hux/http.py`; SO-46 and + SO-48 remain open (see the threat model amendment). diff --git a/docs/hux/THREAT-MODEL.md b/docs/hux/THREAT-MODEL.md index 431696a3..0c7f2e87 100644 --- a/docs/hux/THREAT-MODEL.md +++ b/docs/hux/THREAT-MODEL.md @@ -267,3 +267,112 @@ above as a result: open items, not claimed controls. - On-disk documents carry a `revision` field the 1.0.0 fixtures did not have; served bodies strip it where the record schema forbids it (F11). + +## Integration amendments (2026-08-24) + +The staged integration (`services/hermes/chat-statefulset.yaml`, +`chat-pvcs.yaml`, local chain up to `13359769`; staged only — nothing is +deployed, `origin/main` is still `5558c24f`) changes the following. Where +this section conflicts with the body above, this section is current. + +### SO-45 superseded: the boundary is mount scoping, not a uid split + +The service does not run as a distinct uid and there is no `/opt/data/hux`. +Records live on a standalone ReadWriteMany PVC `hermes-chat-hux-data` +shared by all tenant pods; each pod receives only its own subtree through +kubelet `subPathExpr: $(POD_NAME)` mounts, and an init container sets each +subtree root to 0700 owned 10000:10000. All pod containers run uid 10000. +The isolation boundary now is, exactly: + +1. kubelet subPath scoping — no container in pod N has any mount of pod + M's subtree, so another tenant's records are simply absent from the + mount namespace; +2. the agent container mounts none of the store: it gets read-only + `$(POD_NAME)/context` and `$(POD_NAME)/binding` only and reaches + records solely through the loopback API — SO-45's threat ("the agent + edits the audit log; it has write on the PVC") has no filesystem path; +3. the `hux` sidecar keeps `readOnlyRootFilesystem`, drop-ALL and the + `127.0.0.1:8790` bind (SO-01), with port 8790 absent from every Service + and NetworkPolicy (SO-02 holds unchanged). + +Not defended: an attacker who can mount the whole claim (node or cluster +compromise) sees every tenant's records — the same trust in kubelet the +per-tenant `home` PVC model already rested on. Within a pod, same-uid +means an agent-container escape that can rearrange mounts is equivalent to +kubelet compromise; accepted for this increment and recorded here rather +than claimed away. + +### SO-01..SO-04 status + +- SO-01 satisfied: `HUX_BIND=127.0.0.1`, `HUX_PORT=8790`, asserted by the + delivery gate (`test_hermes_hux_delivery.py`) against the manifest. +- SO-02 satisfied: no hux port in any Service or NetworkPolicy. +- SO-03/SO-04: the router (`services/hermes/router/main.go`) deletes every + inbound header matching prefix `x-hux-` (case-insensitive) at the + authenticated boundary before asserting `X-Hermes-Tenant-Identity`; + regression-tested in `main_test.go`. The router itself sets no + `X-Hux-*`: the in-pod WebUI BFF asserts them from trusted context, and + the subject is pinned by the pod-local binding file below. + +### Persistent subject and context identity + +The `init-hux-runtime` init container (root; capabilities +CHOWN/DAC_OVERRIDE/FOWNER only) provisions per pod a persistent random +32-byte key `context/context-key` (0600) and an immutable binding +`binding/subject` (0440, created `O_EXCL`, verified byte-for-byte, mode, +owner and link count on every restart — a mismatch fails the pod): + + usr_ + +Properties: stable across restarts (the key persists on the claim); +derived from a per-pod key, so subjects are unlinkable across tenants; and +never derived from the Keycloak subject, so no OAuth identifier can be +recovered from a stored `usr_` hash (strictly stronger than ADR-0001's +`identityHash("keycloak", subject)` plan). A redaction canary +`context/redaction-canary` (0400) feeds `HUX_CANARY_FILE`. + +### Relay and worker key lifecycle + +`HUX_RELAY_KEY_FILE` and `HUX_WORKER_KEY_FILE` point into `Memory`-medium +emptyDirs written by the init container: 0400, regenerated on every pod +restart, never stored on the PVC. A leaked relay or worker key therefore +expires at the next restart. SO-07's canary obligation extends to these +files. + +### Evidence trust class (HUX-12) + +A fourth trust class `evidence` (`hux/release_security.py`) with these +obligations: + +- file-only key: `HUX_RELEASE_EVIDENCE_KEY_FILE` must be a 0400 regular + file, symlinks refused, key ≥ 32 chars; inline environment secrets are + ignored; +- strict policy allowlist: `HUX_RELEASE_EVIDENCE_POLICY_FILE` + (`hux.release_evidence_policy.v1`), exact field sets, ≤ 16 workloads, + credential-free HTTPS URLs, bounded evidence age; +- fail-closed capability: without both files healthy the HUX-12 capability + is off (`flags.enabled`), not degraded; +- separation: router trust alone creates `reviewed` proposals; evidence + trust alone transitions and may call nothing else (route allowlist in + `hux/http.py`, `api` surface required); SO-08 narrows again — worker + trust is read-only on releases (list/read only). + +### SO-46 / SO-48 / SO-53 actual status (verified in code, 2026-08-24) + +- SO-53 rate limits: IMPLEMENTED. `hux/http.py` `RateLimiter` keeps + per-subject one-minute read and write buckets (defaults 300/30, + overridden by `HUX_READS_PER_MINUTE`/`HUX_WRITES_PER_MINUTE`; the staged + manifest sets 600/120), returns `429 rate_limited` with `Retry-After`, + caps the bucket table at 4096, and runs immediately after identity + resolution, before route matching and flag checks. +- SO-46 hash-chained audit: STILL OPEN for the general audit ledger. + `hux/audit.py` appends plain JSONL outcome rows with no `prev` hash. + The HUX-12 release ledger is hash-chained end to end + (`previous_hash`/`entry_hash`, verified on every read in + `hux/releases.py`) — that covers release honesty, not the audit trail. +- SO-48 single-writer lock: STILL OPEN. No `flock` anywhere in the + package; `hux/store.py` uses in-process `threading.RLock` per family + only. The per-pod `$(POD_NAME)` subtree makes a second writer against + the same tree unlikely (it would need a second pod with the same name), + but the obligation as written — fail at startup on `v1/.lock` — is not + implemented.