atlas-iac/docs/hux/HANDOFF.md

hermes-next: Claude backend handoff ledger

Branch feature/hermes-next-hux (origin, pushed after every card); draft PR #55 atlas/titan-iac#55 (opened through the SCM broker, never merged by Claude). Board hermes-next on the operator Hermes Kanban; cards HUX-01..HUX-12 are parked (blocked) on purpose so the CLI lane runner never auto-claims them — the Claude coordinator and the Codex root session are the only implementers. Contract: hux.v1 1.0.0 (services/hermes/contracts/hux/VERSION, docs/hux/ADR-0001-hux-v1-contract-freeze.md). Threat model: docs/hux/THREAT-MODEL.md (SO-01..SO-54). Data model: docs/hux/DATA-MODEL.md.

Run everything with the CI interpreter: python -m pytest testing/tests/test_hermes_hux_*.py -q and python -m coverage run --branch --source=dockerfiles/hermes-hux-foundation -m pytest testing/tests/test_hermes_hux_*.py.

Codex integration requirements common to every card

(Superseded 2026-08-24 where it conflicts with "Integrated topology" at the end of this file: the manifests, plugin, router headers and Flux wiring are now in-repo, the data root moved off the tenant PVC, and the Worker has no HUX yet.)

• Package dockerfiles/hermes-hux-foundation/hux/ runs as python -m hux.server inside each hermes-chat-tenant pod (and the Worker) on 127.0.0.1:8790. Image, manifest, NetworkPolicy and Flux wiring are Codex's (SO-01, SO-02, SO-45: separate uid, /opt/data/hux mode 0700).
• Env: HUX_DATA_ROOT (tenant PVC, default /opt/data), HUX_TENANT_SLOT (pin, SO-03), HUX_FLAGS (comma list, default empty = everything off), HUX_RELAY_KEY (Telegram relay), HUX_WORKER_KEY (Worker), HUX_BUILD_COMMIT, HUX_IMAGE_DIGEST, HUX_CONTRACT_DIR (schemas path in the image; default resolves relative to the repo layout).
• Router → service headers (SO-04): X-Hermes-Tenant-Identity: slot-N, X-Hux-Subject: usr_<hash>, X-Hux-Surface: chat|telegram|voice|api|worker, optional X-Hux-Trust: relay|worker + X-Hux-Relay-Key. The router must strip any inbound X-Hux-* before setting its own.
• First client call: GET /hux/v1/capabilities → which cards/routes are on. Disabled or unknown → 404 (flag_off vs not_found only in the body).
• Mutations: If-Match: <revision> on revisioned records (409 on mismatch); Idempotency-Key on creates. Errors are hux.error.v1.
• Fixtures Codex codes against: services/hermes/contracts/hux/examples/*.json (validated in CI by test_hermes_hux_contract_schemas.py).

HUX-11 shared foundation — DONE (backend)

Commits	b4145861 (contract freeze 1.0.0), 6a5e0d87 (service core)
Files	services/hermes/contracts/hux/{common,identity}.schema.json + examples, VERSION; dockerfiles/hermes-hux-foundation/hux/{__init__,errors,identity,flags,store,audit,http,foundation,server,contracts,rules}.py; docs/hux/ADR-0001-*.md, THREAT-MODEL.md, DATA-MODEL.md
Tests	testing/tests/test_hermes_hux_contract_schemas.py (61), test_hermes_hux_contract_foundation.py (30): identity header rejection matrix, relay/worker keys, slot pinning, fail-closed flag chains, capabilities record validity, tenant-scoped paths + traversal rejection, revisions/conflicts, torn-ledger recovery, blobs, manifest, concurrent writers, audit rows, flag-off == not-found, 401 never touches storage, real HTTP server JSON + SSE, loopback-only bind, all errors validate, ≤500 LOC guard
Coverage	99% line / 98% branch over the package
Contract	1.0.0
Flag	hux.foundation (root of every dependency chain)
Risks	Rate limiting (SO-53) and hash-chained audit (SO-46) not yet implemented — tracked for the Wave A review; relay callers still carry X-Hux-Subject (deviates from SO-06; the slot→owner mapping lives in the router, so the header is redundant but harmless and lets the service pin the subject)
Codex needs	Wire the service into the tenant pod + Worker, NetworkPolicy, router header contract above, HUX_FLAGS=hux.foundation for the first canary; testing/quality_contract.json may add dockerfiles/hermes-hux-foundation/**/*.py to line_limit_globs and the two moved modules to managed_modules (Claude does not edit that file)

Wave A backends — DONE, awaiting adversarial review and Codex integration

Full suite: testing/tests/test_hermes_hux_*.py = 325 tests, 99% line / 99% branch over dockerfiles/hermes-hux-foundation/hux/ (every family module 99–100%). Every module ≤ 500 LOC (guarded by tests). Contract revised additively to 1.1.0 (ADR-0001).

Card	Commit	Files	Tests	Flag	Codex needs
HUX-01 activity timeline	b8fb72fb	hux/events.py, hux/redaction.py	test_hermes_hux_contract_events.py (34): ordering, idempotency, replay/reconnect, redaction by surface, cancellation receipts, cross-tenant	hux.activity_timeline	Router forwards Last-Event-ID; telegram/voice surfaces get partial redaction; agent hook posts events with Idempotency-Key; HUX_CANARY_FILE=/opt/data/.env so secrets are scrubbed
HUX-02 memory center	b8fb72fb	hux/memory.py	test_hermes_hux_memory_ledger.py (20), test_hermes_hux_memory_retrieval.py (7): no-store, suggest-only, edit/supersede, forget, retrieval removal, export, If-Match	hux.memory_control (needs hux.privacy)	Agent memory tool must call hux.memory.retrieve semantics (tombstones first) instead of the upstream memory file; UI per-message controls call /memory/{id}/{action} with If-Match
HUX-10 privacy	b8fb72fb	hux/privacy.py	test_hermes_hux_privacy_topics.py (19), test_hermes_hux_privacy_retention.py (6)	hux.privacy	Schedule run_retention daily (no thread inside the service by design); surface HUX-Audit-Stale; show notices and post chosen
HUX-04 artifacts	9f0ffe03 (+ upload caps, project check in the 1.1.0 batch)	hux/artifacts.py, hux/diffs.py	test_hermes_hux_artifact_versions.py, test_hermes_hux_artifact_auth.py (immutable versions, concurrency, lineage forgery 404, caps 413, diffs, promotion)	hux.artifacts (needs hux.projects)	Version content served with nosniff + attachment; uploads ≤ 25 MiB content via base64; sharing intentionally not implemented (access.mode always owner)
HUX-08 research	9f0ffe03	hux/research.py	test_hermes_hux_research_citations.py, test_hermes_hux_research_notebook.py (dedupe, integrity, notebook state machine)	hux.research	Service never fetches URIs (SO-19): the web tool records sources/passages after its own fetch; citation strip = GET /messages/{id}/citations (passages + sources embedded)
HUX-05 autonomy	3ebee2cc	hux/policy.py, hux/budgets.py	test_hermes_hux_policy_matrix.py (22), _approvals.py (22), _receipts.py (13)	hux.autonomy (needs hux.activity_timeline)	Agent hook: POST /approvals before any external side effect, POST /runs/{id}/gate immediately before executing with the canonical argument hash, POST /runs/{id}/budget per turn, POST /runs/{id}/stop with process_registry_empty from the real process registry; decisions only from human surfaces (worker trust → 403)
HUX-03 organization API	3ebee2cc	hux/organization.py	test_hermes_hux_contract_organization.py (14)	hux.projects	Codex owns the UI + migration of upstream WebUI projects; search over message_text is a later increment

Known gaps carried to the Wave A review: rate limiting (SO-53), hash-chained audit (SO-46), retention scheduler ownership (Codex cron vs service thread), approval expiry applied lazily on read, linear scans for idempotency/dedupe (fine at documented caps).

Wave A adversarial review — CLOSED

A fresh reviewer attacked HEAD 1b1a14e9 (tenant isolation, privacy, autonomy, rollback) and reported 13 findings; the repro harness lives outside the repo. Every finding is closed by a regression test that cites it:

#	Severity	Finding	Fix commit
F1	critical	worker/api trust could PUT /policy and self-approve	6964a9d8 — policy writes and allow grants are human-surface only
F2	high	worker trust could read every tenant record (SO-08)	6964a9d8 — flags.WORKER_ROUTES allowlist enforced in Router.dispatch; unexpected exceptions become audited 500 records
F3	high	memory edit skipped topic/sensitivity/private-mode gates	681b0408 — one _classify path for proposals and edits
F4	high	a session approval for an external effect released unrelated later effects	6964a9d8 — external effects match only the same run + argument hash; gate conversation from the run, not the body
F5	high	seq duplicated after a crash between append and checkpoint	681b0408 — seq = max(checkpoint, ledger tail + 1) under the lock
F6	medium	gate ignored budget exhaustion	6964a9d8
F7	medium	stale unconditional write could resurrect a forgotten memory	681b0408 — re-read under lock, always write with the loaded revision
F8	medium	any caller could assert process_registry_empty; failed receipts were sticky	6964a9d8 — gateway (worker trust) only; failed receipts supersedable
F9	medium	secrets stored verbatim in titles/passages/claims/notebooks; forget left the title	681b0408 — scrub applied; forget blanks the document. Artifact bodies stay verbatim (user-owned) but are forced restricted and audited (a75ca299)
F10	medium	SO-46/48/53 claimed but absent	threat model amended (74c65499): tracked as open, not claimed
F11	low	stored receipts carry revision the schema forbade	6964a9d8 — optional revision on the receipt
F12	low	capabilities advertised routes without handlers	6964a9d8 — HUX-06/09/12 routes empty until shipped
F13	low	misc (healthz version, ghost conversations from notices, idempotency outside the lock, artifact titles not searchable, unnormalised paths, multi-hash once approvals)	6964a9d8 + 681b0408

Suite after repairs: 377 tests, 99% line / 99% branch over hermes-hux-foundation and hermes-worker-hux. Full repo gate: same 13 pre-existing, unrelated failures as main.

Worker hook library — DONE

Commit	aeef0f84
Files	dockerfiles/hermes-worker-hux/hux_hook/{__init__,client,hooks}.py, NOTES.md (wiring guide for the runtime patch, which is Codex's file)
Tests	test_hermes_hux_policy_hook.py (11), test_hermes_hux_contract_hook.py (10) — real service in-process, end-to-end approval → human decision → gate released once, canary never persisted, unreachable service fails closed for side effects
Codex needs	Agent container env HUX_BASE_URL, HUX_TENANT_SLOT, HUX_SUBJECT, HUX_WORKER_KEY; call order per NOTES.md: before_tool → execute only on proceed → after_tool; record_spend per turn; on_stop returning None means the stop is not done

Conversation privacy state — DONE

GET /hux/v1/conversations/{id}/privacy (HUX-10, worker-callable) returns forgotten / memory_disabled / topics / mode / memory_writes_allowed; hux_hook.memory_gate now reads it and fails closed. Tests: test_hermes_hux_privacy_topics.py::test_conversation_privacy_state_route, hook memory-gate test extended.

Open items (not blockers for Codex integration)

• SO-46 hash-chained audit, SO-48 single-writer lock. (SO-53 rate limits shipped since: hux/http.py RateLimiter, per-subject read/write buckets, 429 + Retry-After, tuned by HUX_READS_PER_MINUTE / HUX_WRITES_PER_MINUTE — see the integrated-topology section.)
• Retention scheduler: privacy.run_retention is on-demand; Codex decides cron vs sidecar.
• Search over message_text; artifact sharing (shared_readonly).
• testing/quality_contract.json line-limit globs / managed modules for the two new package paths (Codex's file).

Integrated topology (2026-08-24, local chain, staged — not deployed)

Everything in this section exists only on local main. origin/main is still 5558c24f; Flux reconciles origin, so none of this is running anywhere. Chat is staged; the Worker is untouched. The chain since origin (oldest first): 964103f0 merge of the hardened foundation, 438180a9 tool runtime gated through HUX, a1070449 vendored runtime hooks, 53d7c2c5 backend families complete, 31c5eae5 chat router image automation, 1eda927b trusted HUX workspace in the WebUI, 47165274 HUX-12 evidence trust isolation, b66c762f HUX card UI models, bd63b568 router strips inbound X-Hux-*, 9e99470f one-or-two WebUI consumers in release rendering, 0b56d04e quality-contract gates, 71c05cf9 staged sidecar activation topology, 659f70ee WebUI integration contract notes, 13359769 all-or-nothing topology gates.

Pod topology (services/hermes/chat-statefulset.yaml)

Each hermes-chat-tenant pod runs a fourth container, hux, on the SAME reviewed WebUI image as the webui container (registry.bstein.dev/bstein/hermes-webui, Flux $imagepolicy hermes:hermes-webui-release), started as python -m hux.server with PYTHONPATH=/opt/hermes-hux. It binds 127.0.0.1:8790 (HUX_BIND, HUX_PORT), runs as uid/gid 10000 with readOnlyRootFilesystem, all capabilities dropped, tmpfs /tmp, and /healthz probes. HUX_TENANT_SLOT=slot-N comes from the pod ordinal, as for the agent. Port 8790 appears in no Service or NetworkPolicy; loopback is the only path (delivery-gated).

Storage: shared RWX claim, kubelet-scoped subtrees

The plan's /opt/data/hux on the tenant home PVC is gone. Storage is a standalone ReadWriteMany PVC hermes-chat-hux-data (storageClass astreae, 10Gi, services/hermes/chat-pvcs.yaml) shared by all tenant pods, with per-pod isolation enforced by kubelet subPathExpr: $(POD_NAME) mounts:

• hux container: /var/lib/hux = its own $(POD_NAME) subtree; HUX_DATA_ROOT=/var/lib/hux/store.
• hermes (agent) container: read-only /run/hermes-hux-context = $(POD_NAME)/context and /run/hermes-hux-subject = $(POD_NAME)/binding only. It has NO mount of /var/lib/hux/store; records are reachable from the agent only through the loopback API.
• webui container: read-only /run/hermes-hux-context only.

No container in any pod mounts another pod's subtree.

Identity provisioning (init-hux-runtime)

A dedicated init container (root, capabilities dropped to CHOWN/DAC_OVERRIDE/FOWNER) provisions, idempotently per pod:

• 0700 roots binding/, context/, store/, owned 10000:10000;
• a persistent 32-byte context key context/context-key (0600), created once and kept across restarts;
• an immutable subject binding binding/subject (0440, created O_EXCL): usr_ + HMAC-SHA256(context-key, "hux.subject.id.v1\0slot-N"). On restart the file is verified (regular file, uid 10000, mode 0440, one hardlink, expected content) or the pod refuses to start;
• a redaction canary context/redaction-canary (0400);
• relay and worker keys in Memory-medium emptyDirs (/run/hermes-webui-hux/relay-key, /run/hermes-hux-worker/worker-key, 0400) — regenerated on every pod restart, never touching the PVC.

The subject is stable across restarts, derived from a per-pod random key, and never derived from the Keycloak subject.

Env contracts

hux sidecar: HUX_FLAGS lists all twelve cards (hux.foundation, hux.activity_timeline, hux.memory_control, hux.projects, hux.artifacts, hux.autonomy, hux.friendly_modes, hux.multimodal, hux.research, hux.onboarding, hux.privacy, hux.release_followthrough); HUX_RELAY_KEY_FILE, HUX_WORKER_KEY_FILE, HUX_SUBJECT_BINDING_FILE, HUX_CONTEXT_KEY_FILE, HUX_CANARY_FILE; rate limits HUX_READS_PER_MINUTE=600, HUX_WRITES_PER_MINUTE=120; and build provenance HUX_IMAGE_TAG / HUX_IMAGE_DIGEST bound by Flux setters ($imagepolicy hermes:hermes-webui-release:tag / :digest comments).

Agent container: HUX_BASE_URL=http://127.0.0.1:8790, HUX_RUNTIME_ENABLED=1, and HUX_TOOL_ENFORCEMENT=0 — the first rollout is observe-only until approval parking/resume is connected to the upstream tool loop and proven live — plus HUX_WORKER_KEY_FILE, HUX_SUBJECT_FILE, HUX_CONTEXT_KEY_FILE, HUX_TIMEOUT_SECONDS=3.

Agent runtime plugin

services/hermes/plugins/hux-runtime/ renders into the hermes-hux-runtime-plugin ConfigMap (services/hermes/kustomization.yaml) and mounts read-only into the agent at /opt/data/plugins/hux-runtime: runtime.py, tool_policy.py, context_ids.py, plugin.yaml (on_session_end hook + tool_execution middleware, fail-closed), and a vendored hux_hook/ package kept in parity with the reviewed worker hook library (parity is delivery-gated).

Router boundary

services/hermes/router/main.go deletes EVERY inbound header whose name starts with x-hux- (case-insensitive) at the authenticated tenant boundary — in the same Director that strips OAuth headers — before asserting X-Hermes-Tenant-Identity: slot-N. Regression tests: services/hermes/router/main_test.go. Defense in depth: the sidecar is loopback-only anyway, and the in-pod WebUI BFF asserts its own X-Hux-* from trusted context.

WebUI image (dockerfiles/Dockerfile.hermes-webui)

The image ships the service and contracts at /opt/hermes-hux (hux/ package + contracts/, HUX_CONTRACT_DIR), the BFF proxy (api/hux_bff.py) and trusted context attachment (api/hux_context.py, hux.webui_context.v1), the bootstrap bridge (static/hux/bootstrap.js/.css) and the wave A/B/C runtime bundles. A build-time gate compiles the package and constructs the router with HUX_FLAGS="": images carry the code but no activation — flags live only in manifests.

Release rendering (ci/scripts/hermes_webui_flux_release.py)

The release script updates one-or-two WebUI image consumers in StatefulSet/hermes-chat-tenant (the webui container, plus the hux sidecar when staged) and exactly one in Deployment/hermes, and binds HUX_IMAGE_TAG/HUX_IMAGE_DIGEST to the released tag and digest whenever the sidecar fields are present — failing when the binding fields are incomplete (exactly one of each, or none at all).

HUX-12 evidence trust (hux/release_security.py, hux/releases.py, 47165274)

Release follow-through has a dedicated fourth trust class, evidence:

• File-only authentication: HUX_RELEASE_EVIDENCE_KEY_FILE must be a 0400 regular file (symlinks refused) holding a key ≥ 32 chars, and HUX_RELEASE_EVIDENCE_POLICY_FILE a strictly validated hux.release_evidence_policy.v1 allowlist (≤ 16 workloads, credential-free HTTPS URLs, exact field sets, bounded evidence age). Without both healthy the HUX-12 capability is off entirely — fail closed.
• Worker trust is read-only on releases (list/read in WORKER_ROUTES); it can no longer create releases or transitions.
• Only router trust creates the single reviewed proposal per scoped workload commit; only evidence trust performs transitions, and its route allowlist in hux/http.py blocks everything else.
• Transition evidence binds exactly: merged the 40-hex merge commit; built the CI URL plus image ref / image digest / Harbor digest all equal; deployed a main@sha1:<40-hex> Flux revision; converged pod digest == image digest; live_verified one passing health receipt; rolled_back an exact target digest.
• Ledger entries are hash-chained (previous_hash/entry_hash) and verified on every read; Idempotency-Key replays return the recorded view with HUX-Replayed: true.

Worker state

worker.bstein.dev (the hermes Deployment and the execution-worker manifests) has NO HUX sidecar, plugin or environment. Cross-surface continuity is not claimed; HUX is chat-staged only.

Test state

Verified 2026-08-24 with the CI interpreter: python -m pytest testing/tests/test_hermes_hux_*.py = 646 passed (20.8 s). Delivery gates in testing/tests/test_hermes_hux_delivery.py assert the staged topology against the manifests themselves: all-or-nothing activation, loopback-only sidecar on the reviewed WebUI image, least-privilege storage and key mounts, init-container identity persistence + transport-key rotation, file-backed authentication inputs, and plugin/vendored-hook render parity. UI card model suites: testing/tests/test_hermes_hux_ui_*.py, each driving a Node harness (test_hermes_hux_ui_*.mjs). Release rendering: test_hermes_webui_release.py (18) and test_hermes_image_automation.py (2).

Superseded statements

• "/opt/data/hux on the tenant PVC; SO-45 separate uid, mode 0700" — superseded. Storage is the shared RWX claim with kubelet subPathExpr isolation above; every container runs uid 10000 and the boundary is mount scoping, not a uid split. See THREAT-MODEL.md, "Integration amendments (2026-08-24)".
• "HUX_DATA_ROOT (tenant PVC, default /opt/data)" — now /var/lib/hux/store on the dedicated claim.
• "runs … inside each hermes-chat-tenant pod (and the Worker)" — the Worker has no HUX yet.
• "Image, manifest, NetworkPolicy and Flux wiring are Codex's" — landed in-repo with the chain above.
• Open item "SO-53 rate limits" — implemented in hux/http.py; SO-46 and SO-48 remain open (see the threat model amendment).