atlas-iac/docs/hux/HANDOFF.md

hermes-next: Claude backend handoff ledger

Branch feature/hermes-next-hux (origin, pushed after every card); draft PR #55 #55 (opened through the SCM broker, never merged by Claude). Board hermes-next on the operator Hermes Kanban; cards HUX-01..HUX-12 are parked (blocked) on purpose so the CLI lane runner never auto-claims them — the Claude coordinator and the Codex root session are the only implementers. Contract: hux.v1 1.0.0 (services/hermes/contracts/hux/VERSION, docs/hux/ADR-0001-hux-v1-contract-freeze.md). Threat model: docs/hux/THREAT-MODEL.md (SO-01..SO-54). Data model: docs/hux/DATA-MODEL.md.

Run everything with the CI interpreter: python -m pytest testing/tests/test_hermes_hux_*.py -q and python -m coverage run --branch --source=dockerfiles/hermes-hux-foundation -m pytest testing/tests/test_hermes_hux_*.py.

Codex integration requirements common to every card

(Superseded 2026-08-24 where it conflicts with "Integrated topology" at the end of this file: the manifests, plugin, router headers and Flux wiring are now in-repo, the data root moved off the tenant PVC, and the Worker has no HUX yet.)

• Package dockerfiles/hermes-hux-foundation/hux/ runs as python -m hux.server inside each hermes-chat-tenant pod (and the Worker) on 127.0.0.1:8790. Image, manifest, NetworkPolicy and Flux wiring are Codex's (SO-01, SO-02, SO-45: separate uid, /opt/data/hux mode 0700).
• Env: HUX_DATA_ROOT (tenant PVC, default /opt/data), HUX_TENANT_SLOT (pin, SO-03), HUX_FLAGS (comma list, default empty = everything off), HUX_RELAY_KEY (Telegram relay), HUX_WORKER_KEY (Worker), HUX_BUILD_COMMIT, HUX_IMAGE_DIGEST, HUX_CONTRACT_DIR (schemas path in the image; default resolves relative to the repo layout).
• Router → service headers (SO-04): X-Hermes-Tenant-Identity: slot-N, X-Hux-Subject: usr_<hash>, X-Hux-Surface: chat|telegram|voice|api|worker, optional X-Hux-Trust: relay|worker + X-Hux-Relay-Key. The router must strip any inbound X-Hux-* before setting its own.
• First client call: GET /hux/v1/capabilities → which cards/routes are on. Disabled or unknown → 404 (flag_off vs not_found only in the body).
• Mutations: If-Match: <revision> on revisioned records (409 on mismatch); Idempotency-Key on creates. Errors are hux.error.v1.
• Fixtures Codex codes against: services/hermes/contracts/hux/examples/*.json (validated in CI by test_hermes_hux_contract_schemas.py).

HUX-11 shared foundation — DONE (backend)

Commits	b4145861 (contract freeze 1.0.0), 6a5e0d87 (service core)
Files	services/hermes/contracts/hux/{common,identity}.schema.json + examples, VERSION; dockerfiles/hermes-hux-foundation/hux/{__init__,errors,identity,flags,store,audit,http,foundation,server,contracts,rules}.py; docs/hux/ADR-0001-*.md, THREAT-MODEL.md, DATA-MODEL.md
Tests	testing/tests/test_hermes_hux_contract_schemas.py (61), test_hermes_hux_contract_foundation.py (30): identity header rejection matrix, relay/worker keys, slot pinning, fail-closed flag chains, capabilities record validity, tenant-scoped paths + traversal rejection, revisions/conflicts, torn-ledger recovery, blobs, manifest, concurrent writers, audit rows, flag-off == not-found, 401 never touches storage, real HTTP server JSON + SSE, loopback-only bind, all errors validate, ≤500 LOC guard
Coverage	99% line / 98% branch over the package
Contract	1.0.0
Flag	hux.foundation (root of every dependency chain)
Risks	Rate limiting (SO-53) and hash-chained audit (SO-46) not yet implemented — tracked for the Wave A review; relay callers still carry X-Hux-Subject (deviates from SO-06; the slot→owner mapping lives in the router, so the header is redundant but harmless and lets the service pin the subject)
Codex needs	Wire the service into the tenant pod + Worker, NetworkPolicy, router header contract above, HUX_FLAGS=hux.foundation for the first canary; testing/quality_contract.json may add dockerfiles/hermes-hux-foundation/**/*.py to line_limit_globs and the two moved modules to managed_modules (Claude does not edit that file)

Wave A backends — DONE, awaiting adversarial review and Codex integration

Full suite: testing/tests/test_hermes_hux_*.py = 325 tests, 99% line / 99% branch over dockerfiles/hermes-hux-foundation/hux/ (every family module 99–100%). Every module ≤ 500 LOC (guarded by tests). Contract revised additively to 1.1.0 (ADR-0001).

Card	Commit	Files	Tests	Flag	Codex needs
HUX-01 activity timeline	b8fb72fb	hux/events.py, hux/redaction.py	test_hermes_hux_contract_events.py (34): ordering, idempotency, replay/reconnect, redaction by surface, cancellation receipts, cross-tenant	hux.activity_timeline	Router forwards Last-Event-ID; telegram/voice surfaces get partial redaction; agent hook posts events with Idempotency-Key; HUX_CANARY_FILE=/opt/data/.env so secrets are scrubbed
HUX-02 memory center	b8fb72fb	hux/memory.py	test_hermes_hux_memory_ledger.py (20), test_hermes_hux_memory_retrieval.py (7): no-store, suggest-only, edit/supersede, forget, retrieval removal, export, If-Match	hux.memory_control (needs hux.privacy)	Agent memory tool must call hux.memory.retrieve semantics (tombstones first) instead of the upstream memory file; UI per-message controls call /memory/{id}/{action} with If-Match
HUX-10 privacy	b8fb72fb	hux/privacy.py	test_hermes_hux_privacy_topics.py (19), test_hermes_hux_privacy_retention.py (6)	hux.privacy	Schedule run_retention daily (no thread inside the service by design); surface HUX-Audit-Stale; show notices and post chosen
HUX-04 artifacts	9f0ffe03 (+ upload caps, project check in the 1.1.0 batch)	hux/artifacts.py, hux/diffs.py	test_hermes_hux_artifact_versions.py, test_hermes_hux_artifact_auth.py (immutable versions, concurrency, lineage forgery 404, caps 413, diffs, promotion)	hux.artifacts (needs hux.projects)	Version content served with nosniff + attachment; uploads ≤ 25 MiB content via base64; sharing intentionally not implemented (access.mode always owner)
HUX-08 research	9f0ffe03	hux/research.py	test_hermes_hux_research_citations.py, test_hermes_hux_research_notebook.py (dedupe, integrity, notebook state machine)	hux.research	Service never fetches URIs (SO-19): the web tool records sources/passages after its own fetch; citation strip = GET /messages/{id}/citations (passages + sources embedded)
HUX-05 autonomy	3ebee2cc	hux/policy.py, hux/budgets.py	test_hermes_hux_policy_matrix.py (22), _approvals.py (22), _receipts.py (13)	hux.autonomy (needs hux.activity_timeline)	Agent hook: POST /approvals before any external side effect, POST /runs/{id}/gate immediately before executing with the canonical argument hash, POST /runs/{id}/budget per turn, POST /runs/{id}/stop with process_registry_empty from the real process registry; decisions only from human surfaces (worker trust → 403)
HUX-03 organization API	3ebee2cc	hux/organization.py	test_hermes_hux_contract_organization.py (14)	hux.projects	Codex owns the UI + migration of upstream WebUI projects; search over message_text is a later increment

Known gaps carried to the Wave A review: rate limiting (SO-53), hash-chained audit (SO-46), retention scheduler ownership (Codex cron vs service thread), approval expiry applied lazily on read, linear scans for idempotency/dedupe (fine at documented caps).

Wave A adversarial review — CLOSED

A fresh reviewer attacked HEAD 1b1a14e9 (tenant isolation, privacy, autonomy, rollback) and reported 13 findings; the repro harness lives outside the repo. Every finding is closed by a regression test that cites it:

#	Severity	Finding	Fix commit
F1	critical	worker/api trust could PUT /policy and self-approve	6964a9d8 — policy writes and allow grants are human-surface only
F2	high	worker trust could read every tenant record (SO-08)	6964a9d8 — flags.WORKER_ROUTES allowlist enforced in Router.dispatch; unexpected exceptions become audited 500 records
F3	high	memory edit skipped topic/sensitivity/private-mode gates	681b0408 — one _classify path for proposals and edits
F4	high	a session approval for an external effect released unrelated later effects	6964a9d8 — external effects match only the same run + argument hash; gate conversation from the run, not the body
F5	high	seq duplicated after a crash between append and checkpoint	681b0408 — seq = max(checkpoint, ledger tail + 1) under the lock
F6	medium	gate ignored budget exhaustion	6964a9d8
F7	medium	stale unconditional write could resurrect a forgotten memory	681b0408 — re-read under lock, always write with the loaded revision
F8	medium	any caller could assert process_registry_empty; failed receipts were sticky	6964a9d8 — gateway (worker trust) only; failed receipts supersedable
F9	medium	secrets stored verbatim in titles/passages/claims/notebooks; forget left the title	681b0408 — scrub applied; forget blanks the document. Artifact bodies stay verbatim (user-owned) but are forced restricted and audited (a75ca299)
F10	medium	SO-46/48/53 claimed but absent	threat model amended (74c65499): tracked as open, not claimed
F11	low	stored receipts carry revision the schema forbade	6964a9d8 — optional revision on the receipt
F12	low	capabilities advertised routes without handlers	6964a9d8 — HUX-06/09/12 routes empty until shipped
F13	low	misc (healthz version, ghost conversations from notices, idempotency outside the lock, artifact titles not searchable, unnormalised paths, multi-hash once approvals)	6964a9d8 + 681b0408

Suite after repairs: 377 tests, 99% line / 99% branch over hermes-hux-foundation and hermes-worker-hux. Full repo gate: same 13 pre-existing, unrelated failures as main.

Worker hook library — DONE

Commit	aeef0f84
Files	dockerfiles/hermes-worker-hux/hux_hook/{__init__,client,hooks}.py, NOTES.md (wiring guide for the runtime patch, which is Codex's file)
Tests	test_hermes_hux_policy_hook.py (11), test_hermes_hux_contract_hook.py (10) — real service in-process, end-to-end approval → human decision → gate released once, canary never persisted, unreachable service fails closed for side effects
Codex needs	Agent container env HUX_BASE_URL, HUX_TENANT_SLOT, HUX_SUBJECT, HUX_WORKER_KEY; call order per NOTES.md: before_tool → execute only on proceed → after_tool; record_spend per turn; on_stop returning None means the stop is not done

Conversation privacy state — DONE

GET /hux/v1/conversations/{id}/privacy (HUX-10, worker-callable) returns forgotten / memory_disabled / topics / mode / memory_writes_allowed; hux_hook.memory_gate now reads it and fails closed. Tests: test_hermes_hux_privacy_topics.py::test_conversation_privacy_state_route, hook memory-gate test extended.

Open items (not blockers for Codex integration)

• SO-46 hash-chained audit, SO-48 single-writer lock. (SO-53 rate limits shipped since: hux/http.py RateLimiter, per-subject read/write buckets, 429 + Retry-After, tuned by HUX_READS_PER_MINUTE / HUX_WRITES_PER_MINUTE — see the integrated-topology section.)
• Retention scheduler: privacy.run_retention is on-demand; Codex decides cron vs sidecar.
• Search over message_text; artifact sharing (shared_readonly).
• testing/quality_contract.json line-limit globs / managed modules for the two new package paths (Codex's file).

Integrated topology (2026-08-24, local chain, staged — not deployed)

Everything in this section exists only on local main. origin/main is still 5558c24f; Flux reconciles origin, so none of this is running anywhere. Chat is staged; the Worker is untouched. The chain since origin (oldest first): 964103f0 merge of the hardened foundation, 438180a9 tool runtime gated through HUX, a1070449 vendored runtime hooks, 53d7c2c5 backend families complete, 31c5eae5 chat router image automation, 1eda927b trusted HUX workspace in the WebUI, 47165274 HUX-12 evidence trust isolation, b66c762f HUX card UI models, bd63b568 router strips inbound X-Hux-*, 9e99470f one-or-two WebUI consumers in release rendering, 0b56d04e quality-contract gates, 71c05cf9 staged sidecar activation topology, 659f70ee WebUI integration contract notes, 13359769 all-or-nothing topology gates.

Pod topology (services/hermes/chat-statefulset.yaml)

Each hermes-chat-tenant pod runs a fourth container, hux, on the SAME reviewed WebUI image as the webui container (registry.bstein.dev/bstein/hermes-webui, Flux $imagepolicy hermes:hermes-webui-release), started as python -m hux.server with PYTHONPATH=/opt/hermes-hux. It binds 127.0.0.1:8790 (HUX_BIND, HUX_PORT), runs as uid/gid 10000 with readOnlyRootFilesystem, all capabilities dropped, tmpfs /tmp, and /healthz probes. HUX_TENANT_SLOT=slot-N comes from the pod ordinal, as for the agent. Port 8790 appears in no Service or NetworkPolicy; loopback is the only path (delivery-gated).

Storage: shared RWX claim, kubelet-scoped subtrees

The plan's /opt/data/hux on the tenant home PVC is gone. Storage is a standalone ReadWriteMany PVC hermes-chat-hux-data (storageClass astreae, 10Gi, services/hermes/chat-pvcs.yaml) shared by all tenant pods, with per-pod isolation enforced by kubelet subPathExpr: $(POD_NAME) mounts:

• hux container: /var/lib/hux = its own $(POD_NAME) subtree; HUX_DATA_ROOT=/var/lib/hux/store.
• hermes (agent) container: read-only /run/hermes-hux-context = $(POD_NAME)/context and /run/hermes-hux-subject = $(POD_NAME)/binding only. It has NO mount of /var/lib/hux/store; records are reachable from the agent only through the loopback API.
• webui container: read-only /run/hermes-hux-context only.

No container in any pod mounts another pod's subtree.

Identity provisioning (init-hux-runtime)

A dedicated init container (root, capabilities dropped to CHOWN/DAC_OVERRIDE/FOWNER) provisions, idempotently per pod:

• 0700 roots binding/, context/, store/, owned 10000:10000;
• a persistent 32-byte context key context/context-key (0600), created once and kept across restarts;
• an immutable subject binding binding/subject (0440, created O_EXCL): usr_ + HMAC-SHA256(context-key, "hux.subject.id.v1\0slot-N"). On restart the file is verified (regular file, uid 10000, mode 0440, one hardlink, expected content) or the pod refuses to start;
• a redaction canary context/redaction-canary (0400);
• relay and worker keys in Memory-medium emptyDirs (/run/hermes-webui-hux/relay-key, /run/hermes-hux-worker/worker-key, 0400) — regenerated on every pod restart, never touching the PVC.

The subject is stable across restarts, derived from a per-pod random key, and never derived from the Keycloak subject.

Env contracts

hux sidecar: HUX_FLAGS lists all twelve cards (hux.foundation, hux.activity_timeline, hux.memory_control, hux.projects, hux.artifacts, hux.autonomy, hux.friendly_modes, hux.multimodal, hux.research, hux.onboarding, hux.privacy, hux.release_followthrough); HUX_RELAY_KEY_FILE, HUX_WORKER_KEY_FILE, HUX_SUBJECT_BINDING_FILE, HUX_CONTEXT_KEY_FILE, HUX_CANARY_FILE; rate limits HUX_READS_PER_MINUTE=600, HUX_WRITES_PER_MINUTE=120; and build provenance HUX_IMAGE_TAG / HUX_IMAGE_DIGEST bound by Flux setters ($imagepolicy hermes:hermes-webui-release:tag / :digest comments).

Agent container: HUX_BASE_URL=http://127.0.0.1:8790, HUX_RUNTIME_ENABLED=1, and HUX_TOOL_ENFORCEMENT=0 — the first rollout is observe-only until approval parking/resume is connected to the upstream tool loop and proven live — plus HUX_WORKER_KEY_FILE, HUX_SUBJECT_FILE, HUX_CONTEXT_KEY_FILE, HUX_TIMEOUT_SECONDS=3.

Agent runtime plugin

services/hermes/plugins/hux-runtime/ renders into the hermes-hux-runtime-plugin ConfigMap (services/hermes/kustomization.yaml) and mounts read-only into the agent at /opt/data/plugins/hux-runtime: runtime.py, tool_policy.py, context_ids.py, plugin.yaml (on_session_end hook + tool_execution middleware, fail-closed), and a vendored hux_hook/ package kept in parity with the reviewed worker hook library (parity is delivery-gated).

Router boundary

services/hermes/router/main.go deletes EVERY inbound header whose name starts with x-hux- (case-insensitive) at the authenticated tenant boundary — in the same Director that strips OAuth headers — before asserting X-Hermes-Tenant-Identity: slot-N. Regression tests: services/hermes/router/main_test.go. Defense in depth: the sidecar is loopback-only anyway, and the in-pod WebUI BFF asserts its own X-Hux-* from trusted context.

WebUI image (dockerfiles/Dockerfile.hermes-webui)

The image ships the service and contracts at /opt/hermes-hux (hux/ package + contracts/, HUX_CONTRACT_DIR), the BFF proxy (api/hux_bff.py) and trusted context attachment (api/hux_context.py, hux.webui_context.v1), the bootstrap bridge (static/hux/bootstrap.js/.css) and the wave A/B/C runtime bundles. A build-time gate compiles the package and constructs the router with HUX_FLAGS="": images carry the code but no activation — flags live only in manifests.

Release rendering (ci/scripts/hermes_webui_flux_release.py)

The release script updates one-or-two WebUI image consumers in StatefulSet/hermes-chat-tenant (the webui container, plus the hux sidecar when staged) and exactly one in Deployment/hermes, and binds HUX_IMAGE_TAG/HUX_IMAGE_DIGEST to the released tag and digest whenever the sidecar fields are present — failing when the binding fields are incomplete (exactly one of each, or none at all).

HUX-12 evidence trust (hux/release_security.py, hux/releases.py, 47165274)

Release follow-through has a dedicated fourth trust class, evidence:

• File-only authentication: HUX_RELEASE_EVIDENCE_KEY_FILE must be a 0400 regular file (symlinks refused) holding a key ≥ 32 chars, and HUX_RELEASE_EVIDENCE_POLICY_FILE a strictly validated hux.release_evidence_policy.v1 allowlist (≤ 16 workloads, credential-free HTTPS URLs, exact field sets, bounded evidence age). Without both healthy the HUX-12 capability is off entirely — fail closed.
• Worker trust is read-only on releases (list/read in WORKER_ROUTES); it can no longer create releases or transitions.
• Only router trust creates the single reviewed proposal per scoped workload commit; only evidence trust performs transitions, and its route allowlist in hux/http.py blocks everything else.
• Transition evidence binds exactly: merged the 40-hex merge commit; built the CI URL plus image ref / image digest / Harbor digest all equal; deployed a main@sha1:<40-hex> Flux revision; converged pod digest == image digest; live_verified one passing health receipt; rolled_back an exact target digest.
• Ledger entries are hash-chained (previous_hash/entry_hash) and verified on every read; Idempotency-Key replays return the recorded view with HUX-Replayed: true.

Worker state

worker.bstein.dev (the hermes Deployment and the execution-worker manifests) has NO HUX sidecar, plugin or environment. Cross-surface continuity is not claimed; HUX is chat-staged only.

Test state

Verified 2026-08-24 with the CI interpreter: python -m pytest testing/tests/test_hermes_hux_*.py = 646 passed (20.8 s). Delivery gates in testing/tests/test_hermes_hux_delivery.py assert the staged topology against the manifests themselves: all-or-nothing activation, loopback-only sidecar on the reviewed WebUI image, least-privilege storage and key mounts, init-container identity persistence + transport-key rotation, file-backed authentication inputs, and plugin/vendored-hook render parity. UI card model suites: testing/tests/test_hermes_hux_ui_*.py, each driving a Node harness (test_hermes_hux_ui_*.mjs). Release rendering: test_hermes_webui_release.py (18) and test_hermes_image_automation.py (2).

Superseded statements

• "/opt/data/hux on the tenant PVC; SO-45 separate uid, mode 0700" — superseded. Storage is the shared RWX claim with kubelet subPathExpr isolation above; every container runs uid 10000 and the boundary is mount scoping, not a uid split. See THREAT-MODEL.md, "Integration amendments (2026-08-24)".
• "HUX_DATA_ROOT (tenant PVC, default /opt/data)" — now /var/lib/hux/store on the dedicated claim.
• "runs … inside each hermes-chat-tenant pod (and the Worker)" — the Worker has no HUX yet.
• "Image, manifest, NetworkPolicy and Flux wiring are Codex's" — landed in-repo with the chain above.
• Open item "SO-53 rate limits" — implemented in hux/http.py; SO-46 and SO-48 remain open (see the threat model amendment).

Completion increments (2026-08-24, later — local chain, staged, not deployed)

Everything below is committed locally on top of the integrated topology and verified by the full gate (826 passed / 0 failed across test_hermes_hux_*.py, test_hermes_webui_hux_*.py, image-automation, webui-release, auto-router and chat-quality suites, after one fieldRef tolerance fix in the voice gate).

• Topology gates are all-or-nothing adaptive (test_hermes_hux_delivery.py, test_hermes_image_automation.py): with no hux sidecar the manifests must contain zero partial HUX wiring; with it, the full strict boundary applies. This lets the source chain merge and build before activation.
• HUX-06 adoption (services/hermes/plugins/auto-router/hux_mode.py): the auto-router consults the loopback HUX service (worker trust, the same hux.context.id.v1 HMAC derivation) for the conversation's selected friendly mode: fast→auto/fast, thoughtful/research→auto/deep, create→auto/balanced, private→the local route only. Explicit UI picks win; every failure falls through to previous behaviour. Delivered through the auto-router ConfigMap with a degrade-to-noop import shim.
• Private mode is enforced upstream (hux/policy.py, hux/budgets.py): network, web_search, send_message, shell and delegate are refused by the approval resolver and re-refused by the gate for a conversation whose stored mode is private, regardless of autonomy level.
• HUX-04/08 emitters (plugins/hux-runtime/emitters.py): after a successful write_files tool the plugin registers the written file as an artifact (create or immutable version, ≤1 MiB, deterministic idempotency); after successful network/web_search tools it records up to three deduplicated web sources from the real output. Fail-open by construction.
• HUX-03 message-text search (hux/organization.py): GET /hux/v1/search?include=message_text scans stored message events of the 100 most recently active candidates; forgotten, private, restricted and fully-redacted content never matches; deterministic cursor pagination. The default search response is unchanged (the shipped UI requires message_text in not_indexed).
• HUX-12 producer (dockerfiles/hermes-hux-foundation/hux_producer/, 100% line+branch, 71 tests): pure verifier binding review URL prefix → Jenkins SUCCESS/revision → immutable Harbor tag/digest → Flux main@sha1: revision with pin containment → desired workload image → every Ready pod imageID → bounded-age health receipt → rollback target; HTTPS-only injectable collectors (SA token for the API); evidence-trust driver posts exactly one If-Match transition. Cluster layer staged in chat-statefulset.yaml (sidecar + tolerant key init), hux-evidence-policy.json, hux-evidence-rbac.yaml (read-only pods/ statefulset + the single named Kustomization), tenant egress to the API ClusterIP and traefik. The key reaches only the hux and hux-evidence-producer containers.

HUX-12 enablement runbook (fail-closed until every step is done)

1. vault kv put kv/atlas/hermes/hux-evidence evidence_key="$(openssl rand -hex 32)"
2. Add to the chat tenant pod annotations (Flux commit): vault.hashicorp.com/agent-inject-secret-hux-evidence-key: kv/data/atlas/hermes/hux-evidence plus the matching template emitting .Data.data.evidence_key.
3. Create ConfigMap hermes-hux-evidence-scope with HUX_PRODUCER_PROJECT_ID / HUX_PRODUCER_CONVERSATION_ID for the release conversation (visible in the UI once a reviewed proposal exists).
4. Only then does release_security.configured turn HUX-12 on; the sidecar loop starts posting verified transitions.

Staged release sequence (agreed)

1. Push the source tip to main: the full local chain plus a hold commit that restores services/hermes/{chat-statefulset,chat-pvcs,chat-configmap,kustomization,networkpolicy}.yaml to the live state (keeping only the auto-router hux_mode.py ConfigMap key) and removes the evidence policy/RBAC files — so Flux applies nothing and CI builds the reviewed WebUI image.
2. Trigger the WebUI build at that exact SHA; wait for Flux digest convergence.
3. Re-apply the activation topology pinned to the new tag/digest with HUX_FLAGS=hux.foundation, HUX_TOOL_ENFORCEMENT=0, and a RollingUpdate partition canarying ordinal 3 first.
4. HUX-11 live gates → card-by-card flag enablement with lifecycle acceptance → enforcement after park/resume + stop receipts pass live → HUX-12 after the runbook above → Worker separately.

Live canary evidence (2026-08-24 ~09:05 UTC)

Foundation canary LIVE and verified on hermes-chat-tenant-3:

• Commits: source 2f535d3a (build 21), Flux digest 1d29c25c, activation 39a54482 (partition 3, HUX_FLAGS=hux.foundation, enforcement 0).
• Image: registry.bstein.dev/bstein/hermes-webui:git-2f535d3a...-build-21-release@sha256:e5b9b2fa8296a7b8a6065c63e7b288cd3305d70a3acc4312055b819bb5f64c04 on webui + hux + hux-evidence-producer (pod imageIDs verified); agent digest unchanged (sha256:c64c06e6...).
• canary_verify: 15/15 PASS (all containers ready, ordinals 0-2 untouched and huxless, loopback /healthz ok, subject binding present, public https://chat.bstein.dev/healthz 200, Flux Ready).
• Live route matrix: unauth 401, bad key 401, wrong slot 401, relay capabilities 200 with only HUX-11 enabled, worker capabilities 200, disabled card 404 flag_off. Subject usr_224348fa... consistent with the HMAC binding.
• Identity hashes for the restart-persistence gate: context-key 8a1c474d..., subject c4ebe353... (compare after the next roll).
• Rollback target: WebUI build-18 digest sha256:df91f8a3... (previous fleet image); activation rollback = revert 39a54482.
• Retention scheduler reports disabled until hux.privacy is enabled (by design).

Next: Wave-A canary flags (activity_timeline, projects, privacy, memory_control) on ordinal 3, lifecycle gates per card, then widen partition.

Wave-A live gates (2026-08-24 ~09:12 UTC) — PASS

Canary rolled to ee330822 flags; restart persistence PROVEN (context-key and subject hashes identical across the roll). Live lifecycle 13/13 on the canary loopback: capabilities advertise exactly HUX-01/02/03/10/11; real project + conversation created; worker-emitted event readable by relay; memory proposed -> approved -> forgotten with the tombstone effective in the listing; privacy policy served; conversation forget tombstones the record; retention scheduler enabled and healthy. Next: Wave-B flags (artifacts, research, friendly_modes, multimodal, onboarding) on the canary.

Wave-B live gates (2026-08-24 ~09:20 UTC) — PASS 18/18

Canary at dc2b3643 flags. Live on the canary loopback: capabilities show the Wave-B set with HUX-07 correctly fail-closed (depends_on HUX-05); artifact create -> immutable version -> content -> diff -> promotion; research source -> passage -> citation attached to an authoritative message id with the citation strip readable; provider-neutral mode catalog, thoughtful selected under If-Match and read back by worker trust; suggestion shown for the real after_artifact context, explicit clicked dismissal recorded, re-show suppressed, suppression state durable. Next: hux.autonomy (+ multimodal unlock) on the canary, enforcement still 0.

Autonomy-wave live gates (2026-08-24 ~09:30 UTC) — PASS 15/15

Canary at f2f54033 flags. Live on the canary loopback: HUX-05 park -> human-only approval (worker 403) -> gate releases exactly once -> consumed on the second ask; denial stays denied; browser cannot write stop receipts; gateway receipts are honest (process_registry_empty=false -> failed_to_cancel, superseded by a true empty-registry cancellation). HUX-07 unlocked by the dependency chain and media registration required a REAL approved artifact_write action before accepting upload metadata.

ENFORCEMENT: park/resume and gateway-owned receipts passed live, so HUX_TOOL_ENFORCEMENT=1 ships to the canary (partition still 3; ordinals 0-2 remain untouched). Tenant-3 tool calls now require HUX release for real. Remaining: real-traffic validation of the approvals UX on tenant-3, then partition 3 -> 0 to take flags+enforcement fleet-wide; HUX-12 needs the Vault evidence key (runbook above); Worker integration separate.

Enforcement live on the canary (2026-08-24 ~09:35 UTC)

Canary at 4a6d4fdf+17037773: HUX_TOOL_ENFORCEMENT=1 and HUX_RUNTIME_ENABLED=1 verified in the live hermes container; identity hashes unchanged across a third roll (8a1c474d/c4ebe353); /healthz ok with retention healthy; all five containers ready, zero restarts. The earlier hux exit-137s were transient probe kills under roll churn (threaded server, no OOM/liveness events, clean logs) — bump the hux exec probe timeoutSeconds 3->5 with the partition-widening commit. Voice fixes pushed as 91cfb901 (continuous mic, barge stitching, 1.15x TTS); build 22 queued — its digest bump rolls the fleet including this canary.

FLEET LIVE (2026-08-24 ~12:55 UTC) — HUX-01..11 live on all four tenants

Widening commit 5fc3e870 applied, Flux Ready=True. Verification 39/39: every tenant 5/5 ready on webui build-22 (git-91cfb901...-build-22-release@sha256:2313d0d5...) for webui + hux + producer; agent digest unchanged everywhere; hux /healthz ok per pod; HUX_TOOL_ENFORCEMENT=1 fleet-wide; HUX_IMAGE_TAG env now consistent with the image (block-style setters); four DISTINCT HMAC subjects (per-pod isolation proven); public /healthz 200; zero container restarts under the 5s probe budgets (the 2s exec budget had produced 405 readiness timeouts in 135min and stalled the digest roll — root-caused and fixed). Voice fixes (continuous mic, barge stitching, 1.15x TTS) shipped in the same image.

Remaining, deliberately gated:

• HUX-12 enablement: Vault evidence key + scope ConfigMap (runbook above); everything else is live and fail-closed.
• Worker (worker.bstein.dev): separate integration + canary, not started — cross-surface continuity is NOT claimed.
• Real-traffic approvals-UX feedback from the operator; rollback = one revert of 5fc3e870 (re-partition) and/or HUX_TOOL_ENFORCEMENT=0.

Enforcement rolled back to observe-only (2026-08-24 ~13:3x UTC, 1793903f)

Live traffic on the enforced fleet showed the first defaults blocking core assistant faculties (skills listing and the sandboxed Python fell into the unknown->external_side_effect always-ask bucket, browsing was deny-by- default, and no approval prompt surfaced in the chat flow). Fleet rolled to HUX_TOOL_ENFORCEMENT=0 with a corrected capability map (python_sandbox = internal shell; skills/todo/clarify/vision = reads; browser/web = network medium; image_gen = artifact_write via the trusted broker; unknown tools stay fail-closed). Verified live: all tenants 5/5, enf=0 everywhere.

Re-enforcement checklist (in order):

1. Approvals UX must surface prompts inline in the chat/voice flow.
2. Default grants reviewed against the corrected capability map (network under safe autonomy should ask, not deny, for the chat surface).
3. Canary re-enable on one ordinal first (partition), then fleet.

Round-2 fleet live (2026-08-24 ~14:4x UTC) — build 23 everywhere

All four tenants verified on git-043aa9ee...-build-23-release@sha256:b3b5830d... (webui + hux + producer; agent digest unchanged; per-pod hux /healthz ok; HUX_IMAGE_TAG env consistent with the image — the block-style setters were rewritten by Flux this cycle, confirming the flow-mapping fix; four distinct HMAC subjects; public /healthz 200; enforcement observe-only BY DESIGN pending the approvals-UX work). One transient registry 404 during tenant-1's pull (traefik-VIP flake class) cleared by kubelet retry in ~2 min — manifests verified intact.

Build 23 contents: voice clipping fix (young-utterance endpoint hold, speculate min-speech, unconditional resume, capped noise floor + capture- continuity probe), always-stitch within 20s of a voice barge, HUX drawer theme-token bridge (side-panel dark-box fix), full-screen conversation mode (orb, live captions, mute/exit, a11y), renderer block-style HUX binding.

Definitive remaining list: HUX-12 Vault key + scope ConfigMap (operator runbook above); approvals-UX inline prompts then re-enforcement via canary; Worker (worker.bstein.dev) integration as its own lane; titan-iac main CI baseline confirmation post node-skip hardening.

Build 24 fleet-live + cluster read + worker staged (2026-08-24 ~16:xx UTC)

Build 24 (d041f1d1 -> sha256:1ea5b4ad, promote 04f060db) on all four tenants + the worker agent. Fleet verify 34/34 meaningful (enforcement observe-only by design; Flux Ready=True after reconcile). Worker gates 6/6 with identity PERSISTED across two rolls (hashes 7fa1c7d3/532b9745 unchanged) and the auth matrix exact (unauth/badkey/wrongslot 401, worker caps=HUX-11).

Shipped this cycle:

• Voice round 3: caption body-only extraction, dynamic endpointing, both captions touch-scrollable, errors never spoken + resyncCapture, workspace toggle in the sidebar rail, the false 'session unavailable' toast (409 from a fail-closed profile-alias race - fixed at the cause), barge cut-point marker, full language routing, speaking<->thinking cycling.
• Approvals UX: inline pending-approval dock above the composer; the default grant matrix now asks (not denies) for network/web_search below autonomous, SO-39 intact. Re-enforcement checklist items 1+2 done code-side.
• cluster_read: read-only Kubernetes visibility LIVE and verified in-chat (pods/nodes/flux read; secrets refused; RBAC excludes Secrets). Required a tenant egress fix to the real apiserver endpoints (192.168.22.11-13:6443, which the except-block dropped post-DNAT) and a large-node-listing handler hardening (ed0bc30d).
• Worker staged inert-HUX (foundation only, enforcement 0) - activation is a reviewed one-line flip per WORKER-PLAN.md.

Transient registry 404 during tenant-3's pull recurred (3rd time; VIP/registry flake class) and self-cleared - worth a registry-reliability follow-up.

Remaining: HUX-12 Vault key (operator); re-enforcement canary (approvals UX now exists - operator tries the prompts, then canary re-enable per checklist); worker HUX activation flip (WORKER-PLAN TODO 2); cluster_read handler hardening activates on the next pod roll.

Final webui build-29 live + verified (2026-08-24)

All tenants on webui build-29 (sha256:6887c2ad, source 4d93ef5a), verified in the served bundle: Workspace toggle in the nav (desktop rail + mobile titlebar via navIconToggle/titlebarHost, no composer-obstructing floating pill - fixed the mobile regression that pushed the input controls off screen); conversation orb watermark = the Hermes CHARACTER avatar (hermes-agent-192.png), staff SVG removed; 'hands-free' renamed to 'Conversation mode'; reply-language routing density-gated (resolveReplyLanguage) so English no longer gets a foreign voice. Also carries: interim-ack tail flush, natural thinking fillers, one unified audio sink + output-device selector, the 'Something went wrong' error-veto, and the cluster_read large-listing fix.

STT: large-v3-turbo is live for accuracy (already correcting proper nouns); the beam-2 + name-prompt tuning is pending a successful hermes-voice-image (stt) build - the build keeps failing on an IPv6 route to azureedge while downloading the Whisper models (a more persistent flake than the ghcr.io DNS one). Durable follow-up recommended: mirror the ghcr.io webui base image and the Whisper models into registry.bstein.dev so builds stop depending on external DNS/IPv6 (3 webui + 3 stt builds lost to these flakes).

Gated on the operator: HUX-12 Vault evidence key; re-enabling enforcement (approvals UX is live); worker HUX activation flip.