atlas-iac/docs/hux/HANDOFF.md

# hermes-next: Claude backend handoff ledger

Branch `feature/hermes-next-hux` (origin, pushed after every card); draft PR #55 https://scm.bstein.dev/titan/atlas-iac/pulls/55 (opened through the SCM broker, never merged by Claude). Board
`hermes-next` on the operator Hermes Kanban; cards HUX-01..HUX-12 are parked
(`blocked`) on purpose so the CLI lane runner never auto-claims them — the
Claude coordinator and the Codex root session are the only implementers.
Contract: `hux.v1` 1.0.0 (`services/hermes/contracts/hux/VERSION`,
`docs/hux/ADR-0001-hux-v1-contract-freeze.md`). Threat model:
`docs/hux/THREAT-MODEL.md` (SO-01..SO-54). Data model: `docs/hux/DATA-MODEL.md`.

Run everything with the CI interpreter:
`python -m pytest testing/tests/test_hermes_hux_*.py -q` and
`python -m coverage run --branch --source=dockerfiles/hermes-hux-foundation -m pytest testing/tests/test_hermes_hux_*.py`.

## Codex integration requirements common to every card

(Superseded 2026-08-24 where it conflicts with "Integrated topology" at the
end of this file: the manifests, plugin, router headers and Flux wiring are
now in-repo, the data root moved off the tenant PVC, and the Worker has no
HUX yet.)

- Package `dockerfiles/hermes-hux-foundation/hux/` runs as `python -m hux.server`
  inside each `hermes-chat-tenant` pod (and the Worker) on `127.0.0.1:8790`.
  Image, manifest, NetworkPolicy and Flux wiring are Codex's (SO-01, SO-02,
  SO-45: separate uid, `/opt/data/hux` mode 0700).
- Env: `HUX_DATA_ROOT` (tenant PVC, default `/opt/data`), `HUX_TENANT_SLOT`
  (pin, SO-03), `HUX_FLAGS` (comma list, default empty = everything off),
  `HUX_RELAY_KEY` (Telegram relay), `HUX_WORKER_KEY` (Worker),
  `HUX_BUILD_COMMIT`, `HUX_IMAGE_DIGEST`, `HUX_CONTRACT_DIR` (schemas path in
  the image; default resolves relative to the repo layout).
- Router → service headers (SO-04): `X-Hermes-Tenant-Identity: slot-N`,
  `X-Hux-Subject: usr_<hash>`, `X-Hux-Surface: chat|telegram|voice|api|worker`,
  optional `X-Hux-Trust: relay|worker` + `X-Hux-Relay-Key`. The router must
  strip any inbound `X-Hux-*` before setting its own.
- First client call: `GET /hux/v1/capabilities` → which cards/routes are on.
  Disabled or unknown → `404` (`flag_off` vs `not_found` only in the body).
- Mutations: `If-Match: <revision>` on revisioned records (409 on mismatch);
  `Idempotency-Key` on creates. Errors are `hux.error.v1`.
- Fixtures Codex codes against: `services/hermes/contracts/hux/examples/*.json`
  (validated in CI by `test_hermes_hux_contract_schemas.py`).

## HUX-11 shared foundation — DONE (backend)

| | |
|---|---|
| Commits | `b4145861` (contract freeze 1.0.0), `6a5e0d87` (service core) |
| Files | `services/hermes/contracts/hux/{common,identity}.schema.json` + examples, `VERSION`; `dockerfiles/hermes-hux-foundation/hux/{__init__,errors,identity,flags,store,audit,http,foundation,server,contracts,rules}.py`; `docs/hux/ADR-0001-*.md`, `THREAT-MODEL.md`, `DATA-MODEL.md` |
| Tests | `testing/tests/test_hermes_hux_contract_schemas.py` (61), `test_hermes_hux_contract_foundation.py` (30): identity header rejection matrix, relay/worker keys, slot pinning, fail-closed flag chains, capabilities record validity, tenant-scoped paths + traversal rejection, revisions/conflicts, torn-ledger recovery, blobs, manifest, concurrent writers, audit rows, flag-off == not-found, 401 never touches storage, real HTTP server JSON + SSE, loopback-only bind, all errors validate, ≤500 LOC guard |
| Coverage | 99% line / 98% branch over the package |
| Contract | 1.0.0 |
| Flag | `hux.foundation` (root of every dependency chain) |
| Risks | Rate limiting (SO-53) and hash-chained audit (SO-46) not yet implemented — tracked for the Wave A review; relay callers still carry `X-Hux-Subject` (deviates from SO-06; the slot→owner mapping lives in the router, so the header is redundant but harmless and lets the service pin the subject) |
| Codex needs | Wire the service into the tenant pod + Worker, NetworkPolicy, router header contract above, `HUX_FLAGS=hux.foundation` for the first canary; `testing/quality_contract.json` may add `dockerfiles/hermes-hux-foundation/**/*.py` to `line_limit_globs` and the two moved modules to `managed_modules` (Claude does not edit that file) |

## Wave A backends — DONE, awaiting adversarial review and Codex integration

Full suite: `testing/tests/test_hermes_hux_*.py` = 325 tests, 99% line / 99% branch over `dockerfiles/hermes-hux-foundation/hux/` (every family module 99–100%). Every module ≤ 500 LOC (guarded by tests). Contract revised additively to 1.1.0 (ADR-0001).

| Card | Commit | Files | Tests | Flag | Codex needs |
|---|---|---|---|---|---|
| HUX-01 activity timeline | `b8fb72fb` | `hux/events.py`, `hux/redaction.py` | `test_hermes_hux_contract_events.py` (34): ordering, idempotency, replay/reconnect, redaction by surface, cancellation receipts, cross-tenant | `hux.activity_timeline` | Router forwards `Last-Event-ID`; telegram/voice surfaces get partial redaction; agent hook posts events with `Idempotency-Key`; `HUX_CANARY_FILE=/opt/data/.env` so secrets are scrubbed |
| HUX-02 memory center | `b8fb72fb` | `hux/memory.py` | `test_hermes_hux_memory_ledger.py` (20), `test_hermes_hux_memory_retrieval.py` (7): no-store, suggest-only, edit/supersede, forget, retrieval removal, export, If-Match | `hux.memory_control` (needs `hux.privacy`) | Agent memory tool must call `hux.memory.retrieve` semantics (tombstones first) instead of the upstream memory file; UI per-message controls call `/memory/{id}/{action}` with `If-Match` |
| HUX-10 privacy | `b8fb72fb` | `hux/privacy.py` | `test_hermes_hux_privacy_topics.py` (19), `test_hermes_hux_privacy_retention.py` (6) | `hux.privacy` | Schedule `run_retention` daily (no thread inside the service by design); surface `HUX-Audit-Stale`; show notices and post `chosen` |
| HUX-04 artifacts | `9f0ffe03` (+ upload caps, project check in the 1.1.0 batch) | `hux/artifacts.py`, `hux/diffs.py` | `test_hermes_hux_artifact_versions.py`, `test_hermes_hux_artifact_auth.py` (immutable versions, concurrency, lineage forgery 404, caps 413, diffs, promotion) | `hux.artifacts` (needs `hux.projects`) | Version content served with `nosniff` + attachment; uploads ≤ 25 MiB content via base64; sharing intentionally not implemented (`access.mode` always `owner`) |
| HUX-08 research | `9f0ffe03` | `hux/research.py` | `test_hermes_hux_research_citations.py`, `test_hermes_hux_research_notebook.py` (dedupe, integrity, notebook state machine) | `hux.research` | Service never fetches URIs (SO-19): the web tool records sources/passages after its own fetch; citation strip = `GET /messages/{id}/citations` (passages + sources embedded) |
| HUX-05 autonomy | `3ebee2cc` | `hux/policy.py`, `hux/budgets.py` | `test_hermes_hux_policy_matrix.py` (22), `_approvals.py` (22), `_receipts.py` (13) | `hux.autonomy` (needs `hux.activity_timeline`) | Agent hook: `POST /approvals` before any external side effect, `POST /runs/{id}/gate` immediately before executing with the canonical argument hash, `POST /runs/{id}/budget` per turn, `POST /runs/{id}/stop` with `process_registry_empty` from the real process registry; decisions only from human surfaces (worker trust → 403) |
| HUX-03 organization API | `3ebee2cc` | `hux/organization.py` | `test_hermes_hux_contract_organization.py` (14) | `hux.projects` | Codex owns the UI + migration of upstream WebUI projects; search over `message_text` is a later increment |

Known gaps carried to the Wave A review: rate limiting (SO-53), hash-chained audit (SO-46), retention scheduler ownership (Codex cron vs service thread), approval expiry applied lazily on read, linear scans for idempotency/dedupe (fine at documented caps).

## Wave A adversarial review — CLOSED

A fresh reviewer attacked HEAD `1b1a14e9` (tenant isolation, privacy, autonomy, rollback) and reported 13 findings; the repro harness lives outside the repo. Every finding is closed by a regression test that cites it:

| # | Severity | Finding | Fix commit |
|---|---|---|---|
| F1 | critical | worker/api trust could `PUT /policy` and self-approve | `6964a9d8` — policy writes and allow grants are human-surface only |
| F2 | high | worker trust could read every tenant record (SO-08) | `6964a9d8` — `flags.WORKER_ROUTES` allowlist enforced in `Router.dispatch`; unexpected exceptions become audited 500 records |
| F3 | high | memory `edit` skipped topic/sensitivity/private-mode gates | `681b0408` — one `_classify` path for proposals and edits |
| F4 | high | a `session` approval for an external effect released unrelated later effects | `6964a9d8` — external effects match only the same run + argument hash; gate conversation from the run, not the body |
| F5 | high | seq duplicated after a crash between append and checkpoint | `681b0408` — seq = max(checkpoint, ledger tail + 1) under the lock |
| F6 | medium | gate ignored budget exhaustion | `6964a9d8` |
| F7 | medium | stale unconditional write could resurrect a forgotten memory | `681b0408` — re-read under lock, always write with the loaded revision |
| F8 | medium | any caller could assert `process_registry_empty`; failed receipts were sticky | `6964a9d8` — gateway (worker trust) only; failed receipts supersedable |
| F9 | medium | secrets stored verbatim in titles/passages/claims/notebooks; forget left the title | `681b0408` — scrub applied; forget blanks the document. Artifact bodies stay verbatim (user-owned) but are forced `restricted` and audited (`a75ca299`) |
| F10 | medium | SO-46/48/53 claimed but absent | threat model amended (`74c65499`): tracked as open, not claimed |
| F11 | low | stored receipts carry `revision` the schema forbade | `6964a9d8` — optional `revision` on the receipt |
| F12 | low | capabilities advertised routes without handlers | `6964a9d8` — HUX-06/09/12 routes empty until shipped |
| F13 | low | misc (healthz version, ghost conversations from notices, idempotency outside the lock, artifact titles not searchable, unnormalised paths, multi-hash once approvals) | `6964a9d8` + `681b0408` |

Suite after repairs: 377 tests, 99% line / 99% branch over `hermes-hux-foundation` and `hermes-worker-hux`. Full repo gate: same 13 pre-existing, unrelated failures as `main`.

## Worker hook library — DONE

| | |
|---|---|
| Commit | `aeef0f84` |
| Files | `dockerfiles/hermes-worker-hux/hux_hook/{__init__,client,hooks}.py`, `NOTES.md` (wiring guide for the runtime patch, which is Codex's file) |
| Tests | `test_hermes_hux_policy_hook.py` (11), `test_hermes_hux_contract_hook.py` (10) — real service in-process, end-to-end approval → human decision → gate released once, canary never persisted, unreachable service fails closed for side effects |
| Codex needs | Agent container env `HUX_BASE_URL`, `HUX_TENANT_SLOT`, `HUX_SUBJECT`, `HUX_WORKER_KEY`; call order per `NOTES.md`: `before_tool` → execute only on `proceed` → `after_tool`; `record_spend` per turn; `on_stop` returning `None` means the stop is not done |

## Conversation privacy state — DONE

`GET /hux/v1/conversations/{id}/privacy` (HUX-10, worker-callable) returns forgotten / memory_disabled / topics / mode / `memory_writes_allowed`; `hux_hook.memory_gate` now reads it and fails closed. Tests: `test_hermes_hux_privacy_topics.py::test_conversation_privacy_state_route`, hook memory-gate test extended.

## Open items (not blockers for Codex integration)

- SO-46 hash-chained audit, SO-48 single-writer lock. (SO-53 rate limits
  shipped since: `hux/http.py` `RateLimiter`, per-subject read/write
  buckets, `429` + `Retry-After`, tuned by `HUX_READS_PER_MINUTE` /
  `HUX_WRITES_PER_MINUTE` — see the integrated-topology section.)
- Retention scheduler: `privacy.run_retention` is on-demand; Codex decides cron vs sidecar.
- Search over `message_text`; artifact sharing (`shared_readonly`).
- `testing/quality_contract.json` line-limit globs / managed modules for the two new package paths (Codex's file).

## Integrated topology (2026-08-24, local chain, staged — not deployed)

Everything in this section exists only on local `main`. `origin/main` is
still `5558c24f`; Flux reconciles origin, so none of this is running
anywhere. Chat is staged; the Worker is untouched. The chain since origin
(oldest first): `964103f0` merge of the hardened foundation, `438180a9`
tool runtime gated through HUX, `a1070449` vendored runtime hooks,
`53d7c2c5` backend families complete, `31c5eae5` chat router image
automation, `1eda927b` trusted HUX workspace in the WebUI, `47165274`
HUX-12 evidence trust isolation, `b66c762f` HUX card UI models,
`bd63b568` router strips inbound `X-Hux-*`, `9e99470f` one-or-two WebUI
consumers in release rendering, `0b56d04e` quality-contract gates,
`71c05cf9` staged sidecar activation topology, `659f70ee` WebUI
integration contract notes, `13359769` all-or-nothing topology gates.

### Pod topology (`services/hermes/chat-statefulset.yaml`)

Each `hermes-chat-tenant` pod runs a fourth container, `hux`, on the SAME
reviewed WebUI image as the `webui` container
(`registry.bstein.dev/bstein/hermes-webui`, Flux
`$imagepolicy hermes:hermes-webui-release`), started as
`python -m hux.server` with `PYTHONPATH=/opt/hermes-hux`. It binds
`127.0.0.1:8790` (`HUX_BIND`, `HUX_PORT`), runs as uid/gid 10000 with
`readOnlyRootFilesystem`, all capabilities dropped, tmpfs `/tmp`, and
`/healthz` probes. `HUX_TENANT_SLOT=slot-N` comes from the pod ordinal,
as for the agent. Port 8790 appears in no Service or NetworkPolicy;
loopback is the only path (delivery-gated).

### Storage: shared RWX claim, kubelet-scoped subtrees

The plan's `/opt/data/hux` on the tenant `home` PVC is gone. Storage is a
standalone `ReadWriteMany` PVC `hermes-chat-hux-data` (storageClass
`astreae`, 10Gi, `services/hermes/chat-pvcs.yaml`) shared by all tenant
pods, with per-pod isolation enforced by kubelet
`subPathExpr: $(POD_NAME)` mounts:

- `hux` container: `/var/lib/hux` = its own `$(POD_NAME)` subtree;
  `HUX_DATA_ROOT=/var/lib/hux/store`.
- `hermes` (agent) container: read-only `/run/hermes-hux-context` =
  `$(POD_NAME)/context` and `/run/hermes-hux-subject` =
  `$(POD_NAME)/binding` only. It has NO mount of `/var/lib/hux/store`;
  records are reachable from the agent only through the loopback API.
- `webui` container: read-only `/run/hermes-hux-context` only.

No container in any pod mounts another pod's subtree.

### Identity provisioning (`init-hux-runtime`)

A dedicated init container (root, capabilities dropped to
CHOWN/DAC_OVERRIDE/FOWNER) provisions, idempotently per pod:

- 0700 roots `binding/`, `context/`, `store/`, owned 10000:10000;
- a persistent 32-byte context key `context/context-key` (0600), created
  once and kept across restarts;
- an immutable subject binding `binding/subject` (0440, created `O_EXCL`):
  `usr_` + HMAC-SHA256(context-key, `"hux.subject.id.v1\0slot-N"`). On
  restart the file is verified (regular file, uid 10000, mode 0440, one
  hardlink, expected content) or the pod refuses to start;
- a redaction canary `context/redaction-canary` (0400);
- relay and worker keys in `Memory`-medium emptyDirs
  (`/run/hermes-webui-hux/relay-key`, `/run/hermes-hux-worker/worker-key`,
  0400) — regenerated on every pod restart, never touching the PVC.

The subject is stable across restarts, derived from a per-pod random key,
and never derived from the Keycloak subject.

### Env contracts

`hux` sidecar: `HUX_FLAGS` lists all twelve cards (`hux.foundation,
hux.activity_timeline, hux.memory_control, hux.projects, hux.artifacts,
hux.autonomy, hux.friendly_modes, hux.multimodal, hux.research,
hux.onboarding, hux.privacy, hux.release_followthrough`);
`HUX_RELAY_KEY_FILE`, `HUX_WORKER_KEY_FILE`, `HUX_SUBJECT_BINDING_FILE`,
`HUX_CONTEXT_KEY_FILE`, `HUX_CANARY_FILE`; rate limits
`HUX_READS_PER_MINUTE=600`, `HUX_WRITES_PER_MINUTE=120`; and build
provenance `HUX_IMAGE_TAG` / `HUX_IMAGE_DIGEST` bound by Flux setters
(`$imagepolicy hermes:hermes-webui-release:tag` / `:digest` comments).

Agent container: `HUX_BASE_URL=http://127.0.0.1:8790`,
`HUX_RUNTIME_ENABLED=1`, and `HUX_TOOL_ENFORCEMENT=0` — the first rollout
is observe-only until approval parking/resume is connected to the
upstream tool loop and proven live — plus `HUX_WORKER_KEY_FILE`,
`HUX_SUBJECT_FILE`, `HUX_CONTEXT_KEY_FILE`, `HUX_TIMEOUT_SECONDS=3`.

### Agent runtime plugin

`services/hermes/plugins/hux-runtime/` renders into the
`hermes-hux-runtime-plugin` ConfigMap (`services/hermes/kustomization.yaml`)
and mounts read-only into the agent at `/opt/data/plugins/hux-runtime`:
`runtime.py`, `tool_policy.py`, `context_ids.py`, `plugin.yaml`
(`on_session_end` hook + `tool_execution` middleware, fail-closed), and a
vendored `hux_hook/` package kept in parity with the reviewed worker hook
library (parity is delivery-gated).

### Router boundary

`services/hermes/router/main.go` deletes EVERY inbound header whose name
starts with `x-hux-` (case-insensitive) at the authenticated tenant
boundary — in the same Director that strips OAuth headers — before
asserting `X-Hermes-Tenant-Identity: slot-N`. Regression tests:
`services/hermes/router/main_test.go`. Defense in depth: the sidecar is
loopback-only anyway, and the in-pod WebUI BFF asserts its own `X-Hux-*`
from trusted context.

### WebUI image (`dockerfiles/Dockerfile.hermes-webui`)

The image ships the service and contracts at `/opt/hermes-hux` (`hux/`
package + `contracts/`, `HUX_CONTRACT_DIR`), the BFF proxy
(`api/hux_bff.py`) and trusted context attachment (`api/hux_context.py`,
`hux.webui_context.v1`), the bootstrap bridge
(`static/hux/bootstrap.js/.css`) and the wave A/B/C runtime bundles. A
build-time gate compiles the package and constructs the router with
`HUX_FLAGS=""`: images carry the code but no activation — flags live only
in manifests.

### Release rendering (`ci/scripts/hermes_webui_flux_release.py`)

The release script updates one-or-two WebUI image consumers in
`StatefulSet/hermes-chat-tenant` (the `webui` container, plus the `hux`
sidecar when staged) and exactly one in `Deployment/hermes`, and binds
`HUX_IMAGE_TAG`/`HUX_IMAGE_DIGEST` to the released tag and digest whenever
the sidecar fields are present — failing when the binding fields are
incomplete (exactly one of each, or none at all).

### HUX-12 evidence trust (`hux/release_security.py`, `hux/releases.py`, `47165274`)

Release follow-through has a dedicated fourth trust class, `evidence`:

- File-only authentication: `HUX_RELEASE_EVIDENCE_KEY_FILE` must be a 0400
  regular file (symlinks refused) holding a key ≥ 32 chars, and
  `HUX_RELEASE_EVIDENCE_POLICY_FILE` a strictly validated
  `hux.release_evidence_policy.v1` allowlist (≤ 16 workloads,
  credential-free HTTPS URLs, exact field sets, bounded evidence age).
  Without both healthy the HUX-12 capability is off entirely — fail closed.
- Worker trust is read-only on releases (list/read in `WORKER_ROUTES`); it
  can no longer create releases or transitions.
- Only router trust creates the single `reviewed` proposal per scoped
  workload commit; only evidence trust performs transitions, and its route
  allowlist in `hux/http.py` blocks everything else.
- Transition evidence binds exactly: `merged` the 40-hex merge commit;
  `built` the CI URL plus image ref / image digest / Harbor digest all
  equal; `deployed` a `main@sha1:<40-hex>` Flux revision; `converged`
  pod digest == image digest; `live_verified` one passing health receipt;
  `rolled_back` an exact target digest.
- Ledger entries are hash-chained (`previous_hash`/`entry_hash`) and
  verified on every read; `Idempotency-Key` replays return the recorded
  view with `HUX-Replayed: true`.

### Worker state

`worker.bstein.dev` (the `hermes` Deployment and the execution-worker
manifests) has NO HUX sidecar, plugin or environment. Cross-surface
continuity is not claimed; HUX is chat-staged only.

### Test state

Verified 2026-08-24 with the CI interpreter:
`python -m pytest testing/tests/test_hermes_hux_*.py` = 646 passed
(20.8 s). Delivery gates in `testing/tests/test_hermes_hux_delivery.py`
assert the staged topology against the manifests themselves:
all-or-nothing activation, loopback-only sidecar on the reviewed WebUI
image, least-privilege storage and key mounts, init-container identity
persistence + transport-key rotation, file-backed authentication inputs,
and plugin/vendored-hook render parity. UI card model suites:
`testing/tests/test_hermes_hux_ui_*.py`, each driving a Node harness
(`test_hermes_hux_ui_*.mjs`). Release rendering:
`test_hermes_webui_release.py` (18) and `test_hermes_image_automation.py`
(2).

### Superseded statements

- "`/opt/data/hux` on the tenant PVC; SO-45 separate uid, mode 0700" —
  superseded. Storage is the shared RWX claim with kubelet `subPathExpr`
  isolation above; every container runs uid 10000 and the boundary is
  mount scoping, not a uid split. See `THREAT-MODEL.md`, "Integration
  amendments (2026-08-24)".
- "`HUX_DATA_ROOT` (tenant PVC, default `/opt/data`)" — now
  `/var/lib/hux/store` on the dedicated claim.
- "runs … inside each `hermes-chat-tenant` pod (and the Worker)" — the
  Worker has no HUX yet.
- "Image, manifest, NetworkPolicy and Flux wiring are Codex's" — landed
  in-repo with the chain above.
- Open item "SO-53 rate limits" — implemented in `hux/http.py`; SO-46 and
  SO-48 remain open (see the threat model amendment).


## Completion increments (2026-08-24, later — local chain, staged, not deployed)

Everything below is committed locally on top of the integrated topology and
verified by the full gate (826 passed / 0 failed across
`test_hermes_hux_*.py`, `test_hermes_webui_hux_*.py`, image-automation,
webui-release, auto-router and chat-quality suites, after one fieldRef
tolerance fix in the voice gate).

- **Topology gates are all-or-nothing adaptive** (`test_hermes_hux_delivery.py`,
  `test_hermes_image_automation.py`): with no `hux` sidecar the manifests must
  contain zero partial HUX wiring; with it, the full strict boundary applies.
  This lets the source chain merge and build before activation.
- **HUX-06 adoption** (`services/hermes/plugins/auto-router/hux_mode.py`):
  the auto-router consults the loopback HUX service (worker trust, the same
  `hux.context.id.v1` HMAC derivation) for the conversation's selected
  friendly mode: fast→auto/fast, thoughtful/research→auto/deep,
  create→auto/balanced, private→the local route only. Explicit UI picks win;
  every failure falls through to previous behaviour. Delivered through the
  auto-router ConfigMap with a degrade-to-noop import shim.
- **Private mode is enforced upstream** (`hux/policy.py`, `hux/budgets.py`):
  network, web_search, send_message, shell and delegate are refused by the
  approval resolver and re-refused by the gate for a conversation whose
  stored mode is `private`, regardless of autonomy level.
- **HUX-04/08 emitters** (`plugins/hux-runtime/emitters.py`): after a
  successful `write_files` tool the plugin registers the written file as an
  artifact (create or immutable version, ≤1 MiB, deterministic idempotency);
  after successful `network`/`web_search` tools it records up to three
  deduplicated `web` sources from the real output. Fail-open by construction.
- **HUX-03 message-text search** (`hux/organization.py`):
  `GET /hux/v1/search?include=message_text` scans stored message events of
  the 100 most recently active candidates; forgotten, private, restricted
  and fully-redacted content never matches; deterministic cursor pagination.
  The default search response is unchanged (the shipped UI requires
  `message_text` in `not_indexed`).
- **HUX-12 producer** (`dockerfiles/hermes-hux-foundation/hux_producer/`,
  100% line+branch, 71 tests): pure verifier binding review URL prefix →
  Jenkins SUCCESS/revision → immutable Harbor tag/digest → Flux
  `main@sha1:` revision with pin containment → desired workload image →
  every Ready pod imageID → bounded-age health receipt → rollback target;
  HTTPS-only injectable collectors (SA token for the API); evidence-trust
  driver posts exactly one If-Match transition. Cluster layer staged in
  `chat-statefulset.yaml` (sidecar + tolerant key init),
  `hux-evidence-policy.json`, `hux-evidence-rbac.yaml` (read-only pods/
  statefulset + the single named Kustomization), tenant egress to the API
  ClusterIP and traefik. The key reaches only the `hux` and
  `hux-evidence-producer` containers.

### HUX-12 enablement runbook (fail-closed until every step is done)

1. `vault kv put kv/atlas/hermes/hux-evidence evidence_key="$(openssl rand -hex 32)"`
2. Add to the chat tenant pod annotations (Flux commit):
   `vault.hashicorp.com/agent-inject-secret-hux-evidence-key: kv/data/atlas/hermes/hux-evidence`
   plus the matching template emitting `.Data.data.evidence_key`.
3. Create ConfigMap `hermes-hux-evidence-scope` with
   `HUX_PRODUCER_PROJECT_ID` / `HUX_PRODUCER_CONVERSATION_ID` for the release
   conversation (visible in the UI once a reviewed proposal exists).
4. Only then does `release_security.configured` turn HUX-12 on; the sidecar
   loop starts posting verified transitions.

### Staged release sequence (agreed)

1. Push the source tip to main: the full local chain plus a hold commit that
   restores `services/hermes/{chat-statefulset,chat-pvcs,chat-configmap,kustomization,networkpolicy}.yaml`
   to the live state (keeping only the auto-router `hux_mode.py` ConfigMap
   key) and removes the evidence policy/RBAC files — so Flux applies nothing
   and CI builds the reviewed WebUI image.
2. Trigger the WebUI build at that exact SHA; wait for Flux digest
   convergence.
3. Re-apply the activation topology pinned to the new tag/digest with
   `HUX_FLAGS=hux.foundation`, `HUX_TOOL_ENFORCEMENT=0`, and a RollingUpdate
   partition canarying ordinal 3 first.
4. HUX-11 live gates → card-by-card flag enablement with lifecycle
   acceptance → enforcement after park/resume + stop receipts pass live →
   HUX-12 after the runbook above → Worker separately.


## Live canary evidence (2026-08-24 ~09:05 UTC)

Foundation canary LIVE and verified on hermes-chat-tenant-3:
- Commits: source 2f535d3a (build 21), Flux digest 1d29c25c, activation 39a54482 (partition 3, HUX_FLAGS=hux.foundation, enforcement 0).
- Image: registry.bstein.dev/bstein/hermes-webui:git-2f535d3a...-build-21-release@sha256:e5b9b2fa8296a7b8a6065c63e7b288cd3305d70a3acc4312055b819bb5f64c04 on webui + hux + hux-evidence-producer (pod imageIDs verified); agent digest unchanged (sha256:c64c06e6...).
- canary_verify: 15/15 PASS (all containers ready, ordinals 0-2 untouched and huxless, loopback /healthz ok, subject binding present, public https://chat.bstein.dev/healthz 200, Flux Ready).
- Live route matrix: unauth 401, bad key 401, wrong slot 401, relay capabilities 200 with only HUX-11 enabled, worker capabilities 200, disabled card 404 flag_off. Subject usr_224348fa... consistent with the HMAC binding.
- Identity hashes for the restart-persistence gate: context-key 8a1c474d..., subject c4ebe353... (compare after the next roll).
- Rollback target: WebUI build-18 digest sha256:df91f8a3... (previous fleet image); activation rollback = revert 39a54482.
- Retention scheduler reports disabled until hux.privacy is enabled (by design).

Next: Wave-A canary flags (activity_timeline, projects, privacy, memory_control) on ordinal 3, lifecycle gates per card, then widen partition.


## Wave-A live gates (2026-08-24 ~09:12 UTC) — PASS

Canary rolled to ee330822 flags; restart persistence PROVEN (context-key and
subject hashes identical across the roll). Live lifecycle 13/13 on the
canary loopback: capabilities advertise exactly HUX-01/02/03/10/11; real
project + conversation created; worker-emitted event readable by relay;
memory proposed -> approved -> forgotten with the tombstone effective in the
listing; privacy policy served; conversation forget tombstones the record;
retention scheduler enabled and healthy. Next: Wave-B flags (artifacts,
research, friendly_modes, multimodal, onboarding) on the canary.


## Wave-B live gates (2026-08-24 ~09:20 UTC) — PASS 18/18

Canary at dc2b3643 flags. Live on the canary loopback: capabilities show the
Wave-B set with HUX-07 correctly fail-closed (depends_on HUX-05); artifact
create -> immutable version -> content -> diff -> promotion; research source
-> passage -> citation attached to an authoritative message id with the
citation strip readable; provider-neutral mode catalog, thoughtful selected
under If-Match and read back by worker trust; suggestion shown for the real
after_artifact context, explicit clicked dismissal recorded, re-show
suppressed, suppression state durable. Next: hux.autonomy (+ multimodal
unlock) on the canary, enforcement still 0.


## Autonomy-wave live gates (2026-08-24 ~09:30 UTC) — PASS 15/15

Canary at f2f54033 flags. Live on the canary loopback: HUX-05 park ->
human-only approval (worker 403) -> gate releases exactly once -> consumed on
the second ask; denial stays denied; browser cannot write stop receipts;
gateway receipts are honest (process_registry_empty=false ->
failed_to_cancel, superseded by a true empty-registry cancellation). HUX-07
unlocked by the dependency chain and media registration required a REAL
approved artifact_write action before accepting upload metadata.

ENFORCEMENT: park/resume and gateway-owned receipts passed live, so
HUX_TOOL_ENFORCEMENT=1 ships to the canary (partition still 3; ordinals 0-2
remain untouched). Tenant-3 tool calls now require HUX release for real.
Remaining: real-traffic validation of the approvals UX on tenant-3, then
partition 3 -> 0 to take flags+enforcement fleet-wide; HUX-12 needs the
Vault evidence key (runbook above); Worker integration separate.


## Enforcement live on the canary (2026-08-24 ~09:35 UTC)

Canary at 4a6d4fdf+17037773: HUX_TOOL_ENFORCEMENT=1 and HUX_RUNTIME_ENABLED=1
verified in the live hermes container; identity hashes unchanged across a
third roll (8a1c474d/c4ebe353); /healthz ok with retention healthy; all five
containers ready, zero restarts. The earlier hux exit-137s were transient
probe kills under roll churn (threaded server, no OOM/liveness events, clean
logs) — bump the hux exec probe timeoutSeconds 3->5 with the
partition-widening commit. Voice fixes pushed as 91cfb901 (continuous mic,
barge stitching, 1.15x TTS); build 22 queued — its digest bump rolls the
fleet including this canary.


## FLEET LIVE (2026-08-24 ~12:55 UTC) — HUX-01..11 live on all four tenants

Widening commit 5fc3e870 applied, Flux Ready=True. Verification 39/39:
every tenant 5/5 ready on webui build-22
(git-91cfb901...-build-22-release@sha256:2313d0d5...) for webui + hux +
producer; agent digest unchanged everywhere; hux /healthz ok per pod;
HUX_TOOL_ENFORCEMENT=1 fleet-wide; HUX_IMAGE_TAG env now consistent with the
image (block-style setters); four DISTINCT HMAC subjects (per-pod isolation
proven); public /healthz 200; zero container restarts under the 5s probe
budgets (the 2s exec budget had produced 405 readiness timeouts in 135min and
stalled the digest roll — root-caused and fixed). Voice fixes (continuous
mic, barge stitching, 1.15x TTS) shipped in the same image.

Remaining, deliberately gated:
- HUX-12 enablement: Vault evidence key + scope ConfigMap (runbook above);
  everything else is live and fail-closed.
- Worker (worker.bstein.dev): separate integration + canary, not started —
  cross-surface continuity is NOT claimed.
- Real-traffic approvals-UX feedback from the operator; rollback = one
  revert of 5fc3e870 (re-partition) and/or HUX_TOOL_ENFORCEMENT=0.


## Enforcement rolled back to observe-only (2026-08-24 ~13:3x UTC, 1793903f)

Live traffic on the enforced fleet showed the first defaults blocking core
assistant faculties (skills listing and the sandboxed Python fell into the
unknown->external_side_effect always-ask bucket, browsing was deny-by-
default, and no approval prompt surfaced in the chat flow). Fleet rolled to
HUX_TOOL_ENFORCEMENT=0 with a corrected capability map (python_sandbox =
internal shell; skills/todo/clarify/vision = reads; browser/web = network
medium; image_gen = artifact_write via the trusted broker; unknown tools
stay fail-closed). Verified live: all tenants 5/5, enf=0 everywhere.

Re-enforcement checklist (in order):
1. Approvals UX must surface prompts inline in the chat/voice flow.
2. Default grants reviewed against the corrected capability map (network
   under safe autonomy should ask, not deny, for the chat surface).
3. Canary re-enable on one ordinal first (partition), then fleet.


## Round-2 fleet live (2026-08-24 ~14:4x UTC) — build 23 everywhere

All four tenants verified on
git-043aa9ee...-build-23-release@sha256:b3b5830d... (webui + hux + producer;
agent digest unchanged; per-pod hux /healthz ok; HUX_IMAGE_TAG env consistent
with the image — the block-style setters were rewritten by Flux this cycle,
confirming the flow-mapping fix; four distinct HMAC subjects; public
/healthz 200; enforcement observe-only BY DESIGN pending the approvals-UX
work). One transient registry 404 during tenant-1's pull (traefik-VIP flake
class) cleared by kubelet retry in ~2 min — manifests verified intact.

Build 23 contents: voice clipping fix (young-utterance endpoint hold,
speculate min-speech, unconditional resume, capped noise floor + capture-
continuity probe), always-stitch within 20s of a voice barge, HUX drawer
theme-token bridge (side-panel dark-box fix), full-screen conversation mode
(orb, live captions, mute/exit, a11y), renderer block-style HUX binding.

Definitive remaining list: HUX-12 Vault key + scope ConfigMap (operator
runbook above); approvals-UX inline prompts then re-enforcement via canary;
Worker (worker.bstein.dev) integration as its own lane; titan-iac main CI
baseline confirmation post node-skip hardening.


## Build 24 fleet-live + cluster read + worker staged (2026-08-24 ~16:xx UTC)

Build 24 (d041f1d1 -> sha256:1ea5b4ad, promote 04f060db) on all four
tenants + the worker agent. Fleet verify 34/34 meaningful (enforcement
observe-only by design; Flux Ready=True after reconcile). Worker gates 6/6
with identity PERSISTED across two rolls (hashes 7fa1c7d3/532b9745 unchanged)
and the auth matrix exact (unauth/badkey/wrongslot 401, worker caps=HUX-11).

Shipped this cycle:
- Voice round 3: caption body-only extraction, dynamic endpointing, both
  captions touch-scrollable, errors never spoken + resyncCapture, workspace
  toggle in the sidebar rail, the false 'session unavailable' toast (409 from
  a fail-closed profile-alias race - fixed at the cause), barge cut-point
  marker, full language routing, speaking<->thinking cycling.
- Approvals UX: inline pending-approval dock above the composer; the default
  grant matrix now asks (not denies) for network/web_search below autonomous,
  SO-39 intact. Re-enforcement checklist items 1+2 done code-side.
- cluster_read: read-only Kubernetes visibility LIVE and verified in-chat
  (pods/nodes/flux read; secrets refused; RBAC excludes Secrets). Required a
  tenant egress fix to the real apiserver endpoints (192.168.22.11-13:6443,
  which the except-block dropped post-DNAT) and a large-node-listing handler
  hardening (ed0bc30d).
- Worker staged inert-HUX (foundation only, enforcement 0) - activation is a
  reviewed one-line flip per WORKER-PLAN.md.

Transient registry 404 during tenant-3's pull recurred (3rd time; VIP/registry
flake class) and self-cleared - worth a registry-reliability follow-up.

Remaining: HUX-12 Vault key (operator); re-enforcement canary (approvals UX
now exists - operator tries the prompts, then canary re-enable per checklist);
worker HUX activation flip (WORKER-PLAN TODO 2); cluster_read handler
hardening activates on the next pod roll.


## Final webui build-29 live + verified (2026-08-24)

All tenants on webui build-29 (sha256:6887c2ad, source 4d93ef5a), verified in
the served bundle: Workspace toggle in the nav (desktop rail + mobile
titlebar via navIconToggle/titlebarHost, no composer-obstructing floating
pill - fixed the mobile regression that pushed the input controls off
screen); conversation orb watermark = the Hermes CHARACTER avatar
(hermes-agent-192.png), staff SVG removed; 'hands-free' renamed to
'Conversation mode'; reply-language routing density-gated
(resolveReplyLanguage) so English no longer gets a foreign voice. Also
carries: interim-ack tail flush, natural thinking fillers, one unified audio
sink + output-device selector, the 'Something went wrong' error-veto, and
the cluster_read large-listing fix.

STT: large-v3-turbo is live for accuracy (already correcting proper nouns);
the beam-2 + name-prompt tuning is pending a successful hermes-voice-image
(stt) build - the build keeps failing on an IPv6 route to azureedge while
downloading the Whisper models (a more persistent flake than the ghcr.io
DNS one). Durable follow-up recommended: mirror the ghcr.io webui base image
and the Whisper models into registry.bstein.dev so builds stop depending on
external DNS/IPv6 (3 webui + 3 stt builds lost to these flakes).

Gated on the operator: HUX-12 Vault evidence key; re-enabling enforcement
(approvals UX is live); worker HUX activation flip.