atlas-iac/docs/hux/HANDOFF.md
jenkins f2f54033a2 docs(hux): record Wave-B live gate evidence
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
2026-08-24 06:13:57 -03:00

28 KiB
Raw Blame History

hermes-next: Claude backend handoff ledger

Branch feature/hermes-next-hux (origin, pushed after every card); draft PR #55 atlas/titan-iac#55 (opened through the SCM broker, never merged by Claude). Board hermes-next on the operator Hermes Kanban; cards HUX-01..HUX-12 are parked (blocked) on purpose so the CLI lane runner never auto-claims them — the Claude coordinator and the Codex root session are the only implementers. Contract: hux.v1 1.0.0 (services/hermes/contracts/hux/VERSION, docs/hux/ADR-0001-hux-v1-contract-freeze.md). Threat model: docs/hux/THREAT-MODEL.md (SO-01..SO-54). Data model: docs/hux/DATA-MODEL.md.

Run everything with the CI interpreter: python -m pytest testing/tests/test_hermes_hux_*.py -q and python -m coverage run --branch --source=dockerfiles/hermes-hux-foundation -m pytest testing/tests/test_hermes_hux_*.py.

Codex integration requirements common to every card

(Superseded 2026-08-24 where it conflicts with "Integrated topology" at the end of this file: the manifests, plugin, router headers and Flux wiring are now in-repo, the data root moved off the tenant PVC, and the Worker has no HUX yet.)

  • Package dockerfiles/hermes-hux-foundation/hux/ runs as python -m hux.server inside each hermes-chat-tenant pod (and the Worker) on 127.0.0.1:8790. Image, manifest, NetworkPolicy and Flux wiring are Codex's (SO-01, SO-02, SO-45: separate uid, /opt/data/hux mode 0700).
  • Env: HUX_DATA_ROOT (tenant PVC, default /opt/data), HUX_TENANT_SLOT (pin, SO-03), HUX_FLAGS (comma list, default empty = everything off), HUX_RELAY_KEY (Telegram relay), HUX_WORKER_KEY (Worker), HUX_BUILD_COMMIT, HUX_IMAGE_DIGEST, HUX_CONTRACT_DIR (schemas path in the image; default resolves relative to the repo layout).
  • Router → service headers (SO-04): X-Hermes-Tenant-Identity: slot-N, X-Hux-Subject: usr_<hash>, X-Hux-Surface: chat|telegram|voice|api|worker, optional X-Hux-Trust: relay|worker + X-Hux-Relay-Key. The router must strip any inbound X-Hux-* before setting its own.
  • First client call: GET /hux/v1/capabilities → which cards/routes are on. Disabled or unknown → 404 (flag_off vs not_found only in the body).
  • Mutations: If-Match: <revision> on revisioned records (409 on mismatch); Idempotency-Key on creates. Errors are hux.error.v1.
  • Fixtures Codex codes against: services/hermes/contracts/hux/examples/*.json (validated in CI by test_hermes_hux_contract_schemas.py).

HUX-11 shared foundation — DONE (backend)

Commits b4145861 (contract freeze 1.0.0), 6a5e0d87 (service core)
Files services/hermes/contracts/hux/{common,identity}.schema.json + examples, VERSION; dockerfiles/hermes-hux-foundation/hux/{__init__,errors,identity,flags,store,audit,http,foundation,server,contracts,rules}.py; docs/hux/ADR-0001-*.md, THREAT-MODEL.md, DATA-MODEL.md
Tests testing/tests/test_hermes_hux_contract_schemas.py (61), test_hermes_hux_contract_foundation.py (30): identity header rejection matrix, relay/worker keys, slot pinning, fail-closed flag chains, capabilities record validity, tenant-scoped paths + traversal rejection, revisions/conflicts, torn-ledger recovery, blobs, manifest, concurrent writers, audit rows, flag-off == not-found, 401 never touches storage, real HTTP server JSON + SSE, loopback-only bind, all errors validate, ≤500 LOC guard
Coverage 99% line / 98% branch over the package
Contract 1.0.0
Flag hux.foundation (root of every dependency chain)
Risks Rate limiting (SO-53) and hash-chained audit (SO-46) not yet implemented — tracked for the Wave A review; relay callers still carry X-Hux-Subject (deviates from SO-06; the slot→owner mapping lives in the router, so the header is redundant but harmless and lets the service pin the subject)
Codex needs Wire the service into the tenant pod + Worker, NetworkPolicy, router header contract above, HUX_FLAGS=hux.foundation for the first canary; testing/quality_contract.json may add dockerfiles/hermes-hux-foundation/**/*.py to line_limit_globs and the two moved modules to managed_modules (Claude does not edit that file)

Wave A backends — DONE, awaiting adversarial review and Codex integration

Full suite: testing/tests/test_hermes_hux_*.py = 325 tests, 99% line / 99% branch over dockerfiles/hermes-hux-foundation/hux/ (every family module 99100%). Every module ≤ 500 LOC (guarded by tests). Contract revised additively to 1.1.0 (ADR-0001).

Card Commit Files Tests Flag Codex needs
HUX-01 activity timeline b8fb72fb hux/events.py, hux/redaction.py test_hermes_hux_contract_events.py (34): ordering, idempotency, replay/reconnect, redaction by surface, cancellation receipts, cross-tenant hux.activity_timeline Router forwards Last-Event-ID; telegram/voice surfaces get partial redaction; agent hook posts events with Idempotency-Key; HUX_CANARY_FILE=/opt/data/.env so secrets are scrubbed
HUX-02 memory center b8fb72fb hux/memory.py test_hermes_hux_memory_ledger.py (20), test_hermes_hux_memory_retrieval.py (7): no-store, suggest-only, edit/supersede, forget, retrieval removal, export, If-Match hux.memory_control (needs hux.privacy) Agent memory tool must call hux.memory.retrieve semantics (tombstones first) instead of the upstream memory file; UI per-message controls call /memory/{id}/{action} with If-Match
HUX-10 privacy b8fb72fb hux/privacy.py test_hermes_hux_privacy_topics.py (19), test_hermes_hux_privacy_retention.py (6) hux.privacy Schedule run_retention daily (no thread inside the service by design); surface HUX-Audit-Stale; show notices and post chosen
HUX-04 artifacts 9f0ffe03 (+ upload caps, project check in the 1.1.0 batch) hux/artifacts.py, hux/diffs.py test_hermes_hux_artifact_versions.py, test_hermes_hux_artifact_auth.py (immutable versions, concurrency, lineage forgery 404, caps 413, diffs, promotion) hux.artifacts (needs hux.projects) Version content served with nosniff + attachment; uploads ≤ 25 MiB content via base64; sharing intentionally not implemented (access.mode always owner)
HUX-08 research 9f0ffe03 hux/research.py test_hermes_hux_research_citations.py, test_hermes_hux_research_notebook.py (dedupe, integrity, notebook state machine) hux.research Service never fetches URIs (SO-19): the web tool records sources/passages after its own fetch; citation strip = GET /messages/{id}/citations (passages + sources embedded)
HUX-05 autonomy 3ebee2cc hux/policy.py, hux/budgets.py test_hermes_hux_policy_matrix.py (22), _approvals.py (22), _receipts.py (13) hux.autonomy (needs hux.activity_timeline) Agent hook: POST /approvals before any external side effect, POST /runs/{id}/gate immediately before executing with the canonical argument hash, POST /runs/{id}/budget per turn, POST /runs/{id}/stop with process_registry_empty from the real process registry; decisions only from human surfaces (worker trust → 403)
HUX-03 organization API 3ebee2cc hux/organization.py test_hermes_hux_contract_organization.py (14) hux.projects Codex owns the UI + migration of upstream WebUI projects; search over message_text is a later increment

Known gaps carried to the Wave A review: rate limiting (SO-53), hash-chained audit (SO-46), retention scheduler ownership (Codex cron vs service thread), approval expiry applied lazily on read, linear scans for idempotency/dedupe (fine at documented caps).

Wave A adversarial review — CLOSED

A fresh reviewer attacked HEAD 1b1a14e9 (tenant isolation, privacy, autonomy, rollback) and reported 13 findings; the repro harness lives outside the repo. Every finding is closed by a regression test that cites it:

# Severity Finding Fix commit
F1 critical worker/api trust could PUT /policy and self-approve 6964a9d8 — policy writes and allow grants are human-surface only
F2 high worker trust could read every tenant record (SO-08) 6964a9d8flags.WORKER_ROUTES allowlist enforced in Router.dispatch; unexpected exceptions become audited 500 records
F3 high memory edit skipped topic/sensitivity/private-mode gates 681b0408 — one _classify path for proposals and edits
F4 high a session approval for an external effect released unrelated later effects 6964a9d8 — external effects match only the same run + argument hash; gate conversation from the run, not the body
F5 high seq duplicated after a crash between append and checkpoint 681b0408 — seq = max(checkpoint, ledger tail + 1) under the lock
F6 medium gate ignored budget exhaustion 6964a9d8
F7 medium stale unconditional write could resurrect a forgotten memory 681b0408 — re-read under lock, always write with the loaded revision
F8 medium any caller could assert process_registry_empty; failed receipts were sticky 6964a9d8 — gateway (worker trust) only; failed receipts supersedable
F9 medium secrets stored verbatim in titles/passages/claims/notebooks; forget left the title 681b0408 — scrub applied; forget blanks the document. Artifact bodies stay verbatim (user-owned) but are forced restricted and audited (a75ca299)
F10 medium SO-46/48/53 claimed but absent threat model amended (74c65499): tracked as open, not claimed
F11 low stored receipts carry revision the schema forbade 6964a9d8 — optional revision on the receipt
F12 low capabilities advertised routes without handlers 6964a9d8 — HUX-06/09/12 routes empty until shipped
F13 low misc (healthz version, ghost conversations from notices, idempotency outside the lock, artifact titles not searchable, unnormalised paths, multi-hash once approvals) 6964a9d8 + 681b0408

Suite after repairs: 377 tests, 99% line / 99% branch over hermes-hux-foundation and hermes-worker-hux. Full repo gate: same 13 pre-existing, unrelated failures as main.

Worker hook library — DONE

Commit aeef0f84
Files dockerfiles/hermes-worker-hux/hux_hook/{__init__,client,hooks}.py, NOTES.md (wiring guide for the runtime patch, which is Codex's file)
Tests test_hermes_hux_policy_hook.py (11), test_hermes_hux_contract_hook.py (10) — real service in-process, end-to-end approval → human decision → gate released once, canary never persisted, unreachable service fails closed for side effects
Codex needs Agent container env HUX_BASE_URL, HUX_TENANT_SLOT, HUX_SUBJECT, HUX_WORKER_KEY; call order per NOTES.md: before_tool → execute only on proceedafter_tool; record_spend per turn; on_stop returning None means the stop is not done

Conversation privacy state — DONE

GET /hux/v1/conversations/{id}/privacy (HUX-10, worker-callable) returns forgotten / memory_disabled / topics / mode / memory_writes_allowed; hux_hook.memory_gate now reads it and fails closed. Tests: test_hermes_hux_privacy_topics.py::test_conversation_privacy_state_route, hook memory-gate test extended.

Open items (not blockers for Codex integration)

  • SO-46 hash-chained audit, SO-48 single-writer lock. (SO-53 rate limits shipped since: hux/http.py RateLimiter, per-subject read/write buckets, 429 + Retry-After, tuned by HUX_READS_PER_MINUTE / HUX_WRITES_PER_MINUTE — see the integrated-topology section.)
  • Retention scheduler: privacy.run_retention is on-demand; Codex decides cron vs sidecar.
  • Search over message_text; artifact sharing (shared_readonly).
  • testing/quality_contract.json line-limit globs / managed modules for the two new package paths (Codex's file).

Integrated topology (2026-08-24, local chain, staged — not deployed)

Everything in this section exists only on local main. origin/main is still 5558c24f; Flux reconciles origin, so none of this is running anywhere. Chat is staged; the Worker is untouched. The chain since origin (oldest first): 964103f0 merge of the hardened foundation, 438180a9 tool runtime gated through HUX, a1070449 vendored runtime hooks, 53d7c2c5 backend families complete, 31c5eae5 chat router image automation, 1eda927b trusted HUX workspace in the WebUI, 47165274 HUX-12 evidence trust isolation, b66c762f HUX card UI models, bd63b568 router strips inbound X-Hux-*, 9e99470f one-or-two WebUI consumers in release rendering, 0b56d04e quality-contract gates, 71c05cf9 staged sidecar activation topology, 659f70ee WebUI integration contract notes, 13359769 all-or-nothing topology gates.

Pod topology (services/hermes/chat-statefulset.yaml)

Each hermes-chat-tenant pod runs a fourth container, hux, on the SAME reviewed WebUI image as the webui container (registry.bstein.dev/bstein/hermes-webui, Flux $imagepolicy hermes:hermes-webui-release), started as python -m hux.server with PYTHONPATH=/opt/hermes-hux. It binds 127.0.0.1:8790 (HUX_BIND, HUX_PORT), runs as uid/gid 10000 with readOnlyRootFilesystem, all capabilities dropped, tmpfs /tmp, and /healthz probes. HUX_TENANT_SLOT=slot-N comes from the pod ordinal, as for the agent. Port 8790 appears in no Service or NetworkPolicy; loopback is the only path (delivery-gated).

Storage: shared RWX claim, kubelet-scoped subtrees

The plan's /opt/data/hux on the tenant home PVC is gone. Storage is a standalone ReadWriteMany PVC hermes-chat-hux-data (storageClass astreae, 10Gi, services/hermes/chat-pvcs.yaml) shared by all tenant pods, with per-pod isolation enforced by kubelet subPathExpr: $(POD_NAME) mounts:

  • hux container: /var/lib/hux = its own $(POD_NAME) subtree; HUX_DATA_ROOT=/var/lib/hux/store.
  • hermes (agent) container: read-only /run/hermes-hux-context = $(POD_NAME)/context and /run/hermes-hux-subject = $(POD_NAME)/binding only. It has NO mount of /var/lib/hux/store; records are reachable from the agent only through the loopback API.
  • webui container: read-only /run/hermes-hux-context only.

No container in any pod mounts another pod's subtree.

Identity provisioning (init-hux-runtime)

A dedicated init container (root, capabilities dropped to CHOWN/DAC_OVERRIDE/FOWNER) provisions, idempotently per pod:

  • 0700 roots binding/, context/, store/, owned 10000:10000;
  • a persistent 32-byte context key context/context-key (0600), created once and kept across restarts;
  • an immutable subject binding binding/subject (0440, created O_EXCL): usr_ + HMAC-SHA256(context-key, "hux.subject.id.v1\0slot-N"). On restart the file is verified (regular file, uid 10000, mode 0440, one hardlink, expected content) or the pod refuses to start;
  • a redaction canary context/redaction-canary (0400);
  • relay and worker keys in Memory-medium emptyDirs (/run/hermes-webui-hux/relay-key, /run/hermes-hux-worker/worker-key, 0400) — regenerated on every pod restart, never touching the PVC.

The subject is stable across restarts, derived from a per-pod random key, and never derived from the Keycloak subject.

Env contracts

hux sidecar: HUX_FLAGS lists all twelve cards (hux.foundation, hux.activity_timeline, hux.memory_control, hux.projects, hux.artifacts, hux.autonomy, hux.friendly_modes, hux.multimodal, hux.research, hux.onboarding, hux.privacy, hux.release_followthrough); HUX_RELAY_KEY_FILE, HUX_WORKER_KEY_FILE, HUX_SUBJECT_BINDING_FILE, HUX_CONTEXT_KEY_FILE, HUX_CANARY_FILE; rate limits HUX_READS_PER_MINUTE=600, HUX_WRITES_PER_MINUTE=120; and build provenance HUX_IMAGE_TAG / HUX_IMAGE_DIGEST bound by Flux setters ($imagepolicy hermes:hermes-webui-release:tag / :digest comments).

Agent container: HUX_BASE_URL=http://127.0.0.1:8790, HUX_RUNTIME_ENABLED=1, and HUX_TOOL_ENFORCEMENT=0 — the first rollout is observe-only until approval parking/resume is connected to the upstream tool loop and proven live — plus HUX_WORKER_KEY_FILE, HUX_SUBJECT_FILE, HUX_CONTEXT_KEY_FILE, HUX_TIMEOUT_SECONDS=3.

Agent runtime plugin

services/hermes/plugins/hux-runtime/ renders into the hermes-hux-runtime-plugin ConfigMap (services/hermes/kustomization.yaml) and mounts read-only into the agent at /opt/data/plugins/hux-runtime: runtime.py, tool_policy.py, context_ids.py, plugin.yaml (on_session_end hook + tool_execution middleware, fail-closed), and a vendored hux_hook/ package kept in parity with the reviewed worker hook library (parity is delivery-gated).

Router boundary

services/hermes/router/main.go deletes EVERY inbound header whose name starts with x-hux- (case-insensitive) at the authenticated tenant boundary — in the same Director that strips OAuth headers — before asserting X-Hermes-Tenant-Identity: slot-N. Regression tests: services/hermes/router/main_test.go. Defense in depth: the sidecar is loopback-only anyway, and the in-pod WebUI BFF asserts its own X-Hux-* from trusted context.

WebUI image (dockerfiles/Dockerfile.hermes-webui)

The image ships the service and contracts at /opt/hermes-hux (hux/ package + contracts/, HUX_CONTRACT_DIR), the BFF proxy (api/hux_bff.py) and trusted context attachment (api/hux_context.py, hux.webui_context.v1), the bootstrap bridge (static/hux/bootstrap.js/.css) and the wave A/B/C runtime bundles. A build-time gate compiles the package and constructs the router with HUX_FLAGS="": images carry the code but no activation — flags live only in manifests.

Release rendering (ci/scripts/hermes_webui_flux_release.py)

The release script updates one-or-two WebUI image consumers in StatefulSet/hermes-chat-tenant (the webui container, plus the hux sidecar when staged) and exactly one in Deployment/hermes, and binds HUX_IMAGE_TAG/HUX_IMAGE_DIGEST to the released tag and digest whenever the sidecar fields are present — failing when the binding fields are incomplete (exactly one of each, or none at all).

HUX-12 evidence trust (hux/release_security.py, hux/releases.py, 47165274)

Release follow-through has a dedicated fourth trust class, evidence:

  • File-only authentication: HUX_RELEASE_EVIDENCE_KEY_FILE must be a 0400 regular file (symlinks refused) holding a key ≥ 32 chars, and HUX_RELEASE_EVIDENCE_POLICY_FILE a strictly validated hux.release_evidence_policy.v1 allowlist (≤ 16 workloads, credential-free HTTPS URLs, exact field sets, bounded evidence age). Without both healthy the HUX-12 capability is off entirely — fail closed.
  • Worker trust is read-only on releases (list/read in WORKER_ROUTES); it can no longer create releases or transitions.
  • Only router trust creates the single reviewed proposal per scoped workload commit; only evidence trust performs transitions, and its route allowlist in hux/http.py blocks everything else.
  • Transition evidence binds exactly: merged the 40-hex merge commit; built the CI URL plus image ref / image digest / Harbor digest all equal; deployed a main@sha1:<40-hex> Flux revision; converged pod digest == image digest; live_verified one passing health receipt; rolled_back an exact target digest.
  • Ledger entries are hash-chained (previous_hash/entry_hash) and verified on every read; Idempotency-Key replays return the recorded view with HUX-Replayed: true.

Worker state

worker.bstein.dev (the hermes Deployment and the execution-worker manifests) has NO HUX sidecar, plugin or environment. Cross-surface continuity is not claimed; HUX is chat-staged only.

Test state

Verified 2026-08-24 with the CI interpreter: python -m pytest testing/tests/test_hermes_hux_*.py = 646 passed (20.8 s). Delivery gates in testing/tests/test_hermes_hux_delivery.py assert the staged topology against the manifests themselves: all-or-nothing activation, loopback-only sidecar on the reviewed WebUI image, least-privilege storage and key mounts, init-container identity persistence + transport-key rotation, file-backed authentication inputs, and plugin/vendored-hook render parity. UI card model suites: testing/tests/test_hermes_hux_ui_*.py, each driving a Node harness (test_hermes_hux_ui_*.mjs). Release rendering: test_hermes_webui_release.py (18) and test_hermes_image_automation.py (2).

Superseded statements

  • "/opt/data/hux on the tenant PVC; SO-45 separate uid, mode 0700" — superseded. Storage is the shared RWX claim with kubelet subPathExpr isolation above; every container runs uid 10000 and the boundary is mount scoping, not a uid split. See THREAT-MODEL.md, "Integration amendments (2026-08-24)".
  • "HUX_DATA_ROOT (tenant PVC, default /opt/data)" — now /var/lib/hux/store on the dedicated claim.
  • "runs … inside each hermes-chat-tenant pod (and the Worker)" — the Worker has no HUX yet.
  • "Image, manifest, NetworkPolicy and Flux wiring are Codex's" — landed in-repo with the chain above.
  • Open item "SO-53 rate limits" — implemented in hux/http.py; SO-46 and SO-48 remain open (see the threat model amendment).

Completion increments (2026-08-24, later — local chain, staged, not deployed)

Everything below is committed locally on top of the integrated topology and verified by the full gate (826 passed / 0 failed across test_hermes_hux_*.py, test_hermes_webui_hux_*.py, image-automation, webui-release, auto-router and chat-quality suites, after one fieldRef tolerance fix in the voice gate).

  • Topology gates are all-or-nothing adaptive (test_hermes_hux_delivery.py, test_hermes_image_automation.py): with no hux sidecar the manifests must contain zero partial HUX wiring; with it, the full strict boundary applies. This lets the source chain merge and build before activation.
  • HUX-06 adoption (services/hermes/plugins/auto-router/hux_mode.py): the auto-router consults the loopback HUX service (worker trust, the same hux.context.id.v1 HMAC derivation) for the conversation's selected friendly mode: fast→auto/fast, thoughtful/research→auto/deep, create→auto/balanced, private→the local route only. Explicit UI picks win; every failure falls through to previous behaviour. Delivered through the auto-router ConfigMap with a degrade-to-noop import shim.
  • Private mode is enforced upstream (hux/policy.py, hux/budgets.py): network, web_search, send_message, shell and delegate are refused by the approval resolver and re-refused by the gate for a conversation whose stored mode is private, regardless of autonomy level.
  • HUX-04/08 emitters (plugins/hux-runtime/emitters.py): after a successful write_files tool the plugin registers the written file as an artifact (create or immutable version, ≤1 MiB, deterministic idempotency); after successful network/web_search tools it records up to three deduplicated web sources from the real output. Fail-open by construction.
  • HUX-03 message-text search (hux/organization.py): GET /hux/v1/search?include=message_text scans stored message events of the 100 most recently active candidates; forgotten, private, restricted and fully-redacted content never matches; deterministic cursor pagination. The default search response is unchanged (the shipped UI requires message_text in not_indexed).
  • HUX-12 producer (dockerfiles/hermes-hux-foundation/hux_producer/, 100% line+branch, 71 tests): pure verifier binding review URL prefix → Jenkins SUCCESS/revision → immutable Harbor tag/digest → Flux main@sha1: revision with pin containment → desired workload image → every Ready pod imageID → bounded-age health receipt → rollback target; HTTPS-only injectable collectors (SA token for the API); evidence-trust driver posts exactly one If-Match transition. Cluster layer staged in chat-statefulset.yaml (sidecar + tolerant key init), hux-evidence-policy.json, hux-evidence-rbac.yaml (read-only pods/ statefulset + the single named Kustomization), tenant egress to the API ClusterIP and traefik. The key reaches only the hux and hux-evidence-producer containers.

HUX-12 enablement runbook (fail-closed until every step is done)

  1. vault kv put kv/atlas/hermes/hux-evidence evidence_key="$(openssl rand -hex 32)"
  2. Add to the chat tenant pod annotations (Flux commit): vault.hashicorp.com/agent-inject-secret-hux-evidence-key: kv/data/atlas/hermes/hux-evidence plus the matching template emitting .Data.data.evidence_key.
  3. Create ConfigMap hermes-hux-evidence-scope with HUX_PRODUCER_PROJECT_ID / HUX_PRODUCER_CONVERSATION_ID for the release conversation (visible in the UI once a reviewed proposal exists).
  4. Only then does release_security.configured turn HUX-12 on; the sidecar loop starts posting verified transitions.

Staged release sequence (agreed)

  1. Push the source tip to main: the full local chain plus a hold commit that restores services/hermes/{chat-statefulset,chat-pvcs,chat-configmap,kustomization,networkpolicy}.yaml to the live state (keeping only the auto-router hux_mode.py ConfigMap key) and removes the evidence policy/RBAC files — so Flux applies nothing and CI builds the reviewed WebUI image.
  2. Trigger the WebUI build at that exact SHA; wait for Flux digest convergence.
  3. Re-apply the activation topology pinned to the new tag/digest with HUX_FLAGS=hux.foundation, HUX_TOOL_ENFORCEMENT=0, and a RollingUpdate partition canarying ordinal 3 first.
  4. HUX-11 live gates → card-by-card flag enablement with lifecycle acceptance → enforcement after park/resume + stop receipts pass live → HUX-12 after the runbook above → Worker separately.

Live canary evidence (2026-08-24 ~09:05 UTC)

Foundation canary LIVE and verified on hermes-chat-tenant-3:

  • Commits: source 2f535d3a (build 21), Flux digest 1d29c25c, activation 39a54482 (partition 3, HUX_FLAGS=hux.foundation, enforcement 0).
  • Image: registry.bstein.dev/bstein/hermes-webui:git-2f535d3a...-build-21-release@sha256:e5b9b2fa8296a7b8a6065c63e7b288cd3305d70a3acc4312055b819bb5f64c04 on webui + hux + hux-evidence-producer (pod imageIDs verified); agent digest unchanged (sha256:c64c06e6...).
  • canary_verify: 15/15 PASS (all containers ready, ordinals 0-2 untouched and huxless, loopback /healthz ok, subject binding present, public https://chat.bstein.dev/healthz 200, Flux Ready).
  • Live route matrix: unauth 401, bad key 401, wrong slot 401, relay capabilities 200 with only HUX-11 enabled, worker capabilities 200, disabled card 404 flag_off. Subject usr_224348fa... consistent with the HMAC binding.
  • Identity hashes for the restart-persistence gate: context-key 8a1c474d..., subject c4ebe353... (compare after the next roll).
  • Rollback target: WebUI build-18 digest sha256:df91f8a3... (previous fleet image); activation rollback = revert 39a54482.
  • Retention scheduler reports disabled until hux.privacy is enabled (by design).

Next: Wave-A canary flags (activity_timeline, projects, privacy, memory_control) on ordinal 3, lifecycle gates per card, then widen partition.

Wave-A live gates (2026-08-24 ~09:12 UTC) — PASS

Canary rolled to ee330822 flags; restart persistence PROVEN (context-key and subject hashes identical across the roll). Live lifecycle 13/13 on the canary loopback: capabilities advertise exactly HUX-01/02/03/10/11; real project + conversation created; worker-emitted event readable by relay; memory proposed -> approved -> forgotten with the tombstone effective in the listing; privacy policy served; conversation forget tombstones the record; retention scheduler enabled and healthy. Next: Wave-B flags (artifacts, research, friendly_modes, multimodal, onboarding) on the canary.

Wave-B live gates (2026-08-24 ~09:20 UTC) — PASS 18/18

Canary at dc2b3643 flags. Live on the canary loopback: capabilities show the Wave-B set with HUX-07 correctly fail-closed (depends_on HUX-05); artifact create -> immutable version -> content -> diff -> promotion; research source -> passage -> citation attached to an authoritative message id with the citation strip readable; provider-neutral mode catalog, thoughtful selected under If-Match and read back by worker trust; suggestion shown for the real after_artifact context, explicit clicked dismissal recorded, re-show suppressed, suppression state durable. Next: hux.autonomy (+ multimodal unlock) on the canary, enforcement still 0.