docs(hux): describe the integrated staged topology and trust reality
HANDOFF.md and THREAT-MODEL.md now document the actual HMAC subject/ context identity, the shared RWX PVC with kubelet subPathExpr per-pod isolation and its exact boundary, the uid-10000 no-store-mount agent model, tmpfs transport key rotation, router X-Hux-* stripping, HUX-12 file-only evidence trust, staged-not-deployed status, and the verified SO-46/48 (open) and SO-53 (shipped) states. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
This commit is contained in:
jenkins
parent
b2cfbc94e8
commit
b3d4538b4f
@ -14,6 +14,11 @@ Run everything with the CI interpreter:
|
||||
|
||||
## Codex integration requirements common to every card
|
||||
|
||||
(Superseded 2026-08-24 where it conflicts with "Integrated topology" at the
|
||||
end of this file: the manifests, plugin, router headers and Flux wiring are
|
||||
now in-repo, the data root moved off the tenant PVC, and the Worker has no
|
||||
HUX yet.)
|
||||
|
||||
- Package `dockerfiles/hermes-hux-foundation/hux/` runs as `python -m hux.server`
|
||||
inside each `hermes-chat-tenant` pod (and the Worker) on `127.0.0.1:8790`.
|
||||
Image, manifest, NetworkPolicy and Flux wiring are Codex's (SO-01, SO-02,
|
||||
@ -100,7 +105,195 @@ Suite after repairs: 377 tests, 99% line / 99% branch over `hermes-hux-foundatio
|
||||
|
||||
## Open items (not blockers for Codex integration)
|
||||
|
||||
- SO-46 hash-chained audit, SO-48 single-writer lock, SO-53 rate limits.
|
||||
- SO-46 hash-chained audit, SO-48 single-writer lock. (SO-53 rate limits
|
||||
shipped since: `hux/http.py` `RateLimiter`, per-subject read/write
|
||||
buckets, `429` + `Retry-After`, tuned by `HUX_READS_PER_MINUTE` /
|
||||
`HUX_WRITES_PER_MINUTE` — see the integrated-topology section.)
|
||||
- Retention scheduler: `privacy.run_retention` is on-demand; Codex decides cron vs sidecar.
|
||||
- Search over `message_text`; artifact sharing (`shared_readonly`).
|
||||
- `testing/quality_contract.json` line-limit globs / managed modules for the two new package paths (Codex's file).
|
||||
|
||||
## Integrated topology (2026-08-24, local chain, staged — not deployed)
|
||||
|
||||
Everything in this section exists only on local `main`. `origin/main` is
|
||||
still `5558c24f`; Flux reconciles origin, so none of this is running
|
||||
anywhere. Chat is staged; the Worker is untouched. The chain since origin
|
||||
(oldest first): `964103f0` merge of the hardened foundation, `438180a9`
|
||||
tool runtime gated through HUX, `a1070449` vendored runtime hooks,
|
||||
`53d7c2c5` backend families complete, `31c5eae5` chat router image
|
||||
automation, `1eda927b` trusted HUX workspace in the WebUI, `47165274`
|
||||
HUX-12 evidence trust isolation, `b66c762f` HUX card UI models,
|
||||
`bd63b568` router strips inbound `X-Hux-*`, `9e99470f` one-or-two WebUI
|
||||
consumers in release rendering, `0b56d04e` quality-contract gates,
|
||||
`71c05cf9` staged sidecar activation topology, `659f70ee` WebUI
|
||||
integration contract notes, `13359769` all-or-nothing topology gates.
|
||||
|
||||
### Pod topology (`services/hermes/chat-statefulset.yaml`)
|
||||
|
||||
Each `hermes-chat-tenant` pod runs a fourth container, `hux`, on the SAME
|
||||
reviewed WebUI image as the `webui` container
|
||||
(`registry.bstein.dev/bstein/hermes-webui`, Flux
|
||||
`$imagepolicy hermes:hermes-webui-release`), started as
|
||||
`python -m hux.server` with `PYTHONPATH=/opt/hermes-hux`. It binds
|
||||
`127.0.0.1:8790` (`HUX_BIND`, `HUX_PORT`), runs as uid/gid 10000 with
|
||||
`readOnlyRootFilesystem`, all capabilities dropped, tmpfs `/tmp`, and
|
||||
`/healthz` probes. `HUX_TENANT_SLOT=slot-N` comes from the pod ordinal,
|
||||
as for the agent. Port 8790 appears in no Service or NetworkPolicy;
|
||||
loopback is the only path (delivery-gated).
|
||||
|
||||
### Storage: shared RWX claim, kubelet-scoped subtrees
|
||||
|
||||
The plan's `/opt/data/hux` on the tenant `home` PVC is gone. Storage is a
|
||||
standalone `ReadWriteMany` PVC `hermes-chat-hux-data` (storageClass
|
||||
`astreae`, 10Gi, `services/hermes/chat-pvcs.yaml`) shared by all tenant
|
||||
pods, with per-pod isolation enforced by kubelet
|
||||
`subPathExpr: $(POD_NAME)` mounts:
|
||||
|
||||
- `hux` container: `/var/lib/hux` = its own `$(POD_NAME)` subtree;
|
||||
`HUX_DATA_ROOT=/var/lib/hux/store`.
|
||||
- `hermes` (agent) container: read-only `/run/hermes-hux-context` =
|
||||
`$(POD_NAME)/context` and `/run/hermes-hux-subject` =
|
||||
`$(POD_NAME)/binding` only. It has NO mount of `/var/lib/hux/store`;
|
||||
records are reachable from the agent only through the loopback API.
|
||||
- `webui` container: read-only `/run/hermes-hux-context` only.
|
||||
|
||||
No container in any pod mounts another pod's subtree.
|
||||
|
||||
### Identity provisioning (`init-hux-runtime`)
|
||||
|
||||
A dedicated init container (root, capabilities dropped to
|
||||
CHOWN/DAC_OVERRIDE/FOWNER) provisions, idempotently per pod:
|
||||
|
||||
- 0700 roots `binding/`, `context/`, `store/`, owned 10000:10000;
|
||||
- a persistent 32-byte context key `context/context-key` (0600), created
|
||||
once and kept across restarts;
|
||||
- an immutable subject binding `binding/subject` (0440, created `O_EXCL`):
|
||||
`usr_` + HMAC-SHA256(context-key, `"hux.subject.id.v1\0slot-N"`). On
|
||||
restart the file is verified (regular file, uid 10000, mode 0440, one
|
||||
hardlink, expected content) or the pod refuses to start;
|
||||
- a redaction canary `context/redaction-canary` (0400);
|
||||
- relay and worker keys in `Memory`-medium emptyDirs
|
||||
(`/run/hermes-webui-hux/relay-key`, `/run/hermes-hux-worker/worker-key`,
|
||||
0400) — regenerated on every pod restart, never touching the PVC.
|
||||
|
||||
The subject is stable across restarts, derived from a per-pod random key,
|
||||
and never derived from the Keycloak subject.
|
||||
|
||||
### Env contracts
|
||||
|
||||
`hux` sidecar: `HUX_FLAGS` lists all twelve cards (`hux.foundation,
|
||||
hux.activity_timeline, hux.memory_control, hux.projects, hux.artifacts,
|
||||
hux.autonomy, hux.friendly_modes, hux.multimodal, hux.research,
|
||||
hux.onboarding, hux.privacy, hux.release_followthrough`);
|
||||
`HUX_RELAY_KEY_FILE`, `HUX_WORKER_KEY_FILE`, `HUX_SUBJECT_BINDING_FILE`,
|
||||
`HUX_CONTEXT_KEY_FILE`, `HUX_CANARY_FILE`; rate limits
|
||||
`HUX_READS_PER_MINUTE=600`, `HUX_WRITES_PER_MINUTE=120`; and build
|
||||
provenance `HUX_IMAGE_TAG` / `HUX_IMAGE_DIGEST` bound by Flux setters
|
||||
(`$imagepolicy hermes:hermes-webui-release:tag` / `:digest` comments).
|
||||
|
||||
Agent container: `HUX_BASE_URL=http://127.0.0.1:8790`,
|
||||
`HUX_RUNTIME_ENABLED=1`, and `HUX_TOOL_ENFORCEMENT=0` — the first rollout
|
||||
is observe-only until approval parking/resume is connected to the
|
||||
upstream tool loop and proven live — plus `HUX_WORKER_KEY_FILE`,
|
||||
`HUX_SUBJECT_FILE`, `HUX_CONTEXT_KEY_FILE`, `HUX_TIMEOUT_SECONDS=3`.
|
||||
|
||||
### Agent runtime plugin
|
||||
|
||||
`services/hermes/plugins/hux-runtime/` renders into the
|
||||
`hermes-hux-runtime-plugin` ConfigMap (`services/hermes/kustomization.yaml`)
|
||||
and mounts read-only into the agent at `/opt/data/plugins/hux-runtime`:
|
||||
`runtime.py`, `tool_policy.py`, `context_ids.py`, `plugin.yaml`
|
||||
(`on_session_end` hook + `tool_execution` middleware, fail-closed), and a
|
||||
vendored `hux_hook/` package kept in parity with the reviewed worker hook
|
||||
library (parity is delivery-gated).
|
||||
|
||||
### Router boundary
|
||||
|
||||
`services/hermes/router/main.go` deletes EVERY inbound header whose name
|
||||
starts with `x-hux-` (case-insensitive) at the authenticated tenant
|
||||
boundary — in the same Director that strips OAuth headers — before
|
||||
asserting `X-Hermes-Tenant-Identity: slot-N`. Regression tests:
|
||||
`services/hermes/router/main_test.go`. Defense in depth: the sidecar is
|
||||
loopback-only anyway, and the in-pod WebUI BFF asserts its own `X-Hux-*`
|
||||
from trusted context.
|
||||
|
||||
### WebUI image (`dockerfiles/Dockerfile.hermes-webui`)
|
||||
|
||||
The image ships the service and contracts at `/opt/hermes-hux` (`hux/`
|
||||
package + `contracts/`, `HUX_CONTRACT_DIR`), the BFF proxy
|
||||
(`api/hux_bff.py`) and trusted context attachment (`api/hux_context.py`,
|
||||
`hux.webui_context.v1`), the bootstrap bridge
|
||||
(`static/hux/bootstrap.js/.css`) and the wave A/B/C runtime bundles. A
|
||||
build-time gate compiles the package and constructs the router with
|
||||
`HUX_FLAGS=""`: images carry the code but no activation — flags live only
|
||||
in manifests.
|
||||
|
||||
### Release rendering (`ci/scripts/hermes_webui_flux_release.py`)
|
||||
|
||||
The release script updates one-or-two WebUI image consumers in
|
||||
`StatefulSet/hermes-chat-tenant` (the `webui` container, plus the `hux`
|
||||
sidecar when staged) and exactly one in `Deployment/hermes`, and binds
|
||||
`HUX_IMAGE_TAG`/`HUX_IMAGE_DIGEST` to the released tag and digest whenever
|
||||
the sidecar fields are present — failing when the binding fields are
|
||||
incomplete (exactly one of each, or none at all).
|
||||
|
||||
### HUX-12 evidence trust (`hux/release_security.py`, `hux/releases.py`, `47165274`)
|
||||
|
||||
Release follow-through has a dedicated fourth trust class, `evidence`:
|
||||
|
||||
- File-only authentication: `HUX_RELEASE_EVIDENCE_KEY_FILE` must be a 0400
|
||||
regular file (symlinks refused) holding a key ≥ 32 chars, and
|
||||
`HUX_RELEASE_EVIDENCE_POLICY_FILE` a strictly validated
|
||||
`hux.release_evidence_policy.v1` allowlist (≤ 16 workloads,
|
||||
credential-free HTTPS URLs, exact field sets, bounded evidence age).
|
||||
Without both healthy the HUX-12 capability is off entirely — fail closed.
|
||||
- Worker trust is read-only on releases (list/read in `WORKER_ROUTES`); it
|
||||
can no longer create releases or transitions.
|
||||
- Only router trust creates the single `reviewed` proposal per scoped
|
||||
workload commit; only evidence trust performs transitions, and its route
|
||||
allowlist in `hux/http.py` blocks everything else.
|
||||
- Transition evidence binds exactly: `merged` the 40-hex merge commit;
|
||||
`built` the CI URL plus image ref / image digest / Harbor digest all
|
||||
equal; `deployed` a `main@sha1:<40-hex>` Flux revision; `converged`
|
||||
pod digest == image digest; `live_verified` one passing health receipt;
|
||||
`rolled_back` an exact target digest.
|
||||
- Ledger entries are hash-chained (`previous_hash`/`entry_hash`) and
|
||||
verified on every read; `Idempotency-Key` replays return the recorded
|
||||
view with `HUX-Replayed: true`.
|
||||
|
||||
### Worker state
|
||||
|
||||
`worker.bstein.dev` (the `hermes` Deployment and the execution-worker
|
||||
manifests) has NO HUX sidecar, plugin or environment. Cross-surface
|
||||
continuity is not claimed; HUX is chat-staged only.
|
||||
|
||||
### Test state
|
||||
|
||||
Verified 2026-08-24 with the CI interpreter:
|
||||
`python -m pytest testing/tests/test_hermes_hux_*.py` = 646 passed
|
||||
(20.8 s). Delivery gates in `testing/tests/test_hermes_hux_delivery.py`
|
||||
assert the staged topology against the manifests themselves:
|
||||
all-or-nothing activation, loopback-only sidecar on the reviewed WebUI
|
||||
image, least-privilege storage and key mounts, init-container identity
|
||||
persistence + transport-key rotation, file-backed authentication inputs,
|
||||
and plugin/vendored-hook render parity. UI card model suites:
|
||||
`testing/tests/test_hermes_hux_ui_*.py`, each driving a Node harness
|
||||
(`test_hermes_hux_ui_*.mjs`). Release rendering:
|
||||
`test_hermes_webui_release.py` (18) and `test_hermes_image_automation.py`
|
||||
(2).
|
||||
|
||||
### Superseded statements
|
||||
|
||||
- "`/opt/data/hux` on the tenant PVC; SO-45 separate uid, mode 0700" —
|
||||
superseded. Storage is the shared RWX claim with kubelet `subPathExpr`
|
||||
isolation above; every container runs uid 10000 and the boundary is
|
||||
mount scoping, not a uid split. See `THREAT-MODEL.md`, "Integration
|
||||
amendments (2026-08-24)".
|
||||
- "`HUX_DATA_ROOT` (tenant PVC, default `/opt/data`)" — now
|
||||
`/var/lib/hux/store` on the dedicated claim.
|
||||
- "runs … inside each `hermes-chat-tenant` pod (and the Worker)" — the
|
||||
Worker has no HUX yet.
|
||||
- "Image, manifest, NetworkPolicy and Flux wiring are Codex's" — landed
|
||||
in-repo with the chain above.
|
||||
- Open item "SO-53 rate limits" — implemented in `hux/http.py`; SO-46 and
|
||||
SO-48 remain open (see the threat model amendment).
|
||||
|
||||
@ -267,3 +267,112 @@ above as a result:
|
||||
open items, not claimed controls.
|
||||
- On-disk documents carry a `revision` field the 1.0.0 fixtures did not have;
|
||||
served bodies strip it where the record schema forbids it (F11).
|
||||
|
||||
## Integration amendments (2026-08-24)
|
||||
|
||||
The staged integration (`services/hermes/chat-statefulset.yaml`,
|
||||
`chat-pvcs.yaml`, local chain up to `13359769`; staged only — nothing is
|
||||
deployed, `origin/main` is still `5558c24f`) changes the following. Where
|
||||
this section conflicts with the body above, this section is current.
|
||||
|
||||
### SO-45 superseded: the boundary is mount scoping, not a uid split
|
||||
|
||||
The service does not run as a distinct uid and there is no `/opt/data/hux`.
|
||||
Records live on a standalone ReadWriteMany PVC `hermes-chat-hux-data`
|
||||
shared by all tenant pods; each pod receives only its own subtree through
|
||||
kubelet `subPathExpr: $(POD_NAME)` mounts, and an init container sets each
|
||||
subtree root to 0700 owned 10000:10000. All pod containers run uid 10000.
|
||||
The isolation boundary now is, exactly:
|
||||
|
||||
1. kubelet subPath scoping — no container in pod N has any mount of pod
|
||||
M's subtree, so another tenant's records are simply absent from the
|
||||
mount namespace;
|
||||
2. the agent container mounts none of the store: it gets read-only
|
||||
`$(POD_NAME)/context` and `$(POD_NAME)/binding` only and reaches
|
||||
records solely through the loopback API — SO-45's threat ("the agent
|
||||
edits the audit log; it has write on the PVC") has no filesystem path;
|
||||
3. the `hux` sidecar keeps `readOnlyRootFilesystem`, drop-ALL and the
|
||||
`127.0.0.1:8790` bind (SO-01), with port 8790 absent from every Service
|
||||
and NetworkPolicy (SO-02 holds unchanged).
|
||||
|
||||
Not defended: an attacker who can mount the whole claim (node or cluster
|
||||
compromise) sees every tenant's records — the same trust in kubelet the
|
||||
per-tenant `home` PVC model already rested on. Within a pod, same-uid
|
||||
means an agent-container escape that can rearrange mounts is equivalent to
|
||||
kubelet compromise; accepted for this increment and recorded here rather
|
||||
than claimed away.
|
||||
|
||||
### SO-01..SO-04 status
|
||||
|
||||
- SO-01 satisfied: `HUX_BIND=127.0.0.1`, `HUX_PORT=8790`, asserted by the
|
||||
delivery gate (`test_hermes_hux_delivery.py`) against the manifest.
|
||||
- SO-02 satisfied: no hux port in any Service or NetworkPolicy.
|
||||
- SO-03/SO-04: the router (`services/hermes/router/main.go`) deletes every
|
||||
inbound header matching prefix `x-hux-` (case-insensitive) at the
|
||||
authenticated boundary before asserting `X-Hermes-Tenant-Identity`;
|
||||
regression-tested in `main_test.go`. The router itself sets no
|
||||
`X-Hux-*`: the in-pod WebUI BFF asserts them from trusted context, and
|
||||
the subject is pinned by the pod-local binding file below.
|
||||
|
||||
### Persistent subject and context identity
|
||||
|
||||
The `init-hux-runtime` init container (root; capabilities
|
||||
CHOWN/DAC_OVERRIDE/FOWNER only) provisions per pod a persistent random
|
||||
32-byte key `context/context-key` (0600) and an immutable binding
|
||||
`binding/subject` (0440, created `O_EXCL`, verified byte-for-byte, mode,
|
||||
owner and link count on every restart — a mismatch fails the pod):
|
||||
|
||||
usr_<HMAC-SHA256(context-key, "hux.subject.id.v1\0slot-N")>
|
||||
|
||||
Properties: stable across restarts (the key persists on the claim);
|
||||
derived from a per-pod key, so subjects are unlinkable across tenants; and
|
||||
never derived from the Keycloak subject, so no OAuth identifier can be
|
||||
recovered from a stored `usr_` hash (strictly stronger than ADR-0001's
|
||||
`identityHash("keycloak", subject)` plan). A redaction canary
|
||||
`context/redaction-canary` (0400) feeds `HUX_CANARY_FILE`.
|
||||
|
||||
### Relay and worker key lifecycle
|
||||
|
||||
`HUX_RELAY_KEY_FILE` and `HUX_WORKER_KEY_FILE` point into `Memory`-medium
|
||||
emptyDirs written by the init container: 0400, regenerated on every pod
|
||||
restart, never stored on the PVC. A leaked relay or worker key therefore
|
||||
expires at the next restart. SO-07's canary obligation extends to these
|
||||
files.
|
||||
|
||||
### Evidence trust class (HUX-12)
|
||||
|
||||
A fourth trust class `evidence` (`hux/release_security.py`) with these
|
||||
obligations:
|
||||
|
||||
- file-only key: `HUX_RELEASE_EVIDENCE_KEY_FILE` must be a 0400 regular
|
||||
file, symlinks refused, key ≥ 32 chars; inline environment secrets are
|
||||
ignored;
|
||||
- strict policy allowlist: `HUX_RELEASE_EVIDENCE_POLICY_FILE`
|
||||
(`hux.release_evidence_policy.v1`), exact field sets, ≤ 16 workloads,
|
||||
credential-free HTTPS URLs, bounded evidence age;
|
||||
- fail-closed capability: without both files healthy the HUX-12 capability
|
||||
is off (`flags.enabled`), not degraded;
|
||||
- separation: router trust alone creates `reviewed` proposals; evidence
|
||||
trust alone transitions and may call nothing else (route allowlist in
|
||||
`hux/http.py`, `api` surface required); SO-08 narrows again — worker
|
||||
trust is read-only on releases (list/read only).
|
||||
|
||||
### SO-46 / SO-48 / SO-53 actual status (verified in code, 2026-08-24)
|
||||
|
||||
- SO-53 rate limits: IMPLEMENTED. `hux/http.py` `RateLimiter` keeps
|
||||
per-subject one-minute read and write buckets (defaults 300/30,
|
||||
overridden by `HUX_READS_PER_MINUTE`/`HUX_WRITES_PER_MINUTE`; the staged
|
||||
manifest sets 600/120), returns `429 rate_limited` with `Retry-After`,
|
||||
caps the bucket table at 4096, and runs immediately after identity
|
||||
resolution, before route matching and flag checks.
|
||||
- SO-46 hash-chained audit: STILL OPEN for the general audit ledger.
|
||||
`hux/audit.py` appends plain JSONL outcome rows with no `prev` hash.
|
||||
The HUX-12 release ledger is hash-chained end to end
|
||||
(`previous_hash`/`entry_hash`, verified on every read in
|
||||
`hux/releases.py`) — that covers release honesty, not the audit trail.
|
||||
- SO-48 single-writer lock: STILL OPEN. No `flock` anywhere in the
|
||||
package; `hux/store.py` uses in-process `threading.RLock` per family
|
||||
only. The per-pod `$(POD_NAME)` subtree makes a second writer against
|
||||
the same tree unlikely (it would need a second pod with the same name),
|
||||
but the obligation as written — fail at startup on `v1/.lock` — is not
|
||||
implemented.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user