# HUX foundation: on-PVC data model `hux-foundation` is a Python stdlib service that runs inside every tenant pod and owns the records defined in `services/hermes/contracts/hux/`. It has no database and no storage shared between tenants. Everything it keeps lives on the tenant's own `home` PVC (mounted at `/opt/data`, beside `webui/`, `workspace/`, `home/` and the Telegram media roots `cache/images` and `workspace`). The Go router in `services/hermes/router/` fronts it: the router resolves Keycloak identity to a tenant slot and forwards the `common.schema.json#/$defs/identity` tuple (`tenant_slot`, `subject`, `surface`, `trust`) as headers, and enforces `hux.foundation` before any request reaches this service. The service trusts nothing else about the caller. The conventions below are lifted from what already works in this repo: `hermes_model_routing._atomic_write` (temp file + `os.replace`), `cli_lane_records.atomic_json` (journal first, act second), and `execution_pool_store` (idempotent add by digest, terminal rows only are ever garbage-collected). SQLite is deliberately not used: the WebUI already owns `state.db` on this volume and a second writer with its own WAL is one more thing to recover; JSON and JSONL are greppable during an incident. ## 1. Directory layout Tenant root is `HUX_DATA_ROOT`, default `/opt/data/hux`. The layout version is a directory (`v1`), so a v2 layout can be built beside v1 and swapped by `MANIFEST.json`, never by rewriting v1 in place. ``` /opt/data/hux/ MANIFEST.json hux.manifest.v1 (identity.schema.json): contract and layout versions v1/ .lock advisory lock file (flock) proving single writer per tenant users// one subtree per hashed subject; nothing lives above it profile.json {schema:"hux.user_profile.v1", memory_enabled, default_mode, revision} events/ / events.jsonl append-only hux.event.v1, one per line, ordered by seq seq.json {next_seq, last_event_id, bytes, checkpointed_at} idempotency.jsonl {idempotency_key, event_id, seq, at}; last 10k keys kept memory/ ledger.jsonl append-only hux.memory.v1 snapshots; a status change appends a full record tombstones.jsonl {memory_id, at, reason, purged:bool}; written on forget and on decay index.json retrieval index (section 3), rebuilt from the ledger on demand exports/.json GET /memory/export snapshots, audited, pruned after 7 days projects/ .json hux.project.v1 + revision index.json {items:[{id,name,tags,pinned,archived,updated_at}], revision} conversations/ .json hux.conversation.v1 + revision index.json {items:[{id,project_id,title,tags,pinned,archived,mode,branch,last_message_at}]} search/ .json per-conversation term postings for message_text and artifact_titles artifacts/ .json hux.artifact.v1 + revision (versions[] is the version list) index.json {items:[{id,type,title,conversation_id,project_id,current_version}]} blobs/sha256// content-addressed, immutable, 0400, fan-out on first two hex chars blobs/refs.json {hash: [ "@", ...]} for safe purge research/ sources/.json hux.source.v1 passages/.json hux.passage.v1 (text + locator; hash is the dedupe key) citations/.jsonl hux.citation.v1 per message, append-only notebooks/.json hux.research_notebook.v1 + revision index.json {by_conversation:{conv_id:[nb_id]}, by_message:{msg_id:count}, passage_hashes:{hash:psg_id}} policy/ global.json hux.policy.v1 for scope level global project/.json hux.policy.v1 per project conversation/.json hux.policy.v1 per conversation approvals/.json hux.approval.v1; terminal records never change again approvals/pending.json {items:[apr_id], revision}; the queue the UI polls receipts/.json hux.cancel_receipt.v1 receipts/by_run.json {run_id: rcpt_id} suggestions/ state.json {schema:"hux.suggestion_states.v1", items:{sug_id: hux.suggestion_state.v1}, revision} privacy/ notices.jsonl hux.privacy_notice.v1 as shown, append-only conversation_topics.json {conv_id:{topic, first_seen, decay_at, memory_disabled:bool}} forgotten.jsonl {conv_id, requested_at, purged_at, counts} audit/ outcomes/.jsonl one hux.audit_outcome.v1 per read or mutation (section 6) retention/.json hux.retention_audit.v1 per run catalog/ suggestions.json hux.suggestion.v1 catalog shipped with the image (read-only copy) privacy_policy.json hux.privacy_policy.v1 as served, with the policy version ``` The pod is single-tenant, so the `users/` level is not multi-tenancy; it is the guarantee that every path contains the hashed subject the router asserted and that a second subject on the same pod (operator break-glass, future household sharing) can never see another's tree without a distinct path. Record ids are minted by the service: `_<6 random base32>`, e.g. `evt_m0k3xq9a2bcd7f`, so they sort by creation time and satisfy the `common.schema.json` id pattern. `hux.audit_outcome.v1` and `hux.user_profile.v1` are foundation-internal records; they follow the same provenance and versioning rules but are not contracts the UI codes against. ## 2. Write semantics **Atomic document write.** Serialise with `json.dumps(sort_keys=True, ensure_ascii=False)` plus a trailing newline, write to `....tmp` in the same directory, `flush()`, `os.fsync(fd)`, `os.replace(tmp, final)`, then `os.fsync(dir_fd)` so the rename itself is durable. Temp files that survive a crash are deleted on open. Blobs are written the same way under their hash; an existing blob is never rewritten (compare size and hash; on mismatch refuse and audit). **Append.** JSONL families open with `O_APPEND`, write the whole line in one `os.write`, and `fsync` the file before the HTTP response is sent. The controlling checkpoint (`seq.json`, `index.json`) is written atomically after the append; a checkpoint may lag the log, never lead it. **Crash recovery.** On first access to a family the store validates the log: each line must parse and, for events, `seq` must equal the previous `seq + 1`. A trailing partial line (no newline, or JSON error on the last line only) is truncated to the last good newline; a bad line anywhere else is a hard error (the family is marked read-only, an audit outcome is written, `/healthz` reports `degraded`). The checkpoint is then reconciled from the log: `next_seq` = last good `seq + 1`, indexes rebuilt if `checkpointed_at` is older than the log mtime. **Idempotency.** Every mutating request may carry `Idempotency-Key` (`common.schema.json#/$defs/idempotency_key`, `^[A-Za-z0-9._:-]{8,120}$`). For events the key is stored in `idempotency.jsonl`; a replay returns the original event (same `id`, same `seq`) with `HTTP 200` and `HUX-Replayed: true`. For documents the key is stored in the record's `_meta.idempotency_keys` (last 16) and a replay returns the current record without bumping `revision`. Event `id` is unique per tenant; an append whose `id` already exists in the last checkpoint window is treated as a replay, not a duplicate. **Seq allocation.** The HTTP server is a single process, `ThreadingHTTPServer`. Locks are per path, held in a process-wide `dict[str, threading.RLock]` guarded by one `threading.Lock`. The unit of locking is the family directory for JSONL (`events/`, `memory/`) and the document path for JSON. Seq is allocated inside the conversation lock: read `next_seq` from memory (loaded from `seq.json` once per open), assign, append, fsync, write `seq.json`, release. Since one process owns the PVC, `v1/.lock` is taken with `fcntl.flock(LOCK_EX|LOCK_NB)` at start-up so an accidental second replica fails fast instead of interleaving appends. **Optimistic concurrency.** Every JSON document carries an integer `revision` (starts at 1) beside the contract fields; it is served as the `ETag`. `PATCH`/`PUT` require `If-Match: `; a mismatch returns `409` with a `hux.error.v1` body (`code: conflict`, the current revision in `details`) as `common.schema.json#/$defs/revision` specifies. A missing `If-Match` is accepted only when the request carries an `Idempotency-Key`, and then the write is last-writer-wins with the outcome audited as `why: "unconditional_write"`. Indexes carry their own `revision` and are rewritten under the family lock after the document. **Caps.** Rejected with `hux.error.v1` `too_large` (413, size) or `conflict` (409, count): - event line 64 KiB, `detail` 32 KiB; 50 000 events per conversation, then the conversation is `archived` and further appends need a branch - memory content 2 000 chars (schema), 5 000 live entries, ledger 64 MiB before compaction (section 4) - conversation and project documents 256 KiB; 2 000 conversations, 200 projects, `artifact_ids` 500 (schema) - artifact blob 25 MiB (`tenantMediaLimit` is 50 MiB; half leaves room for the Telegram path), 200 versions per artifact, 2 000 artifacts, blob store 2 GiB per tenant - passage text 4 000 chars, 10 000 sources, 50 000 passages - audit outcomes are never capped by count; they rotate daily and age out ## 3. Indexes Nothing is indexed that a scan cannot rebuild; every `index.json` is a cache of its family and carries `built_from` (log bytes or document count) so a stale index is detected and rebuilt rather than trusted. **Conversation search** (`GET /hux/v1/search?q=`) covers exactly `project.schema.json#/$defs/search_index`: `title`, `tags`, `project_name` come from `conversations/index.json` joined with `projects/index.json` in memory (a few thousand rows, scanned per query); `message_text` and `artifact_titles` come from `conversations/search/.json`, a per conversation bag of lowercased, punctuation-stripped terms with positions, updated when a `message.*` event or `artifact.*` event is appended. Query terms are ANDed; ranking is title hit > tag hit > term frequency > recency. Events with `sensitivity: restricted` or `redaction.level: full` never enter the search bag, so a search result can never leak what the timeline hides. **Memory retrieval** (`memory/index.json`) is a term index over `content` of `active` entries only, keyed by `scope.level`/`scope_id` and `topic`, plus `expires` sorted by the effective expiry (`expires_at` or `created_at + decay_days`) for the retention job. Entries are added on `approved` (automatic or user), removed on `rejected`, `expired`, `forgotten`. Retrieval reads the index, then loads the newest ledger snapshot for each id and drops any whose status is no longer `active`, so a lagging index fails safe. **Tombstones remove from retrieval before content is gone.** `forget` is two writes under the memory lock: append a ledger snapshot with `status: forgotten`, `content: ""`, audit `forgotten`; then append `tombstones.jsonl` `{memory_id, at, reason, purged:false}`. The index write follows. Both the retrieval path and the export path consult the tombstone set (loaded once, appended in memory) before returning anything, which is what makes "do not remember" hold even if the index rebuild is interrupted. The earlier ledger lines that still contain content are what the purge job in section 4 rewrites. The same tombstone file records `disable_memory_here` per conversation via `privacy/conversation_topics.json`, which retrieval also checks: a conversation with `memory_disabled` contributes no entries and receives none. **Research** indexes by hash: `passage_hashes` dedupes passages across sources, `by_message` lets `GET /messages/{id}/citations` open one JSONL without listing a directory, `by_conversation` backs the notebook drawer. Approvals index only the pending queue; terminal approvals are found by id. ## 4. Retention One thread runs the retention job every `retention_audit.interval_days` (1 day) at a jittered hour, and on demand via `POST /hux/v1/admin/retention` (worker surface only). Each run writes `audit/retention/.json` as `hux.retention_audit.v1` with the counts the schema names, so a day without the record is itself a finding. - `expire_memory`: entries whose effective expiry has passed get a new ledger snapshot `status: expired` and leave the index. Decay means the entry expires `decay_days` after `created_at` unless a later ledger snapshot carries a newer `updated_at` from an `approved` audit action, which resets the clock once. - `decay_topic_context`: `privacy/conversation_topics.json` rows past `decay_at` (`PRIVACY_TOPICS[topic].decay_days` after `first_seen`) cause the conversation's events with that `sensitivity` to be rewritten with `redaction.level: full` and `detail` removed; `summary` is replaced by the topic notice text. Seq, ids and provenance are kept, so the timeline stays contiguous. - `purge_forgotten_content`: for every tombstone with `purged:false`, rewrite `ledger.jsonl` (temp + replace, under the memory lock) replacing `content` with `""` on every snapshot of that id, keeping `audit[]`; then set `purged:true`. Forgotten conversations (`POST /conversations/{id}/forget`) are handled the same way: `events.jsonl` is rewritten with `detail` dropped and `redaction.level: full`, `search/.json` is deleted, artifacts owned only by that conversation lose their blobs (via `blobs/refs.json`), and `privacy/forgotten.jsonl` records the counts. The conversation document stays with `archived: true` so branches still resolve their parent. - `report`: bounds check. Per-family bounds for a home cluster (10 Gi PVC shared with the WebUI): events 1 GiB total and 180 days for archived conversations, memory ledger compacted when over 64 MiB (rewrite keeping only the newest snapshot per id plus every snapshot of ids with a tombstone), blobs 2 GiB with unreferenced blobs deleted 7 days after their last ref disappears, research 512 MiB and sources unreferenced by any notebook or citation for 90 days deleted, audit outcomes 90 days, retention audits 400 days, memory exports 7 days, idempotency keys 10 000 per conversation. Terminal approvals and receipts are kept 180 days. Nothing in `audit/` is ever removed by a forget or purge; only age. Private mode conversations (`retention: ephemeral`) are not written to `events/` at all; the service returns `204` to appends and the router's session memory is the only copy. ## 5. Migrations `MANIFEST.json`: ```json {"schema":"hux.manifest.v1","contract_version":"1.0.0","data_layout_version":1, "min_reader_contract_version":"1.0.0","created_at":"...","updated_at":"..."} ``` `contract_version` is `services/hermes/contracts/hux/VERSION`; the service that last opened the tree writes its own version there and bumps `updated_at`. `data_layout_version` names the `v/` directory in use. Rules: - Additive only within a layout. A release may add optional fields to a record, add a new file name, add a new directory, add an enum value that a reader can ignore. It may not rename or remove a field, change a field's type, change the meaning of an existing enum value, change the id pattern, change `seq` semantics, or move a family to a different path. - Every record keeps its contract `schema` value (`hux.event.v1`). Internal bookkeeping is under `_meta` (`revision`, `idempotency_keys`, `built_from`), which is stripped before a record is served and which readers must ignore. - Readers ignore unknown fields and never fail a family because one record has an unknown optional key. A reader from the previous release therefore reads records from the next one; a rollback of the service leaves every file readable because the older code sees the same required fields. - A record whose `schema` is a version the running code does not know is skipped on list and returned `409 hux.schema_unknown` on direct fetch, and logged once per family per process. - The service refuses to open a tree whose `data_layout_version` is greater than the one it was built for, or whose `min_reader_contract_version` is above its own `VERSION`, and it never bumps either on its own; a v2 layout is a separate migration tool that builds `v2/` beside `v1/` and flips the manifest last. A release that only adds optional fields leaves `min_reader_contract_version` alone, which is exactly what lets the previous release read after a rollback. - Index files are never migrated; they are deleted and rebuilt. ## 6. Authorization The router asserts the identity tuple: `tenant_slot` (`^slot-[0-9]{1,3}$`), `subject` (`usr_`), `surface` and `trust` (`router|relay|worker`); the service checks the slot equals its own `HERMES_TENANT_SLOT` and refuses otherwise. Every filesystem path is then built by `store.path_for(user, family, *ids)`, where `user` must match `^usr_[0-9a-f]{16,64}$` and each id must match the `common.schema.json` id pattern `^[a-z]{2,6}_[A-Za-z0-9._-]{4,80}$` (dates in `audit/` match `^\d{4}-\d{2}-\d{2}$`, versions `^\d+$`, hashes `^[0-9a-f]{64}$`). The pattern admits `.` but not `/`, so `..` alone is impossible in an id; the resolved path is still checked with `os.path.commonpath` against the user's subtree, the same belt-and-braces the router applies in `normalizeTenantMediaPath`. No caller-supplied string is ever joined into a path without going through `path_for`. Ownership: records with an `owner` field must equal the asserted user on read and write; records without one (events, approvals, receipts, research) are owned through their `conversation_id`, whose document is loaded and checked first. A lookup that fails ownership returns `404`, not `403`, so ids cannot be probed. Every read and mutation appends one line to `audit/outcomes/.jsonl` shaped as `common.schema.json#/$defs/audit_outcome`, also for denials. The line is the contract object plus an `_meta` envelope the store owns: ```json {"at":"...","identity":{"tenant_slot":"slot-3","subject":"usr_...","surface":"chat","trust":"router"}, "action":"memory.forget","resource":"memory/mem_...","outcome":"allow","reason":"owner_match", "_meta":{"schema":"hux.audit_outcome.v1","id":"aud_...","request_id":"...","idempotency_key":"...", "revision_before":4,"revision_after":5,"build":{"commit":"...","image_digest":"sha256:..."}}} ``` `action` is `.`; `resource` is the family-relative record path; `outcome` is `allow|deny|not_found|conflict|flag_off`; `reason` is a short fixed vocabulary (`owner_match`, `owner_mismatch`, `invalid_id`, `revision_conflict`, `cap_exceeded`, `policy_violation`, `unconditional_write`, `replayed`, `family_readonly`). Audit lines never contain record content. ## 7. Module map All modules live in the `hux` package under `dockerfiles/hermes-hux-foundation/` (where `contracts.py` and `rules.py`, the former `hux_contracts.py` and `hux_policy.py`, already sit), stdlib only, each at most 500 lines, and each tested in `testing/tests/test_hermes_hux_foundation_*.py`. | Module | Responsibility | |---|---| | `identity.py` | Parse and validate router headers (slot, `usr_` hash, request id); id and hash regexes from `common.schema.json`; id minting; ownership check helpers. | | `flags.py` | Wrap `rules.flag_enabled` with per-request evaluation of `HUX_FLAGS`; map each route to its card flag; 404 when the chain is off. | | `store.py` | `path_for`, `MANIFEST.json`, `.lock`, per-path lock registry, atomic document write, fsync'd append, tmp-file cleanup, crash validation and truncation, revision/`If-Match`, idempotency-key storage, size caps. No record semantics. | | `events.py` | Per-conversation JSONL log: seq allocation, `seq.json`, `after_seq` reads, SSE cursor, search-bag updates, redaction on read, ephemeral-mode short circuit. | | `memory.py` | Ledger append, state machine via `rules.MEMORY_TRANSITIONS`, `memory_policy_violations`, tombstones, retrieval index, export snapshots, compaction. | | `privacy.py` | Topic detection hooks, `conversation_topics.json`, notices log, `forget` for a conversation, retention job (`expire_memory`, `decay_topic_context`, `purge_forgotten_content`, `report`) and `hux.retention_audit.v1`. | | `artifacts.py` | Artifact documents, version list, content-addressed blobs and `refs.json`, diff between versions, promotion, lineage, blob GC. | | `research.py` | Sources, passages (hash dedupe), per-message citation logs, notebooks, research index. | | `policy.py` | Policy documents per scope, `effective_decision`, approvals queue and terminal transitions, cancellation receipts and `by_run.json`. | | `organization.py` | Projects and conversations, indexes, branch lineage, search over the `search_index` fields, suggestion state and `suggestion_allowed` gating. | | `audit.py` | `hux.audit_outcome.v1` writer with daily rotation, the `why` vocabulary, structured logging of degraded families, age-based pruning. | | `http.py` | `ThreadingHTTPServer` on the tenant loopback port, route table for `/hux/v1`, JSON and SSE responses, `{"items":[],"next":cursor}` list wrapping, error mapping (`404/409/412/413`), `/healthz`. | Dependency direction is one way: `http` -> family modules -> `store`, with `identity`, `flags` and `audit` used by everyone and importing only `store` and `rules`/`contracts`. Family modules never touch the filesystem directly.