atlas-iac/docs/hux/DATA-MODEL.md

# HUX foundation: on-PVC data model

`hux-foundation` is a Python stdlib service that runs inside every tenant pod
and owns the records defined in `services/hermes/contracts/hux/`. It has no
database and no storage shared between tenants. Everything it keeps lives on
the tenant's own `home` PVC (mounted at `/opt/data`, beside `webui/`,
`workspace/`, `home/` and the Telegram media roots `cache/images` and
`workspace`). The Go router in `services/hermes/router/` fronts it: the router
resolves Keycloak identity to a tenant slot and forwards the
`common.schema.json#/$defs/identity` tuple (`tenant_slot`, `subject`,
`surface`, `trust`) as headers, and enforces `hux.foundation` before any
request reaches this service. The service trusts nothing else about the
caller.

The conventions below are lifted from what already works in this repo:
`hermes_model_routing._atomic_write` (temp file + `os.replace`),
`cli_lane_records.atomic_json` (journal first, act second), and
`execution_pool_store` (idempotent add by digest, terminal rows only are ever
garbage-collected). SQLite is deliberately not used: the WebUI already owns
`state.db` on this volume and a second writer with its own WAL is one more
thing to recover; JSON and JSONL are greppable during an incident.

## 1. Directory layout

Tenant root is `HUX_DATA_ROOT`, default `/opt/data/hux`. The layout version is
a directory (`v1`), so a v2 layout can be built beside v1 and swapped by
`MANIFEST.json`, never by rewriting v1 in place.

```
/opt/data/hux/
  MANIFEST.json                       hux.manifest.v1 (identity.schema.json): contract and layout versions
  v1/
    .lock                             advisory lock file (flock) proving single writer per tenant
    users/<usr_hash>/                 one subtree per hashed subject; nothing lives above it
      profile.json                    {schema:"hux.user_profile.v1", memory_enabled, default_mode, revision}
      events/
        <conv_id>/
          events.jsonl                append-only hux.event.v1, one per line, ordered by seq
          seq.json                    {next_seq, last_event_id, bytes, checkpointed_at}
          idempotency.jsonl           {idempotency_key, event_id, seq, at}; last 10k keys kept
      memory/
        ledger.jsonl                  append-only hux.memory.v1 snapshots; a status change appends a full record
        tombstones.jsonl              {memory_id, at, reason, purged:bool}; written on forget and on decay
        index.json                    retrieval index (section 3), rebuilt from the ledger on demand
        exports/<ts>.json             GET /memory/export snapshots, audited, pruned after 7 days
      projects/
        <prj_id>.json                 hux.project.v1 + revision
        index.json                    {items:[{id,name,tags,pinned,archived,updated_at}], revision}
      conversations/
        <conv_id>.json                hux.conversation.v1 + revision
        index.json                    {items:[{id,project_id,title,tags,pinned,archived,mode,branch,last_message_at}]}
        search/
          <conv_id>.json              per-conversation term postings for message_text and artifact_titles
      artifacts/
        <art_id>.json                 hux.artifact.v1 + revision (versions[] is the version list)
        index.json                    {items:[{id,type,title,conversation_id,project_id,current_version}]}
        blobs/sha256/<aa>/<hash>      content-addressed, immutable, 0400, fan-out on first two hex chars
        blobs/refs.json               {hash: [ "<art_id>@<version>", ...]} for safe purge
      research/
        sources/<src_id>.json         hux.source.v1
        passages/<psg_id>.json        hux.passage.v1 (text + locator; hash is the dedupe key)
        citations/<msg_id>.jsonl      hux.citation.v1 per message, append-only
        notebooks/<nb_id>.json        hux.research_notebook.v1 + revision
        index.json                    {by_conversation:{conv_id:[nb_id]}, by_message:{msg_id:count}, passage_hashes:{hash:psg_id}}
      policy/
        global.json                   hux.policy.v1 for scope level global
        project/<prj_id>.json         hux.policy.v1 per project
        conversation/<conv_id>.json   hux.policy.v1 per conversation
        approvals/<apr_id>.json       hux.approval.v1; terminal records never change again
        approvals/pending.json        {items:[apr_id], revision}; the queue the UI polls
        receipts/<rcpt_id>.json       hux.cancel_receipt.v1
        receipts/by_run.json          {run_id: rcpt_id}
      suggestions/
        state.json                    {schema:"hux.suggestion_states.v1", items:{sug_id: hux.suggestion_state.v1}, revision}
      privacy/
        notices.jsonl                 hux.privacy_notice.v1 as shown, append-only
        conversation_topics.json      {conv_id:{topic, first_seen, decay_at, memory_disabled:bool}}
        forgotten.jsonl               {conv_id, requested_at, purged_at, counts}
      audit/
        outcomes/<YYYY-MM-DD>.jsonl   one hux.audit_outcome.v1 per read or mutation (section 6)
        retention/<YYYY-MM-DD>.json   hux.retention_audit.v1 per run
    catalog/
      suggestions.json                hux.suggestion.v1 catalog shipped with the image (read-only copy)
      privacy_policy.json             hux.privacy_policy.v1 as served, with the policy version
```

The pod is single-tenant, so the `users/` level is not multi-tenancy; it is
the guarantee that every path contains the hashed subject the router asserted
and that a second subject on the same pod (operator break-glass, future
household sharing) can never see another's tree without a distinct path.

Record ids are minted by the service: `<prefix>_<ts36><6 random base32>`, e.g.
`evt_m0k3xq9a2bcd7f`, so they sort by creation time and satisfy the
`common.schema.json` id pattern. `hux.audit_outcome.v1` and
`hux.user_profile.v1` are foundation-internal records; they follow the same
provenance and versioning rules but are not contracts the UI codes against.

## 2. Write semantics

**Atomic document write.** Serialise with `json.dumps(sort_keys=True,
ensure_ascii=False)` plus a trailing newline, write to
`.<name>.<pid>.<counter>.tmp` in the same directory, `flush()`, `os.fsync(fd)`,
`os.replace(tmp, final)`, then `os.fsync(dir_fd)` so the rename itself is
durable. Temp files that survive a crash are deleted on open. Blobs are
written the same way under their hash; an existing blob is never rewritten
(compare size and hash; on mismatch refuse and audit).

**Append.** JSONL families open with `O_APPEND`, write the whole line in one
`os.write`, and `fsync` the file before the HTTP response is sent. The
controlling checkpoint (`seq.json`, `index.json`) is written atomically after
the append; a checkpoint may lag the log, never lead it.

**Crash recovery.** On first access to a family the store validates the log:
each line must parse and, for events, `seq` must equal the previous `seq + 1`.
A trailing partial line (no newline, or JSON error on the last line only) is
truncated to the last good newline; a bad line anywhere else is a hard error
(the family is marked read-only, an audit outcome is written, `/healthz`
reports `degraded`). The checkpoint is then reconciled from the log:
`next_seq` = last good `seq + 1`, indexes rebuilt if `checkpointed_at` is
older than the log mtime.

**Idempotency.** Every mutating request may carry `Idempotency-Key`
(`common.schema.json#/$defs/idempotency_key`, `^[A-Za-z0-9._:-]{8,120}$`). For
events the key is stored in `idempotency.jsonl`; a replay returns the original
event (same `id`, same `seq`) with `HTTP 200` and `HUX-Replayed: true`. For
documents the key is stored in the record's `_meta.idempotency_keys` (last 16)
and a replay returns the current record without bumping `revision`. Event
`id` is unique per tenant; an append whose `id` already exists in the last
checkpoint window is treated as a replay, not a duplicate.

**Seq allocation.** The HTTP server is a single process,
`ThreadingHTTPServer`. Locks are per path, held in a process-wide
`dict[str, threading.RLock]` guarded by one `threading.Lock`. The unit of
locking is the family directory for JSONL (`events/<conv>`, `memory/`) and the
document path for JSON. Seq is allocated inside the conversation lock:
read `next_seq` from memory (loaded from `seq.json` once per open), assign,
append, fsync, write `seq.json`, release. Since one process owns the PVC,
`v1/.lock` is taken with `fcntl.flock(LOCK_EX|LOCK_NB)` at start-up so an
accidental second replica fails fast instead of interleaving appends.

**Optimistic concurrency.** Every JSON document carries an integer
`revision` (starts at 1) beside the contract fields; it is served as the
`ETag`. `PATCH`/`PUT` require `If-Match: <revision>`; a mismatch returns
`409` with a `hux.error.v1` body (`code: conflict`, the current revision in
`details`) as `common.schema.json#/$defs/revision` specifies. A missing `If-Match` is accepted only when the
request carries an `Idempotency-Key`, and then the write is last-writer-wins
with the outcome audited as `why: "unconditional_write"`. Indexes carry their
own `revision` and are rewritten under the family lock after the document.

**Caps.** Rejected with `hux.error.v1` `too_large` (413, size) or `conflict`
(409, count):
- event line 64 KiB, `detail` 32 KiB; 50 000 events per conversation, then
  the conversation is `archived` and further appends need a branch
- memory content 2 000 chars (schema), 5 000 live entries, ledger 64 MiB
  before compaction (section 4)
- conversation and project documents 256 KiB; 2 000 conversations, 200
  projects, `artifact_ids` 500 (schema)
- artifact blob 25 MiB (`tenantMediaLimit` is 50 MiB; half leaves room for
  the Telegram path), 200 versions per artifact, 2 000 artifacts, blob store
  2 GiB per tenant
- passage text 4 000 chars, 10 000 sources, 50 000 passages
- audit outcomes are never capped by count; they rotate daily and age out

## 3. Indexes

Nothing is indexed that a scan cannot rebuild; every `index.json` is a cache
of its family and carries `built_from` (log bytes or document count) so a
stale index is detected and rebuilt rather than trusted.

**Conversation search** (`GET /hux/v1/search?q=`) covers exactly
`project.schema.json#/$defs/search_index`: `title`, `tags`, `project_name`
come from `conversations/index.json` joined with `projects/index.json` in
memory (a few thousand rows, scanned per query); `message_text` and
`artifact_titles` come from `conversations/search/<conv_id>.json`, a per
conversation bag of lowercased, punctuation-stripped terms with positions,
updated when a `message.*` event or `artifact.*` event is appended. Query
terms are ANDed; ranking is title hit > tag hit > term frequency > recency.
Events with `sensitivity: restricted` or `redaction.level: full` never enter
the search bag, so a search result can never leak what the timeline hides.

**Memory retrieval** (`memory/index.json`) is a term index over `content` of
`active` entries only, keyed by `scope.level`/`scope_id` and `topic`, plus
`expires` sorted by the effective expiry (`expires_at` or `created_at +
decay_days`) for the retention job. Entries are added on `approved`
(automatic or user), removed on `rejected`, `expired`, `forgotten`. Retrieval
reads the index, then loads the newest ledger snapshot for each id and drops
any whose status is no longer `active`, so a lagging index fails safe.

**Tombstones remove from retrieval before content is gone.** `forget` is two
writes under the memory lock: append a ledger snapshot with
`status: forgotten`, `content: ""`, audit `forgotten`; then append
`tombstones.jsonl` `{memory_id, at, reason, purged:false}`. The index write
follows. Both the retrieval path and the export path consult the tombstone
set (loaded once, appended in memory) before returning anything, which is
what makes "do not remember" hold even if the index rebuild is interrupted.
The earlier ledger lines that still contain content are what the purge job
in section 4 rewrites. The same tombstone file records `disable_memory_here`
per conversation via `privacy/conversation_topics.json`, which retrieval also
checks: a conversation with `memory_disabled` contributes no entries and
receives none.

**Research** indexes by hash: `passage_hashes` dedupes passages across
sources, `by_message` lets `GET /messages/{id}/citations` open one JSONL
without listing a directory, `by_conversation` backs the notebook drawer.
Approvals index only the pending queue; terminal approvals are found by id.

## 4. Retention

One thread runs the retention job every `retention_audit.interval_days`
(1 day) at a jittered hour, and on demand via `POST /hux/v1/admin/retention`
(worker surface only). Each run writes `audit/retention/<date>.json` as
`hux.retention_audit.v1` with the counts the schema names, so a day without
the record is itself a finding.

- `expire_memory`: entries whose effective expiry has passed get a new ledger
  snapshot `status: expired` and leave the index. Decay means the entry
  expires `decay_days` after `created_at` unless a later ledger snapshot
  carries a newer `updated_at` from an `approved` audit action, which resets
  the clock once.
- `decay_topic_context`: `privacy/conversation_topics.json` rows past
  `decay_at` (`PRIVACY_TOPICS[topic].decay_days` after `first_seen`) cause
  the conversation's events with that `sensitivity` to be rewritten with
  `redaction.level: full` and `detail` removed; `summary` is replaced by the
  topic notice text. Seq, ids and provenance are kept, so the timeline stays
  contiguous.
- `purge_forgotten_content`: for every tombstone with `purged:false`, rewrite
  `ledger.jsonl` (temp + replace, under the memory lock) replacing `content`
  with `""` on every snapshot of that id, keeping `audit[]`; then set
  `purged:true`. Forgotten conversations (`POST /conversations/{id}/forget`)
  are handled the same way: `events.jsonl` is rewritten with `detail`
  dropped and `redaction.level: full`, `search/<conv>.json` is deleted,
  artifacts owned only by that conversation lose their blobs (via
  `blobs/refs.json`), and `privacy/forgotten.jsonl` records the counts. The
  conversation document stays with `archived: true` so branches still
  resolve their parent.
- `report`: bounds check. Per-family bounds for a home cluster (10 Gi PVC
  shared with the WebUI): events 1 GiB total and 180 days for archived
  conversations, memory ledger compacted when over 64 MiB (rewrite keeping
  only the newest snapshot per id plus every snapshot of ids with a
  tombstone), blobs 2 GiB with unreferenced blobs deleted 7 days after their
  last ref disappears, research 512 MiB and sources unreferenced by any
  notebook or citation for 90 days deleted, audit outcomes 90 days,
  retention audits 400 days, memory exports 7 days, idempotency keys 10 000
  per conversation. Terminal approvals and receipts are kept 180 days.
  Nothing in `audit/` is ever removed by a forget or purge; only age.

Private mode conversations (`retention: ephemeral`) are not written to
`events/` at all; the service returns `204` to appends and the router's
session memory is the only copy.

## 5. Migrations

`MANIFEST.json`:

```json
{"schema":"hux.manifest.v1","contract_version":"1.0.0","data_layout_version":1,
 "min_reader_contract_version":"1.0.0","created_at":"...","updated_at":"..."}
```

`contract_version` is `services/hermes/contracts/hux/VERSION`; the service
that last opened the tree writes its own version there and bumps
`updated_at`. `data_layout_version` names the `v<N>/` directory in use.

Rules:
- Additive only within a layout. A release may add optional fields to a
  record, add a new file name, add a new directory, add an enum value that a
  reader can ignore. It may not rename or remove a field, change a field's
  type, change the meaning of an existing enum value, change the id pattern,
  change `seq` semantics, or move a family to a different path.
- Every record keeps its contract `schema` value (`hux.event.v1`). Internal
  bookkeeping is under `_meta` (`revision`, `idempotency_keys`, `built_from`),
  which is stripped before a record is served and which readers must ignore.
- Readers ignore unknown fields and never fail a family because one record
  has an unknown optional key. A reader from the previous release therefore
  reads records from the next one; a rollback of the service leaves every
  file readable because the older code sees the same required fields.
- A record whose `schema` is a version the running code does not know is
  skipped on list and returned `409 hux.schema_unknown` on direct fetch, and
  logged once per family per process.
- The service refuses to open a tree whose `data_layout_version` is
  greater than the one it was built for, or whose
  `min_reader_contract_version` is above its own `VERSION`, and it never
  bumps either on its own; a v2 layout is a separate migration tool that
  builds `v2/` beside `v1/` and flips the manifest last. A release that only
  adds optional fields leaves `min_reader_contract_version` alone, which is
  exactly what lets the previous release read after a rollback.
- Index files are never migrated; they are deleted and rebuilt.

## 6. Authorization

The router asserts the identity tuple: `tenant_slot` (`^slot-[0-9]{1,3}$`),
`subject` (`usr_<hash>`), `surface` and `trust` (`router|relay|worker`); the
service checks the slot equals its own `HERMES_TENANT_SLOT` and refuses
otherwise. Every filesystem path is then built by `store.path_for(user,
family, *ids)`, where `user` must match `^usr_[0-9a-f]{16,64}$` and each id
must match the `common.schema.json` id pattern
`^[a-z]{2,6}_[A-Za-z0-9._-]{4,80}$` (dates in `audit/` match
`^\d{4}-\d{2}-\d{2}$`, versions `^\d+$`, hashes `^[0-9a-f]{64}$`). The
pattern admits `.` but not `/`, so `..` alone is impossible in an id; the
resolved path is still checked with `os.path.commonpath` against the user's
subtree, the same belt-and-braces the router applies in
`normalizeTenantMediaPath`. No caller-supplied string is ever joined into a
path without going through `path_for`.

Ownership: records with an `owner` field must equal the asserted user on
read and write; records without one (events, approvals, receipts, research)
are owned through their `conversation_id`, whose document is loaded and
checked first. A lookup that fails ownership returns `404`, not `403`, so
ids cannot be probed.

Every read and mutation appends one line to `audit/outcomes/<date>.jsonl`
shaped as `common.schema.json#/$defs/audit_outcome`, also for denials. The
line is the contract object plus an `_meta` envelope the store owns:

```json
{"at":"...","identity":{"tenant_slot":"slot-3","subject":"usr_...","surface":"chat","trust":"router"},
 "action":"memory.forget","resource":"memory/mem_...","outcome":"allow","reason":"owner_match",
 "_meta":{"schema":"hux.audit_outcome.v1","id":"aud_...","request_id":"...","idempotency_key":"...",
          "revision_before":4,"revision_after":5,"build":{"commit":"...","image_digest":"sha256:..."}}}
```

`action` is `<family>.<op>`; `resource` is the family-relative record path;
`outcome` is `allow|deny|not_found|conflict|flag_off`; `reason` is a short
fixed vocabulary (`owner_match`, `owner_mismatch`, `invalid_id`,
`revision_conflict`, `cap_exceeded`, `policy_violation`,
`unconditional_write`, `replayed`, `family_readonly`). Audit lines never
contain record content.

## 7. Module map

All modules live in the `hux` package under `dockerfiles/hermes-hux-foundation/`
(where `contracts.py` and `rules.py`, the former `hux_contracts.py` and
`hux_policy.py`, already sit), stdlib only, each at most 500 lines, and each
tested in `testing/tests/test_hermes_hux_foundation_*.py`.

| Module | Responsibility |
|---|---|
| `identity.py` | Parse and validate router headers (slot, `usr_` hash, request id); id and hash regexes from `common.schema.json`; id minting; ownership check helpers. |
| `flags.py` | Wrap `rules.flag_enabled` with per-request evaluation of `HUX_FLAGS`; map each route to its card flag; 404 when the chain is off. |
| `store.py` | `path_for`, `MANIFEST.json`, `.lock`, per-path lock registry, atomic document write, fsync'd append, tmp-file cleanup, crash validation and truncation, revision/`If-Match`, idempotency-key storage, size caps. No record semantics. |
| `events.py` | Per-conversation JSONL log: seq allocation, `seq.json`, `after_seq` reads, SSE cursor, search-bag updates, redaction on read, ephemeral-mode short circuit. |
| `memory.py` | Ledger append, state machine via `rules.MEMORY_TRANSITIONS`, `memory_policy_violations`, tombstones, retrieval index, export snapshots, compaction. |
| `privacy.py` | Topic detection hooks, `conversation_topics.json`, notices log, `forget` for a conversation, retention job (`expire_memory`, `decay_topic_context`, `purge_forgotten_content`, `report`) and `hux.retention_audit.v1`. |
| `artifacts.py` | Artifact documents, version list, content-addressed blobs and `refs.json`, diff between versions, promotion, lineage, blob GC. |
| `research.py` | Sources, passages (hash dedupe), per-message citation logs, notebooks, research index. |
| `policy.py` | Policy documents per scope, `effective_decision`, approvals queue and terminal transitions, cancellation receipts and `by_run.json`. |
| `organization.py` | Projects and conversations, indexes, branch lineage, search over the `search_index` fields, suggestion state and `suggestion_allowed` gating. |
| `audit.py` | `hux.audit_outcome.v1` writer with daily rotation, the `why` vocabulary, structured logging of degraded families, age-based pruning. |
| `http.py` | `ThreadingHTTPServer` on the tenant loopback port, route table for `/hux/v1`, JSON and SSE responses, `{"items":[],"next":cursor}` list wrapping, error mapping (`404/409/412/413`), `/healthz`. |

Dependency direction is one way: `http` -> family modules -> `store`, with
`identity`, `flags` and `audit` used by everyone and importing only `store`
and `rules`/`contracts`. Family modules never touch the filesystem directly.