atlas-iac/docs/hux/DATA-MODEL.md

HUX foundation: on-PVC data model

hux-foundation is a Python stdlib service that runs inside every tenant pod and owns the records defined in services/hermes/contracts/hux/. It has no database and no storage shared between tenants. Everything it keeps lives on the tenant's own home PVC (mounted at /opt/data, beside webui/, workspace/, home/ and the Telegram media roots cache/images and workspace). The Go router in services/hermes/router/ fronts it: the router resolves Keycloak identity to a tenant slot and forwards the common.schema.json#/$defs/identity tuple (tenant_slot, subject, surface, trust) as headers, and enforces hux.foundation before any request reaches this service. The service trusts nothing else about the caller.

The conventions below are lifted from what already works in this repo: hermes_model_routing._atomic_write (temp file + os.replace), cli_lane_records.atomic_json (journal first, act second), and execution_pool_store (idempotent add by digest, terminal rows only are ever garbage-collected). SQLite is deliberately not used: the WebUI already owns state.db on this volume and a second writer with its own WAL is one more thing to recover; JSON and JSONL are greppable during an incident.

1. Directory layout

Tenant root is HUX_DATA_ROOT, default /opt/data/hux. The layout version is a directory (v1), so a v2 layout can be built beside v1 and swapped by MANIFEST.json, never by rewriting v1 in place.

/opt/data/hux/
  MANIFEST.json                       hux.manifest.v1 (identity.schema.json): contract and layout versions
  v1/
    .lock                             advisory lock file (flock) proving single writer per tenant
    users/<usr_hash>/                 one subtree per hashed subject; nothing lives above it
      profile.json                    {schema:"hux.user_profile.v1", memory_enabled, default_mode, revision}
      events/
        <conv_id>/
          events.jsonl                append-only hux.event.v1, one per line, ordered by seq
          seq.json                    {next_seq, last_event_id, bytes, checkpointed_at}
          idempotency.jsonl           {idempotency_key, event_id, seq, at}; last 10k keys kept
      memory/
        ledger.jsonl                  append-only hux.memory.v1 snapshots; a status change appends a full record
        tombstones.jsonl              {memory_id, at, reason, purged:bool}; written on forget and on decay
        index.json                    retrieval index (section 3), rebuilt from the ledger on demand
        exports/<ts>.json             GET /memory/export snapshots, audited, pruned after 7 days
      projects/
        <prj_id>.json                 hux.project.v1 + revision
        index.json                    {items:[{id,name,tags,pinned,archived,updated_at}], revision}
      conversations/
        <conv_id>.json                hux.conversation.v1 + revision
        index.json                    {items:[{id,project_id,title,tags,pinned,archived,mode,branch,last_message_at}]}
        search/
          <conv_id>.json              per-conversation term postings for message_text and artifact_titles
      artifacts/
        <art_id>.json                 hux.artifact.v1 + revision (versions[] is the version list)
        index.json                    {items:[{id,type,title,conversation_id,project_id,current_version}]}
        blobs/sha256/<aa>/<hash>      content-addressed, immutable, 0400, fan-out on first two hex chars
        blobs/refs.json               {hash: [ "<art_id>@<version>", ...]} for safe purge
      research/
        sources/<src_id>.json         hux.source.v1
        passages/<psg_id>.json        hux.passage.v1 (text + locator; hash is the dedupe key)
        citations/<msg_id>.jsonl      hux.citation.v1 per message, append-only
        notebooks/<nb_id>.json        hux.research_notebook.v1 + revision
        index.json                    {by_conversation:{conv_id:[nb_id]}, by_message:{msg_id:count}, passage_hashes:{hash:psg_id}}
      policy/
        global.json                   hux.policy.v1 for scope level global
        project/<prj_id>.json         hux.policy.v1 per project
        conversation/<conv_id>.json   hux.policy.v1 per conversation
        approvals/<apr_id>.json       hux.approval.v1; terminal records never change again
        approvals/pending.json        {items:[apr_id], revision}; the queue the UI polls
        receipts/<rcpt_id>.json       hux.cancel_receipt.v1
        receipts/by_run.json          {run_id: rcpt_id}
      suggestions/
        state.json                    {schema:"hux.suggestion_states.v1", items:{sug_id: hux.suggestion_state.v1}, revision}
      privacy/
        notices.jsonl                 hux.privacy_notice.v1 as shown, append-only
        conversation_topics.json      {conv_id:{topic, first_seen, decay_at, memory_disabled:bool}}
        forgotten.jsonl               {conv_id, requested_at, purged_at, counts}
      audit/
        outcomes/<YYYY-MM-DD>.jsonl   one hux.audit_outcome.v1 per read or mutation (section 6)
        retention/<YYYY-MM-DD>.json   hux.retention_audit.v1 per run
    catalog/
      suggestions.json                hux.suggestion.v1 catalog shipped with the image (read-only copy)
      privacy_policy.json             hux.privacy_policy.v1 as served, with the policy version

The pod is single-tenant, so the users/ level is not multi-tenancy; it is the guarantee that every path contains the hashed subject the router asserted and that a second subject on the same pod (operator break-glass, future household sharing) can never see another's tree without a distinct path.

Record ids are minted by the service: <prefix>_<ts36><6 random base32>, e.g. evt_m0k3xq9a2bcd7f, so they sort by creation time and satisfy the common.schema.json id pattern. hux.audit_outcome.v1 and hux.user_profile.v1 are foundation-internal records; they follow the same provenance and versioning rules but are not contracts the UI codes against.

2. Write semantics

Atomic document write. Serialise with json.dumps(sort_keys=True, ensure_ascii=False) plus a trailing newline, write to .<name>.<pid>.<counter>.tmp in the same directory, flush(), os.fsync(fd), os.replace(tmp, final), then os.fsync(dir_fd) so the rename itself is durable. Temp files that survive a crash are deleted on open. Blobs are written the same way under their hash; an existing blob is never rewritten (compare size and hash; on mismatch refuse and audit).

Append. JSONL families open with O_APPEND, write the whole line in one os.write, and fsync the file before the HTTP response is sent. The controlling checkpoint (seq.json, index.json) is written atomically after the append; a checkpoint may lag the log, never lead it.

Crash recovery. On first access to a family the store validates the log: each line must parse and, for events, seq must equal the previous seq + 1. A trailing partial line (no newline, or JSON error on the last line only) is truncated to the last good newline; a bad line anywhere else is a hard error (the family is marked read-only, an audit outcome is written, /healthz reports degraded). The checkpoint is then reconciled from the log: next_seq = last good seq + 1, indexes rebuilt if checkpointed_at is older than the log mtime.

Idempotency. Every mutating request may carry Idempotency-Key (common.schema.json#/$defs/idempotency_key, ^[A-Za-z0-9._:-]{8,120}$). For events the key is stored in idempotency.jsonl; a replay returns the original event (same id, same seq) with HTTP 200 and HUX-Replayed: true. For documents the key is stored in the record's _meta.idempotency_keys (last 16) and a replay returns the current record without bumping revision. Event id is unique per tenant; an append whose id already exists in the last checkpoint window is treated as a replay, not a duplicate.

Seq allocation. The HTTP server is a single process, ThreadingHTTPServer. Locks are per path, held in a process-wide dict[str, threading.RLock] guarded by one threading.Lock. The unit of locking is the family directory for JSONL (events/<conv>, memory/) and the document path for JSON. Seq is allocated inside the conversation lock: read next_seq from memory (loaded from seq.json once per open), assign, append, fsync, write seq.json, release. Since one process owns the PVC, v1/.lock is taken with fcntl.flock(LOCK_EX|LOCK_NB) at start-up so an accidental second replica fails fast instead of interleaving appends.

Optimistic concurrency. Every JSON document carries an integer revision (starts at 1) beside the contract fields; it is served as the ETag. PATCH/PUT require If-Match: <revision>; a mismatch returns 409 with a hux.error.v1 body (code: conflict, the current revision in details) as common.schema.json#/$defs/revision specifies. A missing If-Match is accepted only when the request carries an Idempotency-Key, and then the write is last-writer-wins with the outcome audited as why: "unconditional_write". Indexes carry their own revision and are rewritten under the family lock after the document.

Caps. Rejected with hux.error.v1 too_large (413, size) or conflict (409, count):

• event line 64 KiB, detail 32 KiB; 50 000 events per conversation, then the conversation is archived and further appends need a branch
• memory content 2 000 chars (schema), 5 000 live entries, ledger 64 MiB before compaction (section 4)
• conversation and project documents 256 KiB; 2 000 conversations, 200 projects, artifact_ids 500 (schema)
• artifact blob 25 MiB (tenantMediaLimit is 50 MiB; half leaves room for the Telegram path), 200 versions per artifact, 2 000 artifacts, blob store 2 GiB per tenant
• passage text 4 000 chars, 10 000 sources, 50 000 passages
• audit outcomes are never capped by count; they rotate daily and age out

3. Indexes

Nothing is indexed that a scan cannot rebuild; every index.json is a cache of its family and carries built_from (log bytes or document count) so a stale index is detected and rebuilt rather than trusted.

Conversation search (GET /hux/v1/search?q=) covers exactly project.schema.json#/$defs/search_index: title, tags, project_name come from conversations/index.json joined with projects/index.json in memory (a few thousand rows, scanned per query); message_text and artifact_titles come from conversations/search/<conv_id>.json, a per conversation bag of lowercased, punctuation-stripped terms with positions, updated when a message.* event or artifact.* event is appended. Query terms are ANDed; ranking is title hit > tag hit > term frequency > recency. Events with sensitivity: restricted or redaction.level: full never enter the search bag, so a search result can never leak what the timeline hides.

Memory retrieval (memory/index.json) is a term index over content of active entries only, keyed by scope.level/scope_id and topic, plus expires sorted by the effective expiry (expires_at or created_at + decay_days) for the retention job. Entries are added on approved (automatic or user), removed on rejected, expired, forgotten. Retrieval reads the index, then loads the newest ledger snapshot for each id and drops any whose status is no longer active, so a lagging index fails safe.

Tombstones remove from retrieval before content is gone. forget is two writes under the memory lock: append a ledger snapshot with status: forgotten, content: "", audit forgotten; then append tombstones.jsonl {memory_id, at, reason, purged:false}. The index write follows. Both the retrieval path and the export path consult the tombstone set (loaded once, appended in memory) before returning anything, which is what makes "do not remember" hold even if the index rebuild is interrupted. The earlier ledger lines that still contain content are what the purge job in section 4 rewrites. The same tombstone file records disable_memory_here per conversation via privacy/conversation_topics.json, which retrieval also checks: a conversation with memory_disabled contributes no entries and receives none.

Research indexes by hash: passage_hashes dedupes passages across sources, by_message lets GET /messages/{id}/citations open one JSONL without listing a directory, by_conversation backs the notebook drawer. Approvals index only the pending queue; terminal approvals are found by id.

4. Retention

One thread runs the retention job every retention_audit.interval_days (1 day) at a jittered hour, and on demand via POST /hux/v1/admin/retention (worker surface only). Each run writes audit/retention/<date>.json as hux.retention_audit.v1 with the counts the schema names, so a day without the record is itself a finding.

• expire_memory: entries whose effective expiry has passed get a new ledger snapshot status: expired and leave the index. Decay means the entry expires decay_days after created_at unless a later ledger snapshot carries a newer updated_at from an approved audit action, which resets the clock once.
• decay_topic_context: privacy/conversation_topics.json rows past decay_at (PRIVACY_TOPICS[topic].decay_days after first_seen) cause the conversation's events with that sensitivity to be rewritten with redaction.level: full and detail removed; summary is replaced by the topic notice text. Seq, ids and provenance are kept, so the timeline stays contiguous.
• purge_forgotten_content: for every tombstone with purged:false, rewrite ledger.jsonl (temp + replace, under the memory lock) replacing content with "" on every snapshot of that id, keeping audit[]; then set purged:true. Forgotten conversations (POST /conversations/{id}/forget) are handled the same way: events.jsonl is rewritten with detail dropped and redaction.level: full, search/<conv>.json is deleted, artifacts owned only by that conversation lose their blobs (via blobs/refs.json), and privacy/forgotten.jsonl records the counts. The conversation document stays with archived: true so branches still resolve their parent.
• report: bounds check. Per-family bounds for a home cluster (10 Gi PVC shared with the WebUI): events 1 GiB total and 180 days for archived conversations, memory ledger compacted when over 64 MiB (rewrite keeping only the newest snapshot per id plus every snapshot of ids with a tombstone), blobs 2 GiB with unreferenced blobs deleted 7 days after their last ref disappears, research 512 MiB and sources unreferenced by any notebook or citation for 90 days deleted, audit outcomes 90 days, retention audits 400 days, memory exports 7 days, idempotency keys 10 000 per conversation. Terminal approvals and receipts are kept 180 days. Nothing in audit/ is ever removed by a forget or purge; only age.

Private mode conversations (retention: ephemeral) are not written to events/ at all; the service returns 204 to appends and the router's session memory is the only copy.

5. Migrations

MANIFEST.json:

{"schema":"hux.manifest.v1","contract_version":"1.0.0","data_layout_version":1,
 "min_reader_contract_version":"1.0.0","created_at":"...","updated_at":"..."}

contract_version is services/hermes/contracts/hux/VERSION; the service that last opened the tree writes its own version there and bumps updated_at. data_layout_version names the v<N>/ directory in use.

Rules:

• Additive only within a layout. A release may add optional fields to a record, add a new file name, add a new directory, add an enum value that a reader can ignore. It may not rename or remove a field, change a field's type, change the meaning of an existing enum value, change the id pattern, change seq semantics, or move a family to a different path.
• Every record keeps its contract schema value (hux.event.v1). Internal bookkeeping is under _meta (revision, idempotency_keys, built_from), which is stripped before a record is served and which readers must ignore.
• Readers ignore unknown fields and never fail a family because one record has an unknown optional key. A reader from the previous release therefore reads records from the next one; a rollback of the service leaves every file readable because the older code sees the same required fields.
• A record whose schema is a version the running code does not know is skipped on list and returned 409 hux.schema_unknown on direct fetch, and logged once per family per process.
• The service refuses to open a tree whose data_layout_version is greater than the one it was built for, or whose min_reader_contract_version is above its own VERSION, and it never bumps either on its own; a v2 layout is a separate migration tool that builds v2/ beside v1/ and flips the manifest last. A release that only adds optional fields leaves min_reader_contract_version alone, which is exactly what lets the previous release read after a rollback.
• Index files are never migrated; they are deleted and rebuilt.

6. Authorization

The router asserts the identity tuple: tenant_slot (^slot-[0-9]{1,3}$), subject (usr_<hash>), surface and trust (router|relay|worker); the service checks the slot equals its own HERMES_TENANT_SLOT and refuses otherwise. Every filesystem path is then built by store.path_for(user, family, *ids), where user must match ^usr_[0-9a-f]{16,64}$ and each id must match the common.schema.json id pattern ^[a-z]{2,6}_[A-Za-z0-9._-]{4,80}$ (dates in audit/ match ^\d{4}-\d{2}-\d{2}$, versions ^\d+$, hashes ^[0-9a-f]{64}$). The pattern admits . but not /, so .. alone is impossible in an id; the resolved path is still checked with os.path.commonpath against the user's subtree, the same belt-and-braces the router applies in normalizeTenantMediaPath. No caller-supplied string is ever joined into a path without going through path_for.

Ownership: records with an owner field must equal the asserted user on read and write; records without one (events, approvals, receipts, research) are owned through their conversation_id, whose document is loaded and checked first. A lookup that fails ownership returns 404, not 403, so ids cannot be probed.

Every read and mutation appends one line to audit/outcomes/<date>.jsonl shaped as common.schema.json#/$defs/audit_outcome, also for denials. The line is the contract object plus an _meta envelope the store owns:

{"at":"...","identity":{"tenant_slot":"slot-3","subject":"usr_...","surface":"chat","trust":"router"},
 "action":"memory.forget","resource":"memory/mem_...","outcome":"allow","reason":"owner_match",
 "_meta":{"schema":"hux.audit_outcome.v1","id":"aud_...","request_id":"...","idempotency_key":"...",
          "revision_before":4,"revision_after":5,"build":{"commit":"...","image_digest":"sha256:..."}}}

action is <family>.<op>; resource is the family-relative record path; outcome is allow|deny|not_found|conflict|flag_off; reason is a short fixed vocabulary (owner_match, owner_mismatch, invalid_id, revision_conflict, cap_exceeded, policy_violation, unconditional_write, replayed, family_readonly). Audit lines never contain record content.

7. Module map

All modules live in the hux package under dockerfiles/hermes-hux-foundation/ (where contracts.py and rules.py, the former hux_contracts.py and hux_policy.py, already sit), stdlib only, each at most 500 lines, and each tested in testing/tests/test_hermes_hux_foundation_*.py.

Module	Responsibility
identity.py	Parse and validate router headers (slot, usr_ hash, request id); id and hash regexes from common.schema.json; id minting; ownership check helpers.
flags.py	Wrap rules.flag_enabled with per-request evaluation of HUX_FLAGS; map each route to its card flag; 404 when the chain is off.
store.py	path_for, MANIFEST.json, .lock, per-path lock registry, atomic document write, fsync'd append, tmp-file cleanup, crash validation and truncation, revision/If-Match, idempotency-key storage, size caps. No record semantics.
events.py	Per-conversation JSONL log: seq allocation, seq.json, after_seq reads, SSE cursor, search-bag updates, redaction on read, ephemeral-mode short circuit.
memory.py	Ledger append, state machine via rules.MEMORY_TRANSITIONS, memory_policy_violations, tombstones, retrieval index, export snapshots, compaction.
privacy.py	Topic detection hooks, conversation_topics.json, notices log, forget for a conversation, retention job (expire_memory, decay_topic_context, purge_forgotten_content, report) and hux.retention_audit.v1.
artifacts.py	Artifact documents, version list, content-addressed blobs and refs.json, diff between versions, promotion, lineage, blob GC.
research.py	Sources, passages (hash dedupe), per-message citation logs, notebooks, research index.
policy.py	Policy documents per scope, effective_decision, approvals queue and terminal transitions, cancellation receipts and by_run.json.
organization.py	Projects and conversations, indexes, branch lineage, search over the search_index fields, suggestion state and suggestion_allowed gating.
audit.py	hux.audit_outcome.v1 writer with daily rotation, the why vocabulary, structured logging of degraded families, age-based pruning.
http.py	ThreadingHTTPServer on the tenant loopback port, route table for /hux/v1, JSON and SSE responses, {"items":[],"next":cursor} list wrapping, error mapping (404/409/412/413), /healthz.

Dependency direction is one way: http -> family modules -> store, with identity, flags and audit used by everyone and importing only store and rules/contracts. Family modules never touch the filesystem directly.