atlas-iac/services/hermes/NOTES.md

# Hermes on Atlas: operator guide

This is the mental model and demonstration script for the operator instance at
`triage.bstein.dev`. Read it once, then prove each section in the live UI. The
consumer instance at `hermes.bstein.dev` is intentionally separate and is not
the place to perform infrastructure triage.

`agent.hermes.bstein.dev` is the owner-only engineering control plane. Its root
serves the stock Hermes dashboard, including embedded chat/TUI and the native
sessions, files, models, logs, Kanban, skills, plugins, MCP, profiles, and
configuration surfaces. `/terminal/` remains the raw full-screen TUI. The
community WebUI is intentionally reserved for isolated consumer chat, where
its conversation-first layout is a better fit.

## Consumer chat and Telegram

`hermes.bstein.dev` uses the pinned Hermes WebUI rather than the operator
dashboard. Keycloak still authenticates every browser request, and the tenant
router permanently assigns each Keycloak subject to one Hermes process and one
PVC. The four slots are an isolation pool, not a provider round robin: every
user starts with the same automatic provider/fallback policy and may change the
model or reasoning effort for their own conversation. The WebUI and API reject
reasoning levels above `xhigh`.

Telegram is optional. The Keycloak bootstrap creates
`kv/atlas/hermes/chat-telegram` with a generated relay key and an empty
`bot_token`. After creating the shared bot with BotFather, set only that field:

```sh
vault kv patch kv/atlas/hermes/chat-telegram bot_token='<telegram bot token>'
```

Restart or reconcile `hermes-chat-router` after changing the token. A user then
signs in to the WebUI, selects `Telegram`, creates a ten-minute code, and sends
the displayed `/link` command to the bot. The router accepts only direct chats,
stores hashed Keycloak and Telegram identities, and forwards the message to
that user's tenant API with the shared relay key. `/unlink` works from Telegram
or the WebUI. Browser chat remains available when `bot_token` is empty.

The bot token and relay key must never be added to Git or a Kubernetes Secret.
The router does not log prompt bodies, raw Telegram IDs, link codes, or tokens.

## The one-sentence explanation

Hermes is the persistent agent runtime and control surface; Codex or the local
GPU model supplies reasoning, while Hermes supplies sessions, tools, skills,
approval rules, identity, and the repeatable procedure that connects the model
to Atlas evidence.

Hermes is not the model. Replacing `gpt-5.6-terra` with `gpt-oss:20b` changes the
reasoning engine, but it does not replace the Hermes sessions, tools, skills,
files, permission boundary, or workflow.

## Request and evidence path

```text
browser
  -> Keycloak login
  -> oauth2-proxy exact-user check
  -> operator Hermes pod and persistent workspace
  -> SOUL.md + AGENTS.md + selected skill
  -> Codex primary model
       -> read-only terminal/web/file tools
       -> Ariadne deterministic evidence
       -> retained Jenkins logs and artifacts
       -> Gitea commits and Flux state
       -> Kubernetes workloads, events, logs, and dependencies
       -> Pushgateway data queried through VictoriaMetrics
       -> Grafana query and alert context
  -> structured finding and proposed repo-side change
  -> human review and approval
  -> Git/Flux delivery outside Hermes

provider failure
  -> hermes-model-gate
       -> Hermes owns titan-24: Ollama serves gpt-oss:20b
       -> Wolf owns titan-24: gate returns 503; local inference yields
```

The operator agent pod runs on an ARM worker and does not reserve the GPU.
Ollama is the component on `titan-24` that owns GPU memory. The Lease
`hermes/titan-24-gpu-owner` controls whether the model gate admits local
inference. Normal operator conversations use independently authenticated Codex
first, so Wolf ownership does not need to block the web agent.

## Who does what

| Component | Responsibility | What it does not prove |
| --- | --- | --- |
| Hermes | Maintains the conversation, chooses tools/skills, reasons over evidence, saves reports, and enforces the approval experience | It is not the inference model and does not automatically have cluster-admin access |
| Ariadne | Collects and normalizes deterministic cluster, Jenkins, and quality evidence into a timestamped bundle; may optionally run a local diagnosis | Its diagnosis is not authoritative when stale, empty, or contradicted by the bundle |
| Jenkins | Runs the suites and retains console/artifact evidence | A final `exit 1` alone does not identify the first failed gate |
| Pushgateway | Receives suite/build quality telemetry | A cumulative counter is not the current build result |
| VictoriaMetrics | Stores and answers PromQL for quality and environment telemetry | A Grafana color is not direct proof of root cause |
| Grafana | Visualizes and alerts on VictoriaMetrics data | A red panel must be traced to its query, labels, time range, and raw series |
| Flux | Applies reviewed Git state to the cluster | A recent reconciliation is correlation until the changed path matches the failure |
| Kubernetes | Supplies workload, event, log, node, storage, and dependency evidence | An unrelated unhealthy pod is not proof that CI failed because of the cluster |

Direct Jenkins artifact requests currently require authorization and can return
HTTP 403. That does not break the workflow: Ariadne retains selected Jenkins
console tails and named artifact contents in its deterministic bundle. A report
must say `retained Ariadne evidence` when that fallback is used; it must not
pretend direct Jenkins access succeeded.

## Publishing an agent image after review

The `hermes-agent-image` Jenkins job is the only supported agent image builder.
It runs daemonless Kaniko v1.23.2 under the unbound `hermes-image-builder`
ServiceAccount without a service-account token, host socket,
privileged container, or writable Git credential. It accepts only an exact
40-character revision that is both the checked-out commit and current
`atlas/titan-iac` `main`, so a human must merge the source PR first.

The build pod requires an ARM64 worker and gives `hardware=rpi5` the maximum
scheduler preference. Healthy rpi4 workers remain valid fallback capacity when
the rpi5 pool cannot fit the full request. Cordoned, unhealthy, amd64, and
reserved last-resort nodes remain excluded by the required placement boundary.

Kaniko runs as UID 0 because it must unpack an image root filesystem and enter
that filesystem for Dockerfile `RUN` instructions. Its capability set is the
minimum proven by an exact ARM64 no-push build of this Dockerfile:
`CHOWN`, `FOWNER`, `DAC_OVERRIDE`, `SETGID`, and `SETUID`. `SETGID` is required
because pinned Kaniko applies the base image's supplementary group list before
each Dockerfile `RUN`; `SETUID` lets apt drop privileges to its `_apt` account
while downloading package indexes. It still has no privilege
escalation, service-account token, host path, daemon socket, or Docker/BuildKit
TCP endpoint and uses the runtime-default seccomp profile. This is residual
root-in-the-build-pod risk, bounded to disposable `emptyDir` storage and a
human-reviewed `main` revision; Dockerfile changes require the same scrutiny as
executable cluster code.

Pinned Kaniko v1.23.2 parses Dockerfile `RUN` heredocs but does not materialize
their inline files when executing a command. The Jenkins lane therefore enables
a bounded compatibility replay after copying the reviewed Dockerfile and runner
from the same checked-out commit. Before every replay, the runner inventories
the entire Dockerfile as logical instructions, including split opcodes,
operators, and either Dockerfile escape character. It rejects every heredoc
form outside the exact nine Node/Python blocks before launching an interpreter.
Each accepted body is then replayed immediately after its corresponding Kaniko
`RUN`, preserving source order before any dependent regression. Docker and
BuildKit keep their native behavior because the compatibility argument defaults
off. The final TypeScript build, Python compile, and source assertions remain
mandatory, so an omitted or drifted replay fails before any image can pass
release verification.

From agent.hermes, trigger that one fixed job with:

```sh
jenkins_image_build_trigger.py '<reviewed full main commit>'
```

The helper has no general Jenkins credential or caller-selectable job name. Its
Vault-projected token is bound by Jenkins only to `hermes-agent-image`. The job
also requires its fixed publish confirmation. Each run claims the unique tag
`git-<reviewed-sha>-build-<jenkins-build-number>` and refuses to overwrite an
existing tag. It cross-checks Kaniko's digest and tagged-image evidence, then
independently reads the pushed tag from Harbor before archiving a JSON record
and Flux digest patch. A successful post-condition then revalidates the exact
six-file evidence set and archives those six explicit paths with empty archives
forbidden. Apply that patch on a new branch and submit it for human review; the
build never changes Git, reconciles Flux, or deploys by itself.
The helper posts only to the fixed HTTPS Build Token Root endpoint and accepts
only its real queue responses: HTTP 201 or a non-followed HTTP 303 with an exact
same-origin `/queue/item/<number>/` location. An unauthenticated request is
denied, and the token cannot select, configure, read, or administer another
Jenkins job. The pipeline independently rejects any commit that is not the
current `origin/main`, preserving the human merge/review boundary.

Harbor independently enforces an enabled immutable-tag rule scoped to only the
`bstein/hermes-agent` repository and `git-*-build-*` tags. A second manifest PUT,
retag, or deletion is rejected by Harbor even if a caller bypasses the Jenkins
preflight. A revisioned, Flux-tracked policy Job creates or verifies that exact
rule with a runtime-only Vault credential; it refuses to alter a conflicting
rule and retries only explicit transport/readiness failures while Harbor
starts. The same Job grants the existing Jenkins publisher only
`immutable-tag:list` on `bstein`, preserving its other project scopes and never
calling Harbor's separate secret-rotation endpoint. The pipeline uses that
read-only permission to require the exact enabled rule before Kaniko starts.
That check also makes an already-running Jenkins controller fail closed during
rollout; Jenkins cannot become Ready on the new revision before the policy Job
succeeds.

The tracked rollout order is deliberate: the revisioned Vault role Job must
complete before the Vault Kustomization becomes Ready. A separate Flux
Kustomization then runs the seed Job under the dedicated
`hermes-jenkins-token-seed` identity, which can only create/read the exact
`kv/data/atlas/hermes/developer-jenkins` path and request random bytes. Jenkins
depends on both that seed and the Harbor policy Job; Hermes depends on Jenkins.
The seeder uses KV-v2 CAS create-only semantics. It never changes an existing
token, never fills a missing field in an existing secret, and fails closed when
it cannot distinguish absence from a read error. The recurring Vault
configuration job reconciles the same narrow role and policy.

Rotate this fixed-job token only as a coordinated operator action:

1. Stop new `hermes-agent-image` triggers and wait for its queue and executor to
   drain.
2. Read the current KV-v2 metadata version for
   `kv/atlas/hermes/developer-jenkins`.
3. Generate a fresh value from Vault and patch only `build_token` with
   `vault kv patch -cas=<current-version>`. Keep the value in a mode-0600
   temporary file or standard input, never a command argument or log.
4. Roll Jenkins first and wait until it is Ready, then roll `hermes-agent` and
   wait until it is Ready. Do not resume triggers between those two consumers.
5. Run one reviewed-main canary. If rollback is required, repeat the CAS patch
   with the previous value as another coordinated rotation; never delete the
   secret to make the seeder recreate it.

## The actual supervised triage algorithm

1. Classify the request as test/build triage, service health, or alert tuning.
2. Read the latest Ariadne diagnosis and deterministic bundle with HTTP GET.
3. Compare their timestamps. The deterministic bundle remains authoritative.
4. Select a terminal failed build. Keep running or unknown builds out of the
   terminal-failure list.
5. Identify the first enforced failed gate in this order:
   `style -> loc -> coverage -> tests -> gate_glue -> sonarqube -> supply_chain`.
6. Cite the smallest decisive Jenkins log/artifact evidence. State whether it
   came directly from Jenkins or from Ariadne retention.
7. Query the matching build/check/test series in VictoriaMetrics. Distinguish a
   current gauge from a cumulative counter and ignore zero-valued failure
   series.
8. Correlate recent Gitea commits and the relevant Flux revision by timestamp
   and affected path. Recency alone is not causation.
9. Check only the Kubernetes resources capable of explaining that failure.
   Classify observations as direct, contributing candidate, or background.
10. Return `Finding`, `Confidence`, `Evidence`, `Likely cause`, `Blast radius`,
    `Next checks`, `Repo-side fix`, and `Approval required`.
11. Stop at a proposal. A human reviews the patch or action; Git and Flux remain
    the delivery path.

## What Hermes can read and what it cannot do

The live `hermes-triage` ServiceAccount can get/list/watch ordinary workload,
log, event, ingress, storage, Flux, and image-automation metadata. It cannot
read Kubernetes Secret values, create exec sessions, patch Deployments, or
patch Flux Kustomizations. The configured command deny list also blocks common
Kubernetes and Flux mutations.

The security boundary is layered:

1. Keycloak authenticates the person.
2. oauth2-proxy restricts the operator surface to Brad.
3. The operator has a separate namespace, PVC, configuration, and ServiceAccount.
4. Kubernetes RBAC is the hard API authorization boundary.
5. NetworkPolicy limits reachable paths where configured.
6. Hermes approvals and instructions provide a user-facing safety layer.

A skill is procedure and context, not a permission grant. Adding a sentence to
a skill cannot bypass Kubernetes RBAC.

## What “Hermes learns the workflow” means

It does not silently retrain model weights on cluster data. Learning here means
that a successful repeated procedure is written as a reusable skill on the
operator PVC or versioned in Git. The skill describes when it should trigger,
the evidence order, interpretation rules, output contract, and safety boundary.

The current workflow has a versioned top-level skill and persisted specialist
skills for:

- orchestration and evidence reporting;
- retained Jenkins evidence;
- quality metrics;
- Gitea/Flux correlation;
- Kubernetes failure classification;
- Grafana metric provenance;
- Soteria backup health;
- approval-required actions.

The model still evaluates fresh variable evidence on every incident. The skill
makes the process repeatable; it does not freeze the answer.

## The four operator skills to remember

- `triage-titan-test-failures`: CI, tests, builds, quality gates, and suspected
  test-environment regressions.
- `triage-atlas-service-health`: active service or cluster incidents and red
  high-level health panels.
- `tune-atlas-alerts`: noisy alerts, impossible metrics, bad PromQL, and
  generator-owned Grafana corrections.
- `master-hermes-on-atlas`: hands-on training, assessments, architecture, and
  claim audits.

Do not start by choosing raw tools. State the operational question and let the
skill route to the narrow evidence source.

## Two live, repeatable proof cases

### Proof 6: Soteria build 272

- Terminal result: failure.
- Tests: 318 passed, zero failed.
- Local coverage: 96.195 percent.
- First enforced failure: SonarQube `new_coverage=0.0` against threshold 80.
- Runtime: Soteria remained Ready.
- Correct conclusion: a Sonar new-code policy/reporting problem, not a test or
  Kubernetes capacity failure.
- Correct restraint: do not weaken the policy until scanner import and baseline
  evidence identifies whether the issue is configuration or legitimate new
  uncovered code.

### Proof 7: Ananke build 242

- Terminal result: failure.
- Local coverage gate: 61.8 percent and failed.
- Other checks: tests, SonarQube, supply-chain, LOC, docs naming, and gate glue
  were healthy.
- Sonar new-code coverage: 93.1 percent, which is a different scope.
- Correct conclusion: local coverage input/scope caused the enforced failure;
  do not call the Sonar number contradictory without comparing inputs.
- Correct restraint: do not lower the threshold until the retained local gate
  inputs are readable.

The full sessions and redacted Markdown exports are in Sessions and
`Files/triage-proof`.

## Five-minute demonstration

1. Open Models and show `openai-codex/gpt-5.6-terra` as primary and
   `gpt-oss:20b` as local fallback.
2. Open Skills and show the four operator skills plus the persisted component
   skills.
3. Open `Proof 6 - Soteria 272 Sonar new coverage triage` in Sessions.
4. Point out the timestamped bundle, exact build/artifact evidence, metric
   corroboration, healthy runtime, fact/inference separation, and approval
   boundary.
5. Open Proof 7 and explain why local coverage and Sonar new-code coverage can
   differ without either number being fabricated.
6. End by showing that Hermes proposes a repo-side correction but cannot patch
   the Deployment or Flux Kustomization with its ServiceAccount.

Use this short explanation:

> Hermes is the persistent, permissioned workflow layer around the model. In
> this cluster Ariadne collects deterministic evidence, Hermes correlates it
> with retained Jenkins artifacts, Git/Flux state, Kubernetes health, and
> quality metrics, and then returns a supervised proposal. Repeated procedures
> become explicit skills. Kubernetes RBAC prevents the agent from turning a
> diagnosis into an unreviewed infrastructure change.

## Prompts that exercise the real system

- `Use $triage-titan-test-failures. Triage the worst current terminal CI failure and link every decisive piece of evidence.`
- `Use $triage-titan-test-failures. Triage Ananke build 242. Separate local coverage from Sonar new-code coverage.`
- `Use $triage-atlas-service-health. Explain the VictoriaMetrics outage from current state and retained events; separate recovered impact from current impact.`
- `Use $tune-atlas-alerts. Trace one currently firing alert to its generated source and raw PromQL, but do not edit it.`
- `Use $master-hermes-on-atlas. Assess me on the request path and permission boundary. One question at a time.`

## Honest limits

- Hermes does not currently apply production or cluster changes autonomously.
- Direct Jenkins console/artifact access may be forbidden; Ariadne retention is
  the current fallback and must be named as provenance.
- Local `gpt-oss:20b` is useful for bounded work but is slower and less reliable
  for multi-source triage than Codex.
- The current Ariadne local diagnosis can return an empty model response. That
  does not invalidate the deterministic bundle or prevent Codex-backed Hermes
  from triaging it.
- A single successful case demonstrates capability, not mastery. Mastery means
  leading different real incidents, improving the skill after failures, and
  teaching the architecture without prompts.

## Distributed execution-pool rollout boundary

- Do not reconcile the pool until the approved credential owner has provisioned
  six distinct Vault fields: `execution_worker_{0,1,2}_claude_credentials_json`
  and `execution_worker_{0,1,2}_codex_auth_json`. Each ordinal needs an
  independent account or refresh-token lineage; copying one rotating refresh
  token into multiple fields recreates the lost-update failure this design
  prevents.
- Worker credential files live on separate ordinal-owned RWO `provider-access`
  claims. Provider refresh updates these durable private copies across Pod
  restarts; they are never synchronized back to Vault. Rotate one bootstrap
  credential at a time through the reviewed Vault workflow and reinitialize
  only that ordinal after human approval.
- Existing task worktrees remain on the single local owner lane. Only tasks
  without `workspace_path` enter the distributed pool, where each ordinal owns
  one RWO checkout. Repository and base-branch identity comes from the canonical
  board registry rather than task-supplied metadata.
- Model Pods have no pool key, broker mount, broker egress, or Kubernetes token.
  Ordinal mediator Deployments are separate network identities that co-mount the
  matching workspace PVC; they alone authenticate exact-run results and reach the
  PR14 SCM broker.
- The ordinal workspace claim is `ReadWriteMany` because the worker and its
  mediator both mount it. With `ReadWriteOnce` a drain or preemption that moved
  only one of them left the other holding the attachment on the old node, and
  the ordinal deadlocked on Multi-Attach until an operator deleted a Pod.
  Colocation is now a scheduling preference and both Pods run at `scavenger`
  priority, so either can be evicted and rescheduled on its own. There is
  deliberately no PodDisruptionBudget: this is a best-effort pool and a budget
  would impede node drains. Claims only one Pod mounts stay `ReadWriteOnce`.
- `lease_failed` is a *retryable* durable state, not a terminal one. It means the
  ordinal was released but Kanban has not confirmed the outcome, so every
  maintenance pass retries it. Only `finalized` and `stale` are terminal, and
  only terminal rows are garbage-collected, so no run is dropped before the pool
  knows what happened to it. If the log keeps reporting a deferred lease park for
  the same run, Kanban is refusing an exact-run park on a task it still reports
  as running: inspect that task rather than deleting the row.
- The broker accepts branch creation only. A pool retry that adds commits
  therefore publishes a new `<branch>-attempt-<n>` (or, failing that,
  `<branch>-<head12>`) ref in the same reviewed namespace instead of updating the
  ref a previous attempt published, and adopts an existing ref that already
  points at the exact head rather than pushing again. Expect one draft per
  published ref; each is still human-reviewed. A submission that cannot land
  blocks the run with the reason and leaves the commits on the ordinal's
  workspace, so prior work is recoverable.
- Provider CLIs install once per pinned version onto a durable per-ordinal
  `tools` claim, re-verified against the real binaries so a pruned cache
  reinstalls. The pool is intentionally absent from the `hermes` Kustomization's
  `healthChecks`: gating that 10m window on a best-effort pool would stall
  `hermes-chat` and `hermes-observer-bindings`, which `dependsOn: hermes`.
- A hashed execution-pool ConfigMap, protocol-version readiness checks, and
  versioned SCM boundary name make code/config changes controlled rollouts. A
  rollout is still a human-reviewed operation; this repository change does not
  reconcile or deploy it.

## Your shortest path to fluency

1. Explain the request diagram without looking.
2. Explain why Ariadne and Hermes are separate components.
3. Reproduce Proof 6 from the raw bundle and metrics.
4. Reproduce Proof 7 and explain the two coverage scopes.
5. Produce the live RBAC allow/deny matrix with `kubectl auth can-i`.
6. Create one small writable skill from a repeated sub-workflow, test a trigger
   and non-trigger case, then improve it.
7. Lead one new incident while Hermes coaches rather than answers for you.
8. Teach the five-minute demonstration to another person.

At that point the interview claims are demonstrable. Continue repeating real
incidents before describing yourself as fully autonomous or the system as
self-correcting.