atlas-iac/services/hermes/NOTES.md

Hermes on Atlas: operator guide

This is the mental model and demonstration script for the operator instance at triage.bstein.dev. Read it once, then prove each section in the live UI. The consumer instance at chat.bstein.dev is intentionally separate and is not the place to perform infrastructure triage.

agent.hermes.bstein.dev is the owner-only engineering control plane. Its root serves the stock Hermes dashboard, including embedded chat/TUI and the native sessions, files, models, logs, Kanban, skills, plugins, MCP, profiles, and configuration surfaces. /terminal/ remains the raw full-screen TUI. The community WebUI is intentionally reserved for isolated consumer chat, where its conversation-first layout is a better fit.

Consumer chat and Telegram

chat.bstein.dev uses the pinned Hermes WebUI rather than the operator dashboard. Keycloak still authenticates every browser request, and the tenant router permanently assigns each Keycloak subject to one Hermes process and one PVC. The four slots are an isolation pool, not a provider round robin: every user starts with the same automatic provider/fallback policy and may change the model or reasoning effort for their own conversation. The WebUI and API reject reasoning levels above xhigh.

Telegram is optional. The Keycloak bootstrap creates kv/atlas/hermes/chat-telegram with a generated relay key and an empty bot_token. After creating the shared bot with BotFather, set only that field:

vault kv patch kv/atlas/hermes/chat-telegram bot_token='<telegram bot token>'

Restart or reconcile hermes-chat-router after changing the token. A user then signs in to the WebUI, selects Telegram, creates a ten-minute code, and sends the displayed /link command to the bot. The router accepts only direct chats, stores hashed Keycloak and Telegram identities, and forwards the message to that user's tenant API with the shared relay key. /unlink works from Telegram or the WebUI. Browser chat remains available when bot_token is empty.

The bot token and relay key must never be added to Git or a Kubernetes Secret. The router does not log prompt bodies, raw Telegram IDs, link codes, or tokens.

The one-sentence explanation

Hermes is the persistent agent runtime and control surface; Codex or the local GPU model supplies reasoning, while Hermes supplies sessions, tools, skills, approval rules, identity, and the repeatable procedure that connects the model to Atlas evidence.

Hermes is not the model. Replacing gpt-5.6-terra with gpt-oss:20b changes the reasoning engine, but it does not replace the Hermes sessions, tools, skills, files, permission boundary, or workflow.

Request and evidence path

browser
  -> Keycloak login
  -> oauth2-proxy exact-user check
  -> operator Hermes pod and persistent workspace
  -> SOUL.md + AGENTS.md + selected skill
  -> Codex primary model
       -> read-only terminal/web/file tools
       -> Ariadne deterministic evidence
       -> retained Jenkins logs and artifacts
       -> Gitea commits and Flux state
       -> Kubernetes workloads, events, logs, and dependencies
       -> Pushgateway data queried through VictoriaMetrics
       -> Grafana query and alert context
  -> structured finding and proposed repo-side change
  -> human review and approval
  -> Git/Flux delivery outside Hermes

provider failure
  -> hermes-model-gate
       -> Hermes owns titan-24: Ollama serves gpt-oss:20b
       -> Wolf owns titan-24: gate returns 503; local inference yields

The operator agent pod runs on an ARM worker and does not reserve the GPU. Ollama is the component on titan-24 that owns GPU memory. The Lease hermes/titan-24-gpu-owner controls whether the model gate admits local inference. Normal operator conversations use independently authenticated Codex first, so Wolf ownership does not need to block the web agent.

Who does what

Component	Responsibility	What it does not prove
Hermes	Maintains the conversation, chooses tools/skills, reasons over evidence, saves reports, and enforces the approval experience	It is not the inference model and does not automatically have cluster-admin access
Ariadne	Collects and normalizes deterministic cluster, Jenkins, and quality evidence into a timestamped bundle; may optionally run a local diagnosis	Its diagnosis is not authoritative when stale, empty, or contradicted by the bundle
Jenkins	Runs the suites and retains console/artifact evidence	A final exit 1 alone does not identify the first failed gate
Pushgateway	Receives suite/build quality telemetry	A cumulative counter is not the current build result
VictoriaMetrics	Stores and answers PromQL for quality and environment telemetry	A Grafana color is not direct proof of root cause
Grafana	Visualizes and alerts on VictoriaMetrics data	A red panel must be traced to its query, labels, time range, and raw series
Flux	Applies reviewed Git state to the cluster	A recent reconciliation is correlation until the changed path matches the failure
Kubernetes	Supplies workload, event, log, node, storage, and dependency evidence	An unrelated unhealthy pod is not proof that CI failed because of the cluster

Direct Jenkins artifact requests currently require authorization and can return HTTP 403. That does not break the workflow: Ariadne retains selected Jenkins console tails and named artifact contents in its deterministic bundle. A report must say retained Ariadne evidence when that fallback is used; it must not pretend direct Jenkins access succeeded.

Publishing an agent image after review

The hermes-agent-image Jenkins job is the only supported agent image builder. It runs daemonless Kaniko v1.23.2 under the unbound hermes-image-builder ServiceAccount without a service-account token, host socket, privileged container, or writable Git credential. It accepts only an exact 40-character revision that is both the checked-out commit and current atlas/titan-iac main, so a human must merge the source PR first.

The build pod requires an ARM64 worker and gives hardware=rpi5 the maximum scheduler preference. Healthy rpi4 workers remain valid fallback capacity when the rpi5 pool cannot fit the full request. Cordoned, unhealthy, amd64, and reserved last-resort nodes remain excluded by the required placement boundary.

Kaniko runs as UID 0 because it must unpack an image root filesystem and enter that filesystem for Dockerfile RUN instructions. Its capability set is the minimum proven by an exact ARM64 no-push build of this Dockerfile: CHOWN, FOWNER, DAC_OVERRIDE, SETGID, and SETUID. SETGID is required because pinned Kaniko applies the base image's supplementary group list before each Dockerfile RUN; SETUID lets apt drop privileges to its _apt account while downloading package indexes. It still has no privilege escalation, service-account token, host path, daemon socket, or Docker/BuildKit TCP endpoint and uses the runtime-default seccomp profile. This is residual root-in-the-build-pod risk, bounded to disposable emptyDir storage and a human-reviewed main revision; Dockerfile changes require the same scrutiny as executable cluster code.

Pinned Kaniko v1.23.2 parses Dockerfile RUN heredocs but does not materialize their inline files when executing a command. The Jenkins lane therefore enables a bounded compatibility replay after copying the reviewed Dockerfile and runner from the same checked-out commit. Before every replay, the runner inventories the entire Dockerfile as logical instructions, including split opcodes, operators, and either Dockerfile escape character. It rejects every heredoc form outside the exact nine Node/Python blocks before launching an interpreter. Each accepted body is then replayed immediately after its corresponding Kaniko RUN, preserving source order before any dependent regression. Docker and BuildKit keep their native behavior because the compatibility argument defaults off. The final TypeScript build, Python compile, and source assertions remain mandatory, so an omitted or drifted replay fails before any image can pass release verification.

From agent.hermes, trigger that one fixed job with:

jenkins_image_build_trigger.py '<reviewed full main commit>'

The helper has no general Jenkins credential or caller-selectable job name. Its Vault-projected token is bound by Jenkins only to hermes-agent-image. The job also requires its fixed publish confirmation. Each run claims the unique tag git-<reviewed-sha>-build-<jenkins-build-number> and refuses to overwrite an existing tag. It cross-checks Kaniko's digest and tagged-image evidence, then independently reads the pushed tag from Harbor before archiving a JSON record and Flux digest patch. A successful post-condition then revalidates the exact six-file evidence set and archives those six explicit paths with empty archives forbidden. Apply that patch on a new branch and submit it for human review; the build never changes Git, reconciles Flux, or deploys by itself. The helper posts only to the fixed HTTPS Build Token Root endpoint and accepts only its real queue responses: HTTP 201 or a non-followed HTTP 303 with an exact same-origin /queue/item/<number>/ location. An unauthenticated request is denied, and the token cannot select, configure, read, or administer another Jenkins job. The pipeline independently rejects any commit that is not the current origin/main, preserving the human merge/review boundary.

Harbor independently enforces an enabled immutable-tag rule scoped to only the bstein/hermes-agent repository and git-*-build-* tags. A second manifest PUT, retag, or deletion is rejected by Harbor even if a caller bypasses the Jenkins preflight. A revisioned, Flux-tracked policy Job creates or verifies that exact rule with a runtime-only Vault credential; it refuses to alter a conflicting rule and retries only explicit transport/readiness failures while Harbor starts. The same Job grants the existing Jenkins publisher only immutable-tag:list on bstein, preserving its other project scopes and never calling Harbor's separate secret-rotation endpoint. The pipeline uses that read-only permission to require the exact enabled rule before Kaniko starts. That check also makes an already-running Jenkins controller fail closed during rollout; Jenkins cannot become Ready on the new revision before the policy Job succeeds.

The tracked rollout order is deliberate: the revisioned Vault role Job must complete before the Vault Kustomization becomes Ready. A separate Flux Kustomization then runs the seed Job under the dedicated hermes-jenkins-token-seed identity, which can only create/read the exact kv/data/atlas/hermes/developer-jenkins path and request random bytes. Jenkins depends on both that seed and the Harbor policy Job; Hermes depends on Jenkins. The seeder uses KV-v2 CAS create-only semantics. It never changes an existing token, never fills a missing field in an existing secret, and fails closed when it cannot distinguish absence from a read error. The recurring Vault configuration job reconciles the same narrow role and policy.

Rotate this fixed-job token only as a coordinated operator action:

1. Stop new hermes-agent-image triggers and wait for its queue and executor to drain.
2. Read the current KV-v2 metadata version for kv/atlas/hermes/developer-jenkins.
3. Generate a fresh value from Vault and patch only build_token with vault kv patch -cas=<current-version>. Keep the value in a mode-0600 temporary file or standard input, never a command argument or log.
4. Roll Jenkins first and wait until it is Ready, then roll hermes-agent and wait until it is Ready. Do not resume triggers between those two consumers.
5. Run one reviewed-main canary. If rollback is required, repeat the CAS patch with the previous value as another coordinated rotation; never delete the secret to make the seeder recreate it.

The actual supervised triage algorithm

1. Classify the request as test/build triage, service health, or alert tuning.
2. Read the latest Ariadne diagnosis and deterministic bundle with HTTP GET.
3. Compare their timestamps. The deterministic bundle remains authoritative.
4. Select a terminal failed build. Keep running or unknown builds out of the terminal-failure list.
5. Identify the first enforced failed gate in this order: style -> loc -> coverage -> tests -> gate_glue -> sonarqube -> supply_chain.
6. Cite the smallest decisive Jenkins log/artifact evidence. State whether it came directly from Jenkins or from Ariadne retention.
7. Query the matching build/check/test series in VictoriaMetrics. Distinguish a current gauge from a cumulative counter and ignore zero-valued failure series.
8. Correlate recent Gitea commits and the relevant Flux revision by timestamp and affected path. Recency alone is not causation.
9. Check only the Kubernetes resources capable of explaining that failure. Classify observations as direct, contributing candidate, or background.
10. Return Finding, Confidence, Evidence, Likely cause, Blast radius, Next checks, Repo-side fix, and Approval required.
11. Stop at a proposal. A human reviews the patch or action; Git and Flux remain the delivery path.

What Hermes can read and what it cannot do

The live hermes-triage ServiceAccount can get/list/watch ordinary workload, log, event, ingress, storage, Flux, and image-automation metadata. It cannot read Kubernetes Secret values, create exec sessions, patch Deployments, or patch Flux Kustomizations. The configured command deny list also blocks common Kubernetes and Flux mutations.

The security boundary is layered:

1. Keycloak authenticates the person.
2. oauth2-proxy restricts the operator surface to Brad.
3. The operator has a separate namespace, PVC, configuration, and ServiceAccount.
4. Kubernetes RBAC is the hard API authorization boundary.
5. NetworkPolicy limits reachable paths where configured.
6. Hermes approvals and instructions provide a user-facing safety layer.

A skill is procedure and context, not a permission grant. Adding a sentence to a skill cannot bypass Kubernetes RBAC.

What “Hermes learns the workflow” means

It does not silently retrain model weights on cluster data. Learning here means that a successful repeated procedure is written as a reusable skill on the operator PVC or versioned in Git. The skill describes when it should trigger, the evidence order, interpretation rules, output contract, and safety boundary.

The current workflow has a versioned top-level skill and persisted specialist skills for:

• orchestration and evidence reporting;
• retained Jenkins evidence;
• quality metrics;
• Gitea/Flux correlation;
• Kubernetes failure classification;
• Grafana metric provenance;
• Soteria backup health;
• approval-required actions.

The model still evaluates fresh variable evidence on every incident. The skill makes the process repeatable; it does not freeze the answer.

The four operator skills to remember

• triage-titan-test-failures: CI, tests, builds, quality gates, and suspected test-environment regressions.
• triage-atlas-service-health: active service or cluster incidents and red high-level health panels.
• tune-atlas-alerts: noisy alerts, impossible metrics, bad PromQL, and generator-owned Grafana corrections.
• master-hermes-on-atlas: hands-on training, assessments, architecture, and claim audits.

Do not start by choosing raw tools. State the operational question and let the skill route to the narrow evidence source.

Two live, repeatable proof cases

Proof 6: Soteria build 272

• Terminal result: failure.
• Tests: 318 passed, zero failed.
• Local coverage: 96.195 percent.
• First enforced failure: SonarQube new_coverage=0.0 against threshold 80.
• Runtime: Soteria remained Ready.
• Correct conclusion: a Sonar new-code policy/reporting problem, not a test or Kubernetes capacity failure.
• Correct restraint: do not weaken the policy until scanner import and baseline evidence identifies whether the issue is configuration or legitimate new uncovered code.

Proof 7: Ananke build 242

• Terminal result: failure.
• Local coverage gate: 61.8 percent and failed.
• Other checks: tests, SonarQube, supply-chain, LOC, docs naming, and gate glue were healthy.
• Sonar new-code coverage: 93.1 percent, which is a different scope.
• Correct conclusion: local coverage input/scope caused the enforced failure; do not call the Sonar number contradictory without comparing inputs.
• Correct restraint: do not lower the threshold until the retained local gate inputs are readable.

The full sessions and redacted Markdown exports are in Sessions and Files/triage-proof.

Five-minute demonstration

1. Open Models and show openai-codex/gpt-5.6-terra as primary and gpt-oss:20b as local fallback.
2. Open Skills and show the four operator skills plus the persisted component skills.
3. Open Proof 6 - Soteria 272 Sonar new coverage triage in Sessions.
4. Point out the timestamped bundle, exact build/artifact evidence, metric corroboration, healthy runtime, fact/inference separation, and approval boundary.
5. Open Proof 7 and explain why local coverage and Sonar new-code coverage can differ without either number being fabricated.
6. End by showing that Hermes proposes a repo-side correction but cannot patch the Deployment or Flux Kustomization with its ServiceAccount.

Use this short explanation:

Hermes is the persistent, permissioned workflow layer around the model. In this cluster Ariadne collects deterministic evidence, Hermes correlates it with retained Jenkins artifacts, Git/Flux state, Kubernetes health, and quality metrics, and then returns a supervised proposal. Repeated procedures become explicit skills. Kubernetes RBAC prevents the agent from turning a diagnosis into an unreviewed infrastructure change.

Prompts that exercise the real system

• Use $triage-titan-test-failures. Triage the worst current terminal CI failure and link every decisive piece of evidence.
• Use $triage-titan-test-failures. Triage Ananke build 242. Separate local coverage from Sonar new-code coverage.
• Use $triage-atlas-service-health. Explain the VictoriaMetrics outage from current state and retained events; separate recovered impact from current impact.
• Use $tune-atlas-alerts. Trace one currently firing alert to its generated source and raw PromQL, but do not edit it.
• Use $master-hermes-on-atlas. Assess me on the request path and permission boundary. One question at a time.

Honest limits

• Hermes does not currently apply production or cluster changes autonomously.
• Direct Jenkins console/artifact access may be forbidden; Ariadne retention is the current fallback and must be named as provenance.
• Local gpt-oss:20b is useful for bounded work but is slower and less reliable for multi-source triage than Codex.
• The current Ariadne local diagnosis can return an empty model response. That does not invalidate the deterministic bundle or prevent Codex-backed Hermes from triaging it.
• A single successful case demonstrates capability, not mastery. Mastery means leading different real incidents, improving the skill after failures, and teaching the architecture without prompts.

Distributed execution-pool rollout boundary

• Do not reconcile the pool until the approved credential owner has provisioned six distinct Vault fields: execution_worker_{0,1,2}_claude_credentials_json and execution_worker_{0,1,2}_codex_auth_json. Each ordinal needs an independent account or refresh-token lineage; copying one rotating refresh token into multiple fields recreates the lost-update failure this design prevents.
• Worker credential files live on separate ordinal-owned RWO provider-access claims. Provider refresh updates these durable private copies across Pod restarts; they are never synchronized back to Vault. Rotate one bootstrap credential at a time through the reviewed Vault workflow and reinitialize only that ordinal after human approval.
• Existing task worktrees remain on the single local owner lane. Only tasks without workspace_path enter the distributed pool, where each ordinal owns one RWO checkout. Repository and base-branch identity comes from the canonical board registry rather than task-supplied metadata.
• Model Pods have no pool key, broker mount, broker egress, or Kubernetes token. Ordinal mediator Deployments are separate network identities that co-mount the matching workspace PVC; they alone authenticate exact-run results and reach the PR14 SCM broker.
• The ordinal workspace claim is ReadWriteMany because the worker and its mediator both mount it. With ReadWriteOnce a drain or preemption that moved only one of them left the other holding the attachment on the old node, and the ordinal deadlocked on Multi-Attach until an operator deleted a Pod. Colocation is now a scheduling preference and both Pods run at scavenger priority, so either can be evicted and rescheduled on its own. There is deliberately no PodDisruptionBudget: this is a best-effort pool and a budget would impede node drains. Claims only one Pod mounts stay ReadWriteOnce.
• lease_failed is a retryable durable state, not a terminal one. It means the ordinal was released but Kanban has not confirmed the outcome, so every maintenance pass retries it. Only finalized and stale are terminal, and only terminal rows are garbage-collected, so no run is dropped before the pool knows what happened to it. If the log keeps reporting a deferred lease park for the same run, Kanban is refusing an exact-run park on a task it still reports as running: inspect that task rather than deleting the row.
• The broker accepts branch creation only. A pool retry that adds commits therefore publishes a new <branch>-attempt-<n> (or, failing that, <branch>-<head12>) ref in the same reviewed namespace instead of updating the ref a previous attempt published, and adopts an existing ref that already points at the exact head rather than pushing again. Expect one draft per published ref; each is still human-reviewed. A submission that cannot land blocks the run with the reason and leaves the commits on the ordinal's workspace, so prior work is recoverable.
• Provider CLIs install once per pinned version onto a durable per-ordinal tools claim, re-verified against the real binaries so a pruned cache reinstalls. The pool is intentionally absent from the hermes Kustomization's healthChecks: gating that 10m window on a best-effort pool would stall hermes-chat and hermes-observer-bindings, which dependsOn: hermes.
• A hashed execution-pool ConfigMap, protocol-version readiness checks, and versioned SCM boundary name make code/config changes controlled rollouts. A rollout is still a human-reviewed operation; this repository change does not reconcile or deploy it.

Your shortest path to fluency

1. Explain the request diagram without looking.
2. Explain why Ariadne and Hermes are separate components.
3. Reproduce Proof 6 from the raw bundle and metrics.
4. Reproduce Proof 7 and explain the two coverage scopes.
5. Produce the live RBAC allow/deny matrix with kubectl auth can-i.
6. Create one small writable skill from a repeated sub-workflow, test a trigger and non-trigger case, then improve it.
7. Lead one new incident while Hermes coaches rather than answers for you.
8. Teach the five-minute demonstration to another person.

At that point the interview claims are demonstrable. Continue repeating real incidents before describing yourself as fully autonomous or the system as self-correcting.