atlas-iac/services/hermes/skills/master-hermes-on-atlas/references/architecture.md

# Live Atlas Hermes architecture

Use this reference for deployment-specific facts. Re-check live state before
asserting health, placement, ownership, or current model availability.

## Two isolated agent instances

| Surface | Purpose | Identity boundary | State and permissions |
| --- | --- | --- | --- |
| `agent.bstein.dev` | Brad's operator and triage lab | Keycloak plus an outer oauth2-proxy exact-email allow-list for `brad@bstein.dev` | `hermes` namespace, its own PVC and service account; read-only cluster triage plus approved internal evidence endpoints |
| `chat.bstein.dev` | Consumer chat and research | Keycloak login | `hermes-chat` namespace, separate PVC/config/auth/sessions/skills; read-only cluster observer RBAC and no access to Secrets or Kubernetes mutation APIs |

The instances do not share conversation state, credentials, profiles, skills
created on their PVCs, or Kubernetes identities. They share only the inference
service described below. Consumer users cannot reach the operator dashboard
through the chat Service or Ingress.

## Inference path

```text
operator Hermes ─┐
                 ├─> hermes-model-gate ─> Ollama gpt-oss:20b on titan-24 GPU
consumer Hermes ─┘             │
                               └─ 503 while Wolf owns the GPU

each instance independently ─────> openai-codex/gpt-5.4 fallback
```

- The configured context length is 64,000 tokens.
- `hermes-model-gate` is the stable OpenAI-compatible endpoint.
- A coordination Lease stores whether Hermes or Wolf/Moonlight owns the GPU.
  Ariadne has narrowly scoped permission to update that Lease during handoff.
- Wolf ownership makes the gate return a deliberate unavailable response. The
  Hermes gateway stays alive and can use its configured fallback.
- The two agent pods do not consume titan-24 GPU memory. Ollama does. Agent pods
  are ARM gateway workloads with persistent state on separate PVCs.
- Fallback credentials are deliberately per-instance. Never copy Brad's Codex
  credential store into the consumer PVC.

Verify rather than memorize:

```sh
hermes status
hermes fallback list
kubectl -n hermes get lease titan-24-gpu-owner -o yaml
kubectl -n hermes get deploy hermes hermes-model-gate hermes-ollama -o wide
kubectl -n hermes-chat get deploy hermes-chat -o wide
```

Do not change the GPU owner as part of a lesson.

## Retained OpenClaw replacement evidence

OpenClaw is not part of the current inference path, but its deployment history
is retained:

- `1bc58e10` added the OpenClaw testing-triage workspace, workload, model,
  ingress, RBAC, and storage.
- follow-up commits wired local diagnosis, OIDC, persistence, and the gateway.
- `97ecb36b` replaced OpenClaw with Hermes and moved the agent-facing resources.
- `bb5e286f` decoupled Hermes from the retired OpenClaw Flux dependency.
- the `openclaw-home` PVC remains bound as retained state while no OpenClaw
  workload is running.

Have Brad verify the commits and live resource shape. This establishes the
cluster deployment and replacement sequence; Brad's direct experience remains
the evidence for how he personally used the old agent.

## Operator evidence path

The operator role is designed for supervised test-failure triage:

1. Ariadne deterministic bundle and optional diagnosis.
2. Retained Jenkins logs and artifacts.
3. Recent Git changes and Flux revision/state.
4. Kubernetes workload, node, event, storage, DNS, and dependency health.
5. Pushgateway quality metrics and VictoriaMetrics/Grafana context.
6. A concise finding, confidence, evidence, blast radius, next checks,
   repo-side fix, and approval-required section.

The versioned `$triage-titan-test-failures` skill encodes this workflow. The
operator has internal URLs for Ariadne, Jenkins, and VictoriaMetrics. The
consumer instance intentionally does not.

## Security layers and their limits

Security is layered rather than delegated to a prompt:

1. Keycloak authenticates the human.
2. The operator proxy limits `agent.bstein.dev` to Brad's exact email.
3. Separate namespaces, PVCs, service accounts, configs, and ingresses isolate
   operator and consumer state.
4. Kubernetes RBAC grants the consumer get/list/watch only and excludes Secrets,
   exec, attach, port-forward, and mutation.
5. NetworkPolicy permits the consumer to reach DNS, the shared model gate, the
   Kubernetes API, and public IPv4 while blocking arbitrary private service
   access.
6. Hermes deny patterns and instructions add user-facing guardrails.

Prompt rules and shell deny patterns are not the primary security boundary.
RBAC, network policy, credential separation, and routing isolation are. The
consumer can still disclose non-secret cluster metadata it is authorized to
read; that is an intentional requirement and should be described honestly.

## CLI-to-web mental model

The pinned Hermes build exposes chat, models and fallback, auth, sessions,
profiles, skills and bundles, plugins, tools, MCP, gateway/channels, pairing,
webhooks, cron, kanban, projects, hooks, memory/journey, logs, security, backup,
and diagnostics. The web interface is a control surface over many of these
same persisted capabilities.

Use `<command> --help` and the web Documentation page for the installed version.
Do not rely on screenshots or online docs from a different release when a live
command can settle the question.