atlas-iac/knowledge/runbooks/hermes-triage-demo.md

# Hermes Automated Triage Demo — Runbook

How to arm, run, watch, and roll back the automated failure-to-repair demo.
Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`.

## What the loop does

```text
You arm the failure (SEED_FAILURE=true)
  -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
  -> test-runner pod also writes the incident JSON to stdout
     (Fluent Bit ships it to OpenSearch kube-*)
  -> Ariadne polls the job every minute, opens incident <job>/<build>
  -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
  -> Ariadne calls the Hermes Agent API (/v1/runs) with
     $triage-titan-test-failures
  -> Hermes returns schema-valid diagnosis + requested_action
  -> Ariadne authorizes (twelve gates) and, if remediation is enabled,
     patches the fixture ConfigMap back to "healthy" in process
  -> Ariadne triggers ONE rebuild with SEED_FAILURE=false
  -> rebuild passes -> incident resolved
Anything else -> issue in the service repo + human_required metric
```

The model behind Hermes is `anthropic/claude-opus-5` as of 2026-08-06, with
`openai-codex/gpt-5.6-terra` as first fallback and a local `gpt-oss:20b`
behind that, so an expired Anthropic credential degrades rather than stops.

Two behaviours worth knowing before you demo, because both look like nothing
happening:

- **A build that never finishes** is escalated once it passes
  `ARIADNE_HERMES_HUNG_BUILD_MINUTES` (default 45). No model is consulted -
  the console is still being written - so the issue says only that the build
  overran and is holding an agent slot.
- **Alerting no longer fires on every escalation.** The issue in the service
  repository is the durable artifact. Email now means either a repair ran and
  failed, or an escalation has sat untouched for six hours.

## Arming the demo

Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with
Parameters* → check `SEED_FAILURE` → Build.

CLI (any Jenkins user API token):

```bash
curl -u <user>:<token> -X POST \
  "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"
```

That is the only manual step. Everything after the red build is automatic.

Run `scripts/ops/hermes_triage_demo.sh preflight` first. The check that most
often decides whether a rehearsal holds its timings is the agent pool: the
Kubernetes cloud caps concurrent agent pods at `containerCapStr: "5"`, and when
real CI has taken all five the demo build sits in the queue reporting *"All
nodes of label ... are offline"* with no other symptom. Observed 2026-08-06: a
run armed at 02:13:42 did not start seeding until 02:19:37, close to six
minutes of dead air, purely because five other builds held the pool. Wait for a
free slot before starting, or quiesce CI.

## Expected timings (measured live 2026-08-06, Ariadne 0.1.0-402)

Time the demo from the moment the build goes **red**, not from arming. Arming
only queues a Jenkins build, and that leg is at the mercy of the agent pool.

- Red → fixture patched back to healthy: **25s** (≤60 s detection tick, ~16 s
  Hermes diagnosis, then a single in-process Kubernetes API call).
- Red → rebuild triggered: **38s**.
- Red → rebuild green and incident resolved: **1m04s**.

The whole automated leg is just over a minute, and there is no silent phase
longer than the detection tick. Budgeted Hermes timeout is 420 s; observed
diagnosis runs are 15–21 s.

Arming → red was **6m21s** on this run, but 5m35s of that was queue wait
behind a saturated agent pool. On an idle pool expect roughly 1m45s.

The earlier 2026-08-05 figures (1m15s red→repaired, 4m00s total) were measured
when the repair spawned its own Kubernetes Job. Converting the repair to an
in-process call removed a whole pod launch from the critical path.

## Watching it live

- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N,
  then green build N+1 with `SEED_FAILURE=false`).
- Ariadne incident state:
  `GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or
  `/api/admin/audit/events` (Keycloak JWT) — event types
  `hermes_autotriage_incident`, `hermes_autotriage_diagnosis`,
  `hermes_autotriage_action`. Status flow:
  `detected → diagnosed → repairing → awaiting_rebuild → resolved`.
- Metrics (VictoriaMetrics / Grafana Explore):
  `ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`,
  `ariadne_hermes_triage_action_total`,
  `ariadne_hermes_triage_duration_seconds`.
- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows
  `hermes-demo-test-<N>` (failed) and `hermes-demo-repair-<N>` (succeeded);
  both TTL-clean after 1 h.
- Hermes side: the run appears in the dashboard at
  `https://agent.hermes.bstein.dev` (session/run history).
- Escalation path: alert `HermesTriageHumanRequired` in vmalert
  (`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`)
  fires to Alertmanager for any `human_required` incident. Gauges are
  republished every tick from stored incident state (restart-safe), and
  the alert self-clears once a newer build of the same job is green.
  Note: Alertmanager's default receiver is currently null — the alert is
  visible in vmalert/Alertmanager/Grafana but pushes no notification.

## Demonstrating safe escalation (second path)

Any failure that does not match the demo-fixture signature — or any
invalid/low-confidence/unknown-action Hermes response — ends as
`status="human_required"` with **no mutation**. The simplest live demo:
temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below)
and arm the failure; Ariadne diagnoses fully but executes nothing, and the
alert fires instead.

## Kill switch and rollback

- Instant behavioral off-switch (Flux-managed, in
  `services/maintenance/apps/ariadne-deployment.yaml`):
  `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only.
  `ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off.
- The automatic loop can only ever: create Jobs named
  `hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger
  rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`,
  currently `hermes-triage-demo` only). One action + one rebuild per
  incident, ever (storage-backed idempotency).
- Full teardown: remove `hermes-triage-demo` from
  `clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes
  the namespace) and delete the `pipelineJob('hermes-triage-demo')` block
  from `services/jenkins/configmap-jcasc.yaml`.

## Credentials

- Ariadne → Hermes: `Authorization: Bearer` key shared via the
  `hermes-api-server-key` Secret present in both `hermes` and `maintenance`
  namespaces (Hermes's init container seeds it into the persistent `.env`).
  NOTE: currently manually created (Vault migration pending — see plan
  handoff); rotating = write new value to both Secrets, restart hermes
  deployment and ariadne deployment.
- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault
  (`atlas/maintenance/ariadne-db`).