titan-iac/services/comms/knowledge/hermes-triage-demo.md

Hermes Automated Triage Demo — Runbook

How to arm, run, watch, and roll back the automated failure-to-repair demo. Architecture background: knowledge/hermes-automated-triage-24h-plan.md.

What the loop does

You arm the failure (SEED_FAILURE=true)
  -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
  -> test-runner pod also writes the incident JSON to stdout
     (Fluent Bit ships it to OpenSearch kube-*)
  -> Ariadne polls the job every minute, opens incident <job>/<build>
  -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
  -> Ariadne calls the Hermes Agent API (/v1/runs) with
     $triage-titan-test-failures
  -> Hermes returns schema-valid diagnosis + requested_action
  -> Ariadne authorizes (twelve gates) and, if remediation is enabled,
     patches the fixture ConfigMap back to "healthy" in process
  -> Ariadne triggers ONE rebuild with SEED_FAILURE=false
  -> rebuild passes -> incident resolved
Anything else -> issue in the service repo + human_required metric

The model behind Hermes is anthropic/claude-opus-5 as of 2026-08-06, with openai-codex/gpt-5.6-terra as first fallback and a local gpt-oss:20b behind that, so an expired Anthropic credential degrades rather than stops.

Two behaviours worth knowing before you demo, because both look like nothing happening:

• A build that never finishes is escalated once it passes ARIADNE_HERMES_HUNG_BUILD_MINUTES (default 45). No model is consulted - the console is still being written - so the issue says only that the build overran and is holding an agent slot.
• Alerting no longer fires on every escalation. The issue in the service repository is the durable artifact. Email now means either a repair ran and failed, or an escalation has sat untouched for six hours.

Arming the demo

Jenkins UI: https://ci.bstein.dev/job/hermes-triage-demo/ → Build with Parameters → check SEED_FAILURE → Build.

CLI (any Jenkins user API token):

curl -u <user>:<token> -X POST \
  "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"

That is the only manual step. Everything after the red build is automatic.

Run scripts/ops/hermes_triage_demo.sh preflight first. The check that most often decides whether a rehearsal holds its timings is the agent pool: the Kubernetes cloud caps concurrent agent pods at containerCapStr: "5", and when real CI has taken all five the demo build sits in the queue reporting "All nodes of label ... are offline" with no other symptom. Observed 2026-08-06: a run armed at 02:13:42 did not start seeding until 02:19:37, close to six minutes of dead air, purely because five other builds held the pool. Wait for a free slot before starting, or quiesce CI.

Expected timings (measured live 2026-08-06, Ariadne 0.1.0-402)

Time the demo from the moment the build goes red, not from arming. Arming only queues a Jenkins build, and that leg is at the mercy of the agent pool.

• Red → fixture patched back to healthy: 25s (≤60 s detection tick, ~16 s Hermes diagnosis, then a single in-process Kubernetes API call).
• Red → rebuild triggered: 38s.
• Red → rebuild green and incident resolved: 1m04s.

The whole automated leg is just over a minute, and there is no silent phase longer than the detection tick. Budgeted Hermes timeout is 420 s; observed diagnosis runs are 15–21 s.

Arming → red was 6m21s on this run, but 5m35s of that was queue wait behind a saturated agent pool. On an idle pool expect roughly 1m45s.

The earlier 2026-08-05 figures (1m15s red→repaired, 4m00s total) were measured when the repair spawned its own Kubernetes Job. Converting the repair to an in-process call removed a whole pod launch from the critical path.

Watching it live

• Jenkins: https://ci.bstein.dev/job/hermes-triage-demo/ (red build N, then green build N+1 with SEED_FAILURE=false).
• Ariadne incident state: GET http://ariadne.maintenance/api/internal/audit/events (in-cluster) or /api/admin/audit/events (Keycloak JWT) — event types hermes_autotriage_incident, hermes_autotriage_diagnosis, hermes_autotriage_action. Status flow: detected → diagnosed → repairing → awaiting_rebuild → resolved.
• Metrics (VictoriaMetrics / Grafana Explore): ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}, ariadne_hermes_triage_action_total, ariadne_hermes_triage_duration_seconds.
• Repair evidence: kubectl -n hermes-triage-demo get jobs shows hermes-demo-test-<N> (failed) and hermes-demo-repair-<N> (succeeded); both TTL-clean after 1 h.
• Hermes side: the run appears in the dashboard at https://agent.bstein.dev (session/run history).
• Escalation path: alert HermesTriageHumanRequired in vmalert (vmalert-atlas-availability deployment, 1 m interval, for: 2m) fires to Alertmanager for any human_required incident. Gauges are republished every tick from stored incident state (restart-safe), and the alert self-clears once a newer build of the same job is green. Note: Alertmanager's default receiver is currently null — the alert is visible in vmalert/Alertmanager/Grafana but pushes no notification.

Demonstrating safe escalation (second path)

Any failure that does not match the demo-fixture signature — or any invalid/low-confidence/unknown-action Hermes response — ends as status="human_required" with no mutation. The simplest live demo: temporarily set ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false (see below) and arm the failure; Ariadne diagnoses fully but executes nothing, and the alert fires instead.

Kill switch and rollback

• Instant behavioral off-switch (Flux-managed, in services/maintenance/apps/ariadne-deployment.yaml): ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false → diagnose-only. ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false → fully off.
• The automatic loop can only ever: create Jobs named hermes-demo-repair-* in namespace hermes-triage-demo, and trigger rebuilds of allowlisted jobs (ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST, currently hermes-triage-demo only). One action + one rebuild per incident, ever (storage-backed idempotency).
• Full teardown: remove hermes-triage-demo from clusters/atlas/flux-system/applications/kustomization.yaml (Flux prunes the namespace) and delete the pipelineJob('hermes-triage-demo') block from services/jenkins/configmap-jcasc.yaml.

Credentials

• Ariadne → Hermes: Authorization: Bearer key shared via the hermes-api-server-key Secret present in both hermes and maintenance namespaces (Hermes's init container seeds it into the persistent .env). NOTE: currently manually created (Vault migration pending — see plan handoff); rotating = write new value to both Secrets, restart hermes deployment and ariadne deployment.
• Ariadne → Jenkins: existing JENKINS_API_USER/TOKEN from Vault (atlas/maintenance/ariadne-db).