titan-iac/services/comms/knowledge/runbooks/hermes-triage-demo.md

Hermes Automated Triage Demo — Runbook

How to arm, run, watch, and roll back the automated failure-to-repair demo. Architecture background: knowledge/hermes-automated-triage-24h-plan.md.

What the loop does

You arm the failure (SEED_FAILURE=true)
  -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
  -> test-runner pod also writes the incident JSON to stdout
     (Fluent Bit ships it to OpenSearch kube-*)
  -> Ariadne polls the job every minute, opens incident <job>/<build>
  -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
  -> Ariadne calls the Hermes Agent API (/v1/runs) with
     $triage-titan-test-failures
  -> Hermes returns schema-valid diagnosis + requested_action
  -> Ariadne authorizes (nine gates) and, if remediation is enabled,
     creates Job hermes-demo-repair-<build> (writes "healthy")
  -> Ariadne triggers ONE rebuild with SEED_FAILURE=false
  -> rebuild passes -> incident resolved
Anything else -> human_required metric -> vmalert HermesTriageHumanRequired

Arming the demo

Jenkins UI: https://ci.bstein.dev/job/hermes-triage-demo/ → Build with Parameters → check SEED_FAILURE → Build.

CLI (any Jenkins user API token):

curl -u <user>:<token> -X POST \
  "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"

That is the only manual step. Everything after the red build is automatic.

Expected timings (measured live, first full loop 2026-08-05)

• Armed → seeded build red: 1m49s (mostly agent-pod scheduling; longer if the Jenkins agent capacity of 5 is busy with other CI builds).
• Red → repair Job succeeded: 1m15s (≤60 s detection tick + ~16 s Hermes diagnosis + authorization + repair pod).
• Repair → rebuild green: 32s.
• Green → incident resolved: 24s (next detection tick).
• Total armed-to-resolved: 4m00s. Budgeted Hermes timeout is 420 s; observed diagnosis runs were 15–21 s on gpt-5.6-terra. The only "silent" phases are the two Jenkins agent-pod waits.

Watching it live

• Jenkins: https://ci.bstein.dev/job/hermes-triage-demo/ (red build N, then green build N+1 with SEED_FAILURE=false).
• Ariadne incident state: GET http://ariadne.maintenance/api/internal/audit/events (in-cluster) or /api/admin/audit/events (Keycloak JWT) — event types hermes_autotriage_incident, hermes_autotriage_diagnosis, hermes_autotriage_action. Status flow: detected → diagnosed → repairing → awaiting_rebuild → resolved.
• Metrics (VictoriaMetrics / Grafana Explore): ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}, ariadne_hermes_triage_action_total, ariadne_hermes_triage_duration_seconds.
• Repair evidence: kubectl -n hermes-triage-demo get jobs shows hermes-demo-test-<N> (failed) and hermes-demo-repair-<N> (succeeded); both TTL-clean after 1 h.
• Hermes side: the run appears in the dashboard at https://agent.bstein.dev (session/run history).
• Escalation path: alert HermesTriageHumanRequired in vmalert (vmalert-atlas-availability deployment, 1 m interval, for: 2m) fires to Alertmanager for any human_required incident. Gauges are republished every tick from stored incident state (restart-safe), and the alert self-clears once a newer build of the same job is green. Note: Alertmanager's default receiver is currently null — the alert is visible in vmalert/Alertmanager/Grafana but pushes no notification.

Demonstrating safe escalation (second path)

Any failure that does not match the demo-fixture signature — or any invalid/low-confidence/unknown-action Hermes response — ends as status="human_required" with no mutation. The simplest live demo: temporarily set ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false (see below) and arm the failure; Ariadne diagnoses fully but executes nothing, and the alert fires instead.

Kill switch and rollback

• Instant behavioral off-switch (Flux-managed, in services/maintenance/apps/ariadne-deployment.yaml): ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false → diagnose-only. ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false → fully off.
• The automatic loop can only ever: create Jobs named hermes-demo-repair-* in namespace hermes-triage-demo, and trigger rebuilds of allowlisted jobs (ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST, currently hermes-triage-demo only). One action + one rebuild per incident, ever (storage-backed idempotency).
• Full teardown: remove hermes-triage-demo from clusters/atlas/flux-system/applications/kustomization.yaml (Flux prunes the namespace) and delete the pipelineJob('hermes-triage-demo') block from services/jenkins/configmap-jcasc.yaml.

Credentials

• Ariadne → Hermes: Authorization: Bearer key shared via the hermes-api-server-key Secret present in both hermes and maintenance namespaces (Hermes's init container seeds it into the persistent .env). NOTE: currently manually created (Vault migration pending — see plan handoff); rotating = write new value to both Secrets, restart hermes deployment and ariadne deployment.
• Ariadne → Jenkins: existing JENKINS_API_USER/TOKEN from Vault (atlas/maintenance/ariadne-db).