diff --git a/services/comms/knowledge/runbooks/hermes-triage-demo.md b/services/comms/knowledge/runbooks/hermes-triage-demo.md index 066c8205f..2dc292a9c 100644 --- a/services/comms/knowledge/runbooks/hermes-triage-demo.md +++ b/services/comms/knowledge/runbooks/hermes-triage-demo.md @@ -36,17 +36,17 @@ curl -u : -X POST \ That is the only manual step. Everything after the red build is automatic. -## Expected timings +## Expected timings (measured live, first full loop 2026-08-05) -- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if - the Jenkins agent capacity of 5 is busy with other CI builds). -- Ariadne detection: ≤60 s after the build turns red (1-minute schedule). -- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured - ~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s). -- Repair Job: seconds. -- Rebuild to green: ~1–3 min (agent scheduling again). -- Total failure-to-green: **~5 minutes typical**; the only "silent" phases - are the two Jenkins agent-pod waits. +- Armed → seeded build red: **1m49s** (mostly agent-pod scheduling; longer + if the Jenkins agent capacity of 5 is busy with other CI builds). +- Red → repair Job succeeded: **1m15s** (≤60 s detection tick + ~16 s + Hermes diagnosis + authorization + repair pod). +- Repair → rebuild green: **32s**. +- Green → incident resolved: **24s** (next detection tick). +- Total armed-to-resolved: **4m00s**. Budgeted Hermes timeout is 420 s; + observed diagnosis runs were 15–21 s on gpt-5.6-terra. The only "silent" + phases are the two Jenkins agent-pod waits. ## Watching it live @@ -69,7 +69,11 @@ That is the only manual step. Everything after the red build is automatic. `https://agent.bstein.dev` (session/run history). - Escalation path: alert `HermesTriageHumanRequired` in vmalert (`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`) - fires to Alertmanager for any `human_required` incident. + fires to Alertmanager for any `human_required` incident. Gauges are + republished every tick from stored incident state (restart-safe), and + the alert self-clears once a newer build of the same job is green. + Note: Alertmanager's default receiver is currently null — the alert is + visible in vmalert/Alertmanager/Grafana but pushes no notification. ## Demonstrating safe escalation (second path)