bstein/titan-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(comms): resync runbook mirror

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jenkins 2026-08-05 18:53:37 -03:00
parent 70d8f5f35f
commit 0e9ee04cc4
1 changed files with 15 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
services/comms/knowledge/runbooks/hermes-triage-demo.md
View File

@ -36,17 +36,17 @@ curl -u <user>:<token> -X POST \
That is the only manual step. Everything after the red build is automatic.
## Expected timings
## Expected timings (measured live, first full loop 2026-08-05)
- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if
the Jenkins agent capacity of 5 is busy with other CI builds).
- Ariadne detection: ≤60 s after the build turns red (1-minute schedule).
- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured
~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s).
- Repair Job: seconds.
- Rebuild to green: ~1–3 min (agent scheduling again).
- Total failure-to-green: **~5 minutes typical**; the only "silent" phases
are the two Jenkins agent-pod waits.
- Armed → seeded build red: **1m49s** (mostly agent-pod scheduling; longer
if the Jenkins agent capacity of 5 is busy with other CI builds).
- Red → repair Job succeeded: **1m15s** (≤60 s detection tick + ~16 s
Hermes diagnosis + authorization + repair pod).
- Repair → rebuild green: **32s**.
- Green → incident resolved: **24s** (next detection tick).
- Total armed-to-resolved: **4m00s**. Budgeted Hermes timeout is 420 s;
observed diagnosis runs were 15–21 s on gpt-5.6-terra. The only "silent"
phases are the two Jenkins agent-pod waits.
## Watching it live
@ -69,7 +69,11 @@ That is the only manual step. Everything after the red build is automatic.
`https://agent.bstein.dev` (session/run history).
- Escalation path: alert `HermesTriageHumanRequired` in vmalert
(`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`)
fires to Alertmanager for any `human_required` incident.
fires to Alertmanager for any `human_required` incident. Gauges are
republished every tick from stored incident state (restart-safe), and
the alert self-clears once a newer build of the same job is green.
Note: Alertmanager's default receiver is currently null — the alert is
visible in vmalert/Alertmanager/Grafana but pushes no notification.
## Demonstrating safe escalation (second path)