docs(runbook): record the faster in-process repair timings
Red-build to green-rebuild is 1m04s on Ariadne 0.1.0-402, down from 4m00s armed-to-resolved when the repair spawned its own Kubernetes Job. Reframe the timings around the red build, since the arming leg depends on the agent pool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
jenkins
parent
f400c0116e
commit
b9b1634833
@ -45,17 +45,26 @@ run armed at 02:13:42 did not start seeding until 02:19:37, close to six
|
||||
minutes of dead air, purely because five other builds held the pool. Wait for a
|
||||
free slot before starting, or quiesce CI.
|
||||
|
||||
## Expected timings (measured live, first full loop 2026-08-05)
|
||||
## Expected timings (measured live 2026-08-06, Ariadne 0.1.0-402)
|
||||
|
||||
- Armed → seeded build red: **1m49s** (mostly agent-pod scheduling; longer
|
||||
if the Jenkins agent capacity of 5 is busy with other CI builds).
|
||||
- Red → repair Job succeeded: **1m15s** (≤60 s detection tick + ~16 s
|
||||
Hermes diagnosis + authorization + repair pod).
|
||||
- Repair → rebuild green: **32s**.
|
||||
- Green → incident resolved: **24s** (next detection tick).
|
||||
- Total armed-to-resolved: **4m00s**. Budgeted Hermes timeout is 420 s;
|
||||
observed diagnosis runs were 15–21 s on gpt-5.6-terra. The only "silent"
|
||||
phases are the two Jenkins agent-pod waits.
|
||||
Time the demo from the moment the build goes **red**, not from arming. Arming
|
||||
only queues a Jenkins build, and that leg is at the mercy of the agent pool.
|
||||
|
||||
- Red → fixture patched back to healthy: **25s** (≤60 s detection tick, ~16 s
|
||||
Hermes diagnosis, then a single in-process Kubernetes API call).
|
||||
- Red → rebuild triggered: **38s**.
|
||||
- Red → rebuild green and incident resolved: **1m04s**.
|
||||
|
||||
The whole automated leg is just over a minute, and there is no silent phase
|
||||
longer than the detection tick. Budgeted Hermes timeout is 420 s; observed
|
||||
diagnosis runs are 15–21 s.
|
||||
|
||||
Arming → red was **6m21s** on this run, but 5m35s of that was queue wait
|
||||
behind a saturated agent pool. On an idle pool expect roughly 1m45s.
|
||||
|
||||
The earlier 2026-08-05 figures (1m15s red→repaired, 4m00s total) were measured
|
||||
when the repair spawned its own Kubernetes Job. Converting the repair to an
|
||||
in-process call removed a whole pod launch from the critical path.
|
||||
|
||||
## Watching it live
|
||||
|
||||
|
||||
130
services/comms/knowledge/hermes-triage-demo.md
Normal file
130
services/comms/knowledge/hermes-triage-demo.md
Normal file
@ -0,0 +1,130 @@
|
||||
# Hermes Automated Triage Demo — Runbook
|
||||
|
||||
How to arm, run, watch, and roll back the automated failure-to-repair demo.
|
||||
Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`.
|
||||
|
||||
## What the loop does
|
||||
|
||||
```text
|
||||
You arm the failure (SEED_FAILURE=true)
|
||||
-> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
|
||||
-> test-runner pod also writes the incident JSON to stdout
|
||||
(Fluent Bit ships it to OpenSearch kube-*)
|
||||
-> Ariadne polls the job every minute, opens incident <job>/<build>
|
||||
-> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
|
||||
-> Ariadne calls the Hermes Agent API (/v1/runs) with
|
||||
$triage-titan-test-failures
|
||||
-> Hermes returns schema-valid diagnosis + requested_action
|
||||
-> Ariadne authorizes (nine gates) and, if remediation is enabled,
|
||||
creates Job hermes-demo-repair-<build> (writes "healthy")
|
||||
-> Ariadne triggers ONE rebuild with SEED_FAILURE=false
|
||||
-> rebuild passes -> incident resolved
|
||||
Anything else -> human_required metric -> vmalert HermesTriageHumanRequired
|
||||
```
|
||||
|
||||
## Arming the demo
|
||||
|
||||
Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with
|
||||
Parameters* → check `SEED_FAILURE` → Build.
|
||||
|
||||
CLI (any Jenkins user API token):
|
||||
|
||||
```bash
|
||||
curl -u <user>:<token> -X POST \
|
||||
"https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"
|
||||
```
|
||||
|
||||
That is the only manual step. Everything after the red build is automatic.
|
||||
|
||||
Run `scripts/ops/hermes_triage_demo.sh preflight` first. The check that most
|
||||
often decides whether a rehearsal holds its timings is the agent pool: the
|
||||
Kubernetes cloud caps concurrent agent pods at `containerCapStr: "5"`, and when
|
||||
real CI has taken all five the demo build sits in the queue reporting *"All
|
||||
nodes of label ... are offline"* with no other symptom. Observed 2026-08-06: a
|
||||
run armed at 02:13:42 did not start seeding until 02:19:37, close to six
|
||||
minutes of dead air, purely because five other builds held the pool. Wait for a
|
||||
free slot before starting, or quiesce CI.
|
||||
|
||||
## Expected timings (measured live 2026-08-06, Ariadne 0.1.0-402)
|
||||
|
||||
Time the demo from the moment the build goes **red**, not from arming. Arming
|
||||
only queues a Jenkins build, and that leg is at the mercy of the agent pool.
|
||||
|
||||
- Red → fixture patched back to healthy: **25s** (≤60 s detection tick, ~16 s
|
||||
Hermes diagnosis, then a single in-process Kubernetes API call).
|
||||
- Red → rebuild triggered: **38s**.
|
||||
- Red → rebuild green and incident resolved: **1m04s**.
|
||||
|
||||
The whole automated leg is just over a minute, and there is no silent phase
|
||||
longer than the detection tick. Budgeted Hermes timeout is 420 s; observed
|
||||
diagnosis runs are 15–21 s.
|
||||
|
||||
Arming → red was **6m21s** on this run, but 5m35s of that was queue wait
|
||||
behind a saturated agent pool. On an idle pool expect roughly 1m45s.
|
||||
|
||||
The earlier 2026-08-05 figures (1m15s red→repaired, 4m00s total) were measured
|
||||
when the repair spawned its own Kubernetes Job. Converting the repair to an
|
||||
in-process call removed a whole pod launch from the critical path.
|
||||
|
||||
## Watching it live
|
||||
|
||||
- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N,
|
||||
then green build N+1 with `SEED_FAILURE=false`).
|
||||
- Ariadne incident state:
|
||||
`GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or
|
||||
`/api/admin/audit/events` (Keycloak JWT) — event types
|
||||
`hermes_autotriage_incident`, `hermes_autotriage_diagnosis`,
|
||||
`hermes_autotriage_action`. Status flow:
|
||||
`detected → diagnosed → repairing → awaiting_rebuild → resolved`.
|
||||
- Metrics (VictoriaMetrics / Grafana Explore):
|
||||
`ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`,
|
||||
`ariadne_hermes_triage_action_total`,
|
||||
`ariadne_hermes_triage_duration_seconds`.
|
||||
- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows
|
||||
`hermes-demo-test-<N>` (failed) and `hermes-demo-repair-<N>` (succeeded);
|
||||
both TTL-clean after 1 h.
|
||||
- Hermes side: the run appears in the dashboard at
|
||||
`https://agent.bstein.dev` (session/run history).
|
||||
- Escalation path: alert `HermesTriageHumanRequired` in vmalert
|
||||
(`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`)
|
||||
fires to Alertmanager for any `human_required` incident. Gauges are
|
||||
republished every tick from stored incident state (restart-safe), and
|
||||
the alert self-clears once a newer build of the same job is green.
|
||||
Note: Alertmanager's default receiver is currently null — the alert is
|
||||
visible in vmalert/Alertmanager/Grafana but pushes no notification.
|
||||
|
||||
## Demonstrating safe escalation (second path)
|
||||
|
||||
Any failure that does not match the demo-fixture signature — or any
|
||||
invalid/low-confidence/unknown-action Hermes response — ends as
|
||||
`status="human_required"` with **no mutation**. The simplest live demo:
|
||||
temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below)
|
||||
and arm the failure; Ariadne diagnoses fully but executes nothing, and the
|
||||
alert fires instead.
|
||||
|
||||
## Kill switch and rollback
|
||||
|
||||
- Instant behavioral off-switch (Flux-managed, in
|
||||
`services/maintenance/apps/ariadne-deployment.yaml`):
|
||||
`ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only.
|
||||
`ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off.
|
||||
- The automatic loop can only ever: create Jobs named
|
||||
`hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger
|
||||
rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`,
|
||||
currently `hermes-triage-demo` only). One action + one rebuild per
|
||||
incident, ever (storage-backed idempotency).
|
||||
- Full teardown: remove `hermes-triage-demo` from
|
||||
`clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes
|
||||
the namespace) and delete the `pipelineJob('hermes-triage-demo')` block
|
||||
from `services/jenkins/configmap-jcasc.yaml`.
|
||||
|
||||
## Credentials
|
||||
|
||||
- Ariadne → Hermes: `Authorization: Bearer` key shared via the
|
||||
`hermes-api-server-key` Secret present in both `hermes` and `maintenance`
|
||||
namespaces (Hermes's init container seeds it into the persistent `.env`).
|
||||
NOTE: currently manually created (Vault migration pending — see plan
|
||||
handoff); rotating = write new value to both Secrets, restart hermes
|
||||
deployment and ariadne deployment.
|
||||
- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault
|
||||
(`atlas/maintenance/ariadne-db`).
|
||||
Loading…
x
Reference in New Issue
Block a user