109 lines
4.8 KiB
Markdown
109 lines
4.8 KiB
Markdown
|
# Hermes Automated Triage Demo — Runbook
|
|||
|
|
|
|||
|
|
How to arm, run, watch, and roll back the automated failure-to-repair demo.
|
|||
|
|
Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`.
|
|||
|
|
|
|||
|
|
## What the loop does
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
You arm the failure (SEED_FAILURE=true)
|
|||
|
|
-> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
|
|||
|
|
-> test-runner pod also writes the incident JSON to stdout
|
|||
|
|
(Fluent Bit ships it to OpenSearch kube-*)
|
|||
|
|
-> Ariadne polls the job every minute, opens incident <job>/<build>
|
|||
|
|
-> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
|
|||
|
|
-> Ariadne calls the Hermes Agent API (/v1/runs) with
|
|||
|
|
$triage-titan-test-failures
|
|||
|
|
-> Hermes returns schema-valid diagnosis + requested_action
|
|||
|
|
-> Ariadne authorizes (nine gates) and, if remediation is enabled,
|
|||
|
|
creates Job hermes-demo-repair-<build> (writes "healthy")
|
|||
|
|
-> Ariadne triggers ONE rebuild with SEED_FAILURE=false
|
|||
|
|
-> rebuild passes -> incident resolved
|
|||
|
|
Anything else -> human_required metric -> vmalert HermesTriageHumanRequired
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Arming the demo
|
|||
|
|
|
|||
|
|
Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with
|
|||
|
|
Parameters* → check `SEED_FAILURE` → Build.
|
|||
|
|
|
|||
|
|
CLI (any Jenkins user API token):
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
curl -u <user>:<token> -X POST \
|
|||
|
|
"https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
That is the only manual step. Everything after the red build is automatic.
|
|||
|
|
|
|||
|
|
## Expected timings
|
|||
|
|
|
|||
|
|
- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if
|
|||
|
|
the Jenkins agent capacity of 5 is busy with other CI builds).
|
|||
|
|
- Ariadne detection: ≤60 s after the build turns red (1-minute schedule).
|
|||
|
|
- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured
|
|||
|
|
~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s).
|
|||
|
|
- Repair Job: seconds.
|
|||
|
|
- Rebuild to green: ~1–3 min (agent scheduling again).
|
|||
|
|
- Total failure-to-green: **~5 minutes typical**; the only "silent" phases
|
|||
|
|
are the two Jenkins agent-pod waits.
|
|||
|
|
|
|||
|
|
## Watching it live
|
|||
|
|
|
|||
|
|
- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N,
|
|||
|
|
then green build N+1 with `SEED_FAILURE=false`).
|
|||
|
|
- Ariadne incident state:
|
|||
|
|
`GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or
|
|||
|
|
`/api/admin/audit/events` (Keycloak JWT) — event types
|
|||
|
|
`hermes_autotriage_incident`, `hermes_autotriage_diagnosis`,
|
|||
|
|
`hermes_autotriage_action`. Status flow:
|
|||
|
|
`detected → diagnosed → repairing → awaiting_rebuild → resolved`.
|
|||
|
|
- Metrics (VictoriaMetrics / Grafana Explore):
|
|||
|
|
`ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`,
|
|||
|
|
`ariadne_hermes_triage_action_total`,
|
|||
|
|
`ariadne_hermes_triage_duration_seconds`.
|
|||
|
|
- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows
|
|||
|
|
`hermes-demo-test-<N>` (failed) and `hermes-demo-repair-<N>` (succeeded);
|
|||
|
|
both TTL-clean after 1 h.
|
|||
|
|
- Hermes side: the run appears in the dashboard at
|
|||
|
|
`https://agent.bstein.dev` (session/run history).
|
|||
|
|
- Escalation path: alert `HermesTriageHumanRequired` in vmalert
|
|||
|
|
(`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`)
|
|||
|
|
fires to Alertmanager for any `human_required` incident.
|
|||
|
|
|
|||
|
|
## Demonstrating safe escalation (second path)
|
|||
|
|
|
|||
|
|
Any failure that does not match the demo-fixture signature — or any
|
|||
|
|
invalid/low-confidence/unknown-action Hermes response — ends as
|
|||
|
|
`status="human_required"` with **no mutation**. The simplest live demo:
|
|||
|
|
temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below)
|
|||
|
|
and arm the failure; Ariadne diagnoses fully but executes nothing, and the
|
|||
|
|
alert fires instead.
|
|||
|
|
|
|||
|
|
## Kill switch and rollback
|
|||
|
|
|
|||
|
|
- Instant behavioral off-switch (Flux-managed, in
|
|||
|
|
`services/maintenance/apps/ariadne-deployment.yaml`):
|
|||
|
|
`ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only.
|
|||
|
|
`ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off.
|
|||
|
|
- The automatic loop can only ever: create Jobs named
|
|||
|
|
`hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger
|
|||
|
|
rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`,
|
|||
|
|
currently `hermes-triage-demo` only). One action + one rebuild per
|
|||
|
|
incident, ever (storage-backed idempotency).
|
|||
|
|
- Full teardown: remove `hermes-triage-demo` from
|
|||
|
|
`clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes
|
|||
|
|
the namespace) and delete the `pipelineJob('hermes-triage-demo')` block
|
|||
|
|
from `services/jenkins/configmap-jcasc.yaml`.
|
|||
|
|
|
|||
|
|
## Credentials
|
|||
|
|
|
|||
|
|
- Ariadne → Hermes: `Authorization: Bearer` key shared via the
|
|||
|
|
`hermes-api-server-key` Secret present in both `hermes` and `maintenance`
|
|||
|
|
namespaces (Hermes's init container seeds it into the persistent `.env`).
|
|||
|
|
NOTE: currently manually created (Vault migration pending — see plan
|
|||
|
|
handoff); rotating = write new value to both Secrets, restart hermes
|
|||
|
|
deployment and ariadne deployment.
|
|||
|
|
- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault
|
|||
|
|
(`atlas/maintenance/ariadne-db`).
|