diff --git a/knowledge/runbooks/hermes-triage-demo.md b/knowledge/runbooks/hermes-triage-demo.md new file mode 100644 index 000000000..066c8205f --- /dev/null +++ b/knowledge/runbooks/hermes-triage-demo.md @@ -0,0 +1,108 @@ +# Hermes Automated Triage Demo — Runbook + +How to arm, run, watch, and roll back the automated failure-to-repair demo. +Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`. + +## What the loop does + +```text +You arm the failure (SEED_FAILURE=true) + -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy") + -> test-runner pod also writes the incident JSON to stdout + (Fluent Bit ships it to OpenSearch kube-*) + -> Ariadne polls the job every minute, opens incident / + -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts + -> Ariadne calls the Hermes Agent API (/v1/runs) with + $triage-titan-test-failures + -> Hermes returns schema-valid diagnosis + requested_action + -> Ariadne authorizes (nine gates) and, if remediation is enabled, + creates Job hermes-demo-repair- (writes "healthy") + -> Ariadne triggers ONE rebuild with SEED_FAILURE=false + -> rebuild passes -> incident resolved +Anything else -> human_required metric -> vmalert HermesTriageHumanRequired +``` + +## Arming the demo + +Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with +Parameters* → check `SEED_FAILURE` → Build. + +CLI (any Jenkins user API token): + +```bash +curl -u : -X POST \ + "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true" +``` + +That is the only manual step. Everything after the red build is automatic. + +## Expected timings + +- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if + the Jenkins agent capacity of 5 is busy with other CI builds). +- Ariadne detection: ≤60 s after the build turns red (1-minute schedule). +- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured + ~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s). +- Repair Job: seconds. +- Rebuild to green: ~1–3 min (agent scheduling again). +- Total failure-to-green: **~5 minutes typical**; the only "silent" phases + are the two Jenkins agent-pod waits. + +## Watching it live + +- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N, + then green build N+1 with `SEED_FAILURE=false`). +- Ariadne incident state: + `GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or + `/api/admin/audit/events` (Keycloak JWT) — event types + `hermes_autotriage_incident`, `hermes_autotriage_diagnosis`, + `hermes_autotriage_action`. Status flow: + `detected → diagnosed → repairing → awaiting_rebuild → resolved`. +- Metrics (VictoriaMetrics / Grafana Explore): + `ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`, + `ariadne_hermes_triage_action_total`, + `ariadne_hermes_triage_duration_seconds`. +- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows + `hermes-demo-test-` (failed) and `hermes-demo-repair-` (succeeded); + both TTL-clean after 1 h. +- Hermes side: the run appears in the dashboard at + `https://agent.bstein.dev` (session/run history). +- Escalation path: alert `HermesTriageHumanRequired` in vmalert + (`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`) + fires to Alertmanager for any `human_required` incident. + +## Demonstrating safe escalation (second path) + +Any failure that does not match the demo-fixture signature — or any +invalid/low-confidence/unknown-action Hermes response — ends as +`status="human_required"` with **no mutation**. The simplest live demo: +temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below) +and arm the failure; Ariadne diagnoses fully but executes nothing, and the +alert fires instead. + +## Kill switch and rollback + +- Instant behavioral off-switch (Flux-managed, in + `services/maintenance/apps/ariadne-deployment.yaml`): + `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only. + `ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off. +- The automatic loop can only ever: create Jobs named + `hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger + rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`, + currently `hermes-triage-demo` only). One action + one rebuild per + incident, ever (storage-backed idempotency). +- Full teardown: remove `hermes-triage-demo` from + `clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes + the namespace) and delete the `pipelineJob('hermes-triage-demo')` block + from `services/jenkins/configmap-jcasc.yaml`. + +## Credentials + +- Ariadne → Hermes: `Authorization: Bearer` key shared via the + `hermes-api-server-key` Secret present in both `hermes` and `maintenance` + namespaces (Hermes's init container seeds it into the persistent `.env`). + NOTE: currently manually created (Vault migration pending — see plan + handoff); rotating = write new value to both Secrets, restart hermes + deployment and ariadne deployment. +- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault + (`atlas/maintenance/ariadne-db`). diff --git a/services/comms/knowledge/runbooks/hermes-triage-demo.md b/services/comms/knowledge/runbooks/hermes-triage-demo.md new file mode 100644 index 000000000..066c8205f --- /dev/null +++ b/services/comms/knowledge/runbooks/hermes-triage-demo.md @@ -0,0 +1,108 @@ +# Hermes Automated Triage Demo — Runbook + +How to arm, run, watch, and roll back the automated failure-to-repair demo. +Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`. + +## What the loop does + +```text +You arm the failure (SEED_FAILURE=true) + -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy") + -> test-runner pod also writes the incident JSON to stdout + (Fluent Bit ships it to OpenSearch kube-*) + -> Ariadne polls the job every minute, opens incident / + -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts + -> Ariadne calls the Hermes Agent API (/v1/runs) with + $triage-titan-test-failures + -> Hermes returns schema-valid diagnosis + requested_action + -> Ariadne authorizes (nine gates) and, if remediation is enabled, + creates Job hermes-demo-repair- (writes "healthy") + -> Ariadne triggers ONE rebuild with SEED_FAILURE=false + -> rebuild passes -> incident resolved +Anything else -> human_required metric -> vmalert HermesTriageHumanRequired +``` + +## Arming the demo + +Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with +Parameters* → check `SEED_FAILURE` → Build. + +CLI (any Jenkins user API token): + +```bash +curl -u : -X POST \ + "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true" +``` + +That is the only manual step. Everything after the red build is automatic. + +## Expected timings + +- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if + the Jenkins agent capacity of 5 is busy with other CI builds). +- Ariadne detection: ≤60 s after the build turns red (1-minute schedule). +- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured + ~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s). +- Repair Job: seconds. +- Rebuild to green: ~1–3 min (agent scheduling again). +- Total failure-to-green: **~5 minutes typical**; the only "silent" phases + are the two Jenkins agent-pod waits. + +## Watching it live + +- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N, + then green build N+1 with `SEED_FAILURE=false`). +- Ariadne incident state: + `GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or + `/api/admin/audit/events` (Keycloak JWT) — event types + `hermes_autotriage_incident`, `hermes_autotriage_diagnosis`, + `hermes_autotriage_action`. Status flow: + `detected → diagnosed → repairing → awaiting_rebuild → resolved`. +- Metrics (VictoriaMetrics / Grafana Explore): + `ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`, + `ariadne_hermes_triage_action_total`, + `ariadne_hermes_triage_duration_seconds`. +- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows + `hermes-demo-test-` (failed) and `hermes-demo-repair-` (succeeded); + both TTL-clean after 1 h. +- Hermes side: the run appears in the dashboard at + `https://agent.bstein.dev` (session/run history). +- Escalation path: alert `HermesTriageHumanRequired` in vmalert + (`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`) + fires to Alertmanager for any `human_required` incident. + +## Demonstrating safe escalation (second path) + +Any failure that does not match the demo-fixture signature — or any +invalid/low-confidence/unknown-action Hermes response — ends as +`status="human_required"` with **no mutation**. The simplest live demo: +temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below) +and arm the failure; Ariadne diagnoses fully but executes nothing, and the +alert fires instead. + +## Kill switch and rollback + +- Instant behavioral off-switch (Flux-managed, in + `services/maintenance/apps/ariadne-deployment.yaml`): + `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only. + `ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off. +- The automatic loop can only ever: create Jobs named + `hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger + rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`, + currently `hermes-triage-demo` only). One action + one rebuild per + incident, ever (storage-backed idempotency). +- Full teardown: remove `hermes-triage-demo` from + `clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes + the namespace) and delete the `pipelineJob('hermes-triage-demo')` block + from `services/jenkins/configmap-jcasc.yaml`. + +## Credentials + +- Ariadne → Hermes: `Authorization: Bearer` key shared via the + `hermes-api-server-key` Secret present in both `hermes` and `maintenance` + namespaces (Hermes's init container seeds it into the persistent `.env`). + NOTE: currently manually created (Vault migration pending — see plan + handoff); rotating = write new value to both Secrets, restart hermes + deployment and ariadne deployment. +- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault + (`atlas/maintenance/ariadne-db`). diff --git a/services/jenkins/configmap-jcasc.yaml b/services/jenkins/configmap-jcasc.yaml index 508f8e463..f10ae29a4 100644 --- a/services/jenkins/configmap-jcasc.yaml +++ b/services/jenkins/configmap-jcasc.yaml @@ -512,6 +512,8 @@ data: ' atlas.bstein.dev/role: demo-test-runner', ' spec:', ' restartPolicy: Never', + ' nodeSelector:', + ' node-role.kubernetes.io/worker: "true"', ' containers:', ' - name: test-runner', ' image: busybox:1.37',