fix(hermes-triage-demo): pin test-runner to Longhorn-ready workers; add demo runbook

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 17:18:12 -03:00 · 2026-08-05 17:18:12 -03:00 · 3787095d3c
commit 3787095d3c
parent 239a98c5c9
3 changed files with 218 additions and 0 deletions
--- a/knowledge/runbooks/hermes-triage-demo.md
+++ b/knowledge/runbooks/hermes-triage-demo.md
@ -0,0 +1,108 @@
+# Hermes Automated Triage Demo — Runbook
+
+How to arm, run, watch, and roll back the automated failure-to-repair demo.
+Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`.
+
+## What the loop does
+
+```text
+You arm the failure (SEED_FAILURE=true)
+  -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
+  -> test-runner pod also writes the incident JSON to stdout
+     (Fluent Bit ships it to OpenSearch kube-*)
+  -> Ariadne polls the job every minute, opens incident <job>/<build>
+  -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
+  -> Ariadne calls the Hermes Agent API (/v1/runs) with
+     $triage-titan-test-failures
+  -> Hermes returns schema-valid diagnosis + requested_action
+  -> Ariadne authorizes (nine gates) and, if remediation is enabled,
+     creates Job hermes-demo-repair-<build> (writes "healthy")
+  -> Ariadne triggers ONE rebuild with SEED_FAILURE=false
+  -> rebuild passes -> incident resolved
+Anything else -> human_required metric -> vmalert HermesTriageHumanRequired
+```
+
+## Arming the demo
+
+Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with
+Parameters* → check `SEED_FAILURE` → Build.
+
+CLI (any Jenkins user API token):
+
+```bash
+curl -u <user>:<token> -X POST \
+  "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"
+```
+
+That is the only manual step. Everything after the red build is automatic.
+
+## Expected timings
+
+- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if
+  the Jenkins agent capacity of 5 is busy with other CI builds).
+- Ariadne detection: ≤60 s after the build turns red (1-minute schedule).
+- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured
+  ~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s).
+- Repair Job: seconds.
+- Rebuild to green: ~1–3 min (agent scheduling again).
+- Total failure-to-green: **~5 minutes typical**; the only "silent" phases
+  are the two Jenkins agent-pod waits.
+
+## Watching it live
+
+- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N,
+  then green build N+1 with `SEED_FAILURE=false`).
+- Ariadne incident state:
+  `GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or
+  `/api/admin/audit/events` (Keycloak JWT) — event types
+  `hermes_autotriage_incident`, `hermes_autotriage_diagnosis`,
+  `hermes_autotriage_action`. Status flow:
+  `detected → diagnosed → repairing → awaiting_rebuild → resolved`.
+- Metrics (VictoriaMetrics / Grafana Explore):
+  `ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`,
+  `ariadne_hermes_triage_action_total`,
+  `ariadne_hermes_triage_duration_seconds`.
+- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows
+  `hermes-demo-test-<N>` (failed) and `hermes-demo-repair-<N>` (succeeded);
+  both TTL-clean after 1 h.
+- Hermes side: the run appears in the dashboard at
+  `https://agent.bstein.dev` (session/run history).
+- Escalation path: alert `HermesTriageHumanRequired` in vmalert
+  (`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`)
+  fires to Alertmanager for any `human_required` incident.
+
+## Demonstrating safe escalation (second path)
+
+Any failure that does not match the demo-fixture signature — or any
+invalid/low-confidence/unknown-action Hermes response — ends as
+`status="human_required"` with **no mutation**. The simplest live demo:
+temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below)
+and arm the failure; Ariadne diagnoses fully but executes nothing, and the
+alert fires instead.
+
+## Kill switch and rollback
+
+- Instant behavioral off-switch (Flux-managed, in
+  `services/maintenance/apps/ariadne-deployment.yaml`):
+  `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only.
+  `ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off.
+- The automatic loop can only ever: create Jobs named
+  `hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger
+  rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`,
+  currently `hermes-triage-demo` only). One action + one rebuild per
+  incident, ever (storage-backed idempotency).
+- Full teardown: remove `hermes-triage-demo` from
+  `clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes
+  the namespace) and delete the `pipelineJob('hermes-triage-demo')` block
+  from `services/jenkins/configmap-jcasc.yaml`.
+
+## Credentials
+
+- Ariadne → Hermes: `Authorization: Bearer` key shared via the
+  `hermes-api-server-key` Secret present in both `hermes` and `maintenance`
+  namespaces (Hermes's init container seeds it into the persistent `.env`).
+  NOTE: currently manually created (Vault migration pending — see plan
+  handoff); rotating = write new value to both Secrets, restart hermes
+  deployment and ariadne deployment.
+- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault
+  (`atlas/maintenance/ariadne-db`).
--- a/services/comms/knowledge/runbooks/hermes-triage-demo.md
+++ b/services/comms/knowledge/runbooks/hermes-triage-demo.md
@ -0,0 +1,108 @@
+# Hermes Automated Triage Demo — Runbook
+
+How to arm, run, watch, and roll back the automated failure-to-repair demo.
+Architecture background: `knowledge/hermes-automated-triage-24h-plan.md`.
+
+## What the loop does
+
+```text
+You arm the failure (SEED_FAILURE=true)
+  -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
+  -> test-runner pod also writes the incident JSON to stdout
+     (Fluent Bit ships it to OpenSearch kube-*)
+  -> Ariadne polls the job every minute, opens incident <job>/<build>
+  -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
+  -> Ariadne calls the Hermes Agent API (/v1/runs) with
+     $triage-titan-test-failures
+  -> Hermes returns schema-valid diagnosis + requested_action
+  -> Ariadne authorizes (nine gates) and, if remediation is enabled,
+     creates Job hermes-demo-repair-<build> (writes "healthy")
+  -> Ariadne triggers ONE rebuild with SEED_FAILURE=false
+  -> rebuild passes -> incident resolved
+Anything else -> human_required metric -> vmalert HermesTriageHumanRequired
+```
+
+## Arming the demo
+
+Jenkins UI: `https://ci.bstein.dev/job/hermes-triage-demo/` → *Build with
+Parameters* → check `SEED_FAILURE` → Build.
+
+CLI (any Jenkins user API token):
+
+```bash
+curl -u <user>:<token> -X POST \
+  "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"
+```
+
+That is the only manual step. Everything after the red build is automatic.
+
+## Expected timings
+
+- Seeded build goes red: ~1–3 min (mostly agent-pod scheduling; longer if
+  the Jenkins agent capacity of 5 is busy with other CI builds).
+- Ariadne detection: ≤60 s after the build turns red (1-minute schedule).
+- Evidence + Hermes diagnosis: typically 15–60 s (live rehearsal measured
+  ~10 s model time on gpt-5.6-terra; budgeted timeout is 420 s).
+- Repair Job: seconds.
+- Rebuild to green: ~1–3 min (agent scheduling again).
+- Total failure-to-green: **~5 minutes typical**; the only "silent" phases
+  are the two Jenkins agent-pod waits.
+
+## Watching it live
+
+- Jenkins: `https://ci.bstein.dev/job/hermes-triage-demo/` (red build N,
+  then green build N+1 with `SEED_FAILURE=false`).
+- Ariadne incident state:
+  `GET http://ariadne.maintenance/api/internal/audit/events` (in-cluster) or
+  `/api/admin/audit/events` (Keycloak JWT) — event types
+  `hermes_autotriage_incident`, `hermes_autotriage_diagnosis`,
+  `hermes_autotriage_action`. Status flow:
+  `detected → diagnosed → repairing → awaiting_rebuild → resolved`.
+- Metrics (VictoriaMetrics / Grafana Explore):
+  `ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}`,
+  `ariadne_hermes_triage_action_total`,
+  `ariadne_hermes_triage_duration_seconds`.
+- Repair evidence: `kubectl -n hermes-triage-demo get jobs` shows
+  `hermes-demo-test-<N>` (failed) and `hermes-demo-repair-<N>` (succeeded);
+  both TTL-clean after 1 h.
+- Hermes side: the run appears in the dashboard at
+  `https://agent.bstein.dev` (session/run history).
+- Escalation path: alert `HermesTriageHumanRequired` in vmalert
+  (`vmalert-atlas-availability` deployment, 1 m interval, `for: 2m`)
+  fires to Alertmanager for any `human_required` incident.
+
+## Demonstrating safe escalation (second path)
+
+Any failure that does not match the demo-fixture signature — or any
+invalid/low-confidence/unknown-action Hermes response — ends as
+`status="human_required"` with **no mutation**. The simplest live demo:
+temporarily set `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` (see below)
+and arm the failure; Ariadne diagnoses fully but executes nothing, and the
+alert fires instead.
+
+## Kill switch and rollback
+
+- Instant behavioral off-switch (Flux-managed, in
+  `services/maintenance/apps/ariadne-deployment.yaml`):
+  `ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false` → diagnose-only.
+  `ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false` → fully off.
+- The automatic loop can only ever: create Jobs named
+  `hermes-demo-repair-*` in namespace `hermes-triage-demo`, and trigger
+  rebuilds of allowlisted jobs (`ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST`,
+  currently `hermes-triage-demo` only). One action + one rebuild per
+  incident, ever (storage-backed idempotency).
+- Full teardown: remove `hermes-triage-demo` from
+  `clusters/atlas/flux-system/applications/kustomization.yaml` (Flux prunes
+  the namespace) and delete the `pipelineJob('hermes-triage-demo')` block
+  from `services/jenkins/configmap-jcasc.yaml`.
+
+## Credentials
+
+- Ariadne → Hermes: `Authorization: Bearer` key shared via the
+  `hermes-api-server-key` Secret present in both `hermes` and `maintenance`
+  namespaces (Hermes's init container seeds it into the persistent `.env`).
+  NOTE: currently manually created (Vault migration pending — see plan
+  handoff); rotating = write new value to both Secrets, restart hermes
+  deployment and ariadne deployment.
+- Ariadne → Jenkins: existing `JENKINS_API_USER/TOKEN` from Vault
+  (`atlas/maintenance/ariadne-db`).
--- a/services/jenkins/configmap-jcasc.yaml
+++ b/services/jenkins/configmap-jcasc.yaml
@ -512,6 +512,8 @@ data:
                            '        atlas.bstein.dev/role: demo-test-runner',
                            '    spec:',
                            '      restartPolicy: Never',
+                            '      nodeSelector:',
+                            '        node-role.kubernetes.io/worker: "true"',
                            '      containers:',
                            '        - name: test-runner',
                            '          image: busybox:1.37',