Hermes Agent 94106bf252 refactor(hermes): rename the owner agent host to worker.bstein.dev

Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.

The rename is additive, matching the shape #38 restored for chat and triage.
CoreDNS, both agent Ingresses and the hermes-sites certificate now serve BOTH
names, so merging this cannot take away the endpoint the operator uses to
reach the coordinator. Retiring agent.hermes.bstein.dev is a separate,
separately scheduled change. No redirect middleware is added.

What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak agent proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default

What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
  rollback only needs the oauth2-proxy --redirect-url reverted and does not
  require re-running the ensure job

The agent client passes its legacy origin through the optional fourth argument
#38 added to ensure_proxy_client, so no second mechanism is introduced. The
immutable ensure Job goes -11 -> -12 because #38 already consumed -11 and that
run has completed; without a further bump this change would never be applied.
Login on the new host fails until the -12 Job completes.

Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.

#38's public-host continuity test now covers the agent proxy's dual origins
rather than asserting the agent surface was untouched by the rename.

Knowledge catalogs and diagrams regenerated with `make knowledge`.

2026-08-21 10:29:46 +00:00

6.8 KiB

Raw Blame History

Hermes Automated Triage Demo — Runbook

How to arm, run, watch, and roll back the automated failure-to-repair demo. Architecture background: knowledge/hermes-automated-triage-24h-plan.md.

What the loop does

You arm the failure (SEED_FAILURE=true)
  -> Jenkins job hermes-triage-demo fails (fixture reads "unhealthy")
  -> test-runner pod also writes the incident JSON to stdout
     (Fluent Bit ships it to OpenSearch kube-*)
  -> Ariadne polls the job every minute, opens incident <job>/<build>
  -> Ariadne bundles Jenkins evidence + bounded OpenSearch excerpts
  -> Ariadne calls the Hermes Agent API (/v1/runs) with
     $triage-titan-test-failures
  -> Hermes returns schema-valid diagnosis + requested_action
  -> Ariadne authorizes (twelve gates) and, if remediation is enabled,
     patches the fixture ConfigMap back to "healthy" in process
  -> Ariadne triggers ONE rebuild with SEED_FAILURE=false
  -> rebuild passes -> incident resolved
Anything else -> issue in the service repo + human_required metric

The model behind Hermes is anthropic/claude-opus-5 as of 2026-08-06, with openai-codex/gpt-5.6-terra as first fallback and a local gpt-oss:20b behind that, so an expired Anthropic credential degrades rather than stops.

Two behaviours worth knowing before you demo, because both look like nothing happening:

A build that never finishes is escalated once it passes ARIADNE_HERMES_HUNG_BUILD_MINUTES (default 45). No model is consulted - the console is still being written - so the issue says only that the build overran and is holding an agent slot.
Alerting no longer fires on every escalation. The issue in the service repository is the durable artifact. Email now means either a repair ran and failed, or an escalation has sat untouched for six hours.

Arming the demo

Jenkins UI: https://ci.bstein.dev/job/hermes-triage-demo/ → Build with Parameters → check SEED_FAILURE → Build.

CLI (any Jenkins user API token):

curl -u <user>:<token> -X POST \
  "https://ci.bstein.dev/job/hermes-triage-demo/buildWithParameters?SEED_FAILURE=true"

That is the only manual step. Everything after the red build is automatic.

Run scripts/ops/hermes_triage_demo.sh preflight first. The check that most often decides whether a rehearsal holds its timings is the agent pool: the Kubernetes cloud caps concurrent agent pods at containerCapStr: "5", and when real CI has taken all five the demo build sits in the queue reporting "All nodes of label ... are offline" with no other symptom. Observed 2026-08-06: a run armed at 02:13:42 did not start seeding until 02:19:37, close to six minutes of dead air, purely because five other builds held the pool. Wait for a free slot before starting, or quiesce CI.

Expected timings (measured live 2026-08-06, Ariadne 0.1.0-402)

Time the demo from the moment the build goes red, not from arming. Arming only queues a Jenkins build, and that leg is at the mercy of the agent pool.

Red → fixture patched back to healthy: 25s (≤60 s detection tick, ~16 s Hermes diagnosis, then a single in-process Kubernetes API call).
Red → rebuild triggered: 38s.
Red → rebuild green and incident resolved: 1m04s.

The whole automated leg is just over a minute, and there is no silent phase longer than the detection tick. Budgeted Hermes timeout is 420 s; observed diagnosis runs are 15–21 s.

Arming → red was 6m21s on this run, but 5m35s of that was queue wait behind a saturated agent pool. On an idle pool expect roughly 1m45s.

The earlier 2026-08-05 figures (1m15s red→repaired, 4m00s total) were measured when the repair spawned its own Kubernetes Job. Converting the repair to an in-process call removed a whole pod launch from the critical path.

Watching it live

Jenkins: https://ci.bstein.dev/job/hermes-triage-demo/ (red build N, then green build N+1 with SEED_FAILURE=false).
Ariadne incident state: GET http://ariadne.maintenance/api/internal/audit/events (in-cluster) or /api/admin/audit/events (Keycloak JWT) — event types hermes_autotriage_incident, hermes_autotriage_diagnosis, hermes_autotriage_action. Status flow: detected → diagnosed → repairing → awaiting_rebuild → resolved.
Metrics (VictoriaMetrics / Grafana Explore): ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo"}, ariadne_hermes_triage_action_total, ariadne_hermes_triage_duration_seconds.
Repair evidence: kubectl -n hermes-triage-demo get jobs shows hermes-demo-test-<N> (failed) and hermes-demo-repair-<N> (succeeded); both TTL-clean after 1 h.
Hermes side: the run appears in the dashboard at https://worker.bstein.dev (session/run history).
Escalation path: alert HermesTriageHumanRequired in vmalert (vmalert-atlas-availability deployment, 1 m interval, for: 2m) fires to Alertmanager for any human_required incident. Gauges are republished every tick from stored incident state (restart-safe), and the alert self-clears once a newer build of the same job is green. Note: Alertmanager's default receiver is currently null — the alert is visible in vmalert/Alertmanager/Grafana but pushes no notification.

Demonstrating safe escalation (second path)

Any failure that does not match the demo-fixture signature — or any invalid/low-confidence/unknown-action Hermes response — ends as status="human_required" with no mutation. The simplest live demo: temporarily set ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false (see below) and arm the failure; Ariadne diagnoses fully but executes nothing, and the alert fires instead.

Kill switch and rollback

Instant behavioral off-switch (Flux-managed, in services/maintenance/apps/ariadne-deployment.yaml): ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false → diagnose-only. ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false → fully off.
The automatic loop can only ever: create Jobs named hermes-demo-repair-* in namespace hermes-triage-demo, and trigger rebuilds of allowlisted jobs (ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST, currently hermes-triage-demo only). One action + one rebuild per incident, ever (storage-backed idempotency).
Full teardown: remove hermes-triage-demo from clusters/atlas/flux-system/applications/kustomization.yaml (Flux prunes the namespace) and delete the pipelineJob('hermes-triage-demo') block from services/jenkins/configmap-jcasc.yaml.

Credentials

Ariadne → Hermes: Authorization: Bearer key from kv/atlas/hermes/triage-api. Vault Agent injects it into both workloads at runtime; it is not stored in either persistent Hermes state or a manually maintained Kubernetes Secret. Rotate the Vault value and roll both tracked deployments through Flux.
Ariadne → Jenkins: existing JENKINS_API_USER/TOKEN from Vault (atlas/maintenance/ariadne-db).

6.8 KiB Raw Blame History Unescape Escape