Every human_required escalation already files an issue in the failing
service's own repository, and mailing on each one made the inbox the loudest
and least informative output of the system. Replace the blanket alert with
two narrow ones: a repair that ran and failed, which is the only case where
the automation acted and left things no better, and an escalation still
untouched after six hours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Red-build to green-rebuild is 1m04s on Ariadne 0.1.0-402, down from 4m00s
armed-to-resolved when the repair spawned its own Kubernetes Job. Reframe the
timings around the red build, since the arming leg depends on the agent pool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two conditions silently break a rehearsal. A saturated Kubernetes agent pool
leaves the demo build queued reporting that all nodes are offline, and an open
hermes-repair PR makes the duplicate guard refuse a new proposal. Report both
so the operator sees them before starting rather than mid-demo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The classification->action registry already mapped transient_infra_failure to
retry_transient_infra, but the action allowlist held only repair_demo_fixture,
so that route always died at the action_not_allowlisted gate. Add the action
and state the registry explicitly rather than relying on the code default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The webhook is failurePolicy: Ignore, so with one replica any pod created
during an injector restart is admitted without its Vault agent sidecar and
then crash-loops forever on a missing /vault/secrets file, with nothing to
indicate injection was skipped. Hit twice while rolling ariadne.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fixture repair now runs in-process inside Ariadne, so the
hermes-demo-repair service account and its Role/RoleBinding have no
remaining user.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.
Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Ariadne: per-repo code config for metis, lesavka, soteria,
bstein-dev-home and ariadne, each with its own base branch, source path
prefixes and file suffixes so a proposal can only touch that repo's
source tree
- Alertmanager: the only receiver was an empty "default", so every alert
fired into a void. HermesTriageHumanRequired now routes to an email
receiver via Mailu's in-cluster local-domain relay, with resolved
notices; scoped to service=hermes-triage so nothing else mails yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds ariadne, lesavka, soteria, metis, ananke, pegasus, atlasbot,
bstein-dev-home and data-prepper to the autotriage allowlist. These get
detection, evidence, Hermes diagnosis and human-required escalation only:
no allowlisted repair action applies to them, so nothing is mutated.
Verified on lesavka #582 with the new console failure-region evidence:
classification coverage_quality_gate_failure, confidence 0.93, first
failed gate coverage (previously undetermined).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Indexing succeeded but discovered no branches without an explicit
gitBranchDiscovery trait, so hermes-repair/* never got a validation build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds demo driver script and branch-level test gate so a Hermes-proposed
pull request carries a green build before human merge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rehearsal confirmed: emptied allowlist -> incident 14 human_required
(action_not_allowlisted) -> HermesTriageHumanRequired FIRING in vmalert and
active in Alertmanager.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverted immediately after the rehearsal confirms HermesTriageHumanRequired
fires and self-clears.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Observe-mode rehearsal passed: incident hermes-triage-demo/9 diagnosed
correctly (0.99 confidence, known_demo_fixture_failure) and stopped only
at this kill switch. Scope stays: allowlisted job, one allowlisted
action, one action per incident, isolated namespace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Pipeline ensures the fixture ConfigMap and seeds unhealthy via patch-file
- Test-runner mounts the ConfigMap read-only; no storage attach involved
- New hermes-demo-repair SA + Role scoped to patching only the fixture CM
- Removes the fixture PVC (Flux prunes it)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
