The earlier probe fix addressed slow /health responses under load, but the
restarts continued with a different signature: connection refused rather than
timeout, meaning the app was not listening yet. Ariadne runs migrations and
builds its cron schedule before binding, which can outlast what liveness
allows from initialDelaySeconds, so the kubelet kept restarting a pod that
was merely still starting.
Add a startupProbe granting up to five minutes to come up, after which
liveness takes over unchanged. This is the case startupProbe exists for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The liveness probe used the default timeoutSeconds of 1. The auto-triage tick
runs every minute and spends most of it waiting on Jenkins, OpenSearch, Gitea
and Hermes, so against a 500m CPU limit /health occasionally answers in over a
second. Three of those and the container is killed, dropping triage ticks for
the length of a restart. Observed 11 times in 139 minutes, with the pod
sitting 1/2 Ready and restarting repeatedly.
Give both probes a 5s timeout and let liveness tolerate five failures, so a
busy tick is no longer mistaken for a hung process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The classification->action registry already mapped transient_infra_failure to
retry_transient_infra, but the action allowlist held only repair_demo_fixture,
so that route always died at the action_not_allowlisted gate. Add the action
and state the registry explicitly rather than relying on the code default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Ariadne: per-repo code config for metis, lesavka, soteria,
bstein-dev-home and ariadne, each with its own base branch, source path
prefixes and file suffixes so a proposal can only touch that repo's
source tree
- Alertmanager: the only receiver was an empty "default", so every alert
fired into a void. HermesTriageHumanRequired now routes to an email
receiver via Mailu's in-cluster local-domain relay, with resolved
notices; scoped to service=hermes-triage so nothing else mails yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds ariadne, lesavka, soteria, metis, ananke, pegasus, atlasbot,
bstein-dev-home and data-prepper to the autotriage allowlist. These get
detection, evidence, Hermes diagnosis and human-required escalation only:
no allowlisted repair action applies to them, so nothing is mutated.
Verified on lesavka #582 with the new console failure-region evidence:
classification coverage_quality_gate_failure, confidence 0.93, first
failed gate coverage (previously undetermined).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rehearsal confirmed: emptied allowlist -> incident 14 human_required
(action_not_allowlisted) -> HermesTriageHumanRequired FIRING in vmalert and
active in Alertmanager.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverted immediately after the rehearsal confirms HermesTriageHumanRequired
fires and self-clears.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Observe-mode rehearsal passed: incident hermes-triage-demo/9 diagnosed
correctly (0.99 confidence, known_demo_fixture_failure) and stopped only
at this kill switch. Scope stays: allowlisted job, one allowlisted
action, one action per incident, isolated namespace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
