The sweep's token was injected from kv/data/atlas/quality/sonarqube-oidc,
which the maintenance role cannot read. I granted that path on the live policy
and verified the read, but the grant was reverted by whatever manages Vault
policy, and the next rollout wedged: vault-agent-init retries a 403 forever, so
the pod never initializes and the Deployment cannot roll. The old replica kept
serving, which is the only reason this was not an outage.
A template block that depends on a grant outside this repository is the actual
defect. The token now lives beside Ariadne's other credentials in
kv/data/atlas/maintenance/ariadne-db - a path its role has always been able to
read - so no policy change is needed and nothing outside this repo can revoke
it. Existing keys at that path were merged, not replaced.
Guarded with an if, so a deployment whose secret predates the key renders an
empty value and starts normally instead of blocking on a missing field. The
sweep then reports an empty token and skips, which is the right failure: no
sweep is much better than no Ariadne.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Static analysis findings never fail a build, so nothing has ever pulled them
into triage. There are 139 open on Ariadne alone, each already naming its
file, line and rule - better-located evidence than the console text the
code-repair flow usually mines.
Scoped deliberately narrow to start: one project, one proposal per hourly
sweep, and only findings SonarQube itself estimates at 20 minutes or less.
Effort is the filter rather than severity because it is the closest proxy for
the single anchored change the patch validator can actually check. The
64-open-proposal ceiling still applies on top, so the queue cannot grow while
nobody is draining it.
Hotspots are not in the type list and cannot be: SonarQube models them as
needing human review, this instance's quality gate fails on exactly that
condition, and an automation that resolved them would be marking them reviewed
without review.
The token comes from Vault. Ariadne's maintenance role was granted read on
kv/data/atlas/quality/sonarqube-oidc, which it did not previously have.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A build whose agent never started is a distinct failure from one that lost a
connection mid-run, and a plain retry queues behind the same stuck pods. The
remediation clears pods that have already succeeded or failed - Ariadne's
existing scheduled cleanup - and only then rebuilds.
Mapping jenkins_agent_provisioning_failure to it keeps the one-classification-
one-action rule: a diagnosis asking for this action under any other
classification is still refused before anything runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Maps workspace_storage_exhausted to the reclaim action, so a build that failed
on a full workspace volume is remediated rather than escalated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three categories now appear in the deployment next to the action
allowlist, and the monitor prints both at the policy gate so the difference is
visible during a demo rather than asserted: two ids Ariadne may execute on its
own authority, three categories it may only ask Hermes to propose a patch for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ananke, pegasus and atlasbot were triaged but mapped to no repository, so
their failures could produce neither an issue nor a pull request. All three
exist on main and are non-empty. data-prepper is deliberately absent: it has
no repository under bstein.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The earlier probe fix addressed slow /health responses under load, but the
restarts continued with a different signature: connection refused rather than
timeout, meaning the app was not listening yet. Ariadne runs migrations and
builds its cron schedule before binding, which can outlast what liveness
allows from initialDelaySeconds, so the kubelet kept restarting a pod that
was merely still starting.
Add a startupProbe granting up to five minutes to come up, after which
liveness takes over unchanged. This is the case startupProbe exists for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The liveness probe used the default timeoutSeconds of 1. The auto-triage tick
runs every minute and spends most of it waiting on Jenkins, OpenSearch, Gitea
and Hermes, so against a 500m CPU limit /health occasionally answers in over a
second. Three of those and the container is killed, dropping triage ticks for
the length of a restart. Observed 11 times in 139 minutes, with the pod
sitting 1/2 Ready and restarting repeatedly.
Give both probes a 5s timeout and let liveness tolerate five failures, so a
busy tick is no longer mistaken for a hung process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The classification->action registry already mapped transient_infra_failure to
retry_transient_infra, but the action allowlist held only repair_demo_fixture,
so that route always died at the action_not_allowlisted gate. Add the action
and state the registry explicitly rather than relying on the code default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Ariadne: per-repo code config for metis, lesavka, soteria,
bstein-dev-home and ariadne, each with its own base branch, source path
prefixes and file suffixes so a proposal can only touch that repo's
source tree
- Alertmanager: the only receiver was an empty "default", so every alert
fired into a void. HermesTriageHumanRequired now routes to an email
receiver via Mailu's in-cluster local-domain relay, with resolved
notices; scoped to service=hermes-triage so nothing else mails yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds ariadne, lesavka, soteria, metis, ananke, pegasus, atlasbot,
bstein-dev-home and data-prepper to the autotriage allowlist. These get
detection, evidence, Hermes diagnosis and human-required escalation only:
no allowlisted repair action applies to them, so nothing is mutated.
Verified on lesavka #582 with the new console failure-region evidence:
classification coverage_quality_gate_failure, confidence 0.93, first
failed gate coverage (previously undetermined).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rehearsal confirmed: emptied allowlist -> incident 14 human_required
(action_not_allowlisted) -> HermesTriageHumanRequired FIRING in vmalert and
active in Alertmanager.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverted immediately after the rehearsal confirms HermesTriageHumanRequired
fires and self-clears.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Observe-mode rehearsal passed: incident hermes-triage-demo/9 diagnosed
correctly (0.99 confidence, known_demo_fixture_failure) and stopped only
at this kill switch. Scope stays: allowlisted job, one allowlisted
action, one action per incident, isolated namespace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
