Issue #2 on bstein/ariadne cites its evidence as "console_failures marker
'=== FAILURES ===' at line 1949". That is precise and unusable: the
maintainer has to open Jenkins, find build 408, scroll a 2000-line console and
reconstruct what the diagnosis had already read. The whole point of filing an
issue is that someone can act on it without doing that.
The traceback was not merely unrendered - it was never collected. The junit
query asked for errorDetails and not errorStackTrace, so the bundle carried
the assertion that failed but not the line it failed on, and no amount of
rendering would have found it. Both halves are fixed: the trace is fetched and
bounded, and the issue now opens an Evidence section with it.
Console regions are the fallback for a build that published no test results,
clipped to their last lines because the tail of a failure region holds the
failure while the head only approaches it. A fence inside an excerpt is
escaped so raw log text cannot break out of the code block and spill into the
rendered page.
Bounded deliberately. The body has a hard character budget, and an issue that
spends it on console noise buries the one sentence saying why a person is
needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The patch prompt was written for one entry point and inherited whole by the
other. Rendered for a quality sweep it opened with "Use
$triage-titan-test-failures", called the findings "defects the build
reported", and labelled the payload "Failing test evidence bundle" - three
false statements in a row, on a green build, with an empty jenkins section
sitting right below them. The instruction two lines later already said the
build was not failing, so the prompt contradicted itself.
Wrong in the first line the model reads is the expensive kind of wrong: it
frames everything after it. The skill mention is the concrete cost - it sends
Hermes to investigate a build that did not fail, spending a tool call to
explain a failure that does not exist, when the finding and the file are
already inlined below it.
The framing now follows the bundle's origin. A bundle carrying both static
analysis and build evidence is treated as build-driven, because a real failure
is the more urgent framing of the two.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sweep needs the quality token, and the grant was applied to the live
policy. Recording it here keeps the definition and the cluster agreeing about
this path.
Note for whoever reconciles this properly: the live maintenance policy grants
22 paths and this list names 7. sync_k8s_auth replaces a role's policy
wholesale from read_paths, so running it today would revoke nextcloud, wger,
firefly, comms and metis access from Ariadne. Its cron is 0 0 1 1 * - once a
year - which is why the drift has survived rather than been caught. Widening
this list to match reality is worth doing deliberately, not as a side effect
of adding one path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Triage has only ever entered on a failure. Static analysis is the opposite
shape - a standing backlog that never fails a build and so never asks anyone
for attention. On this instance that backlog is 139 open findings on Ariadne
alone, each already naming its file, its line, its rule and what is wrong.
That is better-located evidence than the console text the code-repair flow
normally mines, and it was being thrown away.
This is a second way into the same flow, not a second flow. A scheduled sweep
picks one finding and hands it to the existing proposal path, which is
unchanged: Hermes returns a patch as data, Ariadne validates it against the
file it names, pushes a branch, opens a pull request nobody merges. A finding
arriving from outside the build is not a reason to relax the gates that make a
proposal worth reading, so it does not.
Three deliberate limits. Security hotspots are never fetched: SonarQube models
them as needing human review, the quality gate here fails on exactly that
condition, and an automation that resolved them would be marking them
reviewed without review - defeating the control rather than satisfying it.
Findings already marked won't-fix carry a judgement someone made, and
reopening it produces pull requests that argue with a person. And the sweep
proposes one fix per run by default, because 139 pull requests nobody reads
would make the review gate theatre.
Selection is by SonarQube's own effort estimate rather than severity: effort
is the closest available proxy for the one-anchor change the patch validator
can actually check, so a trivial CRITICAL beats an involved MINOR. An
unparseable estimate is treated as ineligible, not as free.
Off by default. Triage reacts to a failure someone already cares about; this
opens pull requests nobody asked for, and that is a decision an operator makes
deliberately rather than inherits on upgrade.
Branch naming now sanitizes its token, since it arrives from a finding key as
well as a build number and a ref is one of the few places where an unexpected
character stops being cosmetic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every action in the allowlist got there because someone hit the failure by
hand, recognised the pattern, and wired a job for it. That loop only closes if
a person happens to read enough issues to notice the same failure recurring,
so a failure nobody reviews twice never earns an action. Until now a diagnosis
that fit nothing could only say a human was needed; it could not say what the
human should build.
An optional suggested_remediation field closes the other half of the loop.
When nothing in the allowlist fits, Hermes may name the remediation it
believes would work and the evidence that should be required before running it
is safe. It lands in the incident issue under a heading that states plainly
the remediation does not exist and was not performed, and in the audit event,
where the same proposal recurring across unrelated incidents is the evidence
that building it is worth the effort.
The field is inert by construction. No gate reads it, and an id that is not
already allowlisted still fails action_not_allowlisted exactly as before -
naming a remediation and being granted one stay different things, and only the
second needs a human to change a deployment. It is rejected outright alongside
a requested action: the field reports that nothing fit, so something fitting
contradicts it, and allowing both would invite a rationale to be attached to a
request the gates must judge on evidence alone. The key is optional rather
than required so a response written before today still validates unchanged,
and unknown keys are still refused.
Also names reclaim_workspace_storage, clear_stuck_agent_pods and
abort_hung_build in KNOWN_ACTION_LABELS. They were reporting as "unknown" in
metrics on any deployment that had not enabled them, which is the one case
where you most want to see the real name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A build whose agent never started is a distinct failure from one that lost a
connection mid-run: retrying can work, but when the pool is already full of
stuck pods the retry queues behind them and fails identically. Clearing first
is what makes the retry worth making. The clear is Ariadne's existing
scheduled pod cleanup, which only removes pods that have already succeeded or
failed, so nothing running is touched. This is the failure behind lesavka's
open issue and behind two stalled demo runs tonight.
Critically, all three remediations are now described in the triage prompt.
They were wired in Ariadne but absent from what Hermes is told, so Hermes
could never have requested them - the allowlist would have advertised
capability that could not fire. A test now asserts every allowlisted action id
appears in the prompt, so the two cannot drift apart again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two real actions, both for failures that interrupted this project while it was
being built, and both backed by code Ariadne already had.
reclaim_workspace_storage closes a gap left open deliberately: 'no space left
on device' was excluded from the transient retry because a rebuild lands on
the same full volume and either fails identically or hides a capacity problem.
Reclaiming first makes the retry meaningful. The reclaim is the existing
scheduled cleanup, with its own deletion budget, so no new capability is
granted - it is only reachable from triage now. Its signature requires both a
storage marker and a workspace hint, because a full disk elsewhere in the
cluster is a different failure that reclaiming Jenkins workspaces would not
address.
Hung-build detection previously filed an issue and left the build running,
holding one of five Jenkins agent slots and starving every other job - the
actual harm. Ariadne now stops it as well, which is reversible: the job can
simply be built again.
The action executors move to their own module. They are the only code in
triage that changes anything outside Ariadne and should be reviewable as one
unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three defect categories were implicit in code: an operator could neither
see which were active nor switch one off. ARIADNE_HERMES_FIX_CATEGORIES now
names them, an unrecognised entry is ignored rather than trusted, and an empty
setting means all three so the default stays obvious.
Kept separate from ARIADNE_HERMES_ALLOWED_ACTIONS on purpose. That allowlist
gates what Ariadne executes on its own authority, with nothing between the
decision and the change. Everything here becomes a pull request a person
reads. Sharing one list would let a patch clear the same gate as an autonomous
mutation, and that distinction is what makes the autonomous half defensible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Categories two and three of the mechanical-fix work, sharing the seam category
one established.
Undefined or misspelled names are located exactly by the runtime or compiler,
so the fix is one identifier or one import. Python names the file on a
traceback frame above the error, so the most recent frame is carried forward
and attached when the error arrives; a defect with no known file is dropped
because the patcher could not act on it.
Failing assertions now come from the structured test results junit publishes,
carrying the test, its class and the assertion. They deliberately carry no
path: the failing test is the symptom and the defect is usually in the code
under test, so naming the test file would invite weakening the assertion
instead of fixing the cause. The instruction says so explicitly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Category one of the mechanical-fix work. A linter has already located the
defect precisely - file, line, rule, and what is wrong - so passing that
through converts an open-ended 'repair this build' request into a narrow
instruction whose result can be checked against the same evidence.
Recognises ruff/flake8, golangci-lint, and eslint diagnostics from console
evidence, deduplicates them, and bounds the list at twenty. A diagnostic for a
file outside the write allowlist is dropped at extraction rather than later,
so a defect is never reported for a file the patcher could not touch anyway.
Only categories whose fix is mechanical belong here; anything needing a
judgement about intended behaviour stays on the ordinary escalation path.
A build with no linter output is unaffected and falls back to the existing
open-ended request. The patch prompt moves to its own module, as the triage
prompt already had.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two changes to how this reads and behaves on real service repositories.
The fixture rules were stated to Hermes on every job, so it reasoned about
them out loud and that reasoning was published verbatim into service issue
trackers - ariadne/404 opened with 'The job is ariadne, not
hermes-triage-demo, so the reserved demo fixture classification and repair
action are forbidden'. That reads as though the system exists to serve a
demonstration. Those rules are now appended only for the fixture job, so a
real service is never told about them and cannot repeat them; the demo
classification is unreachable elsewhere by construction rather than by
instruction. The prompt also asks for language aimed at a maintainer who
knows nothing about how triage is configured, and points at the structured
test evidence first now that junit publishes it.
The duplicate guard refused a proposal whenever any repair pull request was
open, which meant one unreviewed fix blocked every later one across the
repository. It now enforces a ceiling instead, ARIADNE_HERMES_CODE_MAX_OPEN_PROPOSALS,
default 64. That is a review-capacity limit, not a correctness one: proposals
are cheap to make and expensive to read.
Auto-triage settings move to their own module; they had grown a section's
worth and pushed settings_sections.py past its size budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Log evidence covered the demo namespace plus jenkins. That is right for the
common case, since CI failures happen in Jenkins agent pods, but it means a
build failure that correlates with the service itself being unhealthy carries
no trace of the service at all.
ARIADNE_HERMES_JOB_NAMESPACES maps a job to its own namespace, which is added
alongside jenkins rather than replacing it. Unmapped jobs are unchanged, and a
namespace already in the list is never duplicated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An incident triaged before issue filing existed, or while its repository was
unmapped, is deduped on every later tick and so never reaches the service's
tracker even though the job is still red. soteria and metis sat in exactly
that state: broken, escalated, and permanently unable to file.
The backfill reuses the classification the original diagnosis recorded, so the
existing dedupe applies unchanged - one open issue per job and classification,
refiled only if it is closed while the job still fails. Filing under a guessed
label would miss the open issue and duplicate it, which is what the dedupe
exists to prevent, so an incident with no recorded classification is skipped
rather than guessed at.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A multibranch project is a folder, not a job: its /api/json carries no
lastBuild at all, only a jobs array with one child per branch. Detection read
lastBuild, so hermes-code-demo-branches was returned as skipped on every tick
since it was created and no branch could ever be triaged.
Expand a folder into its branch jobs and process each. Jenkins addresses them
as parent/job/branch, which the existing fetch already builds correctly, and
branch names stay URL-encoded because re-encoding them yields a 404.
Expansion is capped by ARIADNE_HERMES_MAX_BRANCHES (default 5): a repository
with many active branches would otherwise multiply the watched job count
without limit and could open one incident per failing branch in a single tick.
A name containing a slash is refused outright, since it would escape the
parent and address an unrelated job.
The Jenkins transport moves to its own module so the orchestrator reads as
decision logic, and jobs[name] rides along on the existing request rather than
costing a second call per job per tick.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two remaining misattributions in the artifact a human reads first. The summary
line opened with "Hermes auto-triage classified" even when no run happened,
and the hung-build path fell through to the generic "undiagnosed" label
because HUNG_CLASSIFICATION was defined and never wired, so lesavka issue #3
was titled 'undiagnosed' for a build that plainly overran.
The headline now reads "Ariadne recorded incident X as Y" when there was no
run, and a caller may supply an explicit classification that is used only when
no diagnosis produced one - a real diagnosis is never overridden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hung-build path escalates without calling a model, but the issue footer
said unconditionally that the finding came "from a Hermes Agent diagnosis
(run `unknown`)". Live example: lesavka issue #3, filed for a build that
overran, attributed a direct observation to a model conclusion that was never
requested. That misrepresents where the finding came from, which matters most
in exactly the artifact a human reads first.
Attribute to Ariadne when there was no run, and carry the recorded console
text into the body so the issue says what was actually observed instead of
leaving "Why a human is needed: hung_build" as the whole explanation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build 411 failed the per-file coverage contract at 88.89% against the 95%
threshold: the module shipped exercised only indirectly through the tick, so
its defensive branches were never reached. Cover the unreadable, zero,
negative and future timestamps, the disabled cap, the finished build, and
bundle construction from an empty payload. Module now at 100%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection only ever considered builds that finished, so a build wedged on a
network call was invisible: no incident, no issue, no alert, while it held one
of the five Jenkins agent slots. Observed on metis build 272, which sat on
apt-get update for 75 minutes and starved the triage demo of an agent with
nothing anywhere saying so.
A build still running past ARIADNE_HERMES_HUNG_BUILD_MINUTES (default 45) now
files an issue and lands human_required. No model is consulted: the console is
still being written, so a root cause would be invented. Escalates once per
build, and a cap of zero disables the check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bounded patcher could only ever fix defects whose failure output names
the source file. A pytest assertion that fails inside a test names only the
test: on ariadne build 404 the defective file appeared zero times in the whole
174KB console, and candidate selection returned ariadne/app.py, the wrong
file entirely.
Admit test files as readable candidates and follow their absolute imports back
to the module they exercise. Reading widens; writing does not. The patch
validator gates on allowed_path_prefixes alone, so a test file can now be read
for context and still never be patched - which also stops the classic bad fix
of silencing a failing test instead of repairing the code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On a long pipeline the earliest-first region budget was consumed entirely by
successful tool output. Ariadne build 404 retained six regions spanning lines
259-476 - Trivy setup, Sonar banners, and three lines whose only failure
signal was the word 'coverage' - and dropped the actual pytest failure at line
1487. Hermes then correctly refused to diagnose, reporting that the retained
excerpts did not contain the enforced failure.
Split the markers into those that assert a failure and those that only name a
tool or gate that ran, and fill the budget with strong regions first. Also
ignore pytest progress lines ending in a passing verdict: Ariadne's own suite
parametrizes marker detection with strings like 'FAILED tests/a.py::b', and
those passing lines were being collected as failure evidence.
Verified against the real build 404 console: the retained regions now carry
the failing test name, its source file, and the assertion text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hermes classified a real bstein-dev-home build as known_demo_fixture_failure
at 0.99 confidence because a missing node binary superficially resembled the
fixture signature. Ariadne's evidence gate correctly refused the action, but
the filed issue carried a meaningless classification.
Extract the frozen prompt into its own module and interpolate the job under
analysis, so the demo classification and repair_demo_fixture action are
forbidden outright on any job other than hermes-triage-demo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repair action no longer creates a Kubernetes Job and polls it. Ariadne
patches the fixture ConfigMap directly through its own k8s client, which
removes roughly 40 seconds of pod scheduling from the loop, drops the two
failure modes that Job introduced (volume attach and node selection), and
turns an opaque pod log into an Ariadne event.
- execute_repair returns {action, target, succeeded, error} and issues one
merge patch; no Job, no polling, no injectable clock, never raises
- the patch writes the same terminal value every time, so idempotency needs
no duplicate guard; one action per incident is still enforced upstream
- orchestrator records {repair, target}; the state machine, rebuild trigger
and failure path are unchanged
- retires the unused repair-image setting
Ariadne's service account now needs get+patch on that one ConfigMap by name
instead of Job create; the batch/jobs grant and the hermes-demo-repair
service account can be retired.
480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The code path was gated on a single job id, so homegrown services could
never produce a pull request. Escalated incidents on mapped repositories
now additionally attempt a bounded patch proposal, and the filed issue
links it.
- hermes_code_flow.propose_for_incident: additive entry point invoked only
from the escalation branch, so an auto-remediated failure never also
gets a patch and a failed Hermes run never spends tokens on one
- three gates before any HTTP call: code enabled, not the legacy demo job,
and the job has a repo mapping; unmapped jobs make zero calls
- never raises: a failed proposal cannot change the incident outcome or
break the tick
- issue body links the proposal when a pull request was opened
- propose_code_fix added to the bounded action-label set
The legacy demo-job short-circuit is untouched and all of its tests pass
unchanged.
8 new tests; 480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two capabilities that make triage useful outside the demo surface.
Issues: when triage concludes a human is needed, file an issue in the
failing service's own repository carrying classification, confidence, the
facts with their sources, the inferences and a Jenkins link, plus a footer
stating Hermes has no write access and nothing was changed. Opt-in per job
via a repo map, deduplicated by job+classification so a repeatedly failing
job yields one issue per kind of failure rather than one per build, and
capped per tick. Disabled by default.
Real-repo patches: candidate files are selected from the console failure
regions (Python, Rust and JS/TS reference patterns), filtered to each
repo's allowed prefixes and suffixes, ranked earliest-failure-first with
source preferred over test files, and fetched whole - never truncated,
because a patch anchor must match exactly. Per-job owner/repo/base-branch
resolution; the patch is validated against the file the model actually
chose, and an unlisted path is rejected.
Legacy single-repo demo behaviour is preserved unchanged.
131 new tests; 472 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second entry in the action registry, proving it is a real extension point.
No cluster mutation: the action is one Jenkins rebuild.
- hermes_infra_signals: reviewable marker set across DNS/connectivity,
image pull, upstream 5xx and agent-channel loss; Ariadne independently
confirms a marker in the evidence before any retry, and records which
marker justified it. "no space left on device" is deliberately excluded
because a retry lands on the same full volume.
- decision: classification -> action registry (action_classifications),
falling back to the previous single-classification behavior
- repair: retry_build posts to /build for unparameterized real jobs and
buildWithParameters for the fixture demo job
- orchestrator: retry path records requested/accepted/executed and moves
the incident to awaiting_rebuild so the existing success path resolves
it; one action per incident still enforced, so a retry cannot loop
- events layer split out of the orchestrator to stay under the LOC cap
Motivated by real failures tonight: pip DNS resolution and a Gitea 443
connect timeout, plus live incident metis/271 (SCM checkout timeout).
30 new tests; 368 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-service triage returned "undetermined" because the console tail held
only post-build noise; this Jenkins has no junit or stage-view plugin, so
console text is the only structured evidence available.
- hermes_console_evidence: scan the full console for failure markers
(pytest, build, tool gates, k8s/agent), capture context windows, merge
overlaps, collapse repeats, and prefer the earliest regions under a byte
budget; bundle gains jenkins.console_failures and console_truncated
- evidence: fetch the full console (head + tail bounded at 2MB) instead of
the last 8KB; signature detection now also scans regions
- prompt: one line explaining that the earliest region usually holds the
first enforced failure
- code flow: check for an already-open hermes-repair/* pull request before
spending model tokens; fail open so a Gitea error cannot suppress work
Verified on a synthetic 4018-line pipeline: the real failure at line 7 is
now captured where the previous tail-only slice missed it entirely.
95 new tests; 281 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hermes proposes a minimal anchored patch; Ariadne validates it structurally
and opens a pull request for human review. Nothing merges automatically and
Hermes never holds Git credentials or executes anything.
- hermes_code_patch: response parsing + patch validation (path prefix/suffix
allowlist, size and changed-line caps, exact-single-occurrence anchor,
replacement-differs) and exact application
- hermes_code_repair: Gitea contents-API client (fetch, branch push with
hermes-repair/<build> prefix and base-branch refusal, PR creation)
- hermes_code_flow: proposal orchestration + audit event, no patch bodies
or tokens recorded
- hermes_autotriage: code-path branch for the configured repo job; a PR
records human_required/code_fix_proposed, never auto-resolution, and is
kept out of the fixture-action accounting
- settings: ARIADNE_HERMES_CODE_* configuration, disabled by default
74 new tests; 214 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gauges are process-local, so a pod restart could silently drop an active
human_required signal before the vmalert hold window elapsed. Metrics move
to hermes_autotriage_metrics; every tick republishes gauges from stored
incident state, and an incident superseded by a newer successful build is
zeroed so the alert clears once the job is green again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Longhorn RWO attach latency and per-node engine availability made the
PVC fixture unreliable for a fast repeatable demo. The fixture is now a
ConfigMap; the repair Job runs bitnami/kubectl under the least-privilege
hermes-demo-repair SA and patches state=healthy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-worker nodes (titan-2x lane) are not Longhorn-ready; volume attach
fails there. Same constraint as the demo test-runner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
