Detection only ever considered builds that finished, so a build wedged on a
network call was invisible: no incident, no issue, no alert, while it held one
of the five Jenkins agent slots. Observed on metis build 272, which sat on
apt-get update for 75 minutes and starved the triage demo of an agent with
nothing anywhere saying so.
A build still running past ARIADNE_HERMES_HUNG_BUILD_MINUTES (default 45) now
files an issue and lands human_required. No model is consulted: the console is
still being written, so a root cause would be invented. Escalates once per
build, and a cap of zero disables the check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bounded patcher could only ever fix defects whose failure output names
the source file. A pytest assertion that fails inside a test names only the
test: on ariadne build 404 the defective file appeared zero times in the whole
174KB console, and candidate selection returned ariadne/app.py, the wrong
file entirely.
Admit test files as readable candidates and follow their absolute imports back
to the module they exercise. Reading widens; writing does not. The patch
validator gates on allowed_path_prefixes alone, so a test file can now be read
for context and still never be patched - which also stops the classic bad fix
of silencing a failing test instead of repairing the code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On a long pipeline the earliest-first region budget was consumed entirely by
successful tool output. Ariadne build 404 retained six regions spanning lines
259-476 - Trivy setup, Sonar banners, and three lines whose only failure
signal was the word 'coverage' - and dropped the actual pytest failure at line
1487. Hermes then correctly refused to diagnose, reporting that the retained
excerpts did not contain the enforced failure.
Split the markers into those that assert a failure and those that only name a
tool or gate that ran, and fill the budget with strong regions first. Also
ignore pytest progress lines ending in a passing verdict: Ariadne's own suite
parametrizes marker detection with strings like 'FAILED tests/a.py::b', and
those passing lines were being collected as failure evidence.
Verified against the real build 404 console: the retained regions now carry
the failing test name, its source file, and the assertion text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hermes classified a real bstein-dev-home build as known_demo_fixture_failure
at 0.99 confidence because a missing node binary superficially resembled the
fixture signature. Ariadne's evidence gate correctly refused the action, but
the filed issue carried a meaningless classification.
Extract the frozen prompt into its own module and interpolate the job under
analysis, so the demo classification and repair_demo_fixture action are
forbidden outright on any job other than hermes-triage-demo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repair action no longer creates a Kubernetes Job and polls it. Ariadne
patches the fixture ConfigMap directly through its own k8s client, which
removes roughly 40 seconds of pod scheduling from the loop, drops the two
failure modes that Job introduced (volume attach and node selection), and
turns an opaque pod log into an Ariadne event.
- execute_repair returns {action, target, succeeded, error} and issues one
merge patch; no Job, no polling, no injectable clock, never raises
- the patch writes the same terminal value every time, so idempotency needs
no duplicate guard; one action per incident is still enforced upstream
- orchestrator records {repair, target}; the state machine, rebuild trigger
and failure path are unchanged
- retires the unused repair-image setting
Ariadne's service account now needs get+patch on that one ConfigMap by name
instead of Job create; the batch/jobs grant and the hermes-demo-repair
service account can be retired.
480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The code path was gated on a single job id, so homegrown services could
never produce a pull request. Escalated incidents on mapped repositories
now additionally attempt a bounded patch proposal, and the filed issue
links it.
- hermes_code_flow.propose_for_incident: additive entry point invoked only
from the escalation branch, so an auto-remediated failure never also
gets a patch and a failed Hermes run never spends tokens on one
- three gates before any HTTP call: code enabled, not the legacy demo job,
and the job has a repo mapping; unmapped jobs make zero calls
- never raises: a failed proposal cannot change the incident outcome or
break the tick
- issue body links the proposal when a pull request was opened
- propose_code_fix added to the bounded action-label set
The legacy demo-job short-circuit is untouched and all of its tests pass
unchanged.
8 new tests; 480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two capabilities that make triage useful outside the demo surface.
Issues: when triage concludes a human is needed, file an issue in the
failing service's own repository carrying classification, confidence, the
facts with their sources, the inferences and a Jenkins link, plus a footer
stating Hermes has no write access and nothing was changed. Opt-in per job
via a repo map, deduplicated by job+classification so a repeatedly failing
job yields one issue per kind of failure rather than one per build, and
capped per tick. Disabled by default.
Real-repo patches: candidate files are selected from the console failure
regions (Python, Rust and JS/TS reference patterns), filtered to each
repo's allowed prefixes and suffixes, ranked earliest-failure-first with
source preferred over test files, and fetched whole - never truncated,
because a patch anchor must match exactly. Per-job owner/repo/base-branch
resolution; the patch is validated against the file the model actually
chose, and an unlisted path is rejected.
Legacy single-repo demo behaviour is preserved unchanged.
131 new tests; 472 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second entry in the action registry, proving it is a real extension point.
No cluster mutation: the action is one Jenkins rebuild.
- hermes_infra_signals: reviewable marker set across DNS/connectivity,
image pull, upstream 5xx and agent-channel loss; Ariadne independently
confirms a marker in the evidence before any retry, and records which
marker justified it. "no space left on device" is deliberately excluded
because a retry lands on the same full volume.
- decision: classification -> action registry (action_classifications),
falling back to the previous single-classification behavior
- repair: retry_build posts to /build for unparameterized real jobs and
buildWithParameters for the fixture demo job
- orchestrator: retry path records requested/accepted/executed and moves
the incident to awaiting_rebuild so the existing success path resolves
it; one action per incident still enforced, so a retry cannot loop
- events layer split out of the orchestrator to stay under the LOC cap
Motivated by real failures tonight: pip DNS resolution and a Gitea 443
connect timeout, plus live incident metis/271 (SCM checkout timeout).
30 new tests; 368 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-service triage returned "undetermined" because the console tail held
only post-build noise; this Jenkins has no junit or stage-view plugin, so
console text is the only structured evidence available.
- hermes_console_evidence: scan the full console for failure markers
(pytest, build, tool gates, k8s/agent), capture context windows, merge
overlaps, collapse repeats, and prefer the earliest regions under a byte
budget; bundle gains jenkins.console_failures and console_truncated
- evidence: fetch the full console (head + tail bounded at 2MB) instead of
the last 8KB; signature detection now also scans regions
- prompt: one line explaining that the earliest region usually holds the
first enforced failure
- code flow: check for an already-open hermes-repair/* pull request before
spending model tokens; fail open so a Gitea error cannot suppress work
Verified on a synthetic 4018-line pipeline: the real failure at line 7 is
now captured where the previous tail-only slice missed it entirely.
95 new tests; 281 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hermes proposes a minimal anchored patch; Ariadne validates it structurally
and opens a pull request for human review. Nothing merges automatically and
Hermes never holds Git credentials or executes anything.
- hermes_code_patch: response parsing + patch validation (path prefix/suffix
allowlist, size and changed-line caps, exact-single-occurrence anchor,
replacement-differs) and exact application
- hermes_code_repair: Gitea contents-API client (fetch, branch push with
hermes-repair/<build> prefix and base-branch refusal, PR creation)
- hermes_code_flow: proposal orchestration + audit event, no patch bodies
or tokens recorded
- hermes_autotriage: code-path branch for the configured repo job; a PR
records human_required/code_fix_proposed, never auto-resolution, and is
kept out of the fixture-action accounting
- settings: ARIADNE_HERMES_CODE_* configuration, disabled by default
74 new tests; 214 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gauges are process-local, so a pod restart could silently drop an active
human_required signal before the vmalert hold window elapsed. Metrics move
to hermes_autotriage_metrics; every tick republishes gauges from stored
incident state, and an incident superseded by a newer successful build is
zeroed so the alert clears once the job is green again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Longhorn RWO attach latency and per-node engine availability made the
PVC fixture unreliable for a fast repeatable demo. The fixture is now a
ConfigMap; the repair Job runs bitnami/kubectl under the least-privilege
hermes-demo-repair SA and patches state=healthy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-worker nodes (titan-2x lane) are not Longhorn-ready; volume attach
fails there. Same constraint as the demo test-runner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
