Detection only ever considered builds that finished, so a build wedged on a
network call was invisible: no incident, no issue, no alert, while it held one
of the five Jenkins agent slots. Observed on metis build 272, which sat on
apt-get update for 75 minutes and starved the triage demo of an agent with
nothing anywhere saying so.
A build still running past ARIADNE_HERMES_HUNG_BUILD_MINUTES (default 45) now
files an issue and lands human_required. No model is consulted: the console is
still being written, so a root cause would be invented. Escalates once per
build, and a cap of zero disables the check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hermes classified a real bstein-dev-home build as known_demo_fixture_failure
at 0.99 confidence because a missing node binary superficially resembled the
fixture signature. Ariadne's evidence gate correctly refused the action, but
the filed issue carried a meaningless classification.
Extract the frozen prompt into its own module and interpolate the job under
analysis, so the demo classification and repair_demo_fixture action are
forbidden outright on any job other than hermes-triage-demo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repair action no longer creates a Kubernetes Job and polls it. Ariadne
patches the fixture ConfigMap directly through its own k8s client, which
removes roughly 40 seconds of pod scheduling from the loop, drops the two
failure modes that Job introduced (volume attach and node selection), and
turns an opaque pod log into an Ariadne event.
- execute_repair returns {action, target, succeeded, error} and issues one
merge patch; no Job, no polling, no injectable clock, never raises
- the patch writes the same terminal value every time, so idempotency needs
no duplicate guard; one action per incident is still enforced upstream
- orchestrator records {repair, target}; the state machine, rebuild trigger
and failure path are unchanged
- retires the unused repair-image setting
Ariadne's service account now needs get+patch on that one ConfigMap by name
instead of Job create; the batch/jobs grant and the hermes-demo-repair
service account can be retired.
480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two capabilities that make triage useful outside the demo surface.
Issues: when triage concludes a human is needed, file an issue in the
failing service's own repository carrying classification, confidence, the
facts with their sources, the inferences and a Jenkins link, plus a footer
stating Hermes has no write access and nothing was changed. Opt-in per job
via a repo map, deduplicated by job+classification so a repeatedly failing
job yields one issue per kind of failure rather than one per build, and
capped per tick. Disabled by default.
Real-repo patches: candidate files are selected from the console failure
regions (Python, Rust and JS/TS reference patterns), filtered to each
repo's allowed prefixes and suffixes, ranked earliest-failure-first with
source preferred over test files, and fetched whole - never truncated,
because a patch anchor must match exactly. Per-job owner/repo/base-branch
resolution; the patch is validated against the file the model actually
chose, and an unlisted path is rejected.
Legacy single-repo demo behaviour is preserved unchanged.
131 new tests; 472 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second entry in the action registry, proving it is a real extension point.
No cluster mutation: the action is one Jenkins rebuild.
- hermes_infra_signals: reviewable marker set across DNS/connectivity,
image pull, upstream 5xx and agent-channel loss; Ariadne independently
confirms a marker in the evidence before any retry, and records which
marker justified it. "no space left on device" is deliberately excluded
because a retry lands on the same full volume.
- decision: classification -> action registry (action_classifications),
falling back to the previous single-classification behavior
- repair: retry_build posts to /build for unparameterized real jobs and
buildWithParameters for the fixture demo job
- orchestrator: retry path records requested/accepted/executed and moves
the incident to awaiting_rebuild so the existing success path resolves
it; one action per incident still enforced, so a retry cannot loop
- events layer split out of the orchestrator to stay under the LOC cap
Motivated by real failures tonight: pip DNS resolution and a Gitea 443
connect timeout, plus live incident metis/271 (SCM checkout timeout).
30 new tests; 368 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hermes proposes a minimal anchored patch; Ariadne validates it structurally
and opens a pull request for human review. Nothing merges automatically and
Hermes never holds Git credentials or executes anything.
- hermes_code_patch: response parsing + patch validation (path prefix/suffix
allowlist, size and changed-line caps, exact-single-occurrence anchor,
replacement-differs) and exact application
- hermes_code_repair: Gitea contents-API client (fetch, branch push with
hermes-repair/<build> prefix and base-branch refusal, PR creation)
- hermes_code_flow: proposal orchestration + audit event, no patch bodies
or tokens recorded
- hermes_autotriage: code-path branch for the configured repo job; a PR
records human_required/code_fix_proposed, never auto-resolution, and is
kept out of the fixture-action accounting
- settings: ARIADNE_HERMES_CODE_* configuration, disabled by default
74 new tests; 214 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Longhorn RWO attach latency and per-node engine availability made the
PVC fixture unreliable for a fast repeatable demo. The fixture is now a
ConfigMap; the repair Job runs bitnami/kubectl under the least-privilege
hermes-demo-repair SA and patches state=healthy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
