The repair action no longer creates a Kubernetes Job and polls it. Ariadne
patches the fixture ConfigMap directly through its own k8s client, which
removes roughly 40 seconds of pod scheduling from the loop, drops the two
failure modes that Job introduced (volume attach and node selection), and
turns an opaque pod log into an Ariadne event.
- execute_repair returns {action, target, succeeded, error} and issues one
merge patch; no Job, no polling, no injectable clock, never raises
- the patch writes the same terminal value every time, so idempotency needs
no duplicate guard; one action per incident is still enforced upstream
- orchestrator records {repair, target}; the state machine, rebuild trigger
and failure path are unchanged
- retires the unused repair-image setting
Ariadne's service account now needs get+patch on that one ConfigMap by name
instead of Job create; the batch/jobs grant and the hermes-demo-repair
service account can be retired.
480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The code path was gated on a single job id, so homegrown services could
never produce a pull request. Escalated incidents on mapped repositories
now additionally attempt a bounded patch proposal, and the filed issue
links it.
- hermes_code_flow.propose_for_incident: additive entry point invoked only
from the escalation branch, so an auto-remediated failure never also
gets a patch and a failed Hermes run never spends tokens on one
- three gates before any HTTP call: code enabled, not the legacy demo job,
and the job has a repo mapping; unmapped jobs make zero calls
- never raises: a failed proposal cannot change the incident outcome or
break the tick
- issue body links the proposal when a pull request was opened
- propose_code_fix added to the bounded action-label set
The legacy demo-job short-circuit is untouched and all of its tests pass
unchanged.
8 new tests; 480 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two capabilities that make triage useful outside the demo surface.
Issues: when triage concludes a human is needed, file an issue in the
failing service's own repository carrying classification, confidence, the
facts with their sources, the inferences and a Jenkins link, plus a footer
stating Hermes has no write access and nothing was changed. Opt-in per job
via a repo map, deduplicated by job+classification so a repeatedly failing
job yields one issue per kind of failure rather than one per build, and
capped per tick. Disabled by default.
Real-repo patches: candidate files are selected from the console failure
regions (Python, Rust and JS/TS reference patterns), filtered to each
repo's allowed prefixes and suffixes, ranked earliest-failure-first with
source preferred over test files, and fetched whole - never truncated,
because a patch anchor must match exactly. Per-job owner/repo/base-branch
resolution; the patch is validated against the file the model actually
chose, and an unlisted path is rejected.
Legacy single-repo demo behaviour is preserved unchanged.
131 new tests; 472 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second entry in the action registry, proving it is a real extension point.
No cluster mutation: the action is one Jenkins rebuild.
- hermes_infra_signals: reviewable marker set across DNS/connectivity,
image pull, upstream 5xx and agent-channel loss; Ariadne independently
confirms a marker in the evidence before any retry, and records which
marker justified it. "no space left on device" is deliberately excluded
because a retry lands on the same full volume.
- decision: classification -> action registry (action_classifications),
falling back to the previous single-classification behavior
- repair: retry_build posts to /build for unparameterized real jobs and
buildWithParameters for the fixture demo job
- orchestrator: retry path records requested/accepted/executed and moves
the incident to awaiting_rebuild so the existing success path resolves
it; one action per incident still enforced, so a retry cannot loop
- events layer split out of the orchestrator to stay under the LOC cap
Motivated by real failures tonight: pip DNS resolution and a Gitea 443
connect timeout, plus live incident metis/271 (SCM checkout timeout).
30 new tests; 368 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-service triage returned "undetermined" because the console tail held
only post-build noise; this Jenkins has no junit or stage-view plugin, so
console text is the only structured evidence available.
- hermes_console_evidence: scan the full console for failure markers
(pytest, build, tool gates, k8s/agent), capture context windows, merge
overlaps, collapse repeats, and prefer the earliest regions under a byte
budget; bundle gains jenkins.console_failures and console_truncated
- evidence: fetch the full console (head + tail bounded at 2MB) instead of
the last 8KB; signature detection now also scans regions
- prompt: one line explaining that the earliest region usually holds the
first enforced failure
- code flow: check for an already-open hermes-repair/* pull request before
spending model tokens; fail open so a Gitea error cannot suppress work
Verified on a synthetic 4018-line pipeline: the real failure at line 7 is
now captured where the previous tail-only slice missed it entirely.
95 new tests; 281 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hermes proposes a minimal anchored patch; Ariadne validates it structurally
and opens a pull request for human review. Nothing merges automatically and
Hermes never holds Git credentials or executes anything.
- hermes_code_patch: response parsing + patch validation (path prefix/suffix
allowlist, size and changed-line caps, exact-single-occurrence anchor,
replacement-differs) and exact application
- hermes_code_repair: Gitea contents-API client (fetch, branch push with
hermes-repair/<build> prefix and base-branch refusal, PR creation)
- hermes_code_flow: proposal orchestration + audit event, no patch bodies
or tokens recorded
- hermes_autotriage: code-path branch for the configured repo job; a PR
records human_required/code_fix_proposed, never auto-resolution, and is
kept out of the fixture-action accounting
- settings: ARIADNE_HERMES_CODE_* configuration, disabled by default
74 new tests; 214 pass in the hermes suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gauges are process-local, so a pod restart could silently drop an active
human_required signal before the vmalert hold window elapsed. Metrics move
to hermes_autotriage_metrics; every tick republishes gauges from stored
incident state, and an incident superseded by a newer successful build is
zeroed so the alert clears once the job is green again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Longhorn RWO attach latency and per-node engine availability made the
PVC fixture unreliable for a fast repeatable demo. The fixture is now a
ConfigMap; the repair Job runs bitnami/kubectl under the least-privilege
hermes-demo-repair SA and patches state=healthy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-worker nodes (titan-2x lane) are not Longhorn-ready; volume attach
fails there. Same constraint as the demo test-runner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
