135 lines
6.3 KiB
Markdown
135 lines
6.3 KiB
Markdown
# Corrections for `mermaid/TestAutomationV2.mmd`
|
|
|
|
Written 2026-08-06 against the diagram as it stood that morning. The diagram
|
|
is not edited here on purpose: it is being maintained in parallel, so this is
|
|
the change list rather than a patch.
|
|
|
|
The structure holds. The trust boundary, the gate chain, the action registry
|
|
and the output taxonomy are all still accurate. What follows is what tonight's
|
|
work invalidated.
|
|
|
|
## Nodes that are now factually wrong
|
|
|
|
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
|
|
useful failure regions"*. Earliest-first selection was the defect: on a long
|
|
pipeline the byte budget was consumed entirely by successful tool output, and
|
|
the enforced failure never reached Hermes. Replace with:
|
|
|
|
```
|
|
Ariadne evidence collector
|
|
Reads the full console up to 2 MB
|
|
Ranks failure regions by evidence strength
|
|
A definite failure outranks a tool that merely ran
|
|
Ignores passing test lines that quote failures
|
|
Merges repeats and overlapping context
|
|
```
|
|
|
|
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
|
|
repair is an in-process ConfigMap patch; no pod is created.
|
|
|
|
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
|
|
reads the failing test and follows its imports to the module under test, which
|
|
is what made a pull request on a real service repository possible at all:
|
|
|
|
```
|
|
Candidate file selection
|
|
Uses path and line hints from failure regions
|
|
Reads the failing test as context
|
|
Follows its imports to the module under test
|
|
Reading is wider than writing: a test is readable, never patchable
|
|
```
|
|
|
|
**`action_scope`** says *"No real service classification maps to a mutation"*.
|
|
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
|
|
job, so a real service can now receive an action - a Jenkins rebuild.
|
|
|
|
**`timings`** - the fixture loop is not four minutes. Measure from the red
|
|
build, not from arming, because arming depends on the agent pool:
|
|
|
|
```
|
|
Red -> fixture patched healthy: 25s
|
|
Red -> rebuild green and incident resolved: 1m04s
|
|
```
|
|
|
|
**`examples`** - supersede with the stronger 2026-08-06 results, including
|
|
`ariadne/409` opening a real pull request with the correct one-line fix.
|
|
|
|
## Structurally misleading
|
|
|
|
**The `route` decision reads as three exclusive branches.** It is not. An
|
|
escalating incident attempts a code proposal *and* files an issue *and* stays
|
|
`human_required`. A pull request rides along on the escalation; it never
|
|
replaces the issue. As drawn, a reader concludes one substitutes for the
|
|
other.
|
|
|
|
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
|
|
builds branches in the demo repository. A pull request opened against a real
|
|
service repository gets no automatic branch build. Mark it `limited`.
|
|
|
|
**`alert_output` is stale.** Alerting no longer fires on every escalation.
|
|
Two narrow rules remain: a repair that ran and failed, and an escalation
|
|
untouched for six hours. The issue is the durable artifact; email is the
|
|
exception.
|
|
|
|
## Missing, and worth adding
|
|
|
|
- The demo classification and `repair_demo_fixture` are now forbidden on any
|
|
job other than `hermes-triage-demo`. Hermes misapplied that label to real
|
|
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
|
|
and it can no longer be produced at all.
|
|
- **A build that never finishes** is escalated once it passes the time cap,
|
|
with no model call, because its console is still being written. Before this
|
|
existed such a build produced nothing anywhere while holding an agent slot.
|
|
- **A failure superseded by a newer build can be skipped entirely.** Detection
|
|
reads `lastBuild`, so a red build replaced quickly is never triaged.
|
|
|
|
## Second pass, after the remaining work landed
|
|
|
|
Everything previously marked conditional is now done, which changes three
|
|
things structurally rather than cosmetically.
|
|
|
|
**Not every incident reaches Hermes any more.** The diagram's main flow runs
|
|
intake -> hermes_plane -> controls -> response, which reads as though every
|
|
incident is diagnosed by the model. A build that overruns its time cap is now
|
|
escalated by Ariadne alone, with no model call at all, because its console is
|
|
still being written and any root cause would be invented. That needs its own
|
|
edge from the detector straight to the human path, bypassing `hermes_plane`
|
|
and `controls` entirely. It is also the honest picture of the trust boundary:
|
|
some conclusions are Ariadne's own observations, and the issue body now says
|
|
so rather than crediting a diagnosis that never happened.
|
|
|
|
**The Jenkins evidence limitation is gone, not narrowed.** Delete
|
|
`jenkins_limit` and the "Per-test evidence waits on Jenkins plugins" line from
|
|
`current_limits`. `junit` and `pipeline-stage-view` are installed, `testReport`
|
|
and `wfapi` return 200, and the bundle now carries `failed_tests` with the
|
|
failing test name, class and assertion, plus `first_failed_stage`. The
|
|
collector always fetched both and had simply been receiving 404s. The `bundle`
|
|
node should list them as populated fields, because they are what the diagnosis
|
|
now rests on rather than scraped console text.
|
|
|
|
**The detector watches branches, not just jobs.** A multibranch project is a
|
|
folder with no `lastBuild`; it was returned as skipped on every tick. The
|
|
detector now expands a folder into its branch jobs, capped at five, so
|
|
`detector` should say it watches configured jobs *and the branches inside
|
|
multibranch folders*.
|
|
|
|
Smaller corrections to the same nodes:
|
|
|
|
- `evidence_sources` may include a service's own namespace, not only the demo
|
|
namespace and `jenkins`, where the job is mapped to one.
|
|
- The human path files an issue even for an incident triaged before issue
|
|
filing existed, reusing the classification the original diagnosis recorded
|
|
so one open issue per job and classification still holds.
|
|
- Hermes's credential comes from Vault, not a manually created Secret. If the
|
|
diagram ever shows credential provenance, that is now the accurate source.
|
|
|
|
## Conditional on work that was blocked
|
|
|
|
`model_gate` should now read `anthropic/claude-opus-5` primary, with
|
|
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
|
|
|
|
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
|
|
once the `junit` plugin is live. Note the plugin was never blocked on a core
|
|
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
|
|
2.528.3, only the latest release requires 2.533.
|