docs: record the TestAutomationV2 diagram corrections
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed

Written up rather than applied, since the diagram is maintained in parallel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
jenkins 2026-08-06 06:18:16 -03:00
parent 8b0afb672b
commit 5b29be1010
2 changed files with 188 additions and 0 deletions

View File

@ -0,0 +1,94 @@
# Corrections for `mermaid/TestAutomationV2.mmd`
Written 2026-08-06 against the diagram as it stood that morning. The diagram
is not edited here on purpose: it is being maintained in parallel, so this is
the change list rather than a patch.
The structure holds. The trust boundary, the gate chain, the action registry
and the output taxonomy are all still accurate. What follows is what tonight's
work invalidated.
## Nodes that are now factually wrong
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
useful failure regions"*. Earliest-first selection was the defect: on a long
pipeline the byte budget was consumed entirely by successful tool output, and
the enforced failure never reached Hermes. Replace with:
```
Ariadne evidence collector
Reads the full console up to 2 MB
Ranks failure regions by evidence strength
A definite failure outranks a tool that merely ran
Ignores passing test lines that quote failures
Merges repeats and overlapping context
```
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
repair is an in-process ConfigMap patch; no pod is created.
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
reads the failing test and follows its imports to the module under test, which
is what made a pull request on a real service repository possible at all:
```
Candidate file selection
Uses path and line hints from failure regions
Reads the failing test as context
Follows its imports to the module under test
Reading is wider than writing: a test is readable, never patchable
```
**`action_scope`** says *"No real service classification maps to a mutation"*.
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
job, so a real service can now receive an action - a Jenkins rebuild.
**`timings`** - the fixture loop is not four minutes. Measure from the red
build, not from arming, because arming depends on the agent pool:
```
Red -> fixture patched healthy: 25s
Red -> rebuild green and incident resolved: 1m04s
```
**`examples`** - supersede with the stronger 2026-08-06 results, including
`ariadne/409` opening a real pull request with the correct one-line fix.
## Structurally misleading
**The `route` decision reads as three exclusive branches.** It is not. An
escalating incident attempts a code proposal *and* files an issue *and* stays
`human_required`. A pull request rides along on the escalation; it never
replaces the issue. As drawn, a reader concludes one substitutes for the
other.
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
builds branches in the demo repository. A pull request opened against a real
service repository gets no automatic branch build. Mark it `limited`.
**`alert_output` is stale.** Alerting no longer fires on every escalation.
Two narrow rules remain: a repair that ran and failed, and an escalation
untouched for six hours. The issue is the durable artifact; email is the
exception.
## Missing, and worth adding
- The demo classification and `repair_demo_fixture` are now forbidden on any
job other than `hermes-triage-demo`. Hermes misapplied that label to real
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
and it can no longer be produced at all.
- **A build that never finishes** is escalated once it passes the time cap,
with no model call, because its console is still being written. Before this
existed such a build produced nothing anywhere while holding an agent slot.
- **A failure superseded by a newer build can be skipped entirely.** Detection
reads `lastBuild`, so a red build replaced quickly is never triaged.
## Conditional on work that was blocked
`model_gate` should now read `anthropic/claude-opus-5` primary, with
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
once the `junit` plugin is live. Note the plugin was never blocked on a core
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
2.528.3, only the latest release requires 2.533.

View File

@ -0,0 +1,94 @@
# Corrections for `mermaid/TestAutomationV2.mmd`
Written 2026-08-06 against the diagram as it stood that morning. The diagram
is not edited here on purpose: it is being maintained in parallel, so this is
the change list rather than a patch.
The structure holds. The trust boundary, the gate chain, the action registry
and the output taxonomy are all still accurate. What follows is what tonight's
work invalidated.
## Nodes that are now factually wrong
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
useful failure regions"*. Earliest-first selection was the defect: on a long
pipeline the byte budget was consumed entirely by successful tool output, and
the enforced failure never reached Hermes. Replace with:
```
Ariadne evidence collector
Reads the full console up to 2 MB
Ranks failure regions by evidence strength
A definite failure outranks a tool that merely ran
Ignores passing test lines that quote failures
Merges repeats and overlapping context
```
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
repair is an in-process ConfigMap patch; no pod is created.
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
reads the failing test and follows its imports to the module under test, which
is what made a pull request on a real service repository possible at all:
```
Candidate file selection
Uses path and line hints from failure regions
Reads the failing test as context
Follows its imports to the module under test
Reading is wider than writing: a test is readable, never patchable
```
**`action_scope`** says *"No real service classification maps to a mutation"*.
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
job, so a real service can now receive an action - a Jenkins rebuild.
**`timings`** - the fixture loop is not four minutes. Measure from the red
build, not from arming, because arming depends on the agent pool:
```
Red -> fixture patched healthy: 25s
Red -> rebuild green and incident resolved: 1m04s
```
**`examples`** - supersede with the stronger 2026-08-06 results, including
`ariadne/409` opening a real pull request with the correct one-line fix.
## Structurally misleading
**The `route` decision reads as three exclusive branches.** It is not. An
escalating incident attempts a code proposal *and* files an issue *and* stays
`human_required`. A pull request rides along on the escalation; it never
replaces the issue. As drawn, a reader concludes one substitutes for the
other.
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
builds branches in the demo repository. A pull request opened against a real
service repository gets no automatic branch build. Mark it `limited`.
**`alert_output` is stale.** Alerting no longer fires on every escalation.
Two narrow rules remain: a repair that ran and failed, and an escalation
untouched for six hours. The issue is the durable artifact; email is the
exception.
## Missing, and worth adding
- The demo classification and `repair_demo_fixture` are now forbidden on any
job other than `hermes-triage-demo`. Hermes misapplied that label to real
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
and it can no longer be produced at all.
- **A build that never finishes** is escalated once it passes the time cap,
with no model call, because its console is still being written. Before this
existed such a build produced nothing anywhere while holding an agent slot.
- **A failure superseded by a newer build can be skipped entirely.** Detection
reads `lastBuild`, so a red build replaced quickly is never triaged.
## Conditional on work that was blocked
`model_gate` should now read `anthropic/claude-opus-5` primary, with
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
once the `junit` plugin is live. Note the plugin was never blocked on a core
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
2.528.3, only the latest release requires 2.533.