docs: record the TestAutomationV2 diagram corrections
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
Written up rather than applied, since the diagram is maintained in parallel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8b0afb672b
commit
5b29be1010
94
knowledge/hermes-triage-diagram-corrections.md
Normal file
94
knowledge/hermes-triage-diagram-corrections.md
Normal file
@ -0,0 +1,94 @@
|
|||||||
|
# Corrections for `mermaid/TestAutomationV2.mmd`
|
||||||
|
|
||||||
|
Written 2026-08-06 against the diagram as it stood that morning. The diagram
|
||||||
|
is not edited here on purpose: it is being maintained in parallel, so this is
|
||||||
|
the change list rather than a patch.
|
||||||
|
|
||||||
|
The structure holds. The trust boundary, the gate chain, the action registry
|
||||||
|
and the output taxonomy are all still accurate. What follows is what tonight's
|
||||||
|
work invalidated.
|
||||||
|
|
||||||
|
## Nodes that are now factually wrong
|
||||||
|
|
||||||
|
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
|
||||||
|
useful failure regions"*. Earliest-first selection was the defect: on a long
|
||||||
|
pipeline the byte budget was consumed entirely by successful tool output, and
|
||||||
|
the enforced failure never reached Hermes. Replace with:
|
||||||
|
|
||||||
|
```
|
||||||
|
Ariadne evidence collector
|
||||||
|
Reads the full console up to 2 MB
|
||||||
|
Ranks failure regions by evidence strength
|
||||||
|
A definite failure outranks a tool that merely ran
|
||||||
|
Ignores passing test lines that quote failures
|
||||||
|
Merges repeats and overlapping context
|
||||||
|
```
|
||||||
|
|
||||||
|
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
|
||||||
|
repair is an in-process ConfigMap patch; no pod is created.
|
||||||
|
|
||||||
|
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
|
||||||
|
reads the failing test and follows its imports to the module under test, which
|
||||||
|
is what made a pull request on a real service repository possible at all:
|
||||||
|
|
||||||
|
```
|
||||||
|
Candidate file selection
|
||||||
|
Uses path and line hints from failure regions
|
||||||
|
Reads the failing test as context
|
||||||
|
Follows its imports to the module under test
|
||||||
|
Reading is wider than writing: a test is readable, never patchable
|
||||||
|
```
|
||||||
|
|
||||||
|
**`action_scope`** says *"No real service classification maps to a mutation"*.
|
||||||
|
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
|
||||||
|
job, so a real service can now receive an action - a Jenkins rebuild.
|
||||||
|
|
||||||
|
**`timings`** - the fixture loop is not four minutes. Measure from the red
|
||||||
|
build, not from arming, because arming depends on the agent pool:
|
||||||
|
|
||||||
|
```
|
||||||
|
Red -> fixture patched healthy: 25s
|
||||||
|
Red -> rebuild green and incident resolved: 1m04s
|
||||||
|
```
|
||||||
|
|
||||||
|
**`examples`** - supersede with the stronger 2026-08-06 results, including
|
||||||
|
`ariadne/409` opening a real pull request with the correct one-line fix.
|
||||||
|
|
||||||
|
## Structurally misleading
|
||||||
|
|
||||||
|
**The `route` decision reads as three exclusive branches.** It is not. An
|
||||||
|
escalating incident attempts a code proposal *and* files an issue *and* stays
|
||||||
|
`human_required`. A pull request rides along on the escalation; it never
|
||||||
|
replaces the issue. As drawn, a reader concludes one substitutes for the
|
||||||
|
other.
|
||||||
|
|
||||||
|
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
|
||||||
|
builds branches in the demo repository. A pull request opened against a real
|
||||||
|
service repository gets no automatic branch build. Mark it `limited`.
|
||||||
|
|
||||||
|
**`alert_output` is stale.** Alerting no longer fires on every escalation.
|
||||||
|
Two narrow rules remain: a repair that ran and failed, and an escalation
|
||||||
|
untouched for six hours. The issue is the durable artifact; email is the
|
||||||
|
exception.
|
||||||
|
|
||||||
|
## Missing, and worth adding
|
||||||
|
|
||||||
|
- The demo classification and `repair_demo_fixture` are now forbidden on any
|
||||||
|
job other than `hermes-triage-demo`. Hermes misapplied that label to real
|
||||||
|
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
|
||||||
|
and it can no longer be produced at all.
|
||||||
|
- **A build that never finishes** is escalated once it passes the time cap,
|
||||||
|
with no model call, because its console is still being written. Before this
|
||||||
|
existed such a build produced nothing anywhere while holding an agent slot.
|
||||||
|
- **A failure superseded by a newer build can be skipped entirely.** Detection
|
||||||
|
reads `lastBuild`, so a red build replaced quickly is never triaged.
|
||||||
|
|
||||||
|
## Conditional on work that was blocked
|
||||||
|
|
||||||
|
`model_gate` should now read `anthropic/claude-opus-5` primary, with
|
||||||
|
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
|
||||||
|
|
||||||
|
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
|
||||||
|
once the `junit` plugin is live. Note the plugin was never blocked on a core
|
||||||
|
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
|
||||||
|
2.528.3, only the latest release requires 2.533.
|
||||||
@ -0,0 +1,94 @@
|
|||||||
|
# Corrections for `mermaid/TestAutomationV2.mmd`
|
||||||
|
|
||||||
|
Written 2026-08-06 against the diagram as it stood that morning. The diagram
|
||||||
|
is not edited here on purpose: it is being maintained in parallel, so this is
|
||||||
|
the change list rather than a patch.
|
||||||
|
|
||||||
|
The structure holds. The trust boundary, the gate chain, the action registry
|
||||||
|
and the output taxonomy are all still accurate. What follows is what tonight's
|
||||||
|
work invalidated.
|
||||||
|
|
||||||
|
## Nodes that are now factually wrong
|
||||||
|
|
||||||
|
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
|
||||||
|
useful failure regions"*. Earliest-first selection was the defect: on a long
|
||||||
|
pipeline the byte budget was consumed entirely by successful tool output, and
|
||||||
|
the enforced failure never reached Hermes. Replace with:
|
||||||
|
|
||||||
|
```
|
||||||
|
Ariadne evidence collector
|
||||||
|
Reads the full console up to 2 MB
|
||||||
|
Ranks failure regions by evidence strength
|
||||||
|
A definite failure outranks a tool that merely ran
|
||||||
|
Ignores passing test lines that quote failures
|
||||||
|
Merges repeats and overlapping context
|
||||||
|
```
|
||||||
|
|
||||||
|
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
|
||||||
|
repair is an in-process ConfigMap patch; no pod is created.
|
||||||
|
|
||||||
|
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
|
||||||
|
reads the failing test and follows its imports to the module under test, which
|
||||||
|
is what made a pull request on a real service repository possible at all:
|
||||||
|
|
||||||
|
```
|
||||||
|
Candidate file selection
|
||||||
|
Uses path and line hints from failure regions
|
||||||
|
Reads the failing test as context
|
||||||
|
Follows its imports to the module under test
|
||||||
|
Reading is wider than writing: a test is readable, never patchable
|
||||||
|
```
|
||||||
|
|
||||||
|
**`action_scope`** says *"No real service classification maps to a mutation"*.
|
||||||
|
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
|
||||||
|
job, so a real service can now receive an action - a Jenkins rebuild.
|
||||||
|
|
||||||
|
**`timings`** - the fixture loop is not four minutes. Measure from the red
|
||||||
|
build, not from arming, because arming depends on the agent pool:
|
||||||
|
|
||||||
|
```
|
||||||
|
Red -> fixture patched healthy: 25s
|
||||||
|
Red -> rebuild green and incident resolved: 1m04s
|
||||||
|
```
|
||||||
|
|
||||||
|
**`examples`** - supersede with the stronger 2026-08-06 results, including
|
||||||
|
`ariadne/409` opening a real pull request with the correct one-line fix.
|
||||||
|
|
||||||
|
## Structurally misleading
|
||||||
|
|
||||||
|
**The `route` decision reads as three exclusive branches.** It is not. An
|
||||||
|
escalating incident attempts a code proposal *and* files an issue *and* stays
|
||||||
|
`human_required`. A pull request rides along on the escalation; it never
|
||||||
|
replaces the issue. As drawn, a reader concludes one substitutes for the
|
||||||
|
other.
|
||||||
|
|
||||||
|
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
|
||||||
|
builds branches in the demo repository. A pull request opened against a real
|
||||||
|
service repository gets no automatic branch build. Mark it `limited`.
|
||||||
|
|
||||||
|
**`alert_output` is stale.** Alerting no longer fires on every escalation.
|
||||||
|
Two narrow rules remain: a repair that ran and failed, and an escalation
|
||||||
|
untouched for six hours. The issue is the durable artifact; email is the
|
||||||
|
exception.
|
||||||
|
|
||||||
|
## Missing, and worth adding
|
||||||
|
|
||||||
|
- The demo classification and `repair_demo_fixture` are now forbidden on any
|
||||||
|
job other than `hermes-triage-demo`. Hermes misapplied that label to real
|
||||||
|
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
|
||||||
|
and it can no longer be produced at all.
|
||||||
|
- **A build that never finishes** is escalated once it passes the time cap,
|
||||||
|
with no model call, because its console is still being written. Before this
|
||||||
|
existed such a build produced nothing anywhere while holding an agent slot.
|
||||||
|
- **A failure superseded by a newer build can be skipped entirely.** Detection
|
||||||
|
reads `lastBuild`, so a red build replaced quickly is never triaged.
|
||||||
|
|
||||||
|
## Conditional on work that was blocked
|
||||||
|
|
||||||
|
`model_gate` should now read `anthropic/claude-opus-5` primary, with
|
||||||
|
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
|
||||||
|
|
||||||
|
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
|
||||||
|
once the `junit` plugin is live. Note the plugin was never blocked on a core
|
||||||
|
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
|
||||||
|
2.528.3, only the latest release requires 2.533.
|
||||||
Loading…
x
Reference in New Issue
Block a user