docs: record the TestAutomationV2 diagram corrections
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
Written up rather than applied, since the diagram is maintained in parallel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8b0afb672b
commit
5b29be1010
94
knowledge/hermes-triage-diagram-corrections.md
Normal file
94
knowledge/hermes-triage-diagram-corrections.md
Normal file
@ -0,0 +1,94 @@
|
||||
# Corrections for `mermaid/TestAutomationV2.mmd`
|
||||
|
||||
Written 2026-08-06 against the diagram as it stood that morning. The diagram
|
||||
is not edited here on purpose: it is being maintained in parallel, so this is
|
||||
the change list rather than a patch.
|
||||
|
||||
The structure holds. The trust boundary, the gate chain, the action registry
|
||||
and the output taxonomy are all still accurate. What follows is what tonight's
|
||||
work invalidated.
|
||||
|
||||
## Nodes that are now factually wrong
|
||||
|
||||
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
|
||||
useful failure regions"*. Earliest-first selection was the defect: on a long
|
||||
pipeline the byte budget was consumed entirely by successful tool output, and
|
||||
the enforced failure never reached Hermes. Replace with:
|
||||
|
||||
```
|
||||
Ariadne evidence collector
|
||||
Reads the full console up to 2 MB
|
||||
Ranks failure regions by evidence strength
|
||||
A definite failure outranks a tool that merely ran
|
||||
Ignores passing test lines that quote failures
|
||||
Merges repeats and overlapping context
|
||||
```
|
||||
|
||||
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
|
||||
repair is an in-process ConfigMap patch; no pod is created.
|
||||
|
||||
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
|
||||
reads the failing test and follows its imports to the module under test, which
|
||||
is what made a pull request on a real service repository possible at all:
|
||||
|
||||
```
|
||||
Candidate file selection
|
||||
Uses path and line hints from failure regions
|
||||
Reads the failing test as context
|
||||
Follows its imports to the module under test
|
||||
Reading is wider than writing: a test is readable, never patchable
|
||||
```
|
||||
|
||||
**`action_scope`** says *"No real service classification maps to a mutation"*.
|
||||
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
|
||||
job, so a real service can now receive an action - a Jenkins rebuild.
|
||||
|
||||
**`timings`** - the fixture loop is not four minutes. Measure from the red
|
||||
build, not from arming, because arming depends on the agent pool:
|
||||
|
||||
```
|
||||
Red -> fixture patched healthy: 25s
|
||||
Red -> rebuild green and incident resolved: 1m04s
|
||||
```
|
||||
|
||||
**`examples`** - supersede with the stronger 2026-08-06 results, including
|
||||
`ariadne/409` opening a real pull request with the correct one-line fix.
|
||||
|
||||
## Structurally misleading
|
||||
|
||||
**The `route` decision reads as three exclusive branches.** It is not. An
|
||||
escalating incident attempts a code proposal *and* files an issue *and* stays
|
||||
`human_required`. A pull request rides along on the escalation; it never
|
||||
replaces the issue. As drawn, a reader concludes one substitutes for the
|
||||
other.
|
||||
|
||||
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
|
||||
builds branches in the demo repository. A pull request opened against a real
|
||||
service repository gets no automatic branch build. Mark it `limited`.
|
||||
|
||||
**`alert_output` is stale.** Alerting no longer fires on every escalation.
|
||||
Two narrow rules remain: a repair that ran and failed, and an escalation
|
||||
untouched for six hours. The issue is the durable artifact; email is the
|
||||
exception.
|
||||
|
||||
## Missing, and worth adding
|
||||
|
||||
- The demo classification and `repair_demo_fixture` are now forbidden on any
|
||||
job other than `hermes-triage-demo`. Hermes misapplied that label to real
|
||||
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
|
||||
and it can no longer be produced at all.
|
||||
- **A build that never finishes** is escalated once it passes the time cap,
|
||||
with no model call, because its console is still being written. Before this
|
||||
existed such a build produced nothing anywhere while holding an agent slot.
|
||||
- **A failure superseded by a newer build can be skipped entirely.** Detection
|
||||
reads `lastBuild`, so a red build replaced quickly is never triaged.
|
||||
|
||||
## Conditional on work that was blocked
|
||||
|
||||
`model_gate` should now read `anthropic/claude-opus-5` primary, with
|
||||
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
|
||||
|
||||
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
|
||||
once the `junit` plugin is live. Note the plugin was never blocked on a core
|
||||
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
|
||||
2.528.3, only the latest release requires 2.533.
|
||||
@ -0,0 +1,94 @@
|
||||
# Corrections for `mermaid/TestAutomationV2.mmd`
|
||||
|
||||
Written 2026-08-06 against the diagram as it stood that morning. The diagram
|
||||
is not edited here on purpose: it is being maintained in parallel, so this is
|
||||
the change list rather than a patch.
|
||||
|
||||
The structure holds. The trust boundary, the gate chain, the action registry
|
||||
and the output taxonomy are all still accurate. What follows is what tonight's
|
||||
work invalidated.
|
||||
|
||||
## Nodes that are now factually wrong
|
||||
|
||||
**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest
|
||||
useful failure regions"*. Earliest-first selection was the defect: on a long
|
||||
pipeline the byte budget was consumed entirely by successful tool output, and
|
||||
the enforced failure never reached Hermes. Replace with:
|
||||
|
||||
```
|
||||
Ariadne evidence collector
|
||||
Reads the full console up to 2 MB
|
||||
Ranks failure regions by evidence strength
|
||||
A definite failure outranks a tool that merely ran
|
||||
Ignores passing test lines that quote failures
|
||||
Merges repeats and overlapping context
|
||||
```
|
||||
|
||||
**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The
|
||||
repair is an in-process ConfigMap patch; no pod is created.
|
||||
|
||||
**`candidate_files`** says *"Ranks the earliest hints first"*. It now also
|
||||
reads the failing test and follows its imports to the module under test, which
|
||||
is what made a pull request on a real service repository possible at all:
|
||||
|
||||
```
|
||||
Candidate file selection
|
||||
Uses path and line hints from failure regions
|
||||
Reads the failing test as context
|
||||
Follows its imports to the module under test
|
||||
Reading is wider than writing: a test is readable, never patchable
|
||||
```
|
||||
|
||||
**`action_scope`** says *"No real service classification maps to a mutation"*.
|
||||
No longer true. `retry_transient_infra` is allowlisted for every allowlisted
|
||||
job, so a real service can now receive an action - a Jenkins rebuild.
|
||||
|
||||
**`timings`** - the fixture loop is not four minutes. Measure from the red
|
||||
build, not from arming, because arming depends on the agent pool:
|
||||
|
||||
```
|
||||
Red -> fixture patched healthy: 25s
|
||||
Red -> rebuild green and incident resolved: 1m04s
|
||||
```
|
||||
|
||||
**`examples`** - supersede with the stronger 2026-08-06 results, including
|
||||
`ariadne/409` opening a real pull request with the correct one-line fix.
|
||||
|
||||
## Structurally misleading
|
||||
|
||||
**The `route` decision reads as three exclusive branches.** It is not. An
|
||||
escalating incident attempts a code proposal *and* files an issue *and* stays
|
||||
`human_required`. A pull request rides along on the escalation; it never
|
||||
replaces the issue. As drawn, a reader concludes one substitutes for the
|
||||
other.
|
||||
|
||||
**`branch_build` overstates coverage.** `hermes-code-demo-branches` only
|
||||
builds branches in the demo repository. A pull request opened against a real
|
||||
service repository gets no automatic branch build. Mark it `limited`.
|
||||
|
||||
**`alert_output` is stale.** Alerting no longer fires on every escalation.
|
||||
Two narrow rules remain: a repair that ran and failed, and an escalation
|
||||
untouched for six hours. The issue is the durable artifact; email is the
|
||||
exception.
|
||||
|
||||
## Missing, and worth adding
|
||||
|
||||
- The demo classification and `repair_demo_fixture` are now forbidden on any
|
||||
job other than `hermes-triage-demo`. Hermes misapplied that label to real
|
||||
services twice at 0.96 and 0.99 confidence; the gates refused it both times,
|
||||
and it can no longer be produced at all.
|
||||
- **A build that never finishes** is escalated once it passes the time cap,
|
||||
with no model call, because its console is still being written. Before this
|
||||
existed such a build produced nothing anywhere while holding an agent slot.
|
||||
- **A failure superseded by a newer build can be skipped entirely.** Detection
|
||||
reads `lastBuild`, so a red build replaced quickly is never triaged.
|
||||
|
||||
## Conditional on work that was blocked
|
||||
|
||||
`model_gate` should now read `anthropic/claude-opus-5` primary, with
|
||||
`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second.
|
||||
|
||||
`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out
|
||||
once the `junit` plugin is live. Note the plugin was never blocked on a core
|
||||
upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core
|
||||
2.528.3, only the latest release requires 2.533.
|
||||
Loading…
x
Reference in New Issue
Block a user