diff --git a/knowledge/hermes-triage-diagram-corrections.md b/knowledge/hermes-triage-diagram-corrections.md new file mode 100644 index 000000000..53c664e24 --- /dev/null +++ b/knowledge/hermes-triage-diagram-corrections.md @@ -0,0 +1,94 @@ +# Corrections for `mermaid/TestAutomationV2.mmd` + +Written 2026-08-06 against the diagram as it stood that morning. The diagram +is not edited here on purpose: it is being maintained in parallel, so this is +the change list rather than a patch. + +The structure holds. The trust boundary, the gate chain, the action registry +and the output taxonomy are all still accurate. What follows is what tonight's +work invalidated. + +## Nodes that are now factually wrong + +**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest +useful failure regions"*. Earliest-first selection was the defect: on a long +pipeline the byte budget was consumed entirely by successful tool output, and +the enforced failure never reached Hermes. Replace with: + +``` +Ariadne evidence collector +Reads the full console up to 2 MB +Ranks failure regions by evidence strength +A definite failure outranks a tool that merely ran +Ignores passing test lines that quote failures +Merges repeats and overlapping context +``` + +**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The +repair is an in-process ConfigMap patch; no pod is created. + +**`candidate_files`** says *"Ranks the earliest hints first"*. It now also +reads the failing test and follows its imports to the module under test, which +is what made a pull request on a real service repository possible at all: + +``` +Candidate file selection +Uses path and line hints from failure regions +Reads the failing test as context +Follows its imports to the module under test +Reading is wider than writing: a test is readable, never patchable +``` + +**`action_scope`** says *"No real service classification maps to a mutation"*. +No longer true. `retry_transient_infra` is allowlisted for every allowlisted +job, so a real service can now receive an action - a Jenkins rebuild. + +**`timings`** - the fixture loop is not four minutes. Measure from the red +build, not from arming, because arming depends on the agent pool: + +``` +Red -> fixture patched healthy: 25s +Red -> rebuild green and incident resolved: 1m04s +``` + +**`examples`** - supersede with the stronger 2026-08-06 results, including +`ariadne/409` opening a real pull request with the correct one-line fix. + +## Structurally misleading + +**The `route` decision reads as three exclusive branches.** It is not. An +escalating incident attempts a code proposal *and* files an issue *and* stays +`human_required`. A pull request rides along on the escalation; it never +replaces the issue. As drawn, a reader concludes one substitutes for the +other. + +**`branch_build` overstates coverage.** `hermes-code-demo-branches` only +builds branches in the demo repository. A pull request opened against a real +service repository gets no automatic branch build. Mark it `limited`. + +**`alert_output` is stale.** Alerting no longer fires on every escalation. +Two narrow rules remain: a repair that ran and failed, and an escalation +untouched for six hours. The issue is the durable artifact; email is the +exception. + +## Missing, and worth adding + +- The demo classification and `repair_demo_fixture` are now forbidden on any + job other than `hermes-triage-demo`. Hermes misapplied that label to real + services twice at 0.96 and 0.99 confidence; the gates refused it both times, + and it can no longer be produced at all. +- **A build that never finishes** is escalated once it passes the time cap, + with no model call, because its console is still being written. Before this + existed such a build produced nothing anywhere while holding an agent slot. +- **A failure superseded by a newer build can be skipped entirely.** Detection + reads `lastBuild`, so a red build replaced quickly is never triaged. + +## Conditional on work that was blocked + +`model_gate` should now read `anthropic/claude-opus-5` primary, with +`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second. + +`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out +once the `junit` plugin is live. Note the plugin was never blocked on a core +upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core +2.528.3, only the latest release requires 2.533. diff --git a/services/comms/knowledge/hermes-triage-diagram-corrections.md b/services/comms/knowledge/hermes-triage-diagram-corrections.md new file mode 100644 index 000000000..53c664e24 --- /dev/null +++ b/services/comms/knowledge/hermes-triage-diagram-corrections.md @@ -0,0 +1,94 @@ +# Corrections for `mermaid/TestAutomationV2.mmd` + +Written 2026-08-06 against the diagram as it stood that morning. The diagram +is not edited here on purpose: it is being maintained in parallel, so this is +the change list rather than a patch. + +The structure holds. The trust boundary, the gate chain, the action registry +and the output taxonomy are all still accurate. What follows is what tonight's +work invalidated. + +## Nodes that are now factually wrong + +**`collector`** says *"Keeps the head and tail"* and *"Keeps the earliest +useful failure regions"*. Earliest-first selection was the defect: on a long +pipeline the byte budget was consumed entirely by successful tool output, and +the enforced failure never reached Hermes. Replace with: + +``` +Ariadne evidence collector +Reads the full console up to 2 MB +Ranks failure regions by evidence strength +A definite failure outranks a tool that merely ran +Ignores passing test lines that quote failures +Merges repeats and overlapping context +``` + +**`repair_fixture`** says *"Creates one scoped Job"*. It no longer does. The +repair is an in-process ConfigMap patch; no pod is created. + +**`candidate_files`** says *"Ranks the earliest hints first"*. It now also +reads the failing test and follows its imports to the module under test, which +is what made a pull request on a real service repository possible at all: + +``` +Candidate file selection +Uses path and line hints from failure regions +Reads the failing test as context +Follows its imports to the module under test +Reading is wider than writing: a test is readable, never patchable +``` + +**`action_scope`** says *"No real service classification maps to a mutation"*. +No longer true. `retry_transient_infra` is allowlisted for every allowlisted +job, so a real service can now receive an action - a Jenkins rebuild. + +**`timings`** - the fixture loop is not four minutes. Measure from the red +build, not from arming, because arming depends on the agent pool: + +``` +Red -> fixture patched healthy: 25s +Red -> rebuild green and incident resolved: 1m04s +``` + +**`examples`** - supersede with the stronger 2026-08-06 results, including +`ariadne/409` opening a real pull request with the correct one-line fix. + +## Structurally misleading + +**The `route` decision reads as three exclusive branches.** It is not. An +escalating incident attempts a code proposal *and* files an issue *and* stays +`human_required`. A pull request rides along on the escalation; it never +replaces the issue. As drawn, a reader concludes one substitutes for the +other. + +**`branch_build` overstates coverage.** `hermes-code-demo-branches` only +builds branches in the demo repository. A pull request opened against a real +service repository gets no automatic branch build. Mark it `limited`. + +**`alert_output` is stale.** Alerting no longer fires on every escalation. +Two narrow rules remain: a repair that ran and failed, and an escalation +untouched for six hours. The issue is the durable artifact; email is the +exception. + +## Missing, and worth adding + +- The demo classification and `repair_demo_fixture` are now forbidden on any + job other than `hermes-triage-demo`. Hermes misapplied that label to real + services twice at 0.96 and 0.99 confidence; the gates refused it both times, + and it can no longer be produced at all. +- **A build that never finishes** is escalated once it passes the time cap, + with no model call, because its console is still being written. Before this + existed such a build produced nothing anywhere while holding an agent slot. +- **A failure superseded by a newer build can be skipped entirely.** Detection + reads `lastBuild`, so a red build replaced quickly is never triaged. + +## Conditional on work that was blocked + +`model_gate` should now read `anthropic/claude-opus-5` primary, with +`openai-codex/gpt-5.6-terra` first fallback and local `gpt-oss:20b` second. + +`jenkins_limit` and *"Per-test evidence waits on Jenkins plugins"* come out +once the `junit` plugin is live. Note the plugin was never blocked on a core +upgrade as previously recorded: `junit 1369.v15da_00283f06` runs on core +2.528.3, only the latest release requires 2.533.