From d0753c5eb141d947af36654256115e3237c9e9ef Mon Sep 17 00:00:00 2001 From: jenkins Date: Thu, 6 Aug 2026 01:39:51 -0300 Subject: [PATCH] docs: state what each demo proves and record the two selection defects Co-Authored-By: Claude Opus 5 --- knowledge/hermes-automated-triage-24h-plan.md | 42 ++++++++++++++++++- .../hermes-automated-triage-24h-plan.md | 42 ++++++++++++++++++- 2 files changed, 80 insertions(+), 4 deletions(-) diff --git a/knowledge/hermes-automated-triage-24h-plan.md b/knowledge/hermes-automated-triage-24h-plan.md index b7ffa6594..85d72fc15 100644 --- a/knowledge/hermes-automated-triage-24h-plan.md +++ b/knowledge/hermes-automated-triage-24h-plan.md @@ -1004,8 +1004,31 @@ The handoff must state plainly whether the full Hermes Agent or the Ollama fallb | Action registry: `repair_demo_fixture`, `retry_transient_infra` | live | | Bounded source-patch proposal → branch → pull request | live (demo repo) | | Human-required metric, vmalert rule, Alertmanager | live | -| Gitea issue filing for human-required incidents | built, enabling | -| Candidate-file selection for real service repos | built, enabling | +| Gitea issue filing for human-required incidents | live, proven on 3 real repos | +| Candidate-file selection for real service repos | live, PR not yet observed | + +## What the fixture and code demos actually prove + +Worth stating plainly before demonstrating either one, because the two demos +are not equally representative of the real-service path. + +The **fixture demo** is end-to-end real. Ariadne detects a genuine red build, +Hermes diagnoses it from real evidence, the deterministic gates authorize, and +Ariadne executes and verifies via a rebuild. Nothing about it is staged beyond +the fixture itself being deliberately broken. + +The **code demo** is real in its hard parts and staged in one: the repository +is real, the failing tests are real, Hermes genuinely locates the defect and +writes an anchored patch, the validator genuinely checks it, and the pull +request is genuinely opened and awaits review. What is staged is *which file* +Hermes is handed - `ARIADNE_HERMES_CODE_CANDIDATE_PATH=src/discount.py` pins +it. File selection, the step that has to work on a real service, is the one +step the demo does not exercise. + +On real repositories there is no pin, and selection has to work from evidence +alone. Two defects blocked that and were fixed on 2026-08-06 (see below): +successful tool output crowding the real failure out of the evidence budget, +and the patcher being unable to reach a source file the console never names. ## Measured results @@ -1049,6 +1072,21 @@ Real homegrown service diagnoses (no action allowlisted, all escalated): job went green, the alert self-cleared through supersession. - `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with no matching action, so two independent gates blocked it. +- Two defects were found by deliberately seeding a one-line break in a real + repository (`ariadne`) rather than by reasoning about the code. First, the + console evidence budget was filled entirely by *successful* tool output: + build 404 retained six regions spanning lines 259-476 - Trivy setup, Sonar + banners, and three lines whose only failure signal was the word `coverage` - + and dropped the pytest failure at line 1487. Markers are now ranked by + whether they assert a failure or merely name a tool that ran. Second, the + defective file `ariadne/utils/errors.py` appeared *zero times* in the entire + 174 KB console, because a pytest assertion that fails inside a test names + only the test; candidate selection returned the wrong file. Ariadne now + reads the failing test and follows its imports back to the module under + test, widening what may be read without widening what may be written - the + patch validator still gates writes on `allowed_path_prefixes`, so a failing + test can be read for context and never patched, which also forecloses the + classic bad fix of silencing a test instead of repairing the code. - `bstein-dev-home/446` was classified `known_demo_fixture_failure` at 0.99, even though build 445 of the same job had been classified correctly as `sonarqube_node_runtime_missing`. The evidence gate refused it again, so diff --git a/services/comms/knowledge/hermes-automated-triage-24h-plan.md b/services/comms/knowledge/hermes-automated-triage-24h-plan.md index b7ffa6594..85d72fc15 100644 --- a/services/comms/knowledge/hermes-automated-triage-24h-plan.md +++ b/services/comms/knowledge/hermes-automated-triage-24h-plan.md @@ -1004,8 +1004,31 @@ The handoff must state plainly whether the full Hermes Agent or the Ollama fallb | Action registry: `repair_demo_fixture`, `retry_transient_infra` | live | | Bounded source-patch proposal → branch → pull request | live (demo repo) | | Human-required metric, vmalert rule, Alertmanager | live | -| Gitea issue filing for human-required incidents | built, enabling | -| Candidate-file selection for real service repos | built, enabling | +| Gitea issue filing for human-required incidents | live, proven on 3 real repos | +| Candidate-file selection for real service repos | live, PR not yet observed | + +## What the fixture and code demos actually prove + +Worth stating plainly before demonstrating either one, because the two demos +are not equally representative of the real-service path. + +The **fixture demo** is end-to-end real. Ariadne detects a genuine red build, +Hermes diagnoses it from real evidence, the deterministic gates authorize, and +Ariadne executes and verifies via a rebuild. Nothing about it is staged beyond +the fixture itself being deliberately broken. + +The **code demo** is real in its hard parts and staged in one: the repository +is real, the failing tests are real, Hermes genuinely locates the defect and +writes an anchored patch, the validator genuinely checks it, and the pull +request is genuinely opened and awaits review. What is staged is *which file* +Hermes is handed - `ARIADNE_HERMES_CODE_CANDIDATE_PATH=src/discount.py` pins +it. File selection, the step that has to work on a real service, is the one +step the demo does not exercise. + +On real repositories there is no pin, and selection has to work from evidence +alone. Two defects blocked that and were fixed on 2026-08-06 (see below): +successful tool output crowding the real failure out of the evidence budget, +and the patcher being unable to reach a source file the console never names. ## Measured results @@ -1049,6 +1072,21 @@ Real homegrown service diagnoses (no action allowlisted, all escalated): job went green, the alert self-cleared through supersession. - `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with no matching action, so two independent gates blocked it. +- Two defects were found by deliberately seeding a one-line break in a real + repository (`ariadne`) rather than by reasoning about the code. First, the + console evidence budget was filled entirely by *successful* tool output: + build 404 retained six regions spanning lines 259-476 - Trivy setup, Sonar + banners, and three lines whose only failure signal was the word `coverage` - + and dropped the pytest failure at line 1487. Markers are now ranked by + whether they assert a failure or merely name a tool that ran. Second, the + defective file `ariadne/utils/errors.py` appeared *zero times* in the entire + 174 KB console, because a pytest assertion that fails inside a test names + only the test; candidate selection returned the wrong file. Ariadne now + reads the failing test and follows its imports back to the module under + test, widening what may be read without widening what may be written - the + patch validator still gates writes on `allowed_path_prefixes`, so a failing + test can be read for context and never patched, which also forecloses the + classic bad fix of silencing a test instead of repairing the code. - `bstein-dev-home/446` was classified `known_demo_fixture_failure` at 0.99, even though build 445 of the same job had been classified correctly as `sonarqube_node_runtime_missing`. The evidence gate refused it again, so