diff --git a/knowledge/hermes-automated-triage-24h-plan.md b/knowledge/hermes-automated-triage-24h-plan.md index 2daf232dc..83fbae2b8 100644 --- a/knowledge/hermes-automated-triage-24h-plan.md +++ b/knowledge/hermes-automated-triage-24h-plan.md @@ -983,3 +983,109 @@ At H+24, append a short result section containing: - Kill-switch and rollback verification. The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis. + +--- + +# Delivery record + +- Execution window opened 2026-08-05 ~17:20 UTC. +- Delivered tier: **A, plus the optional autonomous-code stretch.** +- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`, + skill `$triage-titan-test-failures`) in every recorded case. The Ollama + fallback described in the plan was never activated. + +## What runs today + +| Capability | State | +|---|---| +| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live | +| Jenkins + bounded OpenSearch evidence | live | +| Hermes Agent diagnosis, schema-validated | live | +| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live | +| Bounded source-patch proposal → branch → pull request | live (demo repo) | +| Human-required metric, vmalert rule, Alertmanager | live | +| Gitea issue filing for human-required incidents | built, enabling | +| Candidate-file selection for real service repos | built, enabling | + +## Measured results + +Closed autonomous loop, twice: + +```text +armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14 + -> rebuild green 21:01:46 -> resolved 21:02:10 (4m00s) +second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08 +``` + +Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s). +Hermes diagnosis latency across all runs: 15-21s. + +Real homegrown service diagnoses (no action allowlisted, all escalated): + +| Incident | Classification | Confidence | +|---|---|---| +| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 | +| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 | +| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 | +| `soteria/276` | `unknown` | 0.58 (below threshold) | + +## Safety behaviour observed in production, not only in tests + +- Hermes classified a pipeline-infrastructure failure as the demo fixture + signature at 0.91 confidence; Ariadne's deterministic evidence gate + refused the action (`evidence_signature_missing`). The model was wrong and + the system still did the right thing. +- With the action allowlist emptied, a genuine fixture failure produced + `human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert + and active in Alertmanager; after an operator fixed the fixture and the + job went green, the alert self-cleared through supersession. +- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with + no matching action, so two independent gates blocked it. + +## Deviations from the plan + +- **Vault**: the shared Hermes API key and the Gitea automation token are + manually created Kubernetes Secrets, not Vault-sourced. `robotuser` + returned 403 on userpass/ldap/oidc and no token was available. The + manifests reference the Secrets by name, so the swap is config-only. +- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn + attach latency and per-node engine availability made a PVC unreliable for + a fast repeatable demo. +- **JUnit evidence**: this Jenkins has no `junit` publisher and no + `pipeline-stage-view` plugin, so `testReport` and `wfapi` return 404 for + every job. `failed_tests` and `first_failed_stage` are therefore always + empty. Console text is the only structured evidence, which drove the + failure-region extractor. Adding those two plugins (the current `junit` + release needs core 2.533; Jenkins here is 2.528.3, so a back-pin is + required) is the largest remaining quality improvement and needs a + Jenkins restart — deliberately not done the night before a demo. +- **Multibranch jobs are not detected**: folder-style jobs expose a null + `lastBuild` at the job root. +- **OpenSearch namespace** is still pinned to the demo namespace plus + `jenkins`; real services get their Jenkins agent logs but not their own + runtime logs. + +## Kill switches and rollback + +```text +ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false # diagnose only +ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false # fully off +ARIADNE_HERMES_ISSUES_ENABLED=false # stop filing issues +ARIADNE_HERMES_CODE_ENABLED=false # stop proposing patches +ARIADNE_HERMES_ALLOWED_ACTIONS="" # accept no action at all +``` + +All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`. +The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in +`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in +explicitly mapped repositories, and push `hermes-repair/*` branches. One +action per incident, ever. + +## Verified rollback and safety properties + +- Emptying the action allowlist was exercised live and produced diagnosis + without mutation. +- Restoring it was exercised live and the loop resumed. +- Hermes holds no Git credentials, no Kubernetes write access, and cannot + read Secrets; its Flux-managed approval deny-list still blocks `kubectl` + mutations, `flux` mutations and secret reads. diff --git a/services/comms/knowledge/hermes-automated-triage-24h-plan.md b/services/comms/knowledge/hermes-automated-triage-24h-plan.md index 2daf232dc..83fbae2b8 100644 --- a/services/comms/knowledge/hermes-automated-triage-24h-plan.md +++ b/services/comms/knowledge/hermes-automated-triage-24h-plan.md @@ -983,3 +983,109 @@ At H+24, append a short result section containing: - Kill-switch and rollback verification. The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis. + +--- + +# Delivery record + +- Execution window opened 2026-08-05 ~17:20 UTC. +- Delivered tier: **A, plus the optional autonomous-code stretch.** +- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`, + skill `$triage-titan-test-failures`) in every recorded case. The Ollama + fallback described in the plan was never activated. + +## What runs today + +| Capability | State | +|---|---| +| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live | +| Jenkins + bounded OpenSearch evidence | live | +| Hermes Agent diagnosis, schema-validated | live | +| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live | +| Bounded source-patch proposal → branch → pull request | live (demo repo) | +| Human-required metric, vmalert rule, Alertmanager | live | +| Gitea issue filing for human-required incidents | built, enabling | +| Candidate-file selection for real service repos | built, enabling | + +## Measured results + +Closed autonomous loop, twice: + +```text +armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14 + -> rebuild green 21:01:46 -> resolved 21:02:10 (4m00s) +second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08 +``` + +Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s). +Hermes diagnosis latency across all runs: 15-21s. + +Real homegrown service diagnoses (no action allowlisted, all escalated): + +| Incident | Classification | Confidence | +|---|---|---| +| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 | +| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 | +| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 | +| `soteria/276` | `unknown` | 0.58 (below threshold) | + +## Safety behaviour observed in production, not only in tests + +- Hermes classified a pipeline-infrastructure failure as the demo fixture + signature at 0.91 confidence; Ariadne's deterministic evidence gate + refused the action (`evidence_signature_missing`). The model was wrong and + the system still did the right thing. +- With the action allowlist emptied, a genuine fixture failure produced + `human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert + and active in Alertmanager; after an operator fixed the fixture and the + job went green, the alert self-cleared through supersession. +- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with + no matching action, so two independent gates blocked it. + +## Deviations from the plan + +- **Vault**: the shared Hermes API key and the Gitea automation token are + manually created Kubernetes Secrets, not Vault-sourced. `robotuser` + returned 403 on userpass/ldap/oidc and no token was available. The + manifests reference the Secrets by name, so the swap is config-only. +- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn + attach latency and per-node engine availability made a PVC unreliable for + a fast repeatable demo. +- **JUnit evidence**: this Jenkins has no `junit` publisher and no + `pipeline-stage-view` plugin, so `testReport` and `wfapi` return 404 for + every job. `failed_tests` and `first_failed_stage` are therefore always + empty. Console text is the only structured evidence, which drove the + failure-region extractor. Adding those two plugins (the current `junit` + release needs core 2.533; Jenkins here is 2.528.3, so a back-pin is + required) is the largest remaining quality improvement and needs a + Jenkins restart — deliberately not done the night before a demo. +- **Multibranch jobs are not detected**: folder-style jobs expose a null + `lastBuild` at the job root. +- **OpenSearch namespace** is still pinned to the demo namespace plus + `jenkins`; real services get their Jenkins agent logs but not their own + runtime logs. + +## Kill switches and rollback + +```text +ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false # diagnose only +ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false # fully off +ARIADNE_HERMES_ISSUES_ENABLED=false # stop filing issues +ARIADNE_HERMES_CODE_ENABLED=false # stop proposing patches +ARIADNE_HERMES_ALLOWED_ACTIONS="" # accept no action at all +``` + +All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`. +The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in +`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in +explicitly mapped repositories, and push `hermes-repair/*` branches. One +action per incident, ever. + +## Verified rollback and safety properties + +- Emptying the action allowlist was exercised live and produced diagnosis + without mutation. +- Restoring it was exercised live and the loop resumed. +- Hermes holds no Git credentials, no Kubernetes write access, and cannot + read Secrets; its Flux-managed approval deny-list still blocks `kubectl` + mutations, `flux` mutations and secret reads. diff --git a/services/maintenance/apps/ariadne-deployment.yaml b/services/maintenance/apps/ariadne-deployment.yaml index b52b1b816..6e1e9eea7 100644 --- a/services/maintenance/apps/ariadne-deployment.yaml +++ b/services/maintenance/apps/ariadne-deployment.yaml @@ -502,6 +502,19 @@ spec: value: "20" - name: ARIADNE_HERMES_GITEA_BASE_URL value: https://scm.bstein.dev + # File a Gitea issue in the failing service's own repository when + # triage concludes a human is needed. Opt-in per job: only jobs + # mapped here can ever receive an issue. Deduplicated by + # job+classification so a repeatedly failing job produces one + # issue per kind of failure, not one per build. + - name: ARIADNE_HERMES_ISSUES_ENABLED + value: "true" + - name: ARIADNE_HERMES_ISSUE_REPOS + value: metis=bstein/metis,lesavka=bstein/lesavka,soteria=bstein/soteria,bstein-dev-home=bstein/bstein-dev-home,ariadne=bstein/ariadne + - name: ARIADNE_HERMES_ISSUE_DEDUPE_SCOPE + value: classification + - name: ARIADNE_HERMES_ISSUE_MAX_PER_TICK + value: "2" - name: ARIADNE_HERMES_GITEA_TOKEN valueFrom: secretKeyRef: