docs(hermes-triage): delivery record + enable Gitea issue filing config

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jenkins 2026-08-05 21:01:46 -03:00
parent ba28379818
commit f0c224e5f9
3 changed files with 225 additions and 0 deletions

View File

@ -983,3 +983,109 @@ At H+24, append a short result section containing:
- Kill-switch and rollback verification. - Kill-switch and rollback verification.
The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis. The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis.
---
# Delivery record
- Execution window opened 2026-08-05 ~17:20 UTC.
- Delivered tier: **A, plus the optional autonomous-code stretch.**
- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`,
skill `$triage-titan-test-failures`) in every recorded case. The Ollama
fallback described in the plan was never activated.
## What runs today
| Capability | State |
|---|---|
| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live |
| Jenkins + bounded OpenSearch evidence | live |
| Hermes Agent diagnosis, schema-validated | live |
| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live |
| Bounded source-patch proposal → branch → pull request | live (demo repo) |
| Human-required metric, vmalert rule, Alertmanager | live |
| Gitea issue filing for human-required incidents | built, enabling |
| Candidate-file selection for real service repos | built, enabling |
## Measured results
Closed autonomous loop, twice:
```text
armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14
-> rebuild green 21:01:46 -> resolved 21:02:10 (4m00s)
second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08
```
Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s).
Hermes diagnosis latency across all runs: 15-21s.
Real homegrown service diagnoses (no action allowlisted, all escalated):
| Incident | Classification | Confidence |
|---|---|---|
| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 |
| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 |
| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 |
| `soteria/276` | `unknown` | 0.58 (below threshold) |
## Safety behaviour observed in production, not only in tests
- Hermes classified a pipeline-infrastructure failure as the demo fixture
signature at 0.91 confidence; Ariadne's deterministic evidence gate
refused the action (`evidence_signature_missing`). The model was wrong and
the system still did the right thing.
- With the action allowlist emptied, a genuine fixture failure produced
`human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert
and active in Alertmanager; after an operator fixed the fixture and the
job went green, the alert self-cleared through supersession.
- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with
no matching action, so two independent gates blocked it.
## Deviations from the plan
- **Vault**: the shared Hermes API key and the Gitea automation token are
manually created Kubernetes Secrets, not Vault-sourced. `robotuser`
returned 403 on userpass/ldap/oidc and no token was available. The
manifests reference the Secrets by name, so the swap is config-only.
- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn
attach latency and per-node engine availability made a PVC unreliable for
a fast repeatable demo.
- **JUnit evidence**: this Jenkins has no `junit` publisher and no
`pipeline-stage-view` plugin, so `testReport` and `wfapi` return 404 for
every job. `failed_tests` and `first_failed_stage` are therefore always
empty. Console text is the only structured evidence, which drove the
failure-region extractor. Adding those two plugins (the current `junit`
release needs core 2.533; Jenkins here is 2.528.3, so a back-pin is
required) is the largest remaining quality improvement and needs a
Jenkins restart — deliberately not done the night before a demo.
- **Multibranch jobs are not detected**: folder-style jobs expose a null
`lastBuild` at the job root.
- **OpenSearch namespace** is still pinned to the demo namespace plus
`jenkins`; real services get their Jenkins agent logs but not their own
runtime logs.
## Kill switches and rollback
```text
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false # diagnose only
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false # fully off
ARIADNE_HERMES_ISSUES_ENABLED=false # stop filing issues
ARIADNE_HERMES_CODE_ENABLED=false # stop proposing patches
ARIADNE_HERMES_ALLOWED_ACTIONS="" # accept no action at all
```
All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`.
The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in
`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in
explicitly mapped repositories, and push `hermes-repair/*` branches. One
action per incident, ever.
## Verified rollback and safety properties
- Emptying the action allowlist was exercised live and produced diagnosis
without mutation.
- Restoring it was exercised live and the loop resumed.
- Hermes holds no Git credentials, no Kubernetes write access, and cannot
read Secrets; its Flux-managed approval deny-list still blocks `kubectl`
mutations, `flux` mutations and secret reads.

View File

@ -983,3 +983,109 @@ At H+24, append a short result section containing:
- Kill-switch and rollback verification. - Kill-switch and rollback verification.
The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis. The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis.
---
# Delivery record
- Execution window opened 2026-08-05 ~17:20 UTC.
- Delivered tier: **A, plus the optional autonomous-code stretch.**
- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`,
skill `$triage-titan-test-failures`) in every recorded case. The Ollama
fallback described in the plan was never activated.
## What runs today
| Capability | State |
|---|---|
| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live |
| Jenkins + bounded OpenSearch evidence | live |
| Hermes Agent diagnosis, schema-validated | live |
| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live |
| Bounded source-patch proposal → branch → pull request | live (demo repo) |
| Human-required metric, vmalert rule, Alertmanager | live |
| Gitea issue filing for human-required incidents | built, enabling |
| Candidate-file selection for real service repos | built, enabling |
## Measured results
Closed autonomous loop, twice:
```text
armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14
-> rebuild green 21:01:46 -> resolved 21:02:10 (4m00s)
second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08
```
Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s).
Hermes diagnosis latency across all runs: 15-21s.
Real homegrown service diagnoses (no action allowlisted, all escalated):
| Incident | Classification | Confidence |
|---|---|---|
| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 |
| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 |
| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 |
| `soteria/276` | `unknown` | 0.58 (below threshold) |
## Safety behaviour observed in production, not only in tests
- Hermes classified a pipeline-infrastructure failure as the demo fixture
signature at 0.91 confidence; Ariadne's deterministic evidence gate
refused the action (`evidence_signature_missing`). The model was wrong and
the system still did the right thing.
- With the action allowlist emptied, a genuine fixture failure produced
`human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert
and active in Alertmanager; after an operator fixed the fixture and the
job went green, the alert self-cleared through supersession.
- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with
no matching action, so two independent gates blocked it.
## Deviations from the plan
- **Vault**: the shared Hermes API key and the Gitea automation token are
manually created Kubernetes Secrets, not Vault-sourced. `robotuser`
returned 403 on userpass/ldap/oidc and no token was available. The
manifests reference the Secrets by name, so the swap is config-only.
- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn
attach latency and per-node engine availability made a PVC unreliable for
a fast repeatable demo.
- **JUnit evidence**: this Jenkins has no `junit` publisher and no
`pipeline-stage-view` plugin, so `testReport` and `wfapi` return 404 for
every job. `failed_tests` and `first_failed_stage` are therefore always
empty. Console text is the only structured evidence, which drove the
failure-region extractor. Adding those two plugins (the current `junit`
release needs core 2.533; Jenkins here is 2.528.3, so a back-pin is
required) is the largest remaining quality improvement and needs a
Jenkins restart — deliberately not done the night before a demo.
- **Multibranch jobs are not detected**: folder-style jobs expose a null
`lastBuild` at the job root.
- **OpenSearch namespace** is still pinned to the demo namespace plus
`jenkins`; real services get their Jenkins agent logs but not their own
runtime logs.
## Kill switches and rollback
```text
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false # diagnose only
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false # fully off
ARIADNE_HERMES_ISSUES_ENABLED=false # stop filing issues
ARIADNE_HERMES_CODE_ENABLED=false # stop proposing patches
ARIADNE_HERMES_ALLOWED_ACTIONS="" # accept no action at all
```
All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`.
The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in
`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in
explicitly mapped repositories, and push `hermes-repair/*` branches. One
action per incident, ever.
## Verified rollback and safety properties
- Emptying the action allowlist was exercised live and produced diagnosis
without mutation.
- Restoring it was exercised live and the loop resumed.
- Hermes holds no Git credentials, no Kubernetes write access, and cannot
read Secrets; its Flux-managed approval deny-list still blocks `kubectl`
mutations, `flux` mutations and secret reads.

View File

@ -502,6 +502,19 @@ spec:
value: "20" value: "20"
- name: ARIADNE_HERMES_GITEA_BASE_URL - name: ARIADNE_HERMES_GITEA_BASE_URL
value: https://scm.bstein.dev value: https://scm.bstein.dev
# File a Gitea issue in the failing service's own repository when
# triage concludes a human is needed. Opt-in per job: only jobs
# mapped here can ever receive an issue. Deduplicated by
# job+classification so a repeatedly failing job produces one
# issue per kind of failure, not one per build.
- name: ARIADNE_HERMES_ISSUES_ENABLED
value: "true"
- name: ARIADNE_HERMES_ISSUE_REPOS
value: metis=bstein/metis,lesavka=bstein/lesavka,soteria=bstein/soteria,bstein-dev-home=bstein/bstein-dev-home,ariadne=bstein/ariadne
- name: ARIADNE_HERMES_ISSUE_DEDUPE_SCOPE
value: classification
- name: ARIADNE_HERMES_ISSUE_MAX_PER_TICK
value: "2"
- name: ARIADNE_HERMES_GITEA_TOKEN - name: ARIADNE_HERMES_GITEA_TOKEN
valueFrom: valueFrom:
secretKeyRef: secretKeyRef: