docs(hermes-triage): delivery record + enable Gitea issue filing config
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jenkins
parent
ba28379818
commit
f0c224e5f9
@ -983,3 +983,109 @@ At H+24, append a short result section containing:
|
||||
- Kill-switch and rollback verification.
|
||||
|
||||
The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis.
|
||||
|
||||
---
|
||||
|
||||
# Delivery record
|
||||
|
||||
- Execution window opened 2026-08-05 ~17:20 UTC.
|
||||
- Delivered tier: **A, plus the optional autonomous-code stretch.**
|
||||
- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`,
|
||||
skill `$triage-titan-test-failures`) in every recorded case. The Ollama
|
||||
fallback described in the plan was never activated.
|
||||
|
||||
## What runs today
|
||||
|
||||
| Capability | State |
|
||||
|---|---|
|
||||
| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live |
|
||||
| Jenkins + bounded OpenSearch evidence | live |
|
||||
| Hermes Agent diagnosis, schema-validated | live |
|
||||
| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live |
|
||||
| Bounded source-patch proposal → branch → pull request | live (demo repo) |
|
||||
| Human-required metric, vmalert rule, Alertmanager | live |
|
||||
| Gitea issue filing for human-required incidents | built, enabling |
|
||||
| Candidate-file selection for real service repos | built, enabling |
|
||||
|
||||
## Measured results
|
||||
|
||||
Closed autonomous loop, twice:
|
||||
|
||||
```text
|
||||
armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14
|
||||
-> rebuild green 21:01:46 -> resolved 21:02:10 (4m00s)
|
||||
second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08
|
||||
```
|
||||
|
||||
Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s).
|
||||
Hermes diagnosis latency across all runs: 15-21s.
|
||||
|
||||
Real homegrown service diagnoses (no action allowlisted, all escalated):
|
||||
|
||||
| Incident | Classification | Confidence |
|
||||
|---|---|---|
|
||||
| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 |
|
||||
| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 |
|
||||
| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 |
|
||||
| `soteria/276` | `unknown` | 0.58 (below threshold) |
|
||||
|
||||
## Safety behaviour observed in production, not only in tests
|
||||
|
||||
- Hermes classified a pipeline-infrastructure failure as the demo fixture
|
||||
signature at 0.91 confidence; Ariadne's deterministic evidence gate
|
||||
refused the action (`evidence_signature_missing`). The model was wrong and
|
||||
the system still did the right thing.
|
||||
- With the action allowlist emptied, a genuine fixture failure produced
|
||||
`human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert
|
||||
and active in Alertmanager; after an operator fixed the fixture and the
|
||||
job went green, the alert self-cleared through supersession.
|
||||
- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with
|
||||
no matching action, so two independent gates blocked it.
|
||||
|
||||
## Deviations from the plan
|
||||
|
||||
- **Vault**: the shared Hermes API key and the Gitea automation token are
|
||||
manually created Kubernetes Secrets, not Vault-sourced. `robotuser`
|
||||
returned 403 on userpass/ldap/oidc and no token was available. The
|
||||
manifests reference the Secrets by name, so the swap is config-only.
|
||||
- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn
|
||||
attach latency and per-node engine availability made a PVC unreliable for
|
||||
a fast repeatable demo.
|
||||
- **JUnit evidence**: this Jenkins has no `junit` publisher and no
|
||||
`pipeline-stage-view` plugin, so `testReport` and `wfapi` return 404 for
|
||||
every job. `failed_tests` and `first_failed_stage` are therefore always
|
||||
empty. Console text is the only structured evidence, which drove the
|
||||
failure-region extractor. Adding those two plugins (the current `junit`
|
||||
release needs core 2.533; Jenkins here is 2.528.3, so a back-pin is
|
||||
required) is the largest remaining quality improvement and needs a
|
||||
Jenkins restart — deliberately not done the night before a demo.
|
||||
- **Multibranch jobs are not detected**: folder-style jobs expose a null
|
||||
`lastBuild` at the job root.
|
||||
- **OpenSearch namespace** is still pinned to the demo namespace plus
|
||||
`jenkins`; real services get their Jenkins agent logs but not their own
|
||||
runtime logs.
|
||||
|
||||
## Kill switches and rollback
|
||||
|
||||
```text
|
||||
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false # diagnose only
|
||||
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false # fully off
|
||||
ARIADNE_HERMES_ISSUES_ENABLED=false # stop filing issues
|
||||
ARIADNE_HERMES_CODE_ENABLED=false # stop proposing patches
|
||||
ARIADNE_HERMES_ALLOWED_ACTIONS="" # accept no action at all
|
||||
```
|
||||
|
||||
All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`.
|
||||
The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in
|
||||
`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in
|
||||
explicitly mapped repositories, and push `hermes-repair/*` branches. One
|
||||
action per incident, ever.
|
||||
|
||||
## Verified rollback and safety properties
|
||||
|
||||
- Emptying the action allowlist was exercised live and produced diagnosis
|
||||
without mutation.
|
||||
- Restoring it was exercised live and the loop resumed.
|
||||
- Hermes holds no Git credentials, no Kubernetes write access, and cannot
|
||||
read Secrets; its Flux-managed approval deny-list still blocks `kubectl`
|
||||
mutations, `flux` mutations and secret reads.
|
||||
|
||||
@ -983,3 +983,109 @@ At H+24, append a short result section containing:
|
||||
- Kill-switch and rollback verification.
|
||||
|
||||
The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis.
|
||||
|
||||
---
|
||||
|
||||
# Delivery record
|
||||
|
||||
- Execution window opened 2026-08-05 ~17:20 UTC.
|
||||
- Delivered tier: **A, plus the optional autonomous-code stretch.**
|
||||
- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`,
|
||||
skill `$triage-titan-test-failures`) in every recorded case. The Ollama
|
||||
fallback described in the plan was never activated.
|
||||
|
||||
## What runs today
|
||||
|
||||
| Capability | State |
|
||||
|---|---|
|
||||
| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live |
|
||||
| Jenkins + bounded OpenSearch evidence | live |
|
||||
| Hermes Agent diagnosis, schema-validated | live |
|
||||
| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live |
|
||||
| Bounded source-patch proposal → branch → pull request | live (demo repo) |
|
||||
| Human-required metric, vmalert rule, Alertmanager | live |
|
||||
| Gitea issue filing for human-required incidents | built, enabling |
|
||||
| Candidate-file selection for real service repos | built, enabling |
|
||||
|
||||
## Measured results
|
||||
|
||||
Closed autonomous loop, twice:
|
||||
|
||||
```text
|
||||
armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14
|
||||
-> rebuild green 21:01:46 -> resolved 21:02:10 (4m00s)
|
||||
second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08
|
||||
```
|
||||
|
||||
Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s).
|
||||
Hermes diagnosis latency across all runs: 15-21s.
|
||||
|
||||
Real homegrown service diagnoses (no action allowlisted, all escalated):
|
||||
|
||||
| Incident | Classification | Confidence |
|
||||
|---|---|---|
|
||||
| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 |
|
||||
| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 |
|
||||
| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 |
|
||||
| `soteria/276` | `unknown` | 0.58 (below threshold) |
|
||||
|
||||
## Safety behaviour observed in production, not only in tests
|
||||
|
||||
- Hermes classified a pipeline-infrastructure failure as the demo fixture
|
||||
signature at 0.91 confidence; Ariadne's deterministic evidence gate
|
||||
refused the action (`evidence_signature_missing`). The model was wrong and
|
||||
the system still did the right thing.
|
||||
- With the action allowlist emptied, a genuine fixture failure produced
|
||||
`human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert
|
||||
and active in Alertmanager; after an operator fixed the fixture and the
|
||||
job went green, the alert self-cleared through supersession.
|
||||
- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with
|
||||
no matching action, so two independent gates blocked it.
|
||||
|
||||
## Deviations from the plan
|
||||
|
||||
- **Vault**: the shared Hermes API key and the Gitea automation token are
|
||||
manually created Kubernetes Secrets, not Vault-sourced. `robotuser`
|
||||
returned 403 on userpass/ldap/oidc and no token was available. The
|
||||
manifests reference the Secrets by name, so the swap is config-only.
|
||||
- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn
|
||||
attach latency and per-node engine availability made a PVC unreliable for
|
||||
a fast repeatable demo.
|
||||
- **JUnit evidence**: this Jenkins has no `junit` publisher and no
|
||||
`pipeline-stage-view` plugin, so `testReport` and `wfapi` return 404 for
|
||||
every job. `failed_tests` and `first_failed_stage` are therefore always
|
||||
empty. Console text is the only structured evidence, which drove the
|
||||
failure-region extractor. Adding those two plugins (the current `junit`
|
||||
release needs core 2.533; Jenkins here is 2.528.3, so a back-pin is
|
||||
required) is the largest remaining quality improvement and needs a
|
||||
Jenkins restart — deliberately not done the night before a demo.
|
||||
- **Multibranch jobs are not detected**: folder-style jobs expose a null
|
||||
`lastBuild` at the job root.
|
||||
- **OpenSearch namespace** is still pinned to the demo namespace plus
|
||||
`jenkins`; real services get their Jenkins agent logs but not their own
|
||||
runtime logs.
|
||||
|
||||
## Kill switches and rollback
|
||||
|
||||
```text
|
||||
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false # diagnose only
|
||||
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false # fully off
|
||||
ARIADNE_HERMES_ISSUES_ENABLED=false # stop filing issues
|
||||
ARIADNE_HERMES_CODE_ENABLED=false # stop proposing patches
|
||||
ARIADNE_HERMES_ALLOWED_ACTIONS="" # accept no action at all
|
||||
```
|
||||
|
||||
All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`.
|
||||
The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in
|
||||
`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in
|
||||
explicitly mapped repositories, and push `hermes-repair/*` branches. One
|
||||
action per incident, ever.
|
||||
|
||||
## Verified rollback and safety properties
|
||||
|
||||
- Emptying the action allowlist was exercised live and produced diagnosis
|
||||
without mutation.
|
||||
- Restoring it was exercised live and the loop resumed.
|
||||
- Hermes holds no Git credentials, no Kubernetes write access, and cannot
|
||||
read Secrets; its Flux-managed approval deny-list still blocks `kubectl`
|
||||
mutations, `flux` mutations and secret reads.
|
||||
|
||||
@ -502,6 +502,19 @@ spec:
|
||||
value: "20"
|
||||
- name: ARIADNE_HERMES_GITEA_BASE_URL
|
||||
value: https://scm.bstein.dev
|
||||
# File a Gitea issue in the failing service's own repository when
|
||||
# triage concludes a human is needed. Opt-in per job: only jobs
|
||||
# mapped here can ever receive an issue. Deduplicated by
|
||||
# job+classification so a repeatedly failing job produces one
|
||||
# issue per kind of failure, not one per build.
|
||||
- name: ARIADNE_HERMES_ISSUES_ENABLED
|
||||
value: "true"
|
||||
- name: ARIADNE_HERMES_ISSUE_REPOS
|
||||
value: metis=bstein/metis,lesavka=bstein/lesavka,soteria=bstein/soteria,bstein-dev-home=bstein/bstein-dev-home,ariadne=bstein/ariadne
|
||||
- name: ARIADNE_HERMES_ISSUE_DEDUPE_SCOPE
|
||||
value: classification
|
||||
- name: ARIADNE_HERMES_ISSUE_MAX_PER_TICK
|
||||
value: "2"
|
||||
- name: ARIADNE_HERMES_GITEA_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user