docs(hermes): publish Atlas triage proofs

This commit is contained in:
jenkins 2026-08-03 04:30:16 -03:00
parent 951139f675
commit fdb74fa9f9
2 changed files with 71 additions and 2 deletions

View File

@ -52,7 +52,7 @@ data:
home_mode: auto
approvals:
mode: manual
mode: smart
deny:
- "*kubectl apply*"
- "*kubectl delete*"
@ -231,9 +231,77 @@ data:
- Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and
CPU alerts use actionable semantics and guardrails.
The evidence and verification for these changes are in
`triage-proof/ATLAS-TRIAGE-PROOFS.md`.
## Good next prompts
- `Use $triage-atlas-service-health. What is broken right now?`
- `Use $triage-titan-test-failures. Triage the worst current suite.`
- `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.`
- `Save this investigation as a reusable skill after I approve the pattern.`
ATLAS-TRIAGE-PROOFS.md: |
# Atlas triage proofs
This file is the short evidence trail for what Hermes can do today. The
operator skills remain read-only: fixes are reviewed and delivered through
Git and Flux.
## Proof 1: Cassandra secret-sync recovery
- Finding: `cassandra-vault-sync` was stuck because the expected
`VELES_BYOK_ENCRYPTION_KEY` migration input was absent from Cassandra's
Vault path.
- Action: the existing Flux-tracked, suspended one-shot migration Job was
enabled, completed successfully, and immediately returned to suspended.
- Verification: Cassandra frontend, backend, PostgreSQL, and Vault sync all
became Ready. No Kubernetes Secret value was read or committed.
- Why this matters: Hermes can correlate workload state, logs, Vault policy
manifests, and migration ownership without confusing Veles residue with
a current Veles outage.
## Proof 2: Grafana alert-noise correction
- Removed retired Veles pods and completed Jobs from high-level incident
counts while preserving them on drill-down dashboards.
- Added 15-minute persistence to Pending, CrashLoopBackOff, and image-pull
summaries.
- Replaced invalid `increase()` use on gauges, bounded CPU percentages,
required meaningful Postmark sample sizes, and scoped Soteria/Ariadne
alerts to configured current work.
- Live verification on 2026-08-03: `Problem Pods=0`,
`CrashLoop/ImagePull=0`, and `Workers Ready=18/18`.
## Proof 3: all-suite testing triage
- Ariadne build 384 passed 676 tests and the exact coverage contract:
82 source files at or above 95%.
- Deployed image: `registry.bstein.dev/bstein/ariadne:0.1.0-384`.
- The deterministic bundle covers Ananke, Ariadne, Atlasbot, bstein_home,
data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac.
- Zero-valued quality metrics are healthy, running Jenkins builds are
in-progress, old failed Jobs are historical, and Veles objects are
migration residue. They no longer inflate the active failure set.
- Failed builds still include direct Jenkins console and retained artifact
links so Hermes can identify the first failed gate and smallest repo fix.
## Proof 4: Cassandra generator failure classification
- Finding: the Cassandra serving path is healthy, but generation Job
`cassandra-generator-3ce198971b` failed.
- Evidence: the primary OpenAI request returned HTTP 429 because its credit
balance was exhausted. Codex CLI fallback ran, but the strict promotion
gate still rejected unresolved high-impact hint claims and blocked the
semantic scenarios.
- Impact: one generation capability/request failed; this is not a Cassandra
registry outage and not a Veles outage.
- Next action: restore provider credits or continue through Codex fallback,
then resolve the reported hint claims before retrying. This needs operator
approval because it can spend money or rerun an expensive generation.
## Use Hermes next
- `What is broken in the cluster right now?`
- `Triage the worst current CI suite and link the evidence.`
- `Audit Grafana alerts from the last 24 hours and separate incidents from noise.`
- `Explain the Cassandra generator failure and give me the smallest safe next step.`

View File

@ -24,7 +24,7 @@ spec:
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
ai.bstein.dev/role: testing-triage
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
ai.bstein.dev/config-rev: "20260803-service-triage-and-alert-skills"
ai.bstein.dev/config-rev: "20260803-triage-proofs-and-smart-approvals"
spec:
serviceAccountName: hermes-triage
automountServiceAccountToken: true
@ -88,6 +88,7 @@ spec:
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md
cp /config/ATLAS-TRIAGE-PROOFS.md /opt/data/workspace/triage-proof/ATLAS-TRIAGE-PROOFS.md
touch /opt/data/.env
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"