docs(hermes): publish Atlas triage proofs

This commit is contained in:
jenkins 2026-08-03 04:30:16 -03:00
parent 951139f675
commit fdb74fa9f9
2 changed files with 71 additions and 2 deletions

View File

@ -52,7 +52,7 @@ data:
home_mode: auto home_mode: auto
approvals: approvals:
mode: manual mode: smart
deny: deny:
- "*kubectl apply*" - "*kubectl apply*"
- "*kubectl delete*" - "*kubectl delete*"
@ -231,9 +231,77 @@ data:
- Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and - Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and
CPU alerts use actionable semantics and guardrails. CPU alerts use actionable semantics and guardrails.
The evidence and verification for these changes are in
`triage-proof/ATLAS-TRIAGE-PROOFS.md`.
## Good next prompts ## Good next prompts
- `Use $triage-atlas-service-health. What is broken right now?` - `Use $triage-atlas-service-health. What is broken right now?`
- `Use $triage-titan-test-failures. Triage the worst current suite.` - `Use $triage-titan-test-failures. Triage the worst current suite.`
- `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.` - `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.`
- `Save this investigation as a reusable skill after I approve the pattern.` - `Save this investigation as a reusable skill after I approve the pattern.`
ATLAS-TRIAGE-PROOFS.md: |
# Atlas triage proofs
This file is the short evidence trail for what Hermes can do today. The
operator skills remain read-only: fixes are reviewed and delivered through
Git and Flux.
## Proof 1: Cassandra secret-sync recovery
- Finding: `cassandra-vault-sync` was stuck because the expected
`VELES_BYOK_ENCRYPTION_KEY` migration input was absent from Cassandra's
Vault path.
- Action: the existing Flux-tracked, suspended one-shot migration Job was
enabled, completed successfully, and immediately returned to suspended.
- Verification: Cassandra frontend, backend, PostgreSQL, and Vault sync all
became Ready. No Kubernetes Secret value was read or committed.
- Why this matters: Hermes can correlate workload state, logs, Vault policy
manifests, and migration ownership without confusing Veles residue with
a current Veles outage.
## Proof 2: Grafana alert-noise correction
- Removed retired Veles pods and completed Jobs from high-level incident
counts while preserving them on drill-down dashboards.
- Added 15-minute persistence to Pending, CrashLoopBackOff, and image-pull
summaries.
- Replaced invalid `increase()` use on gauges, bounded CPU percentages,
required meaningful Postmark sample sizes, and scoped Soteria/Ariadne
alerts to configured current work.
- Live verification on 2026-08-03: `Problem Pods=0`,
`CrashLoop/ImagePull=0`, and `Workers Ready=18/18`.
## Proof 3: all-suite testing triage
- Ariadne build 384 passed 676 tests and the exact coverage contract:
82 source files at or above 95%.
- Deployed image: `registry.bstein.dev/bstein/ariadne:0.1.0-384`.
- The deterministic bundle covers Ananke, Ariadne, Atlasbot, bstein_home,
data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac.
- Zero-valued quality metrics are healthy, running Jenkins builds are
in-progress, old failed Jobs are historical, and Veles objects are
migration residue. They no longer inflate the active failure set.
- Failed builds still include direct Jenkins console and retained artifact
links so Hermes can identify the first failed gate and smallest repo fix.
## Proof 4: Cassandra generator failure classification
- Finding: the Cassandra serving path is healthy, but generation Job
`cassandra-generator-3ce198971b` failed.
- Evidence: the primary OpenAI request returned HTTP 429 because its credit
balance was exhausted. Codex CLI fallback ran, but the strict promotion
gate still rejected unresolved high-impact hint claims and blocked the
semantic scenarios.
- Impact: one generation capability/request failed; this is not a Cassandra
registry outage and not a Veles outage.
- Next action: restore provider credits or continue through Codex fallback,
then resolve the reported hint claims before retrying. This needs operator
approval because it can spend money or rerun an expensive generation.
## Use Hermes next
- `What is broken in the cluster right now?`
- `Triage the worst current CI suite and link the evidence.`
- `Audit Grafana alerts from the last 24 hours and separate incidents from noise.`
- `Explain the Cassandra generator failure and give me the smallest safe next step.`

View File

@ -24,7 +24,7 @@ spec:
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
ai.bstein.dev/role: testing-triage ai.bstein.dev/role: testing-triage
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred) ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
ai.bstein.dev/config-rev: "20260803-service-triage-and-alert-skills" ai.bstein.dev/config-rev: "20260803-triage-proofs-and-smart-approvals"
spec: spec:
serviceAccountName: hermes-triage serviceAccountName: hermes-triage
automountServiceAccountToken: true automountServiceAccountToken: true
@ -88,6 +88,7 @@ spec:
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md
cp /config/ATLAS-TRIAGE-PROOFS.md /opt/data/workspace/triage-proof/ATLAS-TRIAGE-PROOFS.md
touch /opt/data/.env touch /opt/data/.env
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')" api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"