diff --git a/services/hermes/configmap.yaml b/services/hermes/configmap.yaml index 8344000e5..8c397638c 100644 --- a/services/hermes/configmap.yaml +++ b/services/hermes/configmap.yaml @@ -52,7 +52,7 @@ data: home_mode: auto approvals: - mode: manual + mode: smart deny: - "*kubectl apply*" - "*kubectl delete*" @@ -231,9 +231,77 @@ data: - Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and CPU alerts use actionable semantics and guardrails. + The evidence and verification for these changes are in + `triage-proof/ATLAS-TRIAGE-PROOFS.md`. + ## Good next prompts - `Use $triage-atlas-service-health. What is broken right now?` - `Use $triage-titan-test-failures. Triage the worst current suite.` - `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.` - `Save this investigation as a reusable skill after I approve the pattern.` + ATLAS-TRIAGE-PROOFS.md: | + # Atlas triage proofs + + This file is the short evidence trail for what Hermes can do today. The + operator skills remain read-only: fixes are reviewed and delivered through + Git and Flux. + + ## Proof 1: Cassandra secret-sync recovery + + - Finding: `cassandra-vault-sync` was stuck because the expected + `VELES_BYOK_ENCRYPTION_KEY` migration input was absent from Cassandra's + Vault path. + - Action: the existing Flux-tracked, suspended one-shot migration Job was + enabled, completed successfully, and immediately returned to suspended. + - Verification: Cassandra frontend, backend, PostgreSQL, and Vault sync all + became Ready. No Kubernetes Secret value was read or committed. + - Why this matters: Hermes can correlate workload state, logs, Vault policy + manifests, and migration ownership without confusing Veles residue with + a current Veles outage. + + ## Proof 2: Grafana alert-noise correction + + - Removed retired Veles pods and completed Jobs from high-level incident + counts while preserving them on drill-down dashboards. + - Added 15-minute persistence to Pending, CrashLoopBackOff, and image-pull + summaries. + - Replaced invalid `increase()` use on gauges, bounded CPU percentages, + required meaningful Postmark sample sizes, and scoped Soteria/Ariadne + alerts to configured current work. + - Live verification on 2026-08-03: `Problem Pods=0`, + `CrashLoop/ImagePull=0`, and `Workers Ready=18/18`. + + ## Proof 3: all-suite testing triage + + - Ariadne build 384 passed 676 tests and the exact coverage contract: + 82 source files at or above 95%. + - Deployed image: `registry.bstein.dev/bstein/ariadne:0.1.0-384`. + - The deterministic bundle covers Ananke, Ariadne, Atlasbot, bstein_home, + data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac. + - Zero-valued quality metrics are healthy, running Jenkins builds are + in-progress, old failed Jobs are historical, and Veles objects are + migration residue. They no longer inflate the active failure set. + - Failed builds still include direct Jenkins console and retained artifact + links so Hermes can identify the first failed gate and smallest repo fix. + + ## Proof 4: Cassandra generator failure classification + + - Finding: the Cassandra serving path is healthy, but generation Job + `cassandra-generator-3ce198971b` failed. + - Evidence: the primary OpenAI request returned HTTP 429 because its credit + balance was exhausted. Codex CLI fallback ran, but the strict promotion + gate still rejected unresolved high-impact hint claims and blocked the + semantic scenarios. + - Impact: one generation capability/request failed; this is not a Cassandra + registry outage and not a Veles outage. + - Next action: restore provider credits or continue through Codex fallback, + then resolve the reported hint claims before retrying. This needs operator + approval because it can spend money or rerun an expensive generation. + + ## Use Hermes next + + - `What is broken in the cluster right now?` + - `Triage the worst current CI suite and link the evidence.` + - `Audit Grafana alerts from the last 24 hours and separate incidents from noise.` + - `Explain the Cassandra generator failure and give me the smallest safe next step.` diff --git a/services/hermes/deployment.yaml b/services/hermes/deployment.yaml index 9634570c0..8fa2ec2b1 100644 --- a/services/hermes/deployment.yaml +++ b/services/hermes/deployment.yaml @@ -24,7 +24,7 @@ spec: ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback ai.bstein.dev/role: testing-triage ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred) - ai.bstein.dev/config-rev: "20260803-service-triage-and-alert-skills" + ai.bstein.dev/config-rev: "20260803-triage-proofs-and-smart-approvals" spec: serviceAccountName: hermes-triage automountServiceAccountToken: true @@ -88,6 +88,7 @@ spec: cp /config/AGENTS.md /opt/data/workspace/AGENTS.md cp /config/START-HERE.md /opt/data/workspace/START-HERE.md cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md + cp /config/ATLAS-TRIAGE-PROOFS.md /opt/data/workspace/triage-proof/ATLAS-TRIAGE-PROOFS.md touch /opt/data/.env if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"