docs(hermes): publish Atlas triage proofs
This commit is contained in:
parent
951139f675
commit
fdb74fa9f9
@ -52,7 +52,7 @@ data:
|
||||
home_mode: auto
|
||||
|
||||
approvals:
|
||||
mode: manual
|
||||
mode: smart
|
||||
deny:
|
||||
- "*kubectl apply*"
|
||||
- "*kubectl delete*"
|
||||
@ -231,9 +231,77 @@ data:
|
||||
- Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and
|
||||
CPU alerts use actionable semantics and guardrails.
|
||||
|
||||
The evidence and verification for these changes are in
|
||||
`triage-proof/ATLAS-TRIAGE-PROOFS.md`.
|
||||
|
||||
## Good next prompts
|
||||
|
||||
- `Use $triage-atlas-service-health. What is broken right now?`
|
||||
- `Use $triage-titan-test-failures. Triage the worst current suite.`
|
||||
- `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.`
|
||||
- `Save this investigation as a reusable skill after I approve the pattern.`
|
||||
ATLAS-TRIAGE-PROOFS.md: |
|
||||
# Atlas triage proofs
|
||||
|
||||
This file is the short evidence trail for what Hermes can do today. The
|
||||
operator skills remain read-only: fixes are reviewed and delivered through
|
||||
Git and Flux.
|
||||
|
||||
## Proof 1: Cassandra secret-sync recovery
|
||||
|
||||
- Finding: `cassandra-vault-sync` was stuck because the expected
|
||||
`VELES_BYOK_ENCRYPTION_KEY` migration input was absent from Cassandra's
|
||||
Vault path.
|
||||
- Action: the existing Flux-tracked, suspended one-shot migration Job was
|
||||
enabled, completed successfully, and immediately returned to suspended.
|
||||
- Verification: Cassandra frontend, backend, PostgreSQL, and Vault sync all
|
||||
became Ready. No Kubernetes Secret value was read or committed.
|
||||
- Why this matters: Hermes can correlate workload state, logs, Vault policy
|
||||
manifests, and migration ownership without confusing Veles residue with
|
||||
a current Veles outage.
|
||||
|
||||
## Proof 2: Grafana alert-noise correction
|
||||
|
||||
- Removed retired Veles pods and completed Jobs from high-level incident
|
||||
counts while preserving them on drill-down dashboards.
|
||||
- Added 15-minute persistence to Pending, CrashLoopBackOff, and image-pull
|
||||
summaries.
|
||||
- Replaced invalid `increase()` use on gauges, bounded CPU percentages,
|
||||
required meaningful Postmark sample sizes, and scoped Soteria/Ariadne
|
||||
alerts to configured current work.
|
||||
- Live verification on 2026-08-03: `Problem Pods=0`,
|
||||
`CrashLoop/ImagePull=0`, and `Workers Ready=18/18`.
|
||||
|
||||
## Proof 3: all-suite testing triage
|
||||
|
||||
- Ariadne build 384 passed 676 tests and the exact coverage contract:
|
||||
82 source files at or above 95%.
|
||||
- Deployed image: `registry.bstein.dev/bstein/ariadne:0.1.0-384`.
|
||||
- The deterministic bundle covers Ananke, Ariadne, Atlasbot, bstein_home,
|
||||
data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac.
|
||||
- Zero-valued quality metrics are healthy, running Jenkins builds are
|
||||
in-progress, old failed Jobs are historical, and Veles objects are
|
||||
migration residue. They no longer inflate the active failure set.
|
||||
- Failed builds still include direct Jenkins console and retained artifact
|
||||
links so Hermes can identify the first failed gate and smallest repo fix.
|
||||
|
||||
## Proof 4: Cassandra generator failure classification
|
||||
|
||||
- Finding: the Cassandra serving path is healthy, but generation Job
|
||||
`cassandra-generator-3ce198971b` failed.
|
||||
- Evidence: the primary OpenAI request returned HTTP 429 because its credit
|
||||
balance was exhausted. Codex CLI fallback ran, but the strict promotion
|
||||
gate still rejected unresolved high-impact hint claims and blocked the
|
||||
semantic scenarios.
|
||||
- Impact: one generation capability/request failed; this is not a Cassandra
|
||||
registry outage and not a Veles outage.
|
||||
- Next action: restore provider credits or continue through Codex fallback,
|
||||
then resolve the reported hint claims before retrying. This needs operator
|
||||
approval because it can spend money or rerun an expensive generation.
|
||||
|
||||
## Use Hermes next
|
||||
|
||||
- `What is broken in the cluster right now?`
|
||||
- `Triage the worst current CI suite and link the evidence.`
|
||||
- `Audit Grafana alerts from the last 24 hours and separate incidents from noise.`
|
||||
- `Explain the Cassandra generator failure and give me the smallest safe next step.`
|
||||
|
||||
@ -24,7 +24,7 @@ spec:
|
||||
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
|
||||
ai.bstein.dev/role: testing-triage
|
||||
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
|
||||
ai.bstein.dev/config-rev: "20260803-service-triage-and-alert-skills"
|
||||
ai.bstein.dev/config-rev: "20260803-triage-proofs-and-smart-approvals"
|
||||
spec:
|
||||
serviceAccountName: hermes-triage
|
||||
automountServiceAccountToken: true
|
||||
@ -88,6 +88,7 @@ spec:
|
||||
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
|
||||
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
|
||||
cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md
|
||||
cp /config/ATLAS-TRIAGE-PROOFS.md /opt/data/workspace/triage-proof/ATLAS-TRIAGE-PROOFS.md
|
||||
touch /opt/data/.env
|
||||
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
|
||||
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user