docs(hermes): publish Atlas triage proofs
This commit is contained in:
parent
951139f675
commit
fdb74fa9f9
@ -52,7 +52,7 @@ data:
|
|||||||
home_mode: auto
|
home_mode: auto
|
||||||
|
|
||||||
approvals:
|
approvals:
|
||||||
mode: manual
|
mode: smart
|
||||||
deny:
|
deny:
|
||||||
- "*kubectl apply*"
|
- "*kubectl apply*"
|
||||||
- "*kubectl delete*"
|
- "*kubectl delete*"
|
||||||
@ -231,9 +231,77 @@ data:
|
|||||||
- Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and
|
- Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and
|
||||||
CPU alerts use actionable semantics and guardrails.
|
CPU alerts use actionable semantics and guardrails.
|
||||||
|
|
||||||
|
The evidence and verification for these changes are in
|
||||||
|
`triage-proof/ATLAS-TRIAGE-PROOFS.md`.
|
||||||
|
|
||||||
## Good next prompts
|
## Good next prompts
|
||||||
|
|
||||||
- `Use $triage-atlas-service-health. What is broken right now?`
|
- `Use $triage-atlas-service-health. What is broken right now?`
|
||||||
- `Use $triage-titan-test-failures. Triage the worst current suite.`
|
- `Use $triage-titan-test-failures. Triage the worst current suite.`
|
||||||
- `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.`
|
- `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.`
|
||||||
- `Save this investigation as a reusable skill after I approve the pattern.`
|
- `Save this investigation as a reusable skill after I approve the pattern.`
|
||||||
|
ATLAS-TRIAGE-PROOFS.md: |
|
||||||
|
# Atlas triage proofs
|
||||||
|
|
||||||
|
This file is the short evidence trail for what Hermes can do today. The
|
||||||
|
operator skills remain read-only: fixes are reviewed and delivered through
|
||||||
|
Git and Flux.
|
||||||
|
|
||||||
|
## Proof 1: Cassandra secret-sync recovery
|
||||||
|
|
||||||
|
- Finding: `cassandra-vault-sync` was stuck because the expected
|
||||||
|
`VELES_BYOK_ENCRYPTION_KEY` migration input was absent from Cassandra's
|
||||||
|
Vault path.
|
||||||
|
- Action: the existing Flux-tracked, suspended one-shot migration Job was
|
||||||
|
enabled, completed successfully, and immediately returned to suspended.
|
||||||
|
- Verification: Cassandra frontend, backend, PostgreSQL, and Vault sync all
|
||||||
|
became Ready. No Kubernetes Secret value was read or committed.
|
||||||
|
- Why this matters: Hermes can correlate workload state, logs, Vault policy
|
||||||
|
manifests, and migration ownership without confusing Veles residue with
|
||||||
|
a current Veles outage.
|
||||||
|
|
||||||
|
## Proof 2: Grafana alert-noise correction
|
||||||
|
|
||||||
|
- Removed retired Veles pods and completed Jobs from high-level incident
|
||||||
|
counts while preserving them on drill-down dashboards.
|
||||||
|
- Added 15-minute persistence to Pending, CrashLoopBackOff, and image-pull
|
||||||
|
summaries.
|
||||||
|
- Replaced invalid `increase()` use on gauges, bounded CPU percentages,
|
||||||
|
required meaningful Postmark sample sizes, and scoped Soteria/Ariadne
|
||||||
|
alerts to configured current work.
|
||||||
|
- Live verification on 2026-08-03: `Problem Pods=0`,
|
||||||
|
`CrashLoop/ImagePull=0`, and `Workers Ready=18/18`.
|
||||||
|
|
||||||
|
## Proof 3: all-suite testing triage
|
||||||
|
|
||||||
|
- Ariadne build 384 passed 676 tests and the exact coverage contract:
|
||||||
|
82 source files at or above 95%.
|
||||||
|
- Deployed image: `registry.bstein.dev/bstein/ariadne:0.1.0-384`.
|
||||||
|
- The deterministic bundle covers Ananke, Ariadne, Atlasbot, bstein_home,
|
||||||
|
data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac.
|
||||||
|
- Zero-valued quality metrics are healthy, running Jenkins builds are
|
||||||
|
in-progress, old failed Jobs are historical, and Veles objects are
|
||||||
|
migration residue. They no longer inflate the active failure set.
|
||||||
|
- Failed builds still include direct Jenkins console and retained artifact
|
||||||
|
links so Hermes can identify the first failed gate and smallest repo fix.
|
||||||
|
|
||||||
|
## Proof 4: Cassandra generator failure classification
|
||||||
|
|
||||||
|
- Finding: the Cassandra serving path is healthy, but generation Job
|
||||||
|
`cassandra-generator-3ce198971b` failed.
|
||||||
|
- Evidence: the primary OpenAI request returned HTTP 429 because its credit
|
||||||
|
balance was exhausted. Codex CLI fallback ran, but the strict promotion
|
||||||
|
gate still rejected unresolved high-impact hint claims and blocked the
|
||||||
|
semantic scenarios.
|
||||||
|
- Impact: one generation capability/request failed; this is not a Cassandra
|
||||||
|
registry outage and not a Veles outage.
|
||||||
|
- Next action: restore provider credits or continue through Codex fallback,
|
||||||
|
then resolve the reported hint claims before retrying. This needs operator
|
||||||
|
approval because it can spend money or rerun an expensive generation.
|
||||||
|
|
||||||
|
## Use Hermes next
|
||||||
|
|
||||||
|
- `What is broken in the cluster right now?`
|
||||||
|
- `Triage the worst current CI suite and link the evidence.`
|
||||||
|
- `Audit Grafana alerts from the last 24 hours and separate incidents from noise.`
|
||||||
|
- `Explain the Cassandra generator failure and give me the smallest safe next step.`
|
||||||
|
|||||||
@ -24,7 +24,7 @@ spec:
|
|||||||
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
|
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
|
||||||
ai.bstein.dev/role: testing-triage
|
ai.bstein.dev/role: testing-triage
|
||||||
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
|
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
|
||||||
ai.bstein.dev/config-rev: "20260803-service-triage-and-alert-skills"
|
ai.bstein.dev/config-rev: "20260803-triage-proofs-and-smart-approvals"
|
||||||
spec:
|
spec:
|
||||||
serviceAccountName: hermes-triage
|
serviceAccountName: hermes-triage
|
||||||
automountServiceAccountToken: true
|
automountServiceAccountToken: true
|
||||||
@ -88,6 +88,7 @@ spec:
|
|||||||
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
|
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
|
||||||
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
|
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
|
||||||
cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md
|
cp /config/HERMES-CAPABILITIES.md /opt/data/workspace/HERMES-CAPABILITIES.md
|
||||||
|
cp /config/ATLAS-TRIAGE-PROOFS.md /opt/data/workspace/triage-proof/ATLAS-TRIAGE-PROOFS.md
|
||||||
touch /opt/data/.env
|
touch /opt/data/.env
|
||||||
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
|
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
|
||||||
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user