# services/hermes/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: hermes-config namespace: hermes labels: app: hermes data: config.yaml: | model: provider: anthropic default: claude-opus-5 model: claude-opus-5 fallback_providers: - provider: openai-codex model: gpt-5.6-terra - provider: custom model: gpt-oss:20b base_url: http://hermes-model-gate.hermes.svc.cluster.local:11434/v1 api_key: ollama agent: api_max_retries: 1 platform_toolsets: cli: - clarify - file - session_search - skills - terminal - todo - web api_server: - clarify - file - session_search - skills - terminal - todo - web skills: creation_nudge_interval: 15 external_dirs: - /opt/data/workspace/skills terminal: backend: local cwd: /opt/data/workspace timeout: 180 home_mode: auto approvals: mode: smart deny: - "*kubectl apply*" - "*kubectl delete*" - "*kubectl patch*" - "*kubectl scale*" - "*kubectl cordon*" - "*kubectl uncordon*" - "*kubectl drain*" - "*kubectl rollout restart*" - "*flux suspend*" - "*flux resume*" - "*flux reconcile*" - "*vault kv get*" - "*kubectl get secret*" - "*kubectl describe secret*" dashboard: public_url: https://agent.bstein.dev oauth: provider: self-hosted self_hosted: issuer: https://sso.bstein.dev/realms/atlas client_id: hermes-dashboard scopes: openid profile email groups display: compact: true tool_progress: all interim_assistant_messages: true long_running_notifications: true tool_loop_guardrails: warnings_enabled: true hard_stop_enabled: true warn_after: exact_failure: 2 same_tool_failure: 3 idempotent_no_progress: 2 hard_stop_after: exact_failure: 5 same_tool_failure: 8 idempotent_no_progress: 5 updates: pre_update_backup: quick backup_keep: 5 non_interactive_local_changes: stash SOUL.md: | You are Hermes running inside the Titan Kubernetes cluster as a supervised testing and operations triage assistant. Your strongest job is to follow the same evidence path Brad already uses: Ariadne diagnosis first, then Jenkins logs and artifacts, Pushgateway quality metrics, Flux state, Grafana dashboard context, and Kubernetes read-only state. Turn repeated successful triage paths into reusable skills or memory when the pattern is stable. When Brad asks about CI, tests, suites, builds, or quality gates, use triage-titan-test-failures. When he asks what is broken, why a service is unhealthy, or why Grafana is red, use triage-atlas-service-health. When he asks whether an alert is noisy or too aggressive, use tune-atlas-alerts. Do not make him choose a tool, page, or evidence source first. Stay Flux-first. Do not mutate the cluster directly. Explain evidence, recommend the smallest repo-side change, and name the exact verification commands a human should run after Flux reconciles. AGENTS.md: | # Titan Testing Triage You are Hermes running inside the Titan Kubernetes cluster as a read-only testing and operations triage assistant. Route test/build/quality-gate requests to `triage-titan-test-failures`, service/cluster-health requests to `triage-atlas-service-health`, and noisy Grafana alert requests to `tune-atlas-alerts`. Ask only when approval is required for a state-changing evidence refresh. Ariadne owns deterministic evidence collection and local diagnosis. Start every testing triage by reading: - `curl -sS "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/latest"` - if that is missing or stale, ask a human to run: `curl -sS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/run"` - if the diagnosis is unavailable, fall back to: `curl -sS "$ARIADNE_BASE_URL/api/internal/testing/triage/latest"` Treat the Ariadne diagnosis and evidence bundle as the source of truth. Your job is to explain the evidence and propose small Flux/IaC changes, not to rediscover everything from raw shell commands. Useful read-only commands: - `curl -sS "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/latest"` - `curl -sS "$ARIADNE_BASE_URL/api/internal/testing/triage/latest"` - `kubectl get nodes -o wide` - `kubectl get pods -A -o wide` - `kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded -o wide` - `kubectl get events -A --sort-by=.lastTimestamp` - `kubectl -n describe pod ` - `kubectl -n logs --all-containers --tail=200` - `kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io` - `curl -sS "$VICTORIA_METRICS_URL/api/v1/query?query=up"` Avoid grep/awk pipelines for absence checks; a grep exit code of 1 often means "no matches", not a permission problem. Prefer Kubernetes field selectors and quote the actual stderr when a command fails. Do not run mutating commands such as `kubectl apply`, `delete`, `scale`, `patch`, `cordon`, `uncordon`, `drain`, or `rollout restart`. Do not read Kubernetes Secret values. Draft repo changes or operator steps instead. START-HERE.md: | # Hermes on Atlas: start here Use Chat for a new investigation or Sessions to resume a previous one. The shortest useful prompts are: - `Triage the latest test failure.` - `What is broken in the cluster right now?` - `Why is Grafana red, and which alerts are noise?` - `Triage build and save the report.` Hermes will use Ariadne evidence, Jenkins log excerpts, quality metrics, recent Git changes, Flux state, Grafana context, and Kubernetes read-only state. It will separate facts, inference, and unknowns and will ask before triggering a fresh evidence collection. Hermes now has three focused operator skills: 1. `triage-titan-test-failures` for all ten custom CI suites. 2. `triage-atlas-service-health` for any live namespace or service. 3. `tune-atlas-alerts` for evidence-backed alert and dashboard corrections. Validated CI examples are available under Sessions, including: 1. `Proof 6 - Soteria 272 Sonar new coverage triage` 2. `Proof 7 - Ananke 242 local coverage triage` Their saved reports are in Files under `triage-proof/`. Read `HERMES-OPERATOR-RUNBOOK.md` for the system mental model and the operator walkthrough. Hermes may inspect workload and delivery metadata, logs, metrics, and events. Its Kubernetes identity cannot read Secret values or mutate cluster resources. Apply fixes through the titan-iac GitOps workflow after review. HERMES-CAPABILITIES.md: | # What Hermes can do on Atlas ## Working now - Triage all custom CI suites: Ananke, Ariadne, Atlasbot, bstein_home, data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac. - Read Ariadne's deterministic evidence bundle, Jenkins failure logs and retained quality artifacts, VictoriaMetrics, Flux state, Kubernetes events/workloads/logs, and recent Gitea commits. - Investigate any deployed service, group replica symptoms into incidents, and separate active failures from startup grace, historical Jobs, lifetime restart totals, and Veles migration residue. - Audit Grafana alerts for bad counter/gauge math, low sample sizes, stale schedules, duplicate series, missing persistence, and retired scope. - Save concise triage reports and recommend exact repo-side Flux changes. ## Safety boundary Hermes cannot read Kubernetes Secret values and cannot mutate Kubernetes or Flux. It can quote secret-related errors already exposed in events or logs, inspect the surrounding manifests, and tell Brad what approved change is required. This keeps cluster operation supervised. ## Demonstrated corrections - Zero-valued quality series no longer become failed suites. - Running Jenkins builds remain in-progress instead of failed. - Old failed Jobs, lifetime restart totals, and temporary Flux unknowns no longer inflate the active incident count. - Cassandra is authoritative; Veles failures remain visible as migration residue but do not make the high-level Atlas health view red. - Worker readiness uses the actual 18 Kubernetes workers. - Root-disk growth, Soteria backup, Ariadne schedule, Postmark bounce, and CPU alerts use actionable semantics and guardrails. The evidence and verification for these changes are in `triage-proof/ATLAS-TRIAGE-PROOFS.md`. ## Good next prompts - `Use $triage-atlas-service-health. What is broken right now?` - `Use $triage-titan-test-failures. Triage the worst current suite.` - `Use $tune-atlas-alerts. Audit alerts fired in the last 24 hours.` - `Save this investigation as a reusable skill after I approve the pattern.` ATLAS-TRIAGE-PROOFS.md: | # Atlas triage proofs This file is the short evidence trail for what Hermes can do today. The operator skills remain read-only: fixes are reviewed and delivered through Git and Flux. ## Proof 1: Cassandra secret-sync recovery - Finding: `cassandra-vault-sync` was stuck because the expected `VELES_BYOK_ENCRYPTION_KEY` migration input was absent from Cassandra's Vault path. - Action: the existing Flux-tracked, suspended one-shot migration Job was enabled, completed successfully, and immediately returned to suspended. - Verification: Cassandra frontend, backend, PostgreSQL, and Vault sync all became Ready. No Kubernetes Secret value was read or committed. - Why this matters: Hermes can correlate workload state, logs, Vault policy manifests, and migration ownership without confusing Veles residue with a current Veles outage. ## Proof 2: Grafana alert-noise correction - Removed retired Veles pods and completed Jobs from high-level incident counts while preserving them on drill-down dashboards. - Added 15-minute persistence to Pending, CrashLoopBackOff, and image-pull summaries. - Replaced invalid `increase()` use on gauges, bounded CPU percentages, required meaningful Postmark sample sizes, and scoped Soteria/Ariadne alerts to configured current work. - Live verification on 2026-08-03: `Problem Pods=0`, `CrashLoop/ImagePull=0`, and `Workers Ready=18/18`. ## Proof 3: all-suite testing triage - Ariadne build 384 passed 676 tests and the exact coverage contract: 82 source files at or above 95%. - Deployed image: `registry.bstein.dev/bstein/ariadne:0.1.0-384`. - The deterministic bundle covers Ananke, Ariadne, Atlasbot, bstein_home, data_prepper, Lesavka, Metis, Pegasus, Soteria, and titan_iac. - Zero-valued quality metrics are healthy, running Jenkins builds are in-progress, old failed Jobs are historical, and Veles objects are migration residue. They no longer inflate the active failure set. - Failed builds still include direct Jenkins console and retained artifact links so Hermes can identify the first failed gate and smallest repo fix. ## Proof 4: Cassandra generator failure classification - Finding: the Cassandra serving path is healthy, but generation Job `cassandra-generator-3ce198971b` failed. - Evidence: the primary OpenAI request returned HTTP 429 because its credit balance was exhausted. Codex CLI fallback ran, but the strict promotion gate still rejected unresolved high-impact hint claims and blocked the semantic scenarios. - Impact: one generation capability/request failed; this is not a Cassandra registry outage and not a Veles outage. - Next action: restore provider credits or continue through Codex fallback, then resolve the reported hint claims before retrying. This needs operator approval because it can spend money or rerun an expensive generation. ## Proof 5: live Hermes Soteria triage - Hermes read the fresh 2026-08-03 Ariadne bundle without an approval timeout and chose Soteria build 270 as the worst terminal failure. - It proved that all 318 tests passed and local coverage was 96.195%, then isolated the enforced failure to SonarQube reporting new-code coverage as 0.0%. It did not blame the optional supply-chain advisory. - It kept running titan-iac, Data Prepper, and Lesavka builds out of the terminal-failure list and requested approval before any rerun or edit. ## Proof 6: Soteria build 272 Sonar new-coverage triage - Hermes read the fresh 2026-08-04 Ariadne bundle and selected terminal Soteria build 272. - It proved that all 318 tests passed and local coverage was 96.195%, then isolated the enforced failure to SonarQube `new_coverage=0.0` against the threshold of 80. - It corroborated the retained Jenkins evidence with current quality metrics and healthy Soteria/Flux state, and did not blame Kubernetes. - It proposed no change until scanner import and baseline evidence can distinguish configuration error from genuinely uncovered new code. ## Proof 7: Ananke build 242 local-coverage triage - Hermes selected terminal Ananke build 242 and isolated the first failure to the local coverage gate at 61.8%. - It showed that tests, SonarQube, supply chain, LOC, docs naming, gate glue, Flux, and cluster readiness were healthy. - It distinguished the local 61.8% scope from SonarQube's 93.1% new-code scope instead of treating the numbers as interchangeable. - Direct Jenkins artifact reads returned HTTP 403, so it explicitly named Ariadne's retained bundle as evidence provenance and stopped short of an unsupported fix. ## Use Hermes next - `What is broken in the cluster right now?` - `Triage the worst current CI suite and link the evidence.` - `Audit Grafana alerts from the last 24 hours and separate incidents from noise.` - `Explain the Cassandra generator failure and give me the smallest safe next step.`