diff --git a/services/hermes/skills/triage-atlas-service-health/SKILL.md b/services/hermes/skills/triage-atlas-service-health/SKILL.md new file mode 100644 index 000000000..934a29577 --- /dev/null +++ b/services/hermes/skills/triage-atlas-service-health/SKILL.md @@ -0,0 +1,62 @@ +--- +name: triage-atlas-service-health +description: Diagnose active Atlas service and infrastructure problems from Grafana, Ariadne, Flux, Kubernetes events, workload state, and logs. Use when asked what is broken, why a Grafana health panel is red, whether an outage is real, or how to fix a service through titan-iac. Covers every deployed namespace while distinguishing current services from migration residue and historical noise. +--- + +# Triage Atlas Service Health + +Investigate read-only, group symptoms into incidents, and recommend the smallest Flux-tracked fix. + +## Establish the current scope + +1. Read `references/service-map.md` for custom-service ownership and migrations. +2. Discover the live inventory instead of relying only on the map: + + ```sh + kubectl get namespaces + kubectl get deploy,statefulset,daemonset -A + kubectl get ingress -A + kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io + ``` + +3. Treat Cassandra as authoritative for functionality migrated from Veles. Report Veles failures as migration residue unless a live route or dependency proves current impact. + +## Decide whether the signal is actionable + +Use all of these checks before calling something an incident: + +- Current state: prefer an active condition over lifetime restart totals or old failed Jobs. +- Persistence: ignore normal startup states under 15 minutes unless users are already affected. +- Ownership: group replica pods by Deployment, StatefulSet, DaemonSet, Job, or CronJob. +- Delivery: a Flux object with `Ready=null` may be reconciling; `Ready=false` or a suspended required object is actionable. +- Authority: separate current workloads from retired, migrated, or deliberately suspended resources. +- Correlation: confirm a red Grafana panel with the underlying metric and at least one independent source such as events, logs, readiness, or Flux state. + +Run the narrowest relevant checks: + +```sh +kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded -o wide +kubectl get events -A --sort-by=.lastTimestamp +kubectl -n get deploy,statefulset,daemonset,job,cronjob +kubectl -n describe pod +kubectl -n logs --all-containers --tail=200 +kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io +curl -G -fsS --data-urlencode 'query=' "$VICTORIA_METRICS_URL/api/v1/query" +``` + +Never read Kubernetes Secret values. Secret-key errors quoted by a pod event or log are valid evidence; inspect the SecretProviderClass and Vault policy manifests, not the secret contents. + +## Produce one incident per cause + +Deduplicate replica pods and repeated alerts with a signature of service, workload, failing condition, and likely dependency. Return: + +- `Finding`: current incident, degraded-but-serving, migration residue, historical noise, or healthy. +- `Impact`: the user-facing service and affected capability. +- `Evidence`: timestamps, workload condition, event/log excerpt, metric, and Flux revision. +- `Noise removed`: signals inspected but not counted, with the reason. +- `Likely cause`: causal chain, marking inference explicitly. +- `Next checks`: ordered read-only commands. +- `Repo-side fix`: exact titan-iac or application path and smallest proposed change. +- `Approval required`: every write, reconcile, credential, or external-system action. + +Do not claim a proposed change is applied. Do not mutate the cluster; Atlas is Flux-owned. diff --git a/services/hermes/skills/triage-atlas-service-health/references/service-map.md b/services/hermes/skills/triage-atlas-service-health/references/service-map.md new file mode 100644 index 000000000..bd43cd24d --- /dev/null +++ b/services/hermes/skills/triage-atlas-service-health/references/service-map.md @@ -0,0 +1,22 @@ +# Atlas custom-service map + +Use this map for ownership and canonical names. Discover third-party services live from Flux and Kubernetes. + +| Capability | Canonical workload/repository | Notes | +| --- | --- | --- | +| Cluster automation and state analysis | Ariadne / `bstein/Ariadne` | Runs in `maintenance`; provides deterministic triage bundles. | +| Quality policy and CI telemetry | titan-iac / `bstein/titan-iac` | Owns Jenkins configuration, Grafana generators, alert provisioning, and Data Prepper pipeline. | +| Simulation registry and execution | Cassandra / Cassandra application repository | Cassandra is authoritative after the Veles migration. | +| Legacy simulation stack | Veles | Retained migration residue; do not call it a current outage without dependency evidence. | +| Backup and maintenance policy | Soteria / `bstein/soteria` | Backup inventory and maintenance signals run in `maintenance`. | +| Media client | Pegasus / `bstein/pegasus` | Correlate with Jellyfin and OIDC dependencies. | +| Cluster sentinel | Metis / `bstein/metis` | Runs through the maintenance stack. | +| User and service automation | Ananke / `bstein/ananke` | In-scope CI suite and custom software. | +| Chat automation | Atlasbot / `bstein/atlasbot` | In-scope CI suite and custom software. | +| Public site | bstein_home / `bstein/bstein-dev-home` | Canonical metric suite uses underscore form. | +| Data Prepper integration | data_prepper / `bstein/titan-iac` | Pipeline is under `services/logging`. | +| Desktop test application | Lesavka / `bstein/lesavka` | Test suite is in scope; some runtime evidence may come from titan-jh. | +| Hermes operator | `hermes` namespace / `services/hermes` | Read-only cluster operator with Codex primary and local fallback. | +| Hermes consumer chat | `hermes-chat` namespace / `services/hermes-chat` | Isolated from cluster operation; do not use it for infrastructure triage. | + +Canonical CI suites are `ananke`, `ariadne`, `atlasbot`, `bstein_home`, `data_prepper`, `lesavka`, `metis`, `pegasus`, `soteria`, and `titan_iac`. diff --git a/services/hermes/skills/tune-atlas-alerts/SKILL.md b/services/hermes/skills/tune-atlas-alerts/SKILL.md new file mode 100644 index 000000000..74a728663 --- /dev/null +++ b/services/hermes/skills/tune-atlas-alerts/SKILL.md @@ -0,0 +1,39 @@ +--- +name: tune-atlas-alerts +description: Audit and tune noisy Atlas Grafana alerts using live VictoriaMetrics evidence, Grafana notification history, persistence, minimum sample sizes, and generator ownership. Use when alerts are over-aggressive, a panel is red without user impact, PromQL behaves impossibly, or a Flux-tracked alert/dashboard correction is needed. +--- + +# Tune Atlas Alerts + +Reduce false positives without hiding real failures. Read `references/alert-review.md` before recommending a rule change. + +## Audit the signal + +1. Find the provisioned rule in `services/monitoring/grafana-alerting-config.yaml`. +2. If the issue is a dashboard panel, find its Python source in `scripts/dashboards_render_atlas.py`; never hand-edit generated JSON or ConfigMaps. +3. Query the exact PromQL against VictoriaMetrics and inspect the labels and raw inputs. +4. Compare current value, recent history, notification frequency, and a second source such as Kubernetes events or service metrics. +5. Classify the problem as bad math, lifetime-versus-rate confusion, missing persistence, low sample size, duplicate series, retired scope, rollout noise, real backlog, or real incident. + +Useful read-only commands: + +```sh +curl -G -fsS --data-urlencode 'query=' "$VICTORIA_METRICS_URL/api/v1/query" +kubectl -n monitoring logs deploy/grafana --since=24h +kubectl -n monitoring get configmap grafana-alerting-config -o yaml +kubectl -n get events --sort-by=.lastTimestamp +``` + +## Design the correction + +- Use `rate`, `increase`, or `delta` according to counter/gauge semantics; clamp percentages to 0..100. +- Require persistence for startup, rollout, and scheduling conditions. +- Add a minimum denominator and absolute count to percentage alerts with small samples. +- Alert on configured resources that regressed; keep unenrolled inventory visible as backlog instead of paging continuously. +- Exclude retired or migrated resources from high-level health, while retaining them on detailed dashboards. +- Deduplicate replicas and scrape targets using stable service/workload labels. +- Preserve a drill-down path to the raw evidence. + +Validate changed PromQL live, run the dashboard generator, test the generator, render the monitoring kustomization, and use a client-side dry-run. Report before/after values and what failure will still trigger the rule. + +Hermes is read-only in the cluster. Draft or explain repo changes and mark reconcile or deployment actions as approval-required. diff --git a/services/hermes/skills/tune-atlas-alerts/references/alert-review.md b/services/hermes/skills/tune-atlas-alerts/references/alert-review.md new file mode 100644 index 000000000..7468ffa6e --- /dev/null +++ b/services/hermes/skills/tune-atlas-alerts/references/alert-review.md @@ -0,0 +1,16 @@ +# Alert review checklist + +For every alert, record: + +1. User impact or operator action the alert demands. +2. Metric type: counter, gauge, timestamp, state marker, or recording rule. +3. Scope and deduplication labels. +4. Persistence window and startup grace. +5. Minimum sample size for ratios. +6. No-data and query-error behavior. +7. Current value and a recent range query. +8. Notification frequency in Grafana logs. +9. Current, migrated, suspended, or historical resource status. +10. Exact Flux-tracked source, generated artifacts, tests, and rollback condition. + +A rule is useful only when its firing state implies a specific human action. Backlogs and unenrolled resources belong on dashboards unless they have crossed an explicitly accepted operational deadline.