titan-iac/services/hermes/skills/tune-atlas-alerts/SKILL.md

name, description

name	description
tune-atlas-alerts	Audit and tune noisy Atlas Grafana alerts using live VictoriaMetrics evidence, Grafana notification history, persistence, minimum sample sizes, and generator ownership. Use when alerts are over-aggressive, a panel is red without user impact, PromQL behaves impossibly, or a Flux-tracked alert/dashboard correction is needed.

Tune Atlas Alerts

Reduce false positives without hiding real failures. Read references/alert-review.md before recommending a rule change.

Audit the signal

1. Find the provisioned rule in services/monitoring/grafana-alerting-config.yaml.
2. If the issue is a dashboard panel, find its Python source in scripts/dashboards_render_atlas.py; never hand-edit generated JSON or ConfigMaps.
3. Query the exact PromQL against VictoriaMetrics and inspect the labels and raw inputs.
4. Compare current value, recent history, notification frequency, and a second source such as Kubernetes events or service metrics.
5. Classify the problem as bad math, lifetime-versus-rate confusion, missing persistence, low sample size, duplicate series, retired scope, rollout noise, real backlog, or real incident.

Useful read-only commands:

curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
kubectl -n monitoring logs deploy/grafana --since=24h
kubectl -n monitoring get configmap grafana-alerting-config -o yaml
kubectl -n <namespace> get events --sort-by=.lastTimestamp

Design the correction

• Use rate, increase, or delta according to counter/gauge semantics; clamp percentages to 0..100.
• Require persistence for startup, rollout, and scheduling conditions.
• Add a minimum denominator and absolute count to percentage alerts with small samples.
• Alert on configured resources that regressed; keep unenrolled inventory visible as backlog instead of paging continuously.
• Exclude retired or migrated resources from high-level health, while retaining them on detailed dashboards.
• Deduplicate replicas and scrape targets using stable service/workload labels.
• Preserve a drill-down path to the raw evidence.

Validate changed PromQL live, run the dashboard generator, test the generator, render the monitoring kustomization, and use a client-side dry-run. Report before/after values and what failure will still trigger the rule.

Hermes is read-only in the cluster. Draft or explain repo changes and mark reconcile or deployment actions as approval-required.