---
name: tune-atlas-alerts
description: Audit and tune noisy Atlas Grafana alerts using live VictoriaMetrics evidence, Grafana notification history, persistence, minimum sample sizes, and generator ownership. Use when alerts are over-aggressive, a panel is red without user impact, PromQL behaves impossibly, or a Flux-tracked alert/dashboard correction is needed.
---

# Tune Atlas Alerts

Reduce false positives without hiding real failures. Read `references/alert-review.md` before recommending a rule change.

## Audit the signal

1. Find the provisioned rule in `services/monitoring/grafana-alerting-config.yaml`.
2. If the issue is a dashboard panel, find its Python source in `scripts/dashboards_render_atlas.py`; never hand-edit generated JSON or ConfigMaps.
3. Query the exact PromQL against VictoriaMetrics and inspect the labels and raw inputs.
4. Compare current value, recent history, notification frequency, and a second source such as Kubernetes events or service metrics.
5. Classify the problem as bad math, lifetime-versus-rate confusion, missing persistence, low sample size, duplicate series, retired scope, rollout noise, real backlog, or real incident.

Useful read-only commands:

```sh
curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
kubectl -n monitoring logs deploy/grafana --since=24h
kubectl -n monitoring get configmap grafana-alerting-config -o yaml
kubectl -n <namespace> get events --sort-by=.lastTimestamp
```

## Design the correction

- Use `rate`, `increase`, or `delta` according to counter/gauge semantics; clamp percentages to 0..100.
- Require persistence for startup, rollout, and scheduling conditions.
- Add a minimum denominator and absolute count to percentage alerts with small samples.
- Alert on configured resources that regressed; keep unenrolled inventory visible as backlog instead of paging continuously.
- Exclude retired or migrated resources from high-level health, while retaining them on detailed dashboards.
- Deduplicate replicas and scrape targets using stable service/workload labels.
- Preserve a drill-down path to the raw evidence.

Validate changed PromQL live, run the dashboard generator, test the generator, render the monitoring kustomization, and use a client-side dry-run. Report before/after values and what failure will still trigger the rule.

Hermes is read-only in the cluster. Draft or explain repo changes and mark reconcile or deployment actions as approval-required.