40 lines
2.4 KiB
Markdown
40 lines
2.4 KiB
Markdown
---
|
|
name: tune-atlas-alerts
|
|
description: Audit and tune noisy Atlas Grafana alerts using live VictoriaMetrics evidence, Grafana notification history, persistence, minimum sample sizes, and generator ownership. Use when alerts are over-aggressive, a panel is red without user impact, PromQL behaves impossibly, or a Flux-tracked alert/dashboard correction is needed.
|
|
---
|
|
|
|
# Tune Atlas Alerts
|
|
|
|
Reduce false positives without hiding real failures. Read `references/alert-review.md` before recommending a rule change.
|
|
|
|
## Audit the signal
|
|
|
|
1. Find the provisioned rule in `services/monitoring/grafana-alerting-config.yaml`.
|
|
2. If the issue is a dashboard panel, find its Python source in `scripts/dashboards_render_atlas.py`; never hand-edit generated JSON or ConfigMaps.
|
|
3. Query the exact PromQL against VictoriaMetrics and inspect the labels and raw inputs.
|
|
4. Compare current value, recent history, notification frequency, and a second source such as Kubernetes events or service metrics.
|
|
5. Classify the problem as bad math, lifetime-versus-rate confusion, missing persistence, low sample size, duplicate series, retired scope, rollout noise, real backlog, or real incident.
|
|
|
|
Useful read-only commands:
|
|
|
|
```sh
|
|
curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
|
|
kubectl -n monitoring logs deploy/grafana --since=24h
|
|
kubectl -n monitoring get configmap grafana-alerting-config -o yaml
|
|
kubectl -n <namespace> get events --sort-by=.lastTimestamp
|
|
```
|
|
|
|
## Design the correction
|
|
|
|
- Use `rate`, `increase`, or `delta` according to counter/gauge semantics; clamp percentages to 0..100.
|
|
- Require persistence for startup, rollout, and scheduling conditions.
|
|
- Add a minimum denominator and absolute count to percentage alerts with small samples.
|
|
- Alert on configured resources that regressed; keep unenrolled inventory visible as backlog instead of paging continuously.
|
|
- Exclude retired or migrated resources from high-level health, while retaining them on detailed dashboards.
|
|
- Deduplicate replicas and scrape targets using stable service/workload labels.
|
|
- Preserve a drill-down path to the raw evidence.
|
|
|
|
Validate changed PromQL live, run the dashboard generator, test the generator, render the monitoring kustomization, and use a client-side dry-run. Report before/after values and what failure will still trigger the rule.
|
|
|
|
Hermes is read-only in the cluster. Draft or explain repo changes and mark reconcile or deployment actions as approval-required.
|