67 lines
3.6 KiB
Markdown
67 lines
3.6 KiB
Markdown
---
|
|
name: triage-atlas-service-health
|
|
description: Diagnose active Atlas service and infrastructure problems from Grafana, Ariadne, Flux, Kubernetes events, workload state, and logs. Use when asked what is broken, why a Grafana health panel is red, whether an outage is real, or how to fix a service through titan-iac. Covers every deployed namespace while distinguishing current services from migration residue and historical noise.
|
|
---
|
|
|
|
# Triage Atlas Service Health
|
|
|
|
Investigate read-only, group symptoms into incidents, and recommend the smallest Flux-tracked fix.
|
|
|
|
## Establish the current scope
|
|
|
|
1. Read `references/service-map.md` for custom-service ownership and migrations.
|
|
2. Discover the live inventory instead of relying only on the map:
|
|
|
|
```sh
|
|
kubectl get namespaces
|
|
kubectl get deploy,statefulset,daemonset -A
|
|
kubectl get ingress -A
|
|
kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
|
|
```
|
|
|
|
3. Cassandra is authoritative for functionality migrated from Veles. Veles is
|
|
retired migration residue: its failed pods, old Jobs, and legacy Service or
|
|
Ingress objects are not current user impact. Escalate Veles only when direct
|
|
evidence proves Cassandra still depends on it or the Veles-to-Cassandra
|
|
redirect itself is failing.
|
|
|
|
## Decide whether the signal is actionable
|
|
|
|
Use all of these checks before calling something an incident:
|
|
|
|
- Current state: prefer an active condition over lifetime restart totals or old failed Jobs.
|
|
- Persistence: ignore normal startup states under 15 minutes unless users are already affected.
|
|
- Ownership: group replica pods by Deployment, StatefulSet, DaemonSet, Job, or CronJob.
|
|
- Delivery: a Flux object with `Ready=null` may be reconciling; `Ready=false` or a suspended required object is actionable.
|
|
- Authority: separate current workloads from retired, migrated, or deliberately suspended resources. Never describe Veles as unavailable or degraded user-facing service merely because its retained pods are unhealthy.
|
|
- Correlation: confirm a red Grafana panel with the underlying metric and at least one independent source such as events, logs, readiness, or Flux state.
|
|
|
|
Run the narrowest relevant checks:
|
|
|
|
```sh
|
|
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded -o wide
|
|
kubectl get events -A --sort-by=.lastTimestamp
|
|
kubectl -n <namespace> get deploy,statefulset,daemonset,job,cronjob
|
|
kubectl -n <namespace> describe pod <pod>
|
|
kubectl -n <namespace> logs <pod> --all-containers --tail=200
|
|
kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
|
|
curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
|
|
```
|
|
|
|
Never read Kubernetes Secret values. Secret-key errors quoted by a pod event or log are valid evidence; inspect the SecretProviderClass and Vault policy manifests, not the secret contents.
|
|
|
|
## Produce one incident per cause
|
|
|
|
Deduplicate replica pods and repeated alerts with a signature of service, workload, failing condition, and likely dependency. Return:
|
|
|
|
- `Finding`: current incident, degraded-but-serving, migration residue, historical noise, or healthy.
|
|
- `Impact`: the user-facing service and affected capability.
|
|
- `Evidence`: timestamps, workload condition, event/log excerpt, metric, and Flux revision.
|
|
- `Noise removed`: signals inspected but not counted, with the reason.
|
|
- `Likely cause`: causal chain, marking inference explicitly.
|
|
- `Next checks`: ordered read-only commands.
|
|
- `Repo-side fix`: exact titan-iac or application path and smallest proposed change.
|
|
- `Approval required`: every write, reconcile, credential, or external-system action.
|
|
|
|
Do not claim a proposed change is applied. Do not mutate the cluster; Atlas is Flux-owned.
|