titan-iac/services/hermes/skills/triage-atlas-service-health/SKILL.md

---
name: triage-atlas-service-health
description: Diagnose active Atlas service and infrastructure problems from Grafana, Ariadne, Flux, Kubernetes events, workload state, and logs. Use when asked what is broken, why a Grafana health panel is red, whether an outage is real, or how to fix a service through titan-iac. Covers every deployed namespace while distinguishing current services from migration residue and historical noise.
---

# Triage Atlas Service Health

Investigate read-only, group symptoms into incidents, and recommend the smallest Flux-tracked fix.

## Establish the current scope

1. Read `references/service-map.md` for custom-service ownership and migrations.
2. Discover the live inventory instead of relying only on the map:

   ```sh
   kubectl get namespaces
   kubectl get deploy,statefulset,daemonset -A
   kubectl get ingress -A
   kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
   ```

3. Cassandra is authoritative for functionality migrated from Veles. Veles is
   retired migration residue: its failed pods, old Jobs, and legacy Service or
   Ingress objects are not current user impact. Escalate Veles only when direct
   evidence proves Cassandra still depends on it or the Veles-to-Cassandra
   redirect itself is failing.

## Decide whether the signal is actionable

Use all of these checks before calling something an incident:

- Current state: prefer an active condition over lifetime restart totals or old failed Jobs.
- Persistence: ignore normal startup states under 15 minutes unless users are already affected.
- Ownership: group replica pods by Deployment, StatefulSet, DaemonSet, Job, or CronJob.
- Delivery: a Flux object with `Ready=null` may be reconciling; `Ready=false` or a suspended required object is actionable.
- Authority: separate current workloads from retired, migrated, or deliberately suspended resources. Never describe Veles as unavailable or degraded user-facing service merely because its retained pods are unhealthy.
- Correlation: confirm a red Grafana panel with the underlying metric and at least one independent source such as events, logs, readiness, or Flux state.

Run the narrowest relevant checks:

```sh
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded -o wide
kubectl get events -A --sort-by=.lastTimestamp
kubectl -n <namespace> get deploy,statefulset,daemonset,job,cronjob
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> logs <pod> --all-containers --tail=200
kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
```

Never read Kubernetes Secret values. Secret-key errors quoted by a pod event or log are valid evidence; inspect the SecretProviderClass and Vault policy manifests, not the secret contents.

## Produce one incident per cause

Deduplicate replica pods and repeated alerts with a signature of service, workload, failing condition, and likely dependency. Return:

- `Finding`: current incident, degraded-but-serving, migration residue, historical noise, or healthy.
- `Impact`: the user-facing service and affected capability.
- `Evidence`: timestamps, workload condition, event/log excerpt, metric, and Flux revision.
- `Noise removed`: signals inspected but not counted, with the reason.
- `Likely cause`: causal chain, marking inference explicitly.
- `Next checks`: ordered read-only commands.
- `Repo-side fix`: exact titan-iac or application path and smallest proposed change.
- `Approval required`: every write, reconcile, credential, or external-system action.

Do not claim a proposed change is applied. Do not mutate the cluster; Atlas is Flux-owned.