bstein/titan-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(hermes): package Atlas skill instructions

Browse Source
This commit is contained in:
jenkins 2026-08-03 03:59:00 -03:00
parent e8f75f5cc9
commit abb9a39971
4 changed files with 139 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

62
services/hermes/skills/triage-atlas-service-health/SKILL.md Normal file
View File

@ -0,0 +1,62 @@
---
name: triage-atlas-service-health
description: Diagnose active Atlas service and infrastructure problems from Grafana, Ariadne, Flux, Kubernetes events, workload state, and logs. Use when asked what is broken, why a Grafana health panel is red, whether an outage is real, or how to fix a service through titan-iac. Covers every deployed namespace while distinguishing current services from migration residue and historical noise.
---
# Triage Atlas Service Health
Investigate read-only, group symptoms into incidents, and recommend the smallest Flux-tracked fix.
## Establish the current scope
1. Read `references/service-map.md` for custom-service ownership and migrations.
2. Discover the live inventory instead of relying only on the map:
```sh
kubectl get namespaces
kubectl get deploy,statefulset,daemonset -A
kubectl get ingress -A
kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
```
3. Treat Cassandra as authoritative for functionality migrated from Veles. Report Veles failures as migration residue unless a live route or dependency proves current impact.
## Decide whether the signal is actionable
Use all of these checks before calling something an incident:
- Current state: prefer an active condition over lifetime restart totals or old failed Jobs.
- Persistence: ignore normal startup states under 15 minutes unless users are already affected.
- Ownership: group replica pods by Deployment, StatefulSet, DaemonSet, Job, or CronJob.
- Delivery: a Flux object with `Ready=null` may be reconciling; `Ready=false` or a suspended required object is actionable.
- Authority: separate current workloads from retired, migrated, or deliberately suspended resources.
- Correlation: confirm a red Grafana panel with the underlying metric and at least one independent source such as events, logs, readiness, or Flux state.
Run the narrowest relevant checks:
```sh
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded -o wide
kubectl get events -A --sort-by=.lastTimestamp
kubectl -n <namespace> get deploy,statefulset,daemonset,job,cronjob
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> logs <pod> --all-containers --tail=200
kubectl -n flux-system get kustomizations.kustomize.toolkit.fluxcd.io
curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
```
Never read Kubernetes Secret values. Secret-key errors quoted by a pod event or log are valid evidence; inspect the SecretProviderClass and Vault policy manifests, not the secret contents.
## Produce one incident per cause
Deduplicate replica pods and repeated alerts with a signature of service, workload, failing condition, and likely dependency. Return:
- `Finding`: current incident, degraded-but-serving, migration residue, historical noise, or healthy.
- `Impact`: the user-facing service and affected capability.
- `Evidence`: timestamps, workload condition, event/log excerpt, metric, and Flux revision.
- `Noise removed`: signals inspected but not counted, with the reason.
- `Likely cause`: causal chain, marking inference explicitly.
- `Next checks`: ordered read-only commands.
- `Repo-side fix`: exact titan-iac or application path and smallest proposed change.
- `Approval required`: every write, reconcile, credential, or external-system action.
Do not claim a proposed change is applied. Do not mutate the cluster; Atlas is Flux-owned.

22
services/hermes/skills/triage-atlas-service-health/references/service-map.md Normal file
View File

@ -0,0 +1,22 @@
# Atlas custom-service map
Use this map for ownership and canonical names. Discover third-party services live from Flux and Kubernetes.
| Capability | Canonical workload/repository | Notes |
| --- | --- | --- |
| Cluster automation and state analysis | Ariadne / `bstein/Ariadne` | Runs in `maintenance`; provides deterministic triage bundles. |
| Quality policy and CI telemetry | titan-iac / `bstein/titan-iac` | Owns Jenkins configuration, Grafana generators, alert provisioning, and Data Prepper pipeline. |
| Simulation registry and execution | Cassandra / Cassandra application repository | Cassandra is authoritative after the Veles migration. |
| Legacy simulation stack | Veles | Retained migration residue; do not call it a current outage without dependency evidence. |
| Backup and maintenance policy | Soteria / `bstein/soteria` | Backup inventory and maintenance signals run in `maintenance`. |
| Media client | Pegasus / `bstein/pegasus` | Correlate with Jellyfin and OIDC dependencies. |
| Cluster sentinel | Metis / `bstein/metis` | Runs through the maintenance stack. |
| User and service automation | Ananke / `bstein/ananke` | In-scope CI suite and custom software. |
| Chat automation | Atlasbot / `bstein/atlasbot` | In-scope CI suite and custom software. |
| Public site | bstein_home / `bstein/bstein-dev-home` | Canonical metric suite uses underscore form. |
| Data Prepper integration | data_prepper / `bstein/titan-iac` | Pipeline is under `services/logging`. |
| Desktop test application | Lesavka / `bstein/lesavka` | Test suite is in scope; some runtime evidence may come from titan-jh. |
| Hermes operator | `hermes` namespace / `services/hermes` | Read-only cluster operator with Codex primary and local fallback. |
| Hermes consumer chat | `hermes-chat` namespace / `services/hermes-chat` | Isolated from cluster operation; do not use it for infrastructure triage. |
Canonical CI suites are `ananke`, `ariadne`, `atlasbot`, `bstein_home`, `data_prepper`, `lesavka`, `metis`, `pegasus`, `soteria`, and `titan_iac`.

39
services/hermes/skills/tune-atlas-alerts/SKILL.md Normal file
View File

@ -0,0 +1,39 @@
---
name: tune-atlas-alerts
description: Audit and tune noisy Atlas Grafana alerts using live VictoriaMetrics evidence, Grafana notification history, persistence, minimum sample sizes, and generator ownership. Use when alerts are over-aggressive, a panel is red without user impact, PromQL behaves impossibly, or a Flux-tracked alert/dashboard correction is needed.
---
# Tune Atlas Alerts
Reduce false positives without hiding real failures. Read `references/alert-review.md` before recommending a rule change.
## Audit the signal
1. Find the provisioned rule in `services/monitoring/grafana-alerting-config.yaml`.
2. If the issue is a dashboard panel, find its Python source in `scripts/dashboards_render_atlas.py`; never hand-edit generated JSON or ConfigMaps.
3. Query the exact PromQL against VictoriaMetrics and inspect the labels and raw inputs.
4. Compare current value, recent history, notification frequency, and a second source such as Kubernetes events or service metrics.
5. Classify the problem as bad math, lifetime-versus-rate confusion, missing persistence, low sample size, duplicate series, retired scope, rollout noise, real backlog, or real incident.
Useful read-only commands:
```sh
curl -G -fsS --data-urlencode 'query=<promql>' "$VICTORIA_METRICS_URL/api/v1/query"
kubectl -n monitoring logs deploy/grafana --since=24h
kubectl -n monitoring get configmap grafana-alerting-config -o yaml
kubectl -n <namespace> get events --sort-by=.lastTimestamp
```
## Design the correction
- Use `rate`, `increase`, or `delta` according to counter/gauge semantics; clamp percentages to 0..100.
- Require persistence for startup, rollout, and scheduling conditions.
- Add a minimum denominator and absolute count to percentage alerts with small samples.
- Alert on configured resources that regressed; keep unenrolled inventory visible as backlog instead of paging continuously.
- Exclude retired or migrated resources from high-level health, while retaining them on detailed dashboards.
- Deduplicate replicas and scrape targets using stable service/workload labels.
- Preserve a drill-down path to the raw evidence.
Validate changed PromQL live, run the dashboard generator, test the generator, render the monitoring kustomization, and use a client-side dry-run. Report before/after values and what failure will still trigger the rule.
Hermes is read-only in the cluster. Draft or explain repo changes and mark reconcile or deployment actions as approval-required.

16
services/hermes/skills/tune-atlas-alerts/references/alert-review.md Normal file
View File

@ -0,0 +1,16 @@
# Alert review checklist
For every alert, record:
1. User impact or operator action the alert demands.
2. Metric type: counter, gauge, timestamp, state marker, or recording rule.
3. Scope and deduplication labels.
4. Persistence window and startup grace.
5. Minimum sample size for ratios.
6. No-data and query-error behavior.
7. Current value and a recent range query.
8. Notification frequency in Grafana logs.
9. Current, migrated, suspended, or historical resource status.
10. Exact Flux-tracked source, generated artifacts, tests, and rollback condition.
A rule is useful only when its firing state implies a specific human action. Backlogs and unenrolled resources belong on dashboards unless they have crossed an explicitly accepted operational deadline.