titan-iac/knowledge/catalog/runbooks.json

[
  {
    "path": "runbooks/ci-gitea-jenkins.md",
    "title": "CI: Gitea \u2192 Jenkins pipeline",
    "tags": [
      "atlas",
      "ci",
      "gitea",
      "jenkins"
    ],
    "entrypoints": [
      "scm.bstein.dev",
      "ci.bstein.dev"
    ],
    "source_paths": [
      "services/gitea",
      "services/jenkins",
      "scripts/sync/jenkins_cred_sync.sh",
      "scripts/sync/gitea_cred_sync.sh"
    ],
    "body": "# CI: Gitea \u2192 Jenkins pipeline\n\n## What this is\nAtlas uses Gitea for source control and Jenkins for CI. Authentication is via Keycloak (SSO).\n\n## Where it is configured\n- Gitea manifests: `services/gitea/`\n- Jenkins manifests: `services/jenkins/`\n- Credential sync helpers: `scripts/sync/gitea_cred_sync.sh`, `scripts/sync/jenkins_cred_sync.sh`\n\n## What users do (typical flow)\n- Create a repo in Gitea.\n- Create/update a Jenkins job/pipeline that can fetch the repo.\n- Configure a webhook (or SCM polling) so pushes trigger builds.\n\n## Troubleshooting (common)\n- \u201cWebhook not firing\u201d: confirm ingress host, webhook URL, and Jenkins job is reachable.\n- \u201cAuth denied cloning\u201d: confirm Keycloak group membership and that Jenkins has a valid token/credential configured."
  },
  {
    "path": "runbooks/cluster-power-recovery.md",
    "title": "cluster-power-recovery",
    "tags": [],
    "entrypoints": [],
    "source_paths": [],
    "body": "Atlas Cluster Power Recovery (Graceful Shutdown/Startup)\n\nPurpose\n- Provide a safe operator flow for planned power events and cold-boot recovery.\n- Avoid the Flux/Gitea bootstrap deadlock by using a local bootstrap fallback path.\n- Break the Harbor self-hosting deadlock by seeding Harbor runtime images from a control-host bundle.\n- Refuse bootstrap when UPS charge is too low, and fall back to fast shutdown if a second outage hits mid-recovery.\n\nBootstrapping risk to remember\n- Flux source is Git over SSH to `scm.bstein.dev` (Gitea).\n- Gitea itself is a Flux-managed workload and depends on storage + database.\n- Harbor is also critical, but it is not part of the first recovery stage because Harbor serves its own runtime images.\n- On cold boot, if Flux cannot fetch source before Gitea is up, reconciliation can stall.\n- Recovery path: bring control plane and workers up, then locally apply minimal platform stack (`core -> helm -> longhorn -> metallb -> traefik -> vault-csi -> vault-injector -> vault -> postgres -> gitea`), then seed Harbor images onto the Harbor node from a control-host bundle, then resume/reconcile Flux. Harbor is a later recovery stage after storage, Vault, Postgres, and Gitea are back.\n\nScript\n- `scripts/ops/cluster_power_recovery.sh`\n- `scripts/ops/cluster_power_console.sh`\n- Modes:\n  - `prepare`\n  - `shutdown`\n  - `harbor-seed`\n  - `startup`\n  - `status`\n- Default is dry-run. Add `--execute` to actually perform actions.\n\nDry-run examples\n- Shutdown preview:\n  - `scripts/ops/cluster_power_recovery.sh shutdown --skip-etcd-snapshot --skip-drain`\n- Startup preview:\n  - `scripts/ops/cluster_power_recovery.sh startup`\n- Harbor seed preview:\n  - `scripts/ops/cluster_power_recovery.sh harbor-seed`\n\nExecute examples\n- Prepare helper image on every node:\n  - `scripts/ops/cluster_power_recovery.sh prepare --execute`\n- Seed Harbor runtime images onto `titan-05` from the control-host bundle:\n  - `scripts/ops/cluster_power_recovery.sh harbor-seed --execute`\n- Planned shutdown:\n  - `scripts/ops/cluster_power_recovery.sh shutdown --execute`\n- Planned startup (canonical branch):\n  - `scripts/ops/cluster_power_recovery.sh startup --execute --force-flux-branch main`\n\nManual remote console examples\n- Canonical operator hosts:\n  - `titan-db`\n  - `tethys` (`titan-24`)\n- Both hosts now have:\n  - `~/ananke-tools/cluster_power_recovery.sh`\n  - `~/ananke-tools/cluster_power_console.sh`\n  - `~/ananke-tools/bootstrap/recovery-config.env`\n  - `~/ananke-tools/bootstrap/harbor-bootstrap-images.txt`\n  - `~/ananke-tools/kubeconfig`\n  - `~/ananke-cluster-power`\n  - `~/bin/ananke-cluster-power`\n  - `~/ananke-repo/{infrastructure,services,scripts}`\n- Both hosts also keep the Harbor bootstrap bundle at:\n  - `~/.local/share/ananke/bundles/harbor-bootstrap-v2.14.1-arm64.tar.zst`\n- Remote usage:\n  - `ssh titan-db`\n  - `~/ananke-cluster-power status`\n  - `~/ananke-cluster-power prepare --execute`\n  - `~/ananke-cluster-power shutdown --execute`\n  - `~/ananke-cluster-power startup --execute --force-flux-branch main`\n  - `ssh tethys`\n  - `~/ananke-cluster-power status`\n  - `~/ananke-cluster-power prepare --execute`\n  - `~/ananke-cluster-power shutdown --execute`\n  - `~/ananke-cluster-power startup --execute --force-flux-branch main`\n\nUseful options\n- `--shutdown-mode host-poweroff|cluster-only`\n- `--expected-flux-branch main`\n- `--expected-flux-url ssh://git@scm.bstein.dev:2242/bstein/titan-iac.git`\n- `--force-flux-url ssh://git@scm.bstein.dev:2242/bstein/titan-iac.git`\n- `--force-flux-branch main`\n- `--allow-flux-source-mutation` (required with `--force-flux-url`; breakglass only)\n- `--skip-local-bootstrap` (not recommended for cold-start recovery)\n- `--skip-harbor-bootstrap` (skip the Harbor recovery stage if you know Harbor should stay deferred)\n- `--skip-harbor-seed` (skip bundle import if Harbor images are already cached on the target node)\n- `--skip-helper-prewarm`\n- `--min-startup-battery 35`\n- `--ups-host pyrphoros@localhost`\n- `--require-ups-battery`\n- `--drain-timeout 180`\n- `--emergency-drain-timeout 45`\n- `--flux-ready-timeout 1200`\n- `--startup-checklist-timeout 900`\n- `--startup-stability-window 180`\n- `--startup-stability-timeout 900`\n- `--recovery-state-file ~/.local/share/ananke/cluster_power_recovery.state`\n- `--harbor-bundle-file ~/.local/share/ananke/bundles/harbor-bootstrap-v2.14.1-arm64.tar.zst`\n\nControlled drill checklist (recommended)\n- Operator host: use `titan-db` as canonical control host for the drill.\n- On-site coordination:\n  - Have on-site operator ready before shutdown starts.\n  - Confirm they will manually power cluster nodes back on after shutdown completes.\n  - Confirm who will announce \"all nodes powered on\" to resume startup.\n- Preflight on `titan-db`:\n  - `mkdir -p ~/ananke-logs`\n  - `~/ananke-cluster-power status` and verify:\n    - `ups_host=pyrphoros@localhost`\n    - `ups_battery` is numeric\n    - `flux_source_ready=True`\n- Warm helper image just before shutdown:\n  - `~/ananke-cluster-power prepare --execute`\n- Run in a persistent shell and capture logs:\n  - `tmux new -s ananke-drill`\n  - `script -q -a ~/ananke-logs/ananke-drill-$(date +%Y%m%d-%H%M%S).log`\n- Execute controlled shutdown with telemetry enforcement:\n  - `~/ananke-cluster-power shutdown --execute --require-ups-battery`\n- After on-site power-on confirmation, execute startup:\n  - `~/ananke-cluster-power startup --execute --force-flux-branch main --require-ups-battery`\n- Post-check:\n  - `~/ananke-cluster-power status`\n  - Verify critical services (`longhorn`, `vault`, `postgres`, `gitea`, `harbor`, `pegasus`) and no widespread pull/crash failures.\n\nOperational notes\n- The flow suspends Flux Kustomizations/HelmReleases during shutdown to prevent churn.\n- Shutdown behavior is explicit:\n  - `host-poweroff` schedules host poweroff after service stop.\n  - `cluster-only` stops `k3s`/`k3s-agent` without powering hosts off.\n- Worker drain is no longer best-effort only. The script now escalates from normal drain, to `--force`, to `--disable-eviction` once the configured timeout is exhausted.\n- Startup fails fast if Flux source URL/branch drift from expected values (unless branch override is explicitly requested with `--force-flux-branch`).\n- Flux desired-state source remains `titan-iac.git`. Ananke orchestrates runtime recovery and should not be used as the normal Flux source repo.\n- During startup, if Flux source is not `Ready`, local bootstrap fallback is applied first using the repo snapshot under `~/ananke-repo`.\n- Longhorn is reconciled before Vault/Postgres/Gitea so storage-backed services are not racing the volume layer.\n- Harbor is reconciled after the first critical stateful services.\n- Harbor bootstrap is now designed around a control-host bundle:\n  - Build the Harbor bundle locally with `scripts/ops/build_harbor_bootstrap_bundle.sh`.\n  - Stage it on the operator host at `~/.local/share/ananke/bundles/harbor-bootstrap-v2.14.1-arm64.tar.zst`.\n  - Use `harbor-seed --execute` or a full `startup --execute` to stream/import that bundle onto `titan-05`.\n- The Harbor bundle remains arm64-only because Harbor is pinned to arm64 nodes. The node-helper image is multi-arch because Ananke uses it across both arm64 and amd64 nodes during prepare/shutdown operations.\n- Ananke uses a temporary privileged helper pod for host-side operations. The helper image is prewarmed with `prepare --execute` so later shutdown/startup steps do not stall on image pulls.\n- The script persists outage state in `~/.local/share/ananke/cluster_power_recovery.state` by default. If startup is attempted during an outage window and power becomes unstable again, rerunning startup with insufficient UPS charge will flip into the emergency shutdown path instead of continuing to bootstrap.\n- Startup completion is strict now:\n  - all non-optional Flux kustomizations must be `Ready=True`\n  - external service checklist must pass (defaults include Gitea, Grafana, Harbor)\n  - generated ingress reachability checks must pass (default accepted codes: `200,301,302,307,308,401,403,404`)\n  - stability soak must pass with no crashloop/pull-failure churn\n- If Flux hits immutable one-off Job drift during reconcile, Ananke now attempts self-heal by pruning failed Flux-managed Jobs and retrying reconcile.\n- In dry-run mode, the script now skips the live API wait step so preview runs do not stall on an offline cluster.\n- Dry-run mode no longer mutates outage recovery state.\n- `harbor-seed --execute` was validated by:\n  - prewarming the helper image across all nodes\n  - streaming the Harbor bootstrap bundle to `titan-05`\n  - importing Harbor runtime images into host `containerd`\n  - successfully running a Harbor-backed canary pod (`harbor-canary-ok`)\n- After bootstrap, Flux resources are resumed and reconciled.\n- Keep this runbook aligned with `clusters/atlas/flux-system/gotk-sync.yaml`."
  },
  {
    "path": "runbooks/comms-verify.md",
    "title": "Othrys verification checklist",
    "tags": [
      "comms",
      "matrix",
      "element",
      "livekit"
    ],
    "entrypoints": [
      "https://live.bstein.dev",
      "https://matrix.live.bstein.dev"
    ],
    "source_paths": [],
    "body": "1) Guest join:\n- Open a private window and visit:\n  `https://live.bstein.dev/#/room/#othrys:live.bstein.dev?action=join`\n- Confirm the guest join flow works and the displayname becomes `<word>-<word>`.\n\n2) Keycloak login:\n- Log in from `https://live.bstein.dev` and confirm MAS -> Keycloak -> Element redirect.\n\n3) Video rooms:\n- Start an Element Call room and confirm audio/video with a second account.\n- Check that guests can read public rooms but cannot start calls.\n\n4) Well-known:\n- `https://live.bstein.dev/.well-known/matrix/client` returns JSON.\n- `https://matrix.live.bstein.dev/.well-known/matrix/client` returns JSON.\n\n5) TURN reachability:\n- Confirm `turn.live.bstein.dev:3478` and `turns:5349` are reachable from WAN."
  },
  {
    "path": "runbooks/kb-authoring.md",
    "title": "KB authoring: what to write (and what not to)",
    "tags": [
      "atlas",
      "kb",
      "runbooks"
    ],
    "entrypoints": [],
    "source_paths": [
      "knowledge/runbooks",
      "scripts/render/knowledge_render_atlas.py"
    ],
    "body": "# KB authoring: what to write (and what not to)\n\n## The goal\nGive Atlas assistants enough grounded, Atlas-specific context to answer \u201chow do I\u2026?\u201d questions without guessing.\n\n## What to capture (high value)\n- User workflows: \u201cclick here, set X, expected result\u201d\n- Operator workflows: \u201cedit these files, reconcile this kustomization, verify with these commands\u201d\n- Wiring: \u201cthis host routes to this service; this service depends on Postgres/Vault/etc\u201d\n- Failure modes: exact error messages + the 2\u20135 checks that usually resolve them\n- Permissions: Keycloak groups/roles and what they unlock\n\n## What to avoid (low value / fluff)\n- Generic Kubernetes explanations (link to upstream docs instead)\n- Copy-pasting large manifests (prefer file paths + small snippets)\n- Anything that will drift quickly (render it from GitOps instead)\n- Any secret values (reference Secret/Vault locations by name only)\n\n## Document pattern (recommended)\nEach runbook should answer:\n- \u201cWhat is this?\u201d\n- \u201cWhat do users do?\u201d\n- \u201cWhat do operators change (where in Git)?\u201d\n- \u201cHow do we verify it works?\u201d\n- \u201cWhat breaks and how to debug it?\u201d"
  },
  {
    "path": "runbooks/observability.md",
    "title": "Observability: Grafana + VictoriaMetrics (how to query safely)",
    "tags": [
      "atlas",
      "monitoring",
      "grafana",
      "victoriametrics"
    ],
    "entrypoints": [
      "metrics.bstein.dev",
      "alerts.bstein.dev"
    ],
    "source_paths": [
      "services/monitoring"
    ],
    "body": "# Observability: Grafana + VictoriaMetrics (how to query safely)\n\n## Where it is configured\n- `services/monitoring/helmrelease.yaml` (Grafana + Alertmanager + VM values)\n- `services/monitoring/grafana-dashboard-*.yaml` (dashboards and their PromQL)\n\n## Using metrics as a \u201ctool\u201d for Atlas assistants\nThe safest pattern is: map a small set of intents \u2192 fixed PromQL queries, then summarize results.\n\nExamples (intents)\n- \u201cIs the cluster healthy?\u201d \u2192 node readiness + pod restart rate\n- \u201cWhy is Element Call failing?\u201d \u2192 LiveKit/coturn pod restarts + synapse errors + ingress 5xx\n- \u201cIs Jenkins slow?\u201d \u2192 pod CPU/memory + HTTP latency metrics (if exported)\n\n## Why dashboards are not the KB\nDashboards are great references, but the assistant should query VictoriaMetrics directly for live answers and keep the\nKB focused on wiring, runbooks, and stable conventions."
  },
  {
    "path": "runbooks/template.md",
    "title": "<short title>",
    "tags": [
      "atlas",
      "<service>",
      "<topic>"
    ],
    "entrypoints": [
      "<hostnames if relevant>"
    ],
    "source_paths": [
      "services/<svc>",
      "clusters/atlas/<...>"
    ],
    "body": "# <Short title>\n\n## What this is\n\n## For users (how to)\n\n## For operators (where configured)\n\n## Troubleshooting (symptoms \u2192 checks)"
  },
  {
    "path": "software/metis.md",
    "title": "metis",
    "tags": [],
    "entrypoints": [],
    "source_paths": [],
    "body": "# Metis (node recovery)\n\n## Node classes (current map)\n- rpi5 Ubuntu workers: titan-04,05,06,07,08,09,10,11 (Ubuntu 24.04.3, k3s agent)\n- rpi5 control-plane: titan-0a/0b/0c (Ubuntu 24.04.1, k3s server, control-plane taint)\n- rpi4 Armbian longhorn: titan-13/15/17/19 (Armbian 6.6.x, k3s agent, longhorn disks)\n- rpi4 Armbian standard: titan-12/14/18 (Armbian 6.6.x, k3s agent)\n- Jetson workers: titan-20/21 (Ubuntu 20.04.6, k3s agent)\n- amd64 agents: titan-22/24 (Debian 13, k3s agent)\n- Veles storage/simulation worker: titan-23 (Atlas worker with `oceanus` node-pool labels)\n- External/dedicated hosts: tethys, titan-db, titan-jh, plus any newcomers.\n\n### Jetson nodes (titan-20/21)\n- Ubuntu 20.04.6 (Focal), kernel 5.10.104-tegra, CRI containerd 2.0.5-k3s2, arch arm64.\n- Storage: NVMe 232G at / (ext4); onboard mmc partitions present but root on NVMe; 1.9T sda present (unused).\n- k3s agent with drop-in 99-nofile.conf.\n\n## Longhorn disk UUIDs (critical nodes)\n- titan-13: /mnt/astreae UUID=6031fa8b-f28c-45c3-b7bc-6133300e07c6 (ext4); /mnt/asteria UUID=cbd4989d-62b5-4741-8b2a-28fdae259cae (ext4)\n- titan-15: /mnt/astreae UUID=f3362f14-5822-449f-944b-ac570b5cd615 (ext4); /mnt/asteria UUID=9c5316e6-f847-4884-b502-11f2d0d15d6f (ext4)\n- titan-17: /mnt/astreae UUID=1fecdade-08b0-49cb-9ae3-be6c188b0a96 (ext4); /mnt/asteria UUID=2fe9f613-d372-47ca-b84f-82084e4edda0 (ext4)\n- titan-19: /mnt/astreae UUID=4890abb9-dda2-4f4f-9c0f-081ee82849cf (ext4); /mnt/asteria UUID=2b4ea28d-b0e6-4fa3-841b-cd7067ae9153 (ext4)\n\n## Metis repo (~/Development/metis)\n- CLI skeleton in Go (`cmd/metis`), inventory loader (`pkg/inventory`), plan builder (`pkg/plan`).\n- `inventory.example.yaml` shows expected schema (classes + per-node overlay, Longhorn disks, labels, taints).\n\n## Next implementation steps\n- Add per-class golden image refs and checksums (Harbor or file://) when ready.\n- Implement burn execution: download with checksum, write via dd/etcher-equivalent, mount boot/root to inject hostname/IP/k3s tokens/labels/taints, journald/GC drop-ins, and Longhorn fstab entries. Add Windows writer (diskpart + wmic) and Linux writer (dd + sgdisk) paths.\n- Add Keycloak/SSH bootstrap: ensure ssh user, authorized keys, and k3s token/URL injection for agents; control-plane restore path with etcd snapshot selection.\n- Add per-host inventory entries for tethys, titan-db, titan-jh, and future dedicated hosts once audited.\n\n## Node OS/Kernel/CRI snapshot (Jan 2026)\n- titan-04: Ubuntu 24.04.3 LTS, kernel 6.8.0-1031-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-05: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-06: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-07: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-08: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-09: Ubuntu 24.04.3 LTS, kernel 6.8.0-1031-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-0a: Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-0b: Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-0c: Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-10: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-11: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-12: Armbian 24.11.1 noble, kernel 6.6.60-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-13: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-14: Armbian 24.11.1 noble, kernel 6.6.60-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-15: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-17: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-18: Armbian 24.11.1 noble, kernel 6.6.60-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-19: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-20: Ubuntu 20.04.6 LTS, kernel 5.10.104-tegra, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-21: Ubuntu 20.04.6 LTS, kernel 5.10.104-tegra, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-22: Debian 13 (trixie), kernel 6.12.41+deb13-amd64, CRI containerd://2.0.5-k3s2, arch amd64\n- titan-24: Debian 13 (trixie), kernel 6.12.57+deb13-amd64, CRI containerd://2.0.5-k3s2, arch amd64\n\n\n### Dedicated and special-purpose hosts\n- titan-db: Ubuntu 24.10, kernel 6.11.0-1015-raspi, root on /dev/sda2 ext4 (465G), boot vfat /dev/sda1; PostgreSQL service enabled.\n- titan-jh: Arch Linux ARM (rolling), kernel 6.18.4-2-rpi, NVMe root ext4 238G (/), boot vfat 512M; ~495 packages installed (pacman -Q).\n- titan-23: Atlas worker carrying the `oceanus` node-pool labels for Veles storage/simulation placement.\n\n\n### Control plane Pis (titan-0a/0b/0c)\n- Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, containerd 2.0.5-k3s2.\n- Storage: 477G SSD root (/dev/sda2 ext4), /boot/firmware vfat (/dev/sda1). fstab uses LABEL=writable and LABEL=system-boot.\n- k3s server (control-plane taint expected); etcd snapshots not yet cataloged (TODO).\n\n\n## k3s versions\n- rpi5 workers/control-plane: k3s v1.33.3+k3s1 (crictl v1.31.0-k3s2)\n- rpi4 nodes: k3s v1.31.5+k3s1 (crictl v1.31.0-k3s2)\n- Jetson titan-20/21: k3s v1.33.3+k3s1 (per node info), crictl v1.31.0-k3s2"
  }
]