titan-iac/services/comms/knowledge/catalog/runbooks.json

[
  {
    "path": "runbooks/ci-gitea-jenkins.md",
    "title": "CI: Gitea \u2192 Jenkins pipeline",
    "tags": [
      "atlas",
      "ci",
      "gitea",
      "jenkins"
    ],
    "entrypoints": [
      "scm.bstein.dev",
      "ci.bstein.dev"
    ],
    "source_paths": [
      "services/gitea",
      "services/jenkins",
      "scripts/sync/jenkins_cred_sync.sh",
      "scripts/sync/gitea_cred_sync.sh"
    ],
    "body": "# CI: Gitea \u2192 Jenkins pipeline\n\n## What this is\nAtlas uses Gitea for source control and Jenkins for CI. Authentication is via Keycloak (SSO).\n\n## Where it is configured\n- Gitea manifests: `services/gitea/`\n- Jenkins manifests: `services/jenkins/`\n- Credential sync helpers: `scripts/sync/gitea_cred_sync.sh`, `scripts/sync/jenkins_cred_sync.sh`\n\n## What users do (typical flow)\n- Create a repo in Gitea.\n- Create/update a Jenkins job/pipeline that can fetch the repo.\n- Configure a webhook (or SCM polling) so pushes trigger builds.\n\n## Troubleshooting (common)\n- \u201cWebhook not firing\u201d: confirm ingress host, webhook URL, and Jenkins job is reachable.\n- \u201cAuth denied cloning\u201d: confirm Keycloak group membership and that Jenkins has a valid token/credential configured."
  },
  {
    "path": "runbooks/cluster-power-recovery.md",
    "title": "cluster-power-recovery",
    "tags": [],
    "entrypoints": [],
    "source_paths": [],
    "body": "Atlas Cluster Power Recovery (Graceful Shutdown/Startup)\n\nPurpose\n- Provide a safe operator flow for planned power events and cold-boot recovery.\n- Avoid the Flux/Gitea bootstrap deadlock by using a local bootstrap fallback path.\n- Break the Harbor self-hosting deadlock by seeding Harbor runtime images from a control-host bundle.\n- Refuse bootstrap when UPS charge is too low, and fall back to fast shutdown if a second outage hits mid-recovery.\n\nBootstrapping risk to remember\n- Flux source is Git over SSH to `scm.bstein.dev` (Gitea).\n- Gitea itself is a Flux-managed workload and depends on storage + database.\n- Harbor is also critical, but it is not part of the first recovery stage because Harbor serves its own runtime images.\n- On cold boot, if Flux cannot fetch source before Gitea is up, reconciliation can stall.\n- Recovery path: bring control plane and workers up, then locally apply minimal platform stack (`core -> helm -> longhorn -> metallb -> traefik -> vault-csi -> vault-injector -> vault -> postgres -> gitea`), then seed Harbor images onto the Harbor node from a control-host bundle, then resume/reconcile Flux. Harbor is a later recovery stage after storage, Vault, Postgres, and Gitea are back.\n\nScript\n- `scripts/ops/cluster_power_recovery.sh`\n- `scripts/ops/cluster_power_console.sh`\n- Modes:\n  - `prepare`\n  - `shutdown`\n  - `harbor-seed`\n  - `startup`\n  - `status`\n- Default is dry-run. Add `--execute` to actually perform actions.\n\nDry-run examples\n- Shutdown preview:\n  - `scripts/ops/cluster_power_recovery.sh shutdown --skip-etcd-snapshot --skip-drain`\n- Startup preview:\n  - `scripts/ops/cluster_power_recovery.sh startup`\n- Harbor seed preview:\n  - `scripts/ops/cluster_power_recovery.sh harbor-seed`\n\nExecute examples\n- Prepare helper image on every node:\n  - `scripts/ops/cluster_power_recovery.sh prepare --execute`\n- Seed Harbor runtime images onto `titan-05` from the control-host bundle:\n  - `scripts/ops/cluster_power_recovery.sh harbor-seed --execute`\n- Planned shutdown:\n  - `scripts/ops/cluster_power_recovery.sh shutdown --execute`\n- Planned startup (canonical branch):\n  - `scripts/ops/cluster_power_recovery.sh startup --execute --force-flux-branch main`\n\nManual remote console examples\n- Canonical operator hosts:\n  - `titan-db`\n  - `tethys` (`titan-24`)\n- Both hosts now have:\n  - `~/ananke-tools/cluster_power_recovery.sh`\n  - `~/ananke-tools/cluster_power_console.sh`\n  - `~/ananke-tools/bootstrap/recovery-config.env`\n  - `~/ananke-tools/bootstrap/harbor-bootstrap-images.txt`\n  - `~/ananke-tools/kubeconfig`\n  - `~/ananke-cluster-power`\n  - `~/bin/ananke-cluster-power`\n  - `~/ananke-repo/{infrastructure,services,scripts}`\n- Both hosts also keep the Harbor bootstrap bundle at:\n  - `~/.local/share/ananke/bundles/harbor-bootstrap-v2.14.1-arm64.tar.zst`\n- Remote usage:\n  - `ssh titan-db`\n  - `~/ananke-cluster-power status`\n  - `~/ananke-cluster-power prepare --execute`\n  - `~/ananke-cluster-power shutdown --execute`\n  - `~/ananke-cluster-power startup --execute --force-flux-branch main`\n  - `ssh tethys`\n  - `~/ananke-cluster-power status`\n  - `~/ananke-cluster-power prepare --execute`\n  - `~/ananke-cluster-power shutdown --execute`\n  - `~/ananke-cluster-power startup --execute --force-flux-branch main`\n\nUseful options\n- `--shutdown-mode host-poweroff|cluster-only`\n- `--expected-flux-branch main`\n- `--expected-flux-url ssh://git@scm.bstein.dev:2242/bstein/titan-iac.git`\n- `--force-flux-url ssh://git@scm.bstein.dev:2242/bstein/titan-iac.git`\n- `--force-flux-branch main`\n- `--allow-flux-source-mutation` (required with `--force-flux-url`; breakglass only)\n- `--skip-local-bootstrap` (not recommended for cold-start recovery)\n- `--skip-harbor-bootstrap` (skip the Harbor recovery stage if you know Harbor should stay deferred)\n- `--skip-harbor-seed` (skip bundle import if Harbor images are already cached on the target node)\n- `--skip-helper-prewarm`\n- `--min-startup-battery 35`\n- `--ups-host pyrphoros@localhost`\n- `--require-ups-
  },
  {
    "path": "runbooks/comms-verify.md",
    "title": "Othrys verification checklist",
    "tags": [
      "comms",
      "matrix",
      "element",
      "livekit"
    ],
    "entrypoints": [
      "https://live.bstein.dev",
      "https://matrix.live.bstein.dev"
    ],
    "source_paths": [],
    "body": "1) Guest join:\n- Open a private window and visit:\n  `https://live.bstein.dev/#/room/#othrys:live.bstein.dev?action=join`\n- Confirm the guest join flow works and the displayname becomes `<word>-<word>`.\n\n2) Keycloak login:\n- Log in from `https://live.bstein.dev` and confirm MAS -> Keycloak -> Element redirect.\n\n3) Video rooms:\n- Start an Element Call room and confirm audio/video with a second account.\n- Check that guests can read public rooms but cannot start calls.\n\n4) Well-known:\n- `https://live.bstein.dev/.well-known/matrix/client` returns JSON.\n- `https://matrix.live.bstein.dev/.well-known/matrix/client` returns JSON.\n\n5) TURN reachability:\n- Confirm `turn.live.bstein.dev:3478` and `turns:5349` are reachable from WAN."
  },
  {
    "path": "runbooks/kb-authoring.md",
    "title": "KB authoring: what to write (and what not to)",
    "tags": [
      "atlas",
      "kb",
      "runbooks"
    ],
    "entrypoints": [],
    "source_paths": [
      "knowledge/runbooks",
      "scripts/render/knowledge_render_atlas.py"
    ],
    "body": "# KB authoring: what to write (and what not to)\n\n## The goal\nGive Atlas assistants enough grounded, Atlas-specific context to answer \u201chow do I\u2026?\u201d questions without guessing.\n\n## What to capture (high value)\n- User workflows: \u201cclick here, set X, expected result\u201d\n- Operator workflows: \u201cedit these files, reconcile this kustomization, verify with these commands\u201d\n- Wiring: \u201cthis host routes to this service; this service depends on Postgres/Vault/etc\u201d\n- Failure modes: exact error messages + the 2\u20135 checks that usually resolve them\n- Permissions: Keycloak groups/roles and what they unlock\n\n## What to avoid (low value / fluff)\n- Generic Kubernetes explanations (link to upstream docs instead)\n- Copy-pasting large manifests (prefer file paths + small snippets)\n- Anything that will drift quickly (render it from GitOps instead)\n- Any secret values (reference Secret/Vault locations by name only)\n\n## Document pattern (recommended)\nEach runbook should answer:\n- \u201cWhat is this?\u201d\n- \u201cWhat do users do?\u201d\n- \u201cWhat do operators change (where in Git)?\u201d\n- \u201cHow do we verify it works?\u201d\n- \u201cWhat breaks and how to debug it?\u201d"
  },
  {
    "path": "runbooks/observability.md",
    "title": "Observability: Grafana + VictoriaMetrics (how to query safely)",
    "tags": [
      "atlas",
      "monitoring",
      "grafana",
      "victoriametrics"
    ],
    "entrypoints": [
      "metrics.bstein.dev",
      "alerts.bstein.dev"
    ],
    "source_paths": [
      "services/monitoring"
    ],
    "body": "# Observability: Grafana + VictoriaMetrics (how to query safely)\n\n## Where it is configured\n- `services/monitoring/helmrelease.yaml` (Grafana + Alertmanager + VM values)\n- `services/monitoring/grafana-dashboard-*.yaml` (dashboards and their PromQL)\n\n## Using metrics as a \u201ctool\u201d for Atlas assistants\nThe safest pattern is: map a small set of intents \u2192 fixed PromQL queries, then summarize results.\n\nExamples (intents)\n- \u201cIs the cluster healthy?\u201d \u2192 node readiness + pod restart rate\n- \u201cWhy is Element Call failing?\u201d \u2192 LiveKit/coturn pod restarts + synapse errors + ingress 5xx\n- \u201cIs Jenkins slow?\u201d \u2192 pod CPU/memory + HTTP latency metrics (if exported)\n\n## Why dashboards are not the KB\nDashboards are great references, but the assistant should query VictoriaMetrics directly for live answers and keep the\nKB focused on wiring, runbooks, and stable conventions."
  },
  {
    "path": "runbooks/template.md",
    "title": "<short title>",
    "tags": [
      "atlas",
      "<service>",
      "<topic>"
    ],
    "entrypoints": [
      "<hostnames if relevant>"
    ],
    "source_paths": [
      "services/<svc>",
      "clusters/atlas/<...>"
    ],
    "body": "# <Short title>\n\n## What this is\n\n## For users (how to)\n\n## For operators (where configured)\n\n## Troubleshooting (symptoms \u2192 checks)"
  },
  {
    "path": "software/metis.md",
    "title": "metis",
    "tags": [],
    "entrypoints": [],
    "source_paths": [],
    "body": "# Metis (node recovery)\n\n## Node classes (current map)\n- rpi5 Ubuntu workers: titan-04,05,06,07,08,09,10,11 (Ubuntu 24.04.3, k3s agent)\n- rpi5 control-plane: titan-0a/0b/0c (Ubuntu 24.04.1, k3s server, control-plane taint)\n- rpi4 Armbian longhorn: titan-13/15/17/19 (Armbian 6.6.x, k3s agent, longhorn disks)\n- rpi4 Armbian standard: titan-12/14/18 (Armbian 6.6.x, k3s agent)\n- Jetson workers: titan-20/21 (Ubuntu 20.04.6, k3s agent)\n- amd64 agents: titan-22/24 (Debian 13, k3s agent)\n- Veles storage/simulation worker: titan-23 (Atlas worker with `oceanus` node-pool labels)\n- External/dedicated hosts: tethys, titan-db, titan-jh, plus any newcomers.\n\n### Jetson nodes (titan-20/21)\n- Ubuntu 20.04.6 (Focal), kernel 5.10.104-tegra, CRI containerd 2.0.5-k3s2, arch arm64.\n- Storage: NVMe 232G at / (ext4); onboard mmc partitions present but root on NVMe; 1.9T sda present (unused).\n- k3s agent with drop-in 99-nofile.conf.\n\n## Longhorn disk UUIDs (critical nodes)\n- titan-13: /mnt/astreae UUID=6031fa8b-f28c-45c3-b7bc-6133300e07c6 (ext4); /mnt/asteria UUID=cbd4989d-62b5-4741-8b2a-28fdae259cae (ext4)\n- titan-15: /mnt/astreae UUID=f3362f14-5822-449f-944b-ac570b5cd615 (ext4); /mnt/asteria UUID=9c5316e6-f847-4884-b502-11f2d0d15d6f (ext4)\n- titan-17: /mnt/astreae UUID=1fecdade-08b0-49cb-9ae3-be6c188b0a96 (ext4); /mnt/asteria UUID=2fe9f613-d372-47ca-b84f-82084e4edda0 (ext4)\n- titan-19: /mnt/astreae UUID=4890abb9-dda2-4f4f-9c0f-081ee82849cf (ext4); /mnt/asteria UUID=2b4ea28d-b0e6-4fa3-841b-cd7067ae9153 (ext4)\n\n## Metis repo (~/Development/metis)\n- CLI skeleton in Go (`cmd/metis`), inventory loader (`pkg/inventory`), plan builder (`pkg/plan`).\n- `inventory.example.yaml` shows expected schema (classes + per-node overlay, Longhorn disks, labels, taints).\n\n## Next implementation steps\n- Add per-class golden image refs and checksums (Harbor or file://) when ready.\n- Implement burn execution: download with checksum, write via dd/etcher-equivalent, mount boot/root to inject hostname/IP/k3s tokens/labels/taints, journald/GC drop-ins, and Longhorn fstab entries. Add Windows writer (diskpart + wmic) and Linux writer (dd + sgdisk) paths.\n- Add Keycloak/SSH bootstrap: ensure ssh user, authorized keys, and k3s token/URL injection for agents; control-plane restore path with etcd snapshot selection.\n- Add per-host inventory entries for tethys, titan-db, titan-jh, and future dedicated hosts once audited.\n\n## Node OS/Kernel/CRI snapshot (Jan 2026)\n- titan-04: Ubuntu 24.04.3 LTS, kernel 6.8.0-1031-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-05: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-06: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-07: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-08: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-09: Ubuntu 24.04.3 LTS, kernel 6.8.0-1031-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-0a: Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-0b: Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-0c: Ubuntu 24.04.1 LTS, kernel 6.8.0-1038-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-10: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-11: Ubuntu 24.04.3 LTS, kernel 6.8.0-1039-raspi, CRI containerd://2.0.5-k3s2, arch arm64\n- titan-12: Armbian 24.11.1 noble, kernel 6.6.60-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-13: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-14: Armbian 24.11.1 noble, kernel 6.6.60-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-15: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23-k3s2, arch arm64\n- titan-17: Armbian 25.2.1 noble, kernel 6.6.63-current-bcm2711, CRI containerd://1.7.23
  }
]