107 lines
4.0 KiB
Markdown
107 lines
4.0 KiB
Markdown
# ananke
|
|
|
|
`ananke` is the host-side power + bootstrap orchestrator for Titan.
|
|
|
|
It runs outside Kubernetes (systemd on host), so it can:
|
|
- shut the cluster down gracefully before battery/runtime redlines
|
|
- bring the cluster back after power returns
|
|
- recover common Flux/Kustomize startup deadlocks
|
|
- validate service health from the outside before declaring startup done
|
|
|
|
## Why `ananke`
|
|
|
|
I wanted a name that fits Titan/mythology, but also describes what this service actually does.
|
|
|
|
In Greek myth, **Ananke** is inevitability/necessity. That matches this tool: when power events happen, graceful sequencing is not optional.
|
|
|
|
UPS names in this cluster are also part of the story:
|
|
- `Statera`: powers `titan-23`, `titan-24`, `titan-jh`
|
|
- `Pyrphoros`: powers all other nodes
|
|
|
|
## Breakglass reminder
|
|
|
|
Vault unseal breakglass is wired for remote retrieval (magic mirror host). If local key retrieval fails, Ananke can use the configured breakglass command.
|
|
|
|
## What “startup complete” means now
|
|
|
|
Ananke does **not** stop at “Flux says Ready”. Startup only completes when all configured gates pass:
|
|
- node inventory preflight passes (host mapping + ssh user + port sanity)
|
|
- node SSH auth gate passes (real command execution, not just TCP)
|
|
- Flux source drift guard passes (`expected_flux_source_url` + branch expectation)
|
|
- Flux kustomizations are healthy
|
|
- controller convergence is healthy (deployments/statefulsets/daemonsets)
|
|
- ingress checklist passes (all discovered ingress hosts reachable with accepted status)
|
|
- external service checklist passes (Gitea, Grafana, Keycloak OIDC, Harbor registry auth challenge, Longhorn auth redirect)
|
|
- stability soak window passes (no regressions, no CrashLoop/ImagePull failures)
|
|
|
|
During startup, Ananke also auto-heals known failure patterns (stuck controller pods, immutable Flux Jobs, critical workloads scaled to zero) and writes a report:
|
|
- `/var/lib/ananke/last-startup-report.json`
|
|
|
|
If any gate fails, startup is blocked with a concrete reason.
|
|
|
|
## Command quick sheet
|
|
|
|
From `titan-db` (coordinator):
|
|
|
|
```bash
|
|
sudo /usr/local/bin/ananke status --config /etc/ananke/ananke.yaml
|
|
sudo /usr/local/bin/ananke startup --config /etc/ananke/ananke.yaml --execute --force-flux-branch main
|
|
sudo /usr/local/bin/ananke shutdown --config /etc/ananke/ananke.yaml --execute --reason graceful-maintenance --mode cluster-only
|
|
sudo /usr/local/bin/ananke shutdown --config /etc/ananke/ananke.yaml --execute --reason emergency-power --mode poweroff --skip-drain --skip-etcd-snapshot
|
|
```
|
|
|
|
From `titan-24` (`tethys` peer):
|
|
|
|
```bash
|
|
sudo /usr/local/bin/ananke shutdown --config /etc/ananke/ananke.yaml --execute --reason graceful-maintenance --mode cluster-only
|
|
```
|
|
|
|
Systemd:
|
|
|
|
```bash
|
|
sudo systemctl status ananke.service
|
|
sudo systemctl start ananke-bootstrap.service
|
|
sudo systemctl start ananke-update.service
|
|
```
|
|
|
|
## Shutdown modes (explicit)
|
|
|
|
`ananke shutdown` now supports explicit mode selection:
|
|
- default behavior is `cluster-only` (host poweroff is not performed)
|
|
- `--mode config`: use config default (`shutdown.poweroff_enabled`)
|
|
- `--mode cluster-only`: stop cluster services only (no host poweroff)
|
|
- `--mode poweroff`: include host poweroff path (explicit only)
|
|
|
|
This removes ambiguity during drills.
|
|
|
|
## Config file
|
|
|
|
Primary path:
|
|
- `/etc/ananke/ananke.yaml`
|
|
|
|
Core settings to keep accurate:
|
|
- `expected_flux_branch`
|
|
- `expected_flux_source_url`
|
|
- `startup.require_node_ssh_auth`
|
|
- `startup.require_ingress_checklist`
|
|
- `startup.service_checklist`
|
|
- `startup.service_checklist_stability_seconds`
|
|
- `startup.ignore_unavailable_nodes` (for planned temporary node outages)
|
|
- `coordination.role`, `coordination.peer_hosts`
|
|
|
|
## Install / update
|
|
|
|
```bash
|
|
sudo ./scripts/install.sh
|
|
```
|
|
|
|
Installer behavior:
|
|
- builds and installs `/usr/local/bin/ananke`
|
|
- installs `ananke*.service` units
|
|
- migrates and enforces current `ananke` config/state paths
|
|
|
|
## Notes
|
|
|
|
- Apply changes through Git/Flux manifests; avoid manual in-cluster edits for durable changes.
|
|
- For controlled shutdown/startup drills, treat any manual intervention as a bug and fold the logic back into Ananke.
|