208 lines
23 KiB
Markdown
208 lines
23 KiB
Markdown
# Ananke Recovery Notes - 2026-07-07
|
|
|
|
## Context
|
|
|
|
- Titan cluster lost power briefly; Ariadne may have started shutdown behavior.
|
|
- User goal: let one Ananke recovery process bring the cluster back to health, monitor it, and record any manual help or automation gaps.
|
|
- Operator view is from `/home/brad/Development/bstein_dev_home`; Kubernetes API is reachable through `/home/brad/titan/kubeconfig`.
|
|
|
|
## Current Ananke Run
|
|
|
|
- Coordinator host: `titan-db`.
|
|
- `ananke.service` daemon is active.
|
|
- `ananke-bootstrap.service` is active and running:
|
|
- command: `/usr/local/bin/ananke startup --config /etc/ananke/ananke.yaml --execute --force-flux-branch main --auto-peer-failover --peer-wait-seconds 180`
|
|
- current run started: `2026-07-07T17:19:50Z`
|
|
- current phase observed: `critical-workloads`
|
|
- checklist observed: `10 passed / 0 failed / 2 running`
|
|
- auto-heals observed: pod recycling for `VaultInitStuck`, `ContainerRuntimeWedge`, `CrashLoopBackOff`, stale terminating pods, and Longhorn attach blocked on unready nodes.
|
|
- previous run failed at `2026-07-07T17:19:19Z` waiting on `logging/deployment/oauth2-proxy-logs`; systemd restarted the bootstrap service.
|
|
|
|
## Observed Recovery Progress
|
|
|
|
- Latest snapshot at approximately `2026-07-07T17:58Z`:
|
|
- Ananke is still running in `convergence-checks` with `17/19` checks passed, `0` failed, and `2` running.
|
|
- Ananke has recorded `31` auto-heal actions, mostly repeated stuck-pod recycling.
|
|
- `titan-22` and `titan-24` both recovered to `Ready=True` with fresh kubelet transitions (`titan-22` at `2026-07-07T17:56:29Z`, `titan-24` at `2026-07-07T17:55:11Z`).
|
|
- Metrics server is running and `kubectl top nodes` works for recovered nodes; `titan-22` had just rejoined and metrics may lag briefly.
|
|
- Remaining convergence blocker is mostly app workload readiness, led by `finance/firefly`.
|
|
- Ananke completed the startup workflow successfully at `2026-07-07T17:59:00Z`:
|
|
- `startup_status=success`
|
|
- `startup_phase=complete`
|
|
- `startup_checklist_total=20 passed=19 failed=0 running=1`
|
|
- `startup_auto_heals=33`
|
|
- `intent=normal`
|
|
- Vault recovered:
|
|
- `vault/vault-0` is `1/1 Running` on `titan-18`.
|
|
- Vault status reports `initialized=true`, `sealed=false`, `ha_enabled=true`.
|
|
- Vault became active at `2026-07-07T17:05:46Z`.
|
|
- `vault-k8s-auth-config-autoheal-*` completed.
|
|
- Vault injector recovered after `titan-04` kubelet recovery:
|
|
- `vault-injector-agent-injector-*` became `1/1 Running`.
|
|
- Vault service endpoints and injector webhook endpoints are populated.
|
|
- Vault-backed workloads started moving again:
|
|
- Ariadne's Vault init authenticated successfully and rendered `/vault/secrets/ariadne-env.sh`.
|
|
- Ariadne is now `2/2 Running` on `titan-08`.
|
|
- Flux is mostly reconciled, but several app kustomizations are still waiting on app readiness or dependencies.
|
|
- Longhorn no longer shows the earlier broad faulted volume picture; observed state later was `41 attached`, `1 attaching`, `47 detached`.
|
|
|
|
## Current Known Issues
|
|
|
|
- `titan-22` and `titan-24` are still `NotReady` / `Unknown` from Kubernetes.
|
|
- `titan-05` is `Unknown` after power-event recovery and k3s-agent restart attempts:
|
|
- Ananke SSH route on port `2277` reaches the host.
|
|
- `systemctl restart k3s-agent` timed out while stopping the old process.
|
|
- systemd reported `Processes still around after SIGKILL`, then spawned a new `/usr/local/bin/k3s agent`.
|
|
- `k3s-agent.service` remained `activating/start`; local `127.0.0.1:6444` was open but kubelet `127.0.0.1:10250` was closed.
|
|
- Kubernetes node heartbeat stayed stale at `2026-07-07T17:21:57Z` with `node.kubernetes.io/unreachable` taints.
|
|
- `maintenance/k3s-agent-restart-*` on `titan-05` stayed `ContainerCreating`, which confirms the in-cluster helper cannot repair this class once the node runtime is wedged.
|
|
- `titan-15` and `titan-17` are `Ready,SchedulingDisabled`; Ananke cordoned them because encrypted Longhorn mounts exposed missing `cryptsetup`.
|
|
- Ananke tried to install `cryptsetup-bin` on `titan-15` and `titan-17`, but the SSH repair failed because `sudo` required a password for `atlas`.
|
|
- Several workloads are still unready while Ananke recycles stale `VaultInitStuck`, `CrashLoopBackOff`, and `ImagePullBackOff` pods.
|
|
|
|
## Manual Actions Taken
|
|
|
|
- Verified cluster health with `kubectl` and Flux commands.
|
|
- Verified Vault directly with `vault status` inside `vault-0`.
|
|
- Verified Ananke coordinator service state through `titan-db`.
|
|
- Did not start a second Ananke recovery process after discovering the active `ananke-bootstrap.service`.
|
|
- Used the existing `maintenance/k3s-agent-restart` DaemonSet as a targeted host-level nudge by deleting its pods on `titan-04`, `titan-05`, `titan-06`, `titan-08`, and `titan-18`; this reruns the DaemonSet's `k3s-agent` restart command on those nodes.
|
|
- The in-cluster helper successfully reran on `titan-08`, but helper startup/termination was itself wedged on other affected nodes. Queued direct `systemctl --no-block restart k3s-agent` over Ananke SSH for `titan-05`, `titan-06`, and `titan-18`. A direct restart attempt against `titan-04` hung and was stopped.
|
|
- After discovering Ananke's SSH profile uses `atlas@node:2277`, queued `systemctl --no-block restart k3s-agent` on `titan-04` and `titan-05` through the same route.
|
|
- `titan-04` recovered to `Ready` after the non-blocking restart and its `maintenance/k3s-agent-restart-*` pod became `Running`.
|
|
- `titan-05` did not recover: systemd timed out killing the old `k3s-agent`, left containerd shims behind, and started a new agent process without reopening kubelet port `10250` or restoring node heartbeats.
|
|
- A controlled reboot was issued for `titan-05` through Ananke's SSH route at approximately `2026-07-07T17:32Z` because the node met the proposed escalation criteria: stale Kubernetes heartbeat, kubelet `10250` closed, `k3s-agent.service` stuck in `activating/start`, and the in-cluster helper stuck `ContainerCreating`.
|
|
- After the reboot command returned, `titan-05` temporarily stopped responding to ping and SSH, then recovered to Kubernetes `Ready=True` with fresh heartbeats and no `unreachable` taints by approximately `2026-07-07T17:35Z`.
|
|
- Ananke later detected and logged `cordoned container-runtime-wedged node(s): titan-07 pods=4`; direct host checks showed `titan-07` reachable, `k3s-agent` active/running, and kubelet `10250` open. This should become a false-positive/auto-uncordon test case if Kubernetes readiness remains healthy.
|
|
- Later `titan-07` showed a real runtime/API-state split: `oauth2-proxy-logs` reported container state `running`, but `kubectl exec` failed with `cannot exec in a deleted state` and `kubectl logs --previous` could not retrieve the container logs. Manual assist: queued `systemctl --no-block restart k3s-agent` on `titan-07` and deleted the stale `logging/oauth2-proxy-logs-*` pod.
|
|
- `titan-07` recovered to `Ready=True` with `k3s-agent` active and kubelet `10250` open. `logging/oauth2-proxy-logs` was recreated on `titan-12` and became `2/2 Running`.
|
|
- Ananke repeatedly failed and restarted on `logging/deployment/oauth2-proxy-logs` because the replacement pod on `titan-14` hit the same `failed to reserve container name` / `CreateContainerError` runtime wedge signature.
|
|
- Manual assist: queued `systemctl --no-block restart k3s-agent` on `titan-14` through Ananke SSH and deleted the stuck `logging/oauth2-proxy-logs-5b4d4f87db-2tdzd` pod with `--wait=false` so the controller can recreate it after the runtime reset.
|
|
- Follow-up on `titan-14`: the direct SSH command timed out from the operator view, but a later status check showed `k3s-agent.service` had a new MainPID and was `activating/start`; the existing `maintenance/k3s-agent-restart` DaemonSet pod for `titan-14` was also deleted to rerun the in-cluster helper.
|
|
- `titan-14` recovered to fresh `Ready=True` heartbeats and the replacement `maintenance/k3s-agent-restart-*` pod became `Running`.
|
|
- After the `titan-05` reboot, `logging/opensearch-0` recovered to `1/1 Running`; `logging/oauth2-proxy-logs` was recreated on `titan-07` and was still initializing at the time of note.
|
|
- Ananke also entered a short retry loop where `kubectl exec vault-0 ... vault status` was killed even though Vault HTTP health through the Kubernetes pod proxy reported `initialized=true`, `sealed=false`, `standby=false`. This should become a separate robustness fix: startup should not fail outright on a transient killed `kubectl exec` when service endpoints/HTTP health prove Vault is active.
|
|
- After Keycloak completed its slow post-attach startup at approximately `2026-07-07T17:45:55Z`, OIDC-dependent OAuth proxies began recovering.
|
|
- Ananke advanced past `critical-workloads` into `convergence-checks` by approximately `2026-07-07T17:50:50Z`.
|
|
- Manual assist: force-deleted stale pod `finance/firefly-6669f46874-q8rtq` after Ananke repeatedly issued ordinary deletes but the pod remained `Terminating` with no finalizers and continued to block `firefly-storage` attachment. The Firefly Longhorn volume moved from `titan-04` to `titan-06` and reported `attached`/`healthy` on the replacement node.
|
|
- Post-Ananke-completion manual assist: `titan-04` was still `Ready` with kubelet `10250` open, but several newly scheduled pods showed `FailedCreatePodSandBox`, `failed to reserve sandbox name`, and `context deadline exceeded`. Queued `systemctl --no-block restart k3s-agent` on `titan-04` at approximately `2026-07-07T18:02:55Z`.
|
|
- `titan-04` did not recover from the non-blocking `k3s-agent` restart. It moved to Kubernetes `Ready=Unknown` with `node.kubernetes.io/unreachable` taints; host checks showed `k3s-agent.service` stuck in `deactivating/stop-sigkill`, `Result=timeout`, and kubelet `10250` closed. Manual assist: issued a controlled reboot for `titan-04` through Ananke's SSH route shortly after `2026-07-07T18:06Z`.
|
|
- `titan-04` recovered after reboot by approximately `2026-07-07T18:10Z`: Kubernetes reported `Ready=True`, taints cleared, `k3s-agent` was `active/running`, and kubelet `10250` was open.
|
|
- Firefly hit a second stale-owner handoff after the first cleanup. Replacement pod `firefly-6669f46874-pjhg6` on `titan-07` was blocked while terminating pod `firefly-6669f46874-lrz5d` on `titan-06` still had the main `firefly` container running and mounted `firefly-storage`. No force delete was issued for this second handoff because the application container was live and PVC-writing. The old pod later disappeared on its own; Longhorn then attached `pvc-358b4319-60cb-4322-b948-776c34a414a9` to `titan-07` as `attached`/`healthy`.
|
|
- Attempted to inspect container runtime state on `titan-06` with `crictl`, but `atlas` required a sudo password. This should be part of the automation fix: Ananke either needs a noninteractive, bounded runtime-inspection primitive or must report that it cannot prove a PVC-writing container is stopped.
|
|
- Manual assist: uncordoned `titan-15` and `titan-17` after confirming Kubernetes `Ready=True`, Longhorn `allowScheduling=true`, and Longhorn disks ready. The remaining blocker for Longhorn on both nodes was Kubernetes `SchedulingDisabled`; after uncordon, Longhorn `Schedulable=True` on both nodes.
|
|
- Manual assist: resolved the `titan-22` Kubernetes/Longhorn readiness discrepancy. Kubernetes reported `Ready=True`, but Longhorn node `Ready=False` with `ManagerPodMissing`; the Longhorn manager DaemonSet selected `longhorn-host=true`, and `titan-22` was missing that label. Restored `longhorn-host=true`, which started `longhorn-manager-jtsjw` on `titan-22` and brought Longhorn node `Ready=True`.
|
|
- Manual assist: deleted pending controller-owned pod `crypto/wallet-monero-temp-844bd949c7-28r4p` to force a fresh volume-attach retry after the `titan-22` Longhorn manager came up.
|
|
- Manual assist: deleted stale unattached VolumeAttachment `csi-0bac99e18b687f6a50647dd959c26d6e2a1b2bf1a4f844de4f646cf3408d41c5` after it retained the pre-fix "node titan-22 is not ready" Longhorn error. Longhorn then attached `pvc-c6a4bf9b-17c7-4bdf-8a88-ca460ad0da5b` to `titan-22` as `attached`/`healthy`.
|
|
|
|
## Automation Growth Opportunities
|
|
|
|
- Ananke should expose from a remote operator machine whether the coordinator bootstrap is already running, so an operator does not risk starting a duplicate recovery.
|
|
- The local Ananke config path assumes `/etc/ananke/ananke.yaml` and `/var/lib/ananke`; operator-side `status` failed without root. A read-only remote status wrapper would make this smoother.
|
|
- Ananke should handle the `cryptsetup-bin` host repair on managed nodes without requiring interactive sudo, or clearly preflight and report the missing node privilege before a power event.
|
|
- Ananke should separate `Vault pod/data volume unavailable` from `Vault sealed` in status. It did recover correctly, but early status said the unseal check passed while deferring because `vault-0` was pending.
|
|
- Ananke should avoid repeatedly recycling the same ImagePullBackOff pods if the underlying registry/image problem is unchanged; this creates noisy recovery logs.
|
|
- Ananke should surface current blocking workloads in a compact operator summary rather than requiring journal inspection.
|
|
- Ananke should detect pods stuck in `PodInitializing` after init completion, especially when events show `FailedCreatePodSandbox`, `failed to reserve sandbox name`, `FailedKillPod`, or image pulls that never transition to `ImagePullBackOff`.
|
|
- Ananke should be able to rerun the existing `k3s-agent-restart` maintenance primitive on affected nodes when containerd/kubelet sandbox state is wedged after a power event.
|
|
- Ananke should fall back from the in-cluster restart helper to SSH `systemctl --no-block restart k3s-agent` when the helper pod cannot start because kubelet/containerd is already wedged. Blocking `systemctl restart` over SSH can hang the operator path.
|
|
- Ananke should check host and CoreDNS upstream DNS health separately from node SSH reachability. During this incident, node IP reachability passed while hostname resolution and image pulls were intermittently failing.
|
|
- Ananke should stop repeatedly issuing ordinary pod deletes when the same controller pod remains `Terminating` and continues to own exclusive resources. It should escalate to a stale-owner cleanup flow that proves the old workload is not writing, force-removes the orphaned API object when safe, then verifies Longhorn detach/attach and replacement pod startup.
|
|
- Ananke should reconcile Kubernetes node labels against Longhorn manager scheduling requirements. During this incident `titan-22` was Kubernetes `Ready=True`, had CSI/engine Longhorn pods, and had a Longhorn Node CR, but no `longhorn-manager` pod because `longhorn-host=true` was missing. Kubernetes Ready alone was not sufficient for PVC workloads.
|
|
- Ananke should understand that Kubernetes cordon directly affects Longhorn node schedulability. `titan-15` and `titan-17` had Longhorn-ready disks and `allowScheduling=true`, but Longhorn `Schedulable=False` solely because the Kubernetes nodes were cordoned.
|
|
|
|
### Generalized Node Runtime Wedge Case
|
|
|
|
This incident exposed a reusable failure mode that Ananke should treat as a node-runtime recovery problem, not just an app-pod recovery problem.
|
|
|
|
Detection signals:
|
|
|
|
- Node is `Ready=Unknown` or has a stale heartbeat while SSH on the managed Ananke route still works.
|
|
- Host is pingable and local k3s load balancer port `127.0.0.1:6444` may be open, but kubelet port `127.0.0.1:10250` is closed or API proxy calls fail.
|
|
- `systemctl restart k3s-agent` hangs, or `systemctl show k3s-agent` reports `deactivating`, `final-sigterm`, `final-sigkill`, `activating/start`, or `Result=timeout` for longer than a bounded recovery window.
|
|
- `journalctl -u k3s-agent` includes `Processes still around after SIGKILL`, `State 'stop-sigterm' timed out`, or `State 'final-sigterm' timed out`.
|
|
- Pods on the node show `FailedCreatePodSandBox`, `failed to reserve sandbox name`, `CreateContainerError`, `FailedKillPod`, or long-lived `ContainerCreating`/`PodInitializing`.
|
|
- The `maintenance/k3s-agent-restart` DaemonSet pod for that node cannot start or stays `ContainerCreating`, which means Kubernetes-based repair is unavailable.
|
|
|
|
Suggested automated escalation ladder:
|
|
|
|
- Mark the node as in node-runtime recovery and pause noisy repeated pod recycling for pods pinned to that node.
|
|
- Cordon the node or confirm it already has `unreachable` taints before host-level repair.
|
|
- Try a bounded SSH health probe using Ananke's configured SSH user, port, config, and identity.
|
|
- Prefer `systemctl --no-block restart k3s-agent` over blocking `systemctl restart k3s-agent`.
|
|
- After a short wait, require both Kubernetes heartbeat freshness and local kubelet port `10250` to recover.
|
|
- If the service remains `deactivating`, `final-sigkill`, or `activating/start` with stale heartbeats, escalate to a configured host reboot path.
|
|
- After reboot, wait for SSH, k3s-agent active/running, kubelet `10250`, Kubernetes `Ready=True`, and removal of `unreachable` taints before resuming app-pod recycling for that node.
|
|
|
|
Implementation notes:
|
|
|
|
- This should be a generic managed-node repair primitive, not hard-coded to `titan-05`.
|
|
- The primitive should emit one compact operator summary: node, phase, trigger signals, action taken, elapsed time, and next escalation.
|
|
- Tests should cover a node with SSH available but kubelet closed, a blocking restart timeout, leftover shim evidence, a helper DaemonSet stuck in `ContainerCreating`, and successful recovery through reboot.
|
|
- The post-completion `titan-04` recurrence confirms the primitive must continue monitoring after Ananke declares cluster startup success. A Ready node with fresh-enough status can still be unable to create new sandboxes, and a later restart can transition it into the same hard `deactivating/stop-sigkill` state seen on `titan-05`.
|
|
|
|
### Generalized Stale Pod / Volume Ownership Case
|
|
|
|
This incident also exposed a reusable app-convergence failure mode that Ananke should treat as stale Kubernetes/runtime ownership rather than simple pod unhealthiness.
|
|
|
|
Concrete example:
|
|
|
|
- `finance/firefly-6669f46874-q8rtq` entered deletion at `2026-07-07T17:43:55Z` and remained `Terminating` for more than 25 minutes.
|
|
- The pod had no finalizers but still reported a running `vault-agent` sidecar on `titan-04`.
|
|
- The replacement pod `finance/firefly-6669f46874-lrz5d` on `titan-06` stayed `Init:0/2`.
|
|
- Events showed `FailedAttachVolume` / `Multi-Attach` for PVC `firefly-storage` because volume `pvc-358b4319-60cb-4322-b948-776c34a414a9` was still attached to `titan-04`.
|
|
- Longhorn reported the volume `attached`, `healthy`, current node `titan-04`.
|
|
- Ananke repeatedly logged ordinary recycling of the same stuck pod, but did not escalate to clearing the stale owner.
|
|
- Manual force deletion of the stale pod cleared the stale API owner and allowed Longhorn to attach `firefly-storage` to the replacement pod's node.
|
|
|
|
Detection signals:
|
|
|
|
- A controller-owned pod has `metadata.deletionTimestamp` older than a bounded threshold and no finalizers.
|
|
- A replacement pod from the same controller is pending or initializing on another node.
|
|
- Events include `Multi-Attach`, `FailedAttachVolume`, `Volume is already used by pod(s)`, or `Volume is already exclusively attached to one node`.
|
|
- The PVC is `ReadWriteOnce` and the old and new pod nodes differ.
|
|
- Longhorn/CNI/runtime state still shows the old node as the attachment owner.
|
|
- Ordinary `kubectl delete pod --wait=false` is repeated without changing the deletion timestamp or clearing the volume attachment.
|
|
|
|
Suggested automated escalation ladder:
|
|
|
|
- Group the old pod, replacement pod, PVC, VolumeAttachment, Longhorn volume, controller, and node into one recovery incident.
|
|
- Verify whether any application container that mounts the PVC is still running. Sidecars that do not mount the PVC should not block stale-owner cleanup by themselves.
|
|
- Check node health for the old owner node. If kubelet/runtime is unhealthy, route through the managed-node recovery primitive first.
|
|
- If the old owner node is healthy but the pod remains deleting past the threshold, issue a bounded force deletion for the stale pod.
|
|
- Wait for VolumeAttachment/Longhorn ownership to detach from the old node or attach to the replacement node.
|
|
- Recheck the replacement pod init sequence and controller readiness before declaring the app recovered.
|
|
- Emit one compact operator event instead of repeatedly logging the same pod recycle.
|
|
|
|
Implementation notes:
|
|
|
|
- This should be generic across any `ReadWriteOnce` PVC workload, not hard-coded to Firefly.
|
|
- The safety check should reason at container/mount level: a sidecar that only mounts service-account, Vault secret, or emptyDir volumes is materially different from an app container actively mounting the PVC.
|
|
- Tests should cover a terminating pod without finalizers holding an RWO PVC, an old node with healthy kubelet, an old node with wedged kubelet, a sidecar-only survivor, and a replacement pod blocked by `Multi-Attach`.
|
|
- Tests should also cover the unsafe variant where the terminating pod's application container is still running and mounts the PVC. In that case Ananke should wait, stop the container through a controlled runtime path if allowed, or report that it lacks the privilege to prove storage safety; it should not blindly force-delete the API object.
|
|
|
|
### Generalized Longhorn / Kubernetes Readiness Drift Case
|
|
|
|
This incident exposed a reusable storage readiness gap: Kubernetes `Ready=True` is not enough for PVC workloads when Longhorn's node model disagrees.
|
|
|
|
Detection signals:
|
|
|
|
- Kubernetes node is `Ready=True`, untainted, and schedulable, but Longhorn Node `Ready=False`.
|
|
- Longhorn Node reason is `ManagerPodMissing`, or the Longhorn manager DaemonSet has no pod on the node.
|
|
- Longhorn manager DaemonSet selects a node label such as `longhorn-host=true`, and the Kubernetes node is missing it.
|
|
- Longhorn CSI/engine pods or a Longhorn Node CR exist for the node, proving the node participates in Longhorn even though manager scheduling is broken.
|
|
- VolumeAttachment errors say Longhorn cannot attach because the node is not ready, despite Kubernetes Ready=True.
|
|
|
|
Suggested automated escalation ladder:
|
|
|
|
- Compare Kubernetes node readiness, labels, taints, Longhorn Node readiness/schedulability, Longhorn manager DaemonSet selector, manager pod presence, CSI pod presence, and VolumeAttachment errors.
|
|
- If desired-state evidence proves the node should be a Longhorn node, restore the missing manager selector label and wait for the manager pod and Longhorn Node `Ready=True`.
|
|
- If desired-state evidence is missing, report a clear unsafe label-drift blocker rather than guessing.
|
|
- After Longhorn Node `Ready=True`, retry stale VolumeAttachments by recreating the affected controller-owned pending pod or using a safe VolumeAttachment retry policy.
|
|
- For Kubernetes-cordoned Longhorn nodes with `allowScheduling=true` and ready disks, either complete the host repair and uncordon or report the explicit blocker; do not leave Longhorn capacity disabled silently.
|
|
|
|
Implementation notes:
|
|
|
|
- This must be generic across Longhorn nodes and manager DaemonSet selectors, not hard-coded to `titan-22` or `longhorn-host`.
|
|
- Tests should cover missing Longhorn selector labels, manager pod missing, Kubernetes Ready but Longhorn NotReady, Kubernetes cordon causing Longhorn Schedulable=False, and stale attach errors that clear after Longhorn readiness is restored.
|