ananke/docs/ananke-recovery-2026-07-07.md

Ananke Recovery Notes - 2026-07-07

Context

• Titan cluster lost power briefly; Ariadne may have started shutdown behavior.
• User goal: let one Ananke recovery process bring the cluster back to health, monitor it, and record any manual help or automation gaps.
• Operator view is from /home/brad/Development/bstein_dev_home; Kubernetes API is reachable through /home/brad/titan/kubeconfig.

Current Ananke Run

• Coordinator host: titan-db.
• ananke.service daemon is active.
• ananke-bootstrap.service is active and running:
  • command: /usr/local/bin/ananke startup --config /etc/ananke/ananke.yaml --execute --force-flux-branch main --auto-peer-failover --peer-wait-seconds 180
  • current run started: 2026-07-07T17:19:50Z
  • current phase observed: critical-workloads
  • checklist observed: 10 passed / 0 failed / 2 running
  • auto-heals observed: pod recycling for VaultInitStuck, ContainerRuntimeWedge, CrashLoopBackOff, stale terminating pods, and Longhorn attach blocked on unready nodes.
  • previous run failed at 2026-07-07T17:19:19Z waiting on logging/deployment/oauth2-proxy-logs; systemd restarted the bootstrap service.

Observed Recovery Progress

• Latest snapshot at approximately 2026-07-07T17:58Z:
  • Ananke is still running in convergence-checks with 17/19 checks passed, 0 failed, and 2 running.
  • Ananke has recorded 31 auto-heal actions, mostly repeated stuck-pod recycling.
  • titan-22 and titan-24 both recovered to Ready=True with fresh kubelet transitions (titan-22 at 2026-07-07T17:56:29Z, titan-24 at 2026-07-07T17:55:11Z).
  • Metrics server is running and kubectl top nodes works for recovered nodes; titan-22 had just rejoined and metrics may lag briefly.
  • Remaining convergence blocker is mostly app workload readiness, led by finance/firefly.
• Ananke completed the startup workflow successfully at 2026-07-07T17:59:00Z:
  • startup_status=success
  • startup_phase=complete
  • startup_checklist_total=20 passed=19 failed=0 running=1
  • startup_auto_heals=33
  • intent=normal
• Vault recovered:
  • vault/vault-0 is 1/1 Running on titan-18.
  • Vault status reports initialized=true, sealed=false, ha_enabled=true.
  • Vault became active at 2026-07-07T17:05:46Z.
  • vault-k8s-auth-config-autoheal-* completed.
• Vault injector recovered after titan-04 kubelet recovery:
  • vault-injector-agent-injector-* became 1/1 Running.
  • Vault service endpoints and injector webhook endpoints are populated.
• Vault-backed workloads started moving again:
  • Ariadne's Vault init authenticated successfully and rendered /vault/secrets/ariadne-env.sh.
  • Ariadne is now 2/2 Running on titan-08.
• Flux is mostly reconciled, but several app kustomizations are still waiting on app readiness or dependencies.
• Longhorn no longer shows the earlier broad faulted volume picture; observed state later was 41 attached, 1 attaching, 47 detached.

Current Known Issues

• titan-22 and titan-24 are still NotReady / Unknown from Kubernetes.
• titan-05 is Unknown after power-event recovery and k3s-agent restart attempts:
  • Ananke SSH route on port 2277 reaches the host.
  • systemctl restart k3s-agent timed out while stopping the old process.
  • systemd reported Processes still around after SIGKILL, then spawned a new /usr/local/bin/k3s agent.
  • k3s-agent.service remained activating/start; local 127.0.0.1:6444 was open but kubelet 127.0.0.1:10250 was closed.
  • Kubernetes node heartbeat stayed stale at 2026-07-07T17:21:57Z with node.kubernetes.io/unreachable taints.
  • maintenance/k3s-agent-restart-* on titan-05 stayed ContainerCreating, which confirms the in-cluster helper cannot repair this class once the node runtime is wedged.
• titan-15 and titan-17 are Ready,SchedulingDisabled; Ananke cordoned them because encrypted Longhorn mounts exposed missing cryptsetup.
• Ananke tried to install cryptsetup-bin on titan-15 and titan-17, but the SSH repair failed because sudo required a password for atlas.
• Several workloads are still unready while Ananke recycles stale VaultInitStuck, CrashLoopBackOff, and ImagePullBackOff pods.

Manual Actions Taken

• Verified cluster health with kubectl and Flux commands.
• Verified Vault directly with vault status inside vault-0.
• Verified Ananke coordinator service state through titan-db.
• Did not start a second Ananke recovery process after discovering the active ananke-bootstrap.service.
• Used the existing maintenance/k3s-agent-restart DaemonSet as a targeted host-level nudge by deleting its pods on titan-04, titan-05, titan-06, titan-08, and titan-18; this reruns the DaemonSet's k3s-agent restart command on those nodes.
• The in-cluster helper successfully reran on titan-08, but helper startup/termination was itself wedged on other affected nodes. Queued direct systemctl --no-block restart k3s-agent over Ananke SSH for titan-05, titan-06, and titan-18. A direct restart attempt against titan-04 hung and was stopped.
• After discovering Ananke's SSH profile uses atlas@node:2277, queued systemctl --no-block restart k3s-agent on titan-04 and titan-05 through the same route.
• titan-04 recovered to Ready after the non-blocking restart and its maintenance/k3s-agent-restart-* pod became Running.
• titan-05 did not recover: systemd timed out killing the old k3s-agent, left containerd shims behind, and started a new agent process without reopening kubelet port 10250 or restoring node heartbeats.
• A controlled reboot was issued for titan-05 through Ananke's SSH route at approximately 2026-07-07T17:32Z because the node met the proposed escalation criteria: stale Kubernetes heartbeat, kubelet 10250 closed, k3s-agent.service stuck in activating/start, and the in-cluster helper stuck ContainerCreating.
• After the reboot command returned, titan-05 temporarily stopped responding to ping and SSH, then recovered to Kubernetes Ready=True with fresh heartbeats and no unreachable taints by approximately 2026-07-07T17:35Z.
• Ananke later detected and logged cordoned container-runtime-wedged node(s): titan-07 pods=4; direct host checks showed titan-07 reachable, k3s-agent active/running, and kubelet 10250 open. This should become a false-positive/auto-uncordon test case if Kubernetes readiness remains healthy.
• Later titan-07 showed a real runtime/API-state split: oauth2-proxy-logs reported container state running, but kubectl exec failed with cannot exec in a deleted state and kubectl logs --previous could not retrieve the container logs. Manual assist: queued systemctl --no-block restart k3s-agent on titan-07 and deleted the stale logging/oauth2-proxy-logs-* pod.
• titan-07 recovered to Ready=True with k3s-agent active and kubelet 10250 open. logging/oauth2-proxy-logs was recreated on titan-12 and became 2/2 Running.
• Ananke repeatedly failed and restarted on logging/deployment/oauth2-proxy-logs because the replacement pod on titan-14 hit the same failed to reserve container name / CreateContainerError runtime wedge signature.
• Manual assist: queued systemctl --no-block restart k3s-agent on titan-14 through Ananke SSH and deleted the stuck logging/oauth2-proxy-logs-5b4d4f87db-2tdzd pod with --wait=false so the controller can recreate it after the runtime reset.
• Follow-up on titan-14: the direct SSH command timed out from the operator view, but a later status check showed k3s-agent.service had a new MainPID and was activating/start; the existing maintenance/k3s-agent-restart DaemonSet pod for titan-14 was also deleted to rerun the in-cluster helper.
• titan-14 recovered to fresh Ready=True heartbeats and the replacement maintenance/k3s-agent-restart-* pod became Running.
• After the titan-05 reboot, logging/opensearch-0 recovered to 1/1 Running; logging/oauth2-proxy-logs was recreated on titan-07 and was still initializing at the time of note.
• Ananke also entered a short retry loop where kubectl exec vault-0 ... vault status was killed even though Vault HTTP health through the Kubernetes pod proxy reported initialized=true, sealed=false, standby=false. This should become a separate robustness fix: startup should not fail outright on a transient killed kubectl exec when service endpoints/HTTP health prove Vault is active.
• After Keycloak completed its slow post-attach startup at approximately 2026-07-07T17:45:55Z, OIDC-dependent OAuth proxies began recovering.
• Ananke advanced past critical-workloads into convergence-checks by approximately 2026-07-07T17:50:50Z.
• Manual assist: force-deleted stale pod finance/firefly-6669f46874-q8rtq after Ananke repeatedly issued ordinary deletes but the pod remained Terminating with no finalizers and continued to block firefly-storage attachment. The Firefly Longhorn volume moved from titan-04 to titan-06 and reported attached/healthy on the replacement node.
• Post-Ananke-completion manual assist: titan-04 was still Ready with kubelet 10250 open, but several newly scheduled pods showed FailedCreatePodSandBox, failed to reserve sandbox name, and context deadline exceeded. Queued systemctl --no-block restart k3s-agent on titan-04 at approximately 2026-07-07T18:02:55Z.
• titan-04 did not recover from the non-blocking k3s-agent restart. It moved to Kubernetes Ready=Unknown with node.kubernetes.io/unreachable taints; host checks showed k3s-agent.service stuck in deactivating/stop-sigkill, Result=timeout, and kubelet 10250 closed. Manual assist: issued a controlled reboot for titan-04 through Ananke's SSH route shortly after 2026-07-07T18:06Z.
• titan-04 recovered after reboot by approximately 2026-07-07T18:10Z: Kubernetes reported Ready=True, taints cleared, k3s-agent was active/running, and kubelet 10250 was open.
• Firefly hit a second stale-owner handoff after the first cleanup. Replacement pod firefly-6669f46874-pjhg6 on titan-07 was blocked while terminating pod firefly-6669f46874-lrz5d on titan-06 still had the main firefly container running and mounted firefly-storage. No force delete was issued for this second handoff because the application container was live and PVC-writing. The old pod later disappeared on its own; Longhorn then attached pvc-358b4319-60cb-4322-b948-776c34a414a9 to titan-07 as attached/healthy.
• Attempted to inspect container runtime state on titan-06 with crictl, but atlas required a sudo password. This should be part of the automation fix: Ananke either needs a noninteractive, bounded runtime-inspection primitive or must report that it cannot prove a PVC-writing container is stopped.
• Manual assist: uncordoned titan-15 and titan-17 after confirming Kubernetes Ready=True, Longhorn allowScheduling=true, and Longhorn disks ready. The remaining blocker for Longhorn on both nodes was Kubernetes SchedulingDisabled; after uncordon, Longhorn Schedulable=True on both nodes.
• Manual assist: resolved the titan-22 Kubernetes/Longhorn readiness discrepancy. Kubernetes reported Ready=True, but Longhorn node Ready=False with ManagerPodMissing; the Longhorn manager DaemonSet selected longhorn-host=true, and titan-22 was missing that label. Restored longhorn-host=true, which started longhorn-manager-jtsjw on titan-22 and brought Longhorn node Ready=True.
• Manual assist: deleted pending controller-owned pod crypto/wallet-monero-temp-844bd949c7-28r4p to force a fresh volume-attach retry after the titan-22 Longhorn manager came up.
• Manual assist: deleted stale unattached VolumeAttachment csi-0bac99e18b687f6a50647dd959c26d6e2a1b2bf1a4f844de4f646cf3408d41c5 after it retained the pre-fix "node titan-22 is not ready" Longhorn error. Longhorn then attached pvc-c6a4bf9b-17c7-4bdf-8a88-ca460ad0da5b to titan-22 as attached/healthy.

Automation Growth Opportunities

• Ananke should expose from a remote operator machine whether the coordinator bootstrap is already running, so an operator does not risk starting a duplicate recovery.
• The local Ananke config path assumes /etc/ananke/ananke.yaml and /var/lib/ananke; operator-side status failed without root. A read-only remote status wrapper would make this smoother.
• Ananke should handle the cryptsetup-bin host repair on managed nodes without requiring interactive sudo, or clearly preflight and report the missing node privilege before a power event.
• Ananke should separate Vault pod/data volume unavailable from Vault sealed in status. It did recover correctly, but early status said the unseal check passed while deferring because vault-0 was pending.
• Ananke should avoid repeatedly recycling the same ImagePullBackOff pods if the underlying registry/image problem is unchanged; this creates noisy recovery logs.
• Ananke should surface current blocking workloads in a compact operator summary rather than requiring journal inspection.
• Ananke should detect pods stuck in PodInitializing after init completion, especially when events show FailedCreatePodSandbox, failed to reserve sandbox name, FailedKillPod, or image pulls that never transition to ImagePullBackOff.
• Ananke should be able to rerun the existing k3s-agent-restart maintenance primitive on affected nodes when containerd/kubelet sandbox state is wedged after a power event.
• Ananke should fall back from the in-cluster restart helper to SSH systemctl --no-block restart k3s-agent when the helper pod cannot start because kubelet/containerd is already wedged. Blocking systemctl restart over SSH can hang the operator path.
• Ananke should check host and CoreDNS upstream DNS health separately from node SSH reachability. During this incident, node IP reachability passed while hostname resolution and image pulls were intermittently failing.
• Ananke should stop repeatedly issuing ordinary pod deletes when the same controller pod remains Terminating and continues to own exclusive resources. It should escalate to a stale-owner cleanup flow that proves the old workload is not writing, force-removes the orphaned API object when safe, then verifies Longhorn detach/attach and replacement pod startup.
• Ananke should reconcile Kubernetes node labels against Longhorn manager scheduling requirements. During this incident titan-22 was Kubernetes Ready=True, had CSI/engine Longhorn pods, and had a Longhorn Node CR, but no longhorn-manager pod because longhorn-host=true was missing. Kubernetes Ready alone was not sufficient for PVC workloads.
• Ananke should understand that Kubernetes cordon directly affects Longhorn node schedulability. titan-15 and titan-17 had Longhorn-ready disks and allowScheduling=true, but Longhorn Schedulable=False solely because the Kubernetes nodes were cordoned.

Generalized Node Runtime Wedge Case

This incident exposed a reusable failure mode that Ananke should treat as a node-runtime recovery problem, not just an app-pod recovery problem.

Detection signals:

• Node is Ready=Unknown or has a stale heartbeat while SSH on the managed Ananke route still works.
• Host is pingable and local k3s load balancer port 127.0.0.1:6444 may be open, but kubelet port 127.0.0.1:10250 is closed or API proxy calls fail.
• systemctl restart k3s-agent hangs, or systemctl show k3s-agent reports deactivating, final-sigterm, final-sigkill, activating/start, or Result=timeout for longer than a bounded recovery window.
• journalctl -u k3s-agent includes Processes still around after SIGKILL, State 'stop-sigterm' timed out, or State 'final-sigterm' timed out.
• Pods on the node show FailedCreatePodSandBox, failed to reserve sandbox name, CreateContainerError, FailedKillPod, or long-lived ContainerCreating/PodInitializing.
• The maintenance/k3s-agent-restart DaemonSet pod for that node cannot start or stays ContainerCreating, which means Kubernetes-based repair is unavailable.

Suggested automated escalation ladder:

• Mark the node as in node-runtime recovery and pause noisy repeated pod recycling for pods pinned to that node.
• Cordon the node or confirm it already has unreachable taints before host-level repair.
• Try a bounded SSH health probe using Ananke's configured SSH user, port, config, and identity.
• Prefer systemctl --no-block restart k3s-agent over blocking systemctl restart k3s-agent.
• After a short wait, require both Kubernetes heartbeat freshness and local kubelet port 10250 to recover.
• If the service remains deactivating, final-sigkill, or activating/start with stale heartbeats, escalate to a configured host reboot path.
• After reboot, wait for SSH, k3s-agent active/running, kubelet 10250, Kubernetes Ready=True, and removal of unreachable taints before resuming app-pod recycling for that node.

Implementation notes:

• This should be a generic managed-node repair primitive, not hard-coded to titan-05.
• The primitive should emit one compact operator summary: node, phase, trigger signals, action taken, elapsed time, and next escalation.
• Tests should cover a node with SSH available but kubelet closed, a blocking restart timeout, leftover shim evidence, a helper DaemonSet stuck in ContainerCreating, and successful recovery through reboot.
• The post-completion titan-04 recurrence confirms the primitive must continue monitoring after Ananke declares cluster startup success. A Ready node with fresh-enough status can still be unable to create new sandboxes, and a later restart can transition it into the same hard deactivating/stop-sigkill state seen on titan-05.

Generalized Stale Pod / Volume Ownership Case

This incident also exposed a reusable app-convergence failure mode that Ananke should treat as stale Kubernetes/runtime ownership rather than simple pod unhealthiness.

Concrete example:

• finance/firefly-6669f46874-q8rtq entered deletion at 2026-07-07T17:43:55Z and remained Terminating for more than 25 minutes.
• The pod had no finalizers but still reported a running vault-agent sidecar on titan-04.
• The replacement pod finance/firefly-6669f46874-lrz5d on titan-06 stayed Init:0/2.
• Events showed FailedAttachVolume / Multi-Attach for PVC firefly-storage because volume pvc-358b4319-60cb-4322-b948-776c34a414a9 was still attached to titan-04.
• Longhorn reported the volume attached, healthy, current node titan-04.
• Ananke repeatedly logged ordinary recycling of the same stuck pod, but did not escalate to clearing the stale owner.
• Manual force deletion of the stale pod cleared the stale API owner and allowed Longhorn to attach firefly-storage to the replacement pod's node.

Detection signals:

• A controller-owned pod has metadata.deletionTimestamp older than a bounded threshold and no finalizers.
• A replacement pod from the same controller is pending or initializing on another node.
• Events include Multi-Attach, FailedAttachVolume, Volume is already used by pod(s), or Volume is already exclusively attached to one node.
• The PVC is ReadWriteOnce and the old and new pod nodes differ.
• Longhorn/CNI/runtime state still shows the old node as the attachment owner.
• Ordinary kubectl delete pod --wait=false is repeated without changing the deletion timestamp or clearing the volume attachment.

Suggested automated escalation ladder:

• Group the old pod, replacement pod, PVC, VolumeAttachment, Longhorn volume, controller, and node into one recovery incident.
• Verify whether any application container that mounts the PVC is still running. Sidecars that do not mount the PVC should not block stale-owner cleanup by themselves.
• Check node health for the old owner node. If kubelet/runtime is unhealthy, route through the managed-node recovery primitive first.
• If the old owner node is healthy but the pod remains deleting past the threshold, issue a bounded force deletion for the stale pod.
• Wait for VolumeAttachment/Longhorn ownership to detach from the old node or attach to the replacement node.
• Recheck the replacement pod init sequence and controller readiness before declaring the app recovered.
• Emit one compact operator event instead of repeatedly logging the same pod recycle.

Implementation notes:

• This should be generic across any ReadWriteOnce PVC workload, not hard-coded to Firefly.
• The safety check should reason at container/mount level: a sidecar that only mounts service-account, Vault secret, or emptyDir volumes is materially different from an app container actively mounting the PVC.
• Tests should cover a terminating pod without finalizers holding an RWO PVC, an old node with healthy kubelet, an old node with wedged kubelet, a sidecar-only survivor, and a replacement pod blocked by Multi-Attach.
• Tests should also cover the unsafe variant where the terminating pod's application container is still running and mounts the PVC. In that case Ananke should wait, stop the container through a controlled runtime path if allowed, or report that it lacks the privilege to prove storage safety; it should not blindly force-delete the API object.

Generalized Longhorn / Kubernetes Readiness Drift Case

This incident exposed a reusable storage readiness gap: Kubernetes Ready=True is not enough for PVC workloads when Longhorn's node model disagrees.

Detection signals:

• Kubernetes node is Ready=True, untainted, and schedulable, but Longhorn Node Ready=False.
• Longhorn Node reason is ManagerPodMissing, or the Longhorn manager DaemonSet has no pod on the node.
• Longhorn manager DaemonSet selects a node label such as longhorn-host=true, and the Kubernetes node is missing it.
• Longhorn CSI/engine pods or a Longhorn Node CR exist for the node, proving the node participates in Longhorn even though manager scheduling is broken.
• VolumeAttachment errors say Longhorn cannot attach because the node is not ready, despite Kubernetes Ready=True.

Suggested automated escalation ladder:

• Compare Kubernetes node readiness, labels, taints, Longhorn Node readiness/schedulability, Longhorn manager DaemonSet selector, manager pod presence, CSI pod presence, and VolumeAttachment errors.
• If desired-state evidence proves the node should be a Longhorn node, restore the missing manager selector label and wait for the manager pod and Longhorn Node Ready=True.
• If desired-state evidence is missing, report a clear unsafe label-drift blocker rather than guessing.
• After Longhorn Node Ready=True, retry stale VolumeAttachments by recreating the affected controller-owned pending pod or using a safe VolumeAttachment retry policy.
• For Kubernetes-cordoned Longhorn nodes with allowScheduling=true and ready disks, either complete the host repair and uncordon or report the explicit blocker; do not leave Longhorn capacity disabled silently.

Implementation notes:

• This must be generic across Longhorn nodes and manager DaemonSet selectors, not hard-coded to titan-22 or longhorn-host.
• Tests should cover missing Longhorn selector labels, manager pod missing, Kubernetes Ready but Longhorn NotReady, Kubernetes cordon causing Longhorn Schedulable=False, and stale attach errors that clear after Longhorn readiness is restored.