# Prompt For Ananke Recovery Hardening You are a fresh, stateless Codex session working in the Ananke repository. Your goal is to implement, test, and validate generalized recovery improvements in Ananke based on a real Titan cluster power-loss incident on 2026-07-07. Treat this as production reliability work: read the repo first, follow existing code style and architecture, keep changes cohesive, and solve failure classes categorically rather than hard-coding node names, namespaces, or one-off incidents. ## Mission Make Ananke resilient and automatic during post-power-loss cluster recovery. After your changes, one Ananke startup/recovery process should be able to: - Bring Vault and critical dependencies back safely. - Repair or escalate wedged k3s/containerd nodes using existing Ananke SSH/host access. - Use sudo credentials from Vault, securely and noninteractively, for host repair and diagnostics. - Avoid leaving Longhorn storage nodes cordoned indefinitely. - Detect Kubernetes/Longhorn readiness mismatches and repair label/manager/VolumeAttachment drift. - Resolve stale pod/PVC ownership safely, with different behavior for sidecar-only stale owners versus still-running application containers that mount the PVC. - Continue post-success convergence monitoring long enough to catch late node/runtime/storage wedges. - Report concise, actionable blockers without noisy repeated pod recycling. Do not implement special cases for `titan-04`, `titan-05`, `titan-15`, `titan-17`, `titan-22`, Firefly, Jellyfin, or any specific app. Use those names only as incident fixtures/test names. The code should operate on generic managed nodes, generic Longhorn nodes, generic Kubernetes workloads, and generic RWO PVCs. ## Incident Context The Titan cluster lost power for a few minutes. Ariadne may have started shutdown behavior. The operator asked for a single Ananke process to bring the cluster back up and learn from any manual help required. Ananke did a lot correctly: - It recovered Vault and unsealed it. - It recovered critical workloads, Vault injector, Postgres, Keycloak, OpenSearch, oauth2 logs, CoreDNS, metrics-server, and many app pods. - It ran a single startup workflow and eventually completed successfully. - It recycled many stuck pods and ran several checks. But Ananke needed manual assistance in multiple reusable failure classes: - Node runtime wedges where Kubernetes readiness did not fully reflect kubelet/containerd health. - Host repairs blocked by sudo privilege gaps. - Longhorn nodes cordoned by Kubernetes remained unavailable for storage scheduling. - Longhorn readiness diverged from Kubernetes readiness. - Missing node labels caused Longhorn manager pods to disappear. - Stale terminating pods held RWO volumes and blocked replacement pods. - Repeated pod deletes did not resolve storage ownership or runtime reservation issues. - Ananke declared startup success before all late runtime/storage fallout was resolved. ## Live Incident Facts To Encode As Tests Use these as fixtures for tests and documentation, not as hard-coded behavior: - `titan-05`: k3s-agent restart hung. Host SSH worked. `k3s-agent.service` entered `deactivating`, `final-sigkill`, `activating/start`, and `Result=timeout`. Kubelet port `10250` stayed closed. Kubernetes heartbeat was stale with `node.kubernetes.io/unreachable` taints. In-cluster helper pod could not start. Manual controlled reboot recovered it. - `titan-04`: after Ananke reported startup success, the node was Kubernetes `Ready` and kubelet `10250` initially open, but new pods failed with `FailedCreatePodSandBox`, `failed to reserve sandbox name`, and `context deadline exceeded`. A non-blocking `k3s-agent` restart then stuck in `deactivating/stop-sigkill`, kubelet `10250` closed, Kubernetes moved to `Ready=Unknown`, and a controlled reboot recovered it. - `titan-07`: a pod looked running in the API, but `kubectl exec` failed with `cannot exec in a deleted state`; restarting k3s-agent and recreating the pod fixed it. - `titan-14`: a replacement pod hit `failed to reserve container name` / `CreateContainerError`; k3s-agent restart and rerunning the in-cluster helper fixed it. Later another pod on the same node showed the same sandbox reservation pattern, but then progressed after enough time. Tests should distinguish transient progress from a stall. - `titan-15` and `titan-17`: Ananke cordoned them after encrypted Longhorn mounts exposed missing `cryptsetup`. Longhorn itself had `allowScheduling=true` and ready disks, but Longhorn `Schedulable=False` because Kubernetes had cordoned the nodes. Manual `kubectl uncordon titan-15 titan-17` restored Kubernetes scheduling and Longhorn schedulability. Ananke had attempted `cryptsetup-bin` repair but failed because `atlas` sudo required a password. - `titan-22`: Kubernetes `Ready=True`, CSI plugin and engine images were running, but Longhorn node `Ready=False` because `longhorn-manager` pod was missing. The root cause was missing node label `longhorn-host=true`, while the `longhorn-manager` DaemonSet selected `longhorn-host=true`. Manual `kubectl label node titan-22 longhorn-host=true --overwrite` started `longhorn-manager` and Longhorn node Ready became true. A pending wallet pod still carried an old failed VolumeAttachment until the pod was deleted/recreated. - Firefly stale owner, safe-to-force variant: `finance/firefly-...-q8rtq` was terminating for more than 25 minutes, had no finalizers, and only a Vault sidecar remained running. Replacement pod was blocked by `Multi-Attach` for an RWO PVC. Force-deleting the stale pod cleared ownership and allowed Longhorn attach to the replacement. - Firefly stale owner, unsafe-to-force variant: a later terminating Firefly pod still had the main application container running and mounting the RWO PVC. It was not safe to blindly force-delete. Waiting allowed kubelet to finish termination, then Longhorn attached the volume to the replacement. - Manual runtime inspection attempt on `titan-06`: `sudo -n crictl ...` failed because `atlas` required a sudo password. Ananke is expected to have access to sudo passwords in Vault, so this must be designed as a first-class Ananke capability rather than manual operator shell access. - Vault health check robustness: at one point `kubectl exec vault-0 -- vault status` was killed, but HTTP health via Kubernetes pod proxy showed `initialized=true`, `sealed=false`, `standby=false`. Startup should not fail solely on transient `kubectl exec` failure when endpoint and HTTP health prove Vault is available. - Image/DNS issues: several image pulls failed with `lookup registry-1.docker.io`, `lookup auth.docker.io`, or `lookup production.cloudfront.docker.com: Try again`, then later succeeded. Ananke repeatedly recycling unchanged ImagePullBackOff pods created noise. ## Required Implementation Areas ### 1. Secure Sudo Capability From Vault Ananke should be able to run bounded host diagnostics and repairs that require sudo. Requirements: - Discover and use the existing Vault integration and secret naming conventions already present in the repo/config. Do not invent a new hard-coded secret path if the repo has one. - Support noninteractive sudo for configured managed nodes. - Never log sudo passwords, Vault tokens, rendered secrets, command input containing secrets, or full environment dumps. - Provide a preflight check: for each managed node, can Ananke retrieve host sudo material and can it run a harmless command such as `sudo -S -p '' true` with a timeout? - If credentials are missing, wrong, or Vault is unavailable, Ananke should report `host-privilege-unavailable` with node, intended action, and secret lookup class, not spin. - Use strict command allowlists for privileged commands. The recovery path should not become arbitrary remote root execution. - Use bounded timeouts for all host commands and classify timeout versus auth failure versus command failure. - Prefer safe wrappers for: - `systemctl show/is-active/restart --no-block k3s-agent` - `systemctl reboot` when configured and escalated - `crictl ps/pods/inspect/stop` for recovery diagnostics and controlled container stop - package checks/install for `cryptsetup-bin`, `open-iscsi`, `nfs-common`, `dmsetup` - `modprobe dm_crypt` or equivalent kernel-module validation where appropriate Tests: - Vault lookup succeeds and sudo command succeeds. - Vault lookup succeeds but sudo password is wrong. - Vault lookup missing. - Vault unavailable while Vault HTTP health is also degraded. - Vault unavailable but action does not require sudo. - Command times out. - Command returns nonzero. - Logs are scanned to ensure no password appears. ### 2. Managed Node Runtime Recovery Primitive Implement a generic state machine for node runtime recovery. Detection signals: - Kubernetes node `Ready=Unknown` or stale heartbeat. - Kubernetes node `Ready=True` but pods on that node show repeated: - `FailedCreatePodSandBox` - `failed to reserve sandbox name` - `failed to reserve container name` - `CreateContainerError` - `FailedKillPod` - long-lived `ContainerCreating` or `PodInitializing` with no progress - `kubectl exec`/logs failures indicating deleted runtime state - Host SSH works but kubelet port `127.0.0.1:10250` is closed or slow. - `systemctl show k3s-agent` reports `deactivating`, `stop-sigterm`, `stop-sigkill`, `final-sigterm`, `final-sigkill`, `activating/start`, or `Result=timeout`. - Existing in-cluster maintenance helper pod for the node cannot start, stays `ContainerCreating`, or cannot be scheduled because the node runtime is already unhealthy. Escalation ladder: - Open a single node recovery incident and suppress noisy repeated pod recycling for pods on that node while host repair is in progress. - Cordon the node before host-level intervention unless it is already unreachable/tainted. - Try in-cluster maintenance helper if kubelet/runtime can create pods. - If helper cannot run, use Ananke SSH route and sudo from Vault. - Prefer `systemctl --no-block restart k3s-agent`; never use a blocking restart over SSH as the primary recovery action. - Wait a bounded interval and require both: - Kubernetes Ready=True with fresh heartbeat - local kubelet `10250` open - If k3s-agent remains stuck in stop/kill/activating states or heartbeats remain stale, escalate to configured controlled reboot. - After reboot, wait for SSH, k3s-agent active/running, kubelet port open, Kubernetes Ready=True, and unreachable taints cleared. - Uncordon only after the node is healthy and after storage-specific checks pass. - Continue this monitoring after startup success for a configurable post-success convergence window. Tests: - Ready node with repeated sandbox reservation events triggers recovery. - Ready node with one transient sandbox event and then progress does not trigger restart. - Unknown node with SSH unavailable reports external blocker without repeated action. - Unknown node with SSH available, kubelet closed, k3s-agent stuck, helper unavailable escalates to reboot. - Non-blocking restart recovers node. - Non-blocking restart moves service to `deactivating/stop-sigkill`; reboot escalation occurs. - Reboot command accepted but SSH does not return within timeout. - Reboot returns and all readiness conditions clear. - False positive prevention: node Ready=True, kubelet open, events old/stale only, no action. - Idempotency: repeated Ananke runs do not issue repeated restarts/reboots while an incident is active. ### 3. Longhorn And Kubernetes Readiness Reconciliation Ananke must not treat Kubernetes node Ready as sufficient for storage workloads. Implement a Longhorn readiness reconciler: - Read Kubernetes Nodes, Longhorn Node CRs, Longhorn manager DaemonSet, Longhorn manager pods, CSI pods, engine-image pods, Volume CRs, VolumeAttachments, and relevant PVC-bound pods. - Compare: - Kubernetes Ready/SchedulingDisabled/taints - Node labels required by Longhorn manager DaemonSet, especially `longhorn-host=true` - Longhorn Node `Ready` and `Schedulable` conditions - Longhorn Node `spec.allowScheduling` - Longhorn manager pod presence on nodes where Longhorn Node exists or where PVC workloads are allowed - CSI plugin readiness - VolumeAttachment errors that reference stale Longhorn node readiness - Detect label drift: Longhorn node exists and CSI/engine pods exist, but manager pod is missing because the node no longer matches the DaemonSet selector. - Repair label drift only if it is safe: - There is a known desired label source in repo/config/state, or - Longhorn Node CR exists for that Kubernetes node and existing Longhorn metadata indicates the node is intended to be a Longhorn node. - Emit exactly what label will be restored and why. - If safe repair cannot be proven, cordon or avoid scheduling PVC workloads to the node and report a blocker. - After restoring labels or manager pods, wait for Longhorn Node Ready=True. - For VolumeAttachments that failed because Longhorn thought a now-ready node was not ready, trigger a bounded retry: - Prefer deleting/recreating the controller-owned Pending pod if that is the established pattern. - Delete stale failed VolumeAttachment only if Kubernetes controller semantics and repo policy allow it. - Never delete PV/PVC/Longhorn Volume data. Specific expectations from incident: - `titan-22` class: Kubernetes Ready, missing `longhorn-host=true`, no manager pod, Longhorn Node Ready=False ManagerPodMissing. Ananke should restore label or report exact unsafe reason, then wait for manager pod and Longhorn Ready. - `titan-15`/`titan-17` class: Kubernetes cordon makes Longhorn Schedulable=False despite Longhorn `allowScheduling=true` and ready disks. Ananke should not leave them cordoned after the blocking repair condition is resolved or after operator-approved override. Tests: - Longhorn manager selector changes are handled generically. - Missing Longhorn label restored from desired state. - Missing Longhorn label not restored if node lacks Longhorn Node CR and no desired-state evidence exists. - Longhorn Node Ready=False ManagerPodMissing becomes Ready after label restoration and manager pod startup. - VolumeAttachment error due stale Longhorn node readiness is retried after Longhorn Ready. - Kubernetes cordoned node causes Longhorn Schedulable=False and is uncordoned after repair. - Kubernetes cordoned node remains cordoned if required package/kernel preflight still fails and no override is configured. - `allowScheduling=false` is respected; Ananke should not enable Longhorn disk scheduling unless explicitly configured, but it may still ensure manager/CSI readiness for attachment if workloads are allowed there. ### 4. Cordon/Uncordon And Host Package Repair Policy Ananke cordoned `titan-15` and `titan-17` because encrypted Longhorn mounts exposed missing `cryptsetup`; it failed to install because sudo required a password. This must become a closed-loop repair. Requirements: - Distinguish: - Kubernetes scheduling cordon - Longhorn disk scheduling (`allowScheduling`) - Longhorn node readiness/schedulability - ordinary `PreferNoSchedule` taints used as soft placement hints - If Ananke cordons a node, it must record why, when, what remediation is required, and what condition will uncordon it. - If the remediation is package/module installation, use Vault-backed sudo and verify the result. - If package repair succeeds, uncordon automatically and verify Longhorn Schedulable=True. - If package repair fails due missing sudo secret, wrong sudo password, package manager lock, apt/dpkg failure, or network outage, report the exact class and do not leave the node silently disabled. - Support an operator override policy for "uncordon for Longhorn availability even if encrypted workload preflight is not perfect", but this must be explicit/configured and logged. - Do not remove existing soft taints such as `atlas.bstein.dev/spillover=true:PreferNoSchedule` or `longhorn=true:PreferNoSchedule` unless the repo’s desired-state policy says to. Tests: - Missing cryptsetup with valid sudo installs package, loads module if needed, uncordons node. - Missing cryptsetup with missing sudo secret reports privilege blocker and leaves cordon with reason. - Node already has package but missing kernel module: module load path tested. - Node has Longhorn ready disks but is Kubernetes cordoned: Longhorn schedulability blocker is reported. - Uncordon preserves unrelated soft taints. - Re-running recovery is idempotent. ### 5. Stale RWO PVC Owner Recovery Implement a generic stale-owner recovery flow for controller-owned pods and RWO PVCs. Detection: - Old pod has `metadata.deletionTimestamp` older than threshold. - Replacement pod from same controller exists and is pending/initializing. - Replacement events include `Multi-Attach`, `FailedAttachVolume`, `Volume is already used by pod(s)`, or `Volume is already exclusively attached to one node`. - PVC is `ReadWriteOnce` or equivalent single-writer mode. - Old and new pods are on different nodes. - Longhorn Volume/VolumeAttachment shows old node still owns attachment or new attach blocked. - Ordinary pod deletes have been attempted repeatedly without changing ownership. Recovery grouping: - Group old pod, replacement pod, controller, PVC, PV, VolumeAttachment(s), Longhorn Volume, old node, new node, and relevant events into one incident. - Emit one concise operator record. - Stop repeated blind pod deletes while the incident is active. Safety decisions: - Inspect old pod container statuses and volume mounts. - Classify containers: - PVC-writing application containers: mount the blocked PVC. - Sidecars that do not mount the blocked PVC: Vault agent, projected service account, emptyDir-only, etc. - Unknown containers: treat as unsafe. - If only sidecars remain and no container mounts the PVC, force-delete the stale pod after a bounded wait. - If an application container still runs and mounts the PVC, do not blindly force-delete. Instead: - Wait for kubelet termination if it is making progress. - If stalled and host runtime inspection is available through Vault-backed sudo, inspect and optionally stop the specific container through a controlled allowlisted path. - If Ananke cannot prove the PVC-writing container is stopped, report `unsafe-stale-owner` and do not clear the API object. - If old node kubelet/runtime is unhealthy, route through managed node recovery before stale-owner cleanup. Tests: - Sidecar-only stale pod force-clears and replacement attaches. - Live app container with blocked PVC waits and does not force-delete. - Live app container eventually exits; replacement attaches without force-delete. - Host runtime inspection unavailable due sudo privilege reports unsafe blocker. - Host runtime inspection available and controlled container stop succeeds. - Longhorn detach/attach follows cleanup. - Replacement readiness is rechecked after attach. - Multiple PVCs in one pod are handled. - Non-RWO PVC is not processed by this recovery path. ### 6. Vault Health Robustness Startup should not fail because one Vault probe path is flaky if other authoritative health signals prove Vault is usable. Requirements: - Prefer multiple health signals: - pod Running/Ready - service endpoints populated - HTTP `/v1/sys/health` through service or pod proxy - `vault status` through exec as one signal, not the only signal - If `kubectl exec vault status` is killed/transiently fails but HTTP health reports initialized/unsealed/active or standby-acceptable per config, classify as transient exec failure and continue with warning. - If HTTP health and exec disagree, report clearly and retry with bounded backoff. - Never log tokens. Tests: - exec killed but HTTP health unsealed => startup continues with warning. - exec says sealed and HTTP says sealed => startup blocks/unseals. - endpoints missing => startup waits. - HTTP unavailable but exec succeeds => startup may proceed if policy allows. ### 7. Image Pull And DNS Failure Classification Ananke should not repeatedly recycle pods whose only issue is unchanged image pull or DNS failure. Requirements: - Detect ImagePullBackOff/ErrImagePull events caused by DNS/registry lookup. - Run separate DNS checks: - CoreDNS pods ready - in-cluster DNS query from a known diagnostic pod if available - node host DNS or kubelet image pull DNS failures grouped by node - registry reachability by hostname class, not one app - If the image pull failure is unchanged and recent, do not repeatedly delete the pod. - Emit a compact blocker: image, registry host, node, last error, and whether DNS is cluster-wide or node-specific. - Retry only after DNS/registry health changes or after bounded interval. Tests: - Docker Hub auth DNS failure is classified. - Private registry DNS failure is classified. - One-off pull failure that later succeeds is not over-escalated. - Repeated pod recycle is suppressed. ### 8. Startup Completion And Post-Success Monitoring Ananke declared `startup_status=success` while late runtime/storage issues still surfaced. Make completion smarter. Requirements: - Startup can reach "cluster core recovered" while still monitoring app convergence. - Do not mark total recovery complete until: - critical services pass - all managed nodes are either healthy, intentionally excluded, or explicitly blocked - Longhorn node readiness/schedulability matches desired policy - no active node-runtime incidents exist - no unsafe stale-owner incidents exist - Add a post-success watch window for late events: sandbox reservation, FailedKillPod, kubelet metrics scrape failures, Longhorn manager missing, VolumeAttachment errors. - Distinguish statuses: - `core_recovered` - `converging` - `success` - `success_with_noncritical_blockers` - `blocked_requires_operator` - Report app-level blockers separately from cluster-level blockers. Tests: - Critical services pass but node runtime wedge appears during post-success window => not final success until resolved. - Noncritical app image pull blocker can be `success_with_noncritical_blockers`. - Unsafe stale RWO owner blocks full success. - All clear => success. ## Status And Operator Reporting Improve operator output. The user should not need to read journals to know what is happening. Status should include: - Whether a startup/recovery workflow is already running. - Current phase and elapsed time. - Current incidents grouped by type: - node runtime recovery - Longhorn/Kubernetes drift - stale RWO owner - Vault health - image/DNS pull - host privilege/preflight - For each incident: resource, trigger signals, action taken, next action, elapsed time, and whether operator input is needed. - Counts should be accurate and stable. - Repeated noisy details should be summarized with "same blocker still present for X minutes" instead of repeated lines. ## Security Constraints - Do not log sudo passwords, Vault tokens, unsealed keys, or rendered secret contents. - Redact secrets in command arguments and environment. - Use command allowlists for privileged host actions. - Make destructive operations explicit and guarded: - reboot only after escalation criteria - force-delete pod only after stale-owner safety checks - deleting VolumeAttachments only if repo policy allows and after safeguards - Keep audit logs of actions: node/resource, reason, command class, result, elapsed time, but not secrets. ## Implementation Guidance Before editing: - Read Ananke’s existing startup/recovery code, Kubernetes client abstractions, Vault access layer, SSH/host command layer, Longhorn helpers if any, status/progress writer, and tests. - Reuse existing abstractions. Do not invent a parallel framework unless necessary. - Use fake clients and table-driven tests where the repo already uses them. - Keep changes modular: - host privilege provider - node runtime incident detector/reconciler - Longhorn readiness reconciler - stale RWO owner reconciler - image/DNS classifier - status/progress summarizer - Ensure all reconcilers are idempotent and bounded. Suggested model: - Define incident structs with stable keys, timestamps, observed signals, action history, next action, and terminal status. - Every recovery action should require: - detection evidence - policy permission - safety gate - timeout - postcondition check - Avoid "delete pod until healthy" loops. Replace with evidence-driven remediation. ## Validation Plan Run the repository’s standard checks first, then add targeted tests. Required validation: - Unit tests for each new classifier/reconciler. - Tests for redaction/no secret leakage. - Tests for idempotency on repeated reconcile calls. - Tests for progress/status output. - Integration-style tests with fake Kubernetes/Longhorn objects: - nodes, pods, events, PVCs, PVs, VolumeAttachments, Longhorn Node CRs, Longhorn Volume CRs, DaemonSets, and pods. - Host command fake tests: - SSH unavailable - sudo unavailable - sudo succeeds - command timeout - k3s-agent service states - reboot accepted/not accepted - Vault fake tests: - secret found - secret missing - Vault sealed/unavailable - HTTP health fallback - End-to-end scenario tests composed from the incident: - `titan-05` hard runtime wedge -> reboot -> Ready - `titan-04` post-success sandbox wedge -> restart fails -> reboot -> Ready - `titan-15`/`titan-17` cryptsetup/cordon flow -> sudo repair or explicit blocker -> uncordon when safe - `titan-22` missing `longhorn-host` label -> restore label -> manager pod appears -> Longhorn Ready - Firefly sidecar-only stale owner -> force delete -> attach replacement - Firefly live app container stale owner -> wait/report unsafe, no force delete - Vault exec killed but HTTP healthy -> continue - DNS image pull errors -> classify without repeated recycle Run commands appropriate for the repo, likely including: ```bash go test ./... go test -race ./... go vet ./... ``` If the repo uses other tooling, discover and run it. If any expensive/integration tests require cluster credentials, document how to run them and ensure fake-client tests cover the logic by default. ## Acceptance Criteria The work is complete only when: - Ananke can retrieve and use Vault-backed sudo credentials securely or clearly reports why it cannot. - Node runtime wedges are detected and recovered by a bounded escalation ladder. - Ananke can handle Ready nodes that cannot create sandboxes. - Ananke can recover from k3s-agent stop-sigkill/timeout by controlled reboot when configured. - Ananke does not leave Longhorn nodes cordoned indefinitely after repair/override conditions are satisfied. - Ananke detects and fixes or reports Longhorn/Kubernetes drift, including missing manager-pod labels. - Ananke verifies Longhorn node readiness before trusting Kubernetes node readiness for PVC workloads. - Ananke handles stale RWO ownership safely, distinguishing sidecar-only stale pods from live PVC-writing app containers. - Ananke avoids noisy repeated deletion of unchanged image-pull failures. - Vault health checks tolerate transient exec failures when HTTP health is authoritative. - Startup success reflects real cluster recovery, with explicit `success_with_noncritical_blockers` or `blocked_requires_operator` states where appropriate. - All changes are covered by rigorous tests and pass the repo’s standard test suite. ## What Not To Do - Do not hard-code Titan node names, app names, PVC names, or namespaces. - Do not log secrets. - Do not delete PVs, PVCs, or Longhorn Volume data. - Do not force-delete pods with live app containers mounting RWO PVCs unless a controlled, audited stop path has proven storage safety. - Do not repeatedly restart/reboot nodes without incident state and cooldowns. - Do not treat Kubernetes Ready as equivalent to Longhorn Ready. - Do not declare full success while node-runtime or unsafe stale-owner incidents are active. ## Useful Terminology From The Incident - "Node runtime wedge": Kubernetes or host state where kubelet/containerd cannot create/kill sandboxes even if the node appears partially Ready. - "Stale RWO owner": a terminating pod or runtime survivor that still holds an exclusive PVC and blocks replacement attach. - "Sidecar-only stale owner": old pod still exists, but only non-PVC sidecars remain; force deletion may be safe after checks. - "Unsafe stale owner": app container still running and mounting the PVC; wait or controlled stop required. - "Longhorn/Kubernetes drift": Kubernetes node readiness/labels and Longhorn node readiness/manager state disagree. - "Host privilege unavailable": Ananke cannot run necessary sudo actions despite needing host-level repair. Use this prompt to implement production-grade, categorical recovery logic in Ananke, not one-off incident handling.