ananke/docs/ananke-hardening-prompt-2026-07-07.md

Prompt For Ananke Recovery Hardening

You are a fresh, stateless Codex session working in the Ananke repository. Your goal is to implement, test, and validate generalized recovery improvements in Ananke based on a real Titan cluster power-loss incident on 2026-07-07. Treat this as production reliability work: read the repo first, follow existing code style and architecture, keep changes cohesive, and solve failure classes categorically rather than hard-coding node names, namespaces, or one-off incidents.

Mission

Make Ananke resilient and automatic during post-power-loss cluster recovery. After your changes, one Ananke startup/recovery process should be able to:

• Bring Vault and critical dependencies back safely.
• Repair or escalate wedged k3s/containerd nodes using existing Ananke SSH/host access.
• Use sudo credentials from Vault, securely and noninteractively, for host repair and diagnostics.
• Avoid leaving Longhorn storage nodes cordoned indefinitely.
• Detect Kubernetes/Longhorn readiness mismatches and repair label/manager/VolumeAttachment drift.
• Resolve stale pod/PVC ownership safely, with different behavior for sidecar-only stale owners versus still-running application containers that mount the PVC.
• Continue post-success convergence monitoring long enough to catch late node/runtime/storage wedges.
• Report concise, actionable blockers without noisy repeated pod recycling.

Do not implement special cases for titan-04, titan-05, titan-15, titan-17, titan-22, Firefly, Jellyfin, or any specific app. Use those names only as incident fixtures/test names. The code should operate on generic managed nodes, generic Longhorn nodes, generic Kubernetes workloads, and generic RWO PVCs.

Incident Context

The Titan cluster lost power for a few minutes. Ariadne may have started shutdown behavior. The operator asked for a single Ananke process to bring the cluster back up and learn from any manual help required.

Ananke did a lot correctly:

• It recovered Vault and unsealed it.
• It recovered critical workloads, Vault injector, Postgres, Keycloak, OpenSearch, oauth2 logs, CoreDNS, metrics-server, and many app pods.
• It ran a single startup workflow and eventually completed successfully.
• It recycled many stuck pods and ran several checks.

But Ananke needed manual assistance in multiple reusable failure classes:

• Node runtime wedges where Kubernetes readiness did not fully reflect kubelet/containerd health.
• Host repairs blocked by sudo privilege gaps.
• Longhorn nodes cordoned by Kubernetes remained unavailable for storage scheduling.
• Longhorn readiness diverged from Kubernetes readiness.
• Missing node labels caused Longhorn manager pods to disappear.
• Stale terminating pods held RWO volumes and blocked replacement pods.
• Repeated pod deletes did not resolve storage ownership or runtime reservation issues.
• Ananke declared startup success before all late runtime/storage fallout was resolved.

Live Incident Facts To Encode As Tests

Use these as fixtures for tests and documentation, not as hard-coded behavior:

• titan-05: k3s-agent restart hung. Host SSH worked. k3s-agent.service entered deactivating, final-sigkill, activating/start, and Result=timeout. Kubelet port 10250 stayed closed. Kubernetes heartbeat was stale with node.kubernetes.io/unreachable taints. In-cluster helper pod could not start. Manual controlled reboot recovered it.
• titan-04: after Ananke reported startup success, the node was Kubernetes Ready and kubelet 10250 initially open, but new pods failed with FailedCreatePodSandBox, failed to reserve sandbox name, and context deadline exceeded. A non-blocking k3s-agent restart then stuck in deactivating/stop-sigkill, kubelet 10250 closed, Kubernetes moved to Ready=Unknown, and a controlled reboot recovered it.
• titan-07: a pod looked running in the API, but kubectl exec failed with cannot exec in a deleted state; restarting k3s-agent and recreating the pod fixed it.
• titan-14: a replacement pod hit failed to reserve container name / CreateContainerError; k3s-agent restart and rerunning the in-cluster helper fixed it. Later another pod on the same node showed the same sandbox reservation pattern, but then progressed after enough time. Tests should distinguish transient progress from a stall.
• titan-15 and titan-17: Ananke cordoned them after encrypted Longhorn mounts exposed missing cryptsetup. Longhorn itself had allowScheduling=true and ready disks, but Longhorn Schedulable=False because Kubernetes had cordoned the nodes. Manual kubectl uncordon titan-15 titan-17 restored Kubernetes scheduling and Longhorn schedulability. Ananke had attempted cryptsetup-bin repair but failed because atlas sudo required a password.
• titan-22: Kubernetes Ready=True, CSI plugin and engine images were running, but Longhorn node Ready=False because longhorn-manager pod was missing. The root cause was missing node label longhorn-host=true, while the longhorn-manager DaemonSet selected longhorn-host=true. Manual kubectl label node titan-22 longhorn-host=true --overwrite started longhorn-manager and Longhorn node Ready became true. A pending wallet pod still carried an old failed VolumeAttachment until the pod was deleted/recreated.
• Firefly stale owner, safe-to-force variant: finance/firefly-...-q8rtq was terminating for more than 25 minutes, had no finalizers, and only a Vault sidecar remained running. Replacement pod was blocked by Multi-Attach for an RWO PVC. Force-deleting the stale pod cleared ownership and allowed Longhorn attach to the replacement.
• Firefly stale owner, unsafe-to-force variant: a later terminating Firefly pod still had the main application container running and mounting the RWO PVC. It was not safe to blindly force-delete. Waiting allowed kubelet to finish termination, then Longhorn attached the volume to the replacement.
• Manual runtime inspection attempt on titan-06: sudo -n crictl ... failed because atlas required a sudo password. Ananke is expected to have access to sudo passwords in Vault, so this must be designed as a first-class Ananke capability rather than manual operator shell access.
• Vault health check robustness: at one point kubectl exec vault-0 -- vault status was killed, but HTTP health via Kubernetes pod proxy showed initialized=true, sealed=false, standby=false. Startup should not fail solely on transient kubectl exec failure when endpoint and HTTP health prove Vault is available.
• Image/DNS issues: several image pulls failed with lookup registry-1.docker.io, lookup auth.docker.io, or lookup production.cloudfront.docker.com: Try again, then later succeeded. Ananke repeatedly recycling unchanged ImagePullBackOff pods created noise.

Required Implementation Areas

1. Secure Sudo Capability From Vault

Ananke should be able to run bounded host diagnostics and repairs that require sudo.

Requirements:

• Discover and use the existing Vault integration and secret naming conventions already present in the repo/config. Do not invent a new hard-coded secret path if the repo has one.
• Support noninteractive sudo for configured managed nodes.
• Never log sudo passwords, Vault tokens, rendered secrets, command input containing secrets, or full environment dumps.
• Provide a preflight check: for each managed node, can Ananke retrieve host sudo material and can it run a harmless command such as sudo -S -p '' true with a timeout?
• If credentials are missing, wrong, or Vault is unavailable, Ananke should report host-privilege-unavailable with node, intended action, and secret lookup class, not spin.
• Use strict command allowlists for privileged commands. The recovery path should not become arbitrary remote root execution.
• Use bounded timeouts for all host commands and classify timeout versus auth failure versus command failure.
• Prefer safe wrappers for:
  • systemctl show/is-active/restart --no-block k3s-agent
  • systemctl reboot when configured and escalated
  • crictl ps/pods/inspect/stop for recovery diagnostics and controlled container stop
  • package checks/install for cryptsetup-bin, open-iscsi, nfs-common, dmsetup
  • modprobe dm_crypt or equivalent kernel-module validation where appropriate

Tests:

• Vault lookup succeeds and sudo command succeeds.
• Vault lookup succeeds but sudo password is wrong.
• Vault lookup missing.
• Vault unavailable while Vault HTTP health is also degraded.
• Vault unavailable but action does not require sudo.
• Command times out.
• Command returns nonzero.
• Logs are scanned to ensure no password appears.

2. Managed Node Runtime Recovery Primitive

Implement a generic state machine for node runtime recovery.

Detection signals:

• Kubernetes node Ready=Unknown or stale heartbeat.
• Kubernetes node Ready=True but pods on that node show repeated:
  • FailedCreatePodSandBox
  • failed to reserve sandbox name
  • failed to reserve container name
  • CreateContainerError
  • FailedKillPod
  • long-lived ContainerCreating or PodInitializing with no progress
  • kubectl exec/logs failures indicating deleted runtime state
• Host SSH works but kubelet port 127.0.0.1:10250 is closed or slow.
• systemctl show k3s-agent reports deactivating, stop-sigterm, stop-sigkill, final-sigterm, final-sigkill, activating/start, or Result=timeout.
• Existing in-cluster maintenance helper pod for the node cannot start, stays ContainerCreating, or cannot be scheduled because the node runtime is already unhealthy.

Escalation ladder:

• Open a single node recovery incident and suppress noisy repeated pod recycling for pods on that node while host repair is in progress.
• Cordon the node before host-level intervention unless it is already unreachable/tainted.
• Try in-cluster maintenance helper if kubelet/runtime can create pods.
• If helper cannot run, use Ananke SSH route and sudo from Vault.
• Prefer systemctl --no-block restart k3s-agent; never use a blocking restart over SSH as the primary recovery action.
• Wait a bounded interval and require both:
  • Kubernetes Ready=True with fresh heartbeat
  • local kubelet 10250 open
• If k3s-agent remains stuck in stop/kill/activating states or heartbeats remain stale, escalate to configured controlled reboot.
• After reboot, wait for SSH, k3s-agent active/running, kubelet port open, Kubernetes Ready=True, and unreachable taints cleared.
• Uncordon only after the node is healthy and after storage-specific checks pass.
• Continue this monitoring after startup success for a configurable post-success convergence window.

Tests:

• Ready node with repeated sandbox reservation events triggers recovery.
• Ready node with one transient sandbox event and then progress does not trigger restart.
• Unknown node with SSH unavailable reports external blocker without repeated action.
• Unknown node with SSH available, kubelet closed, k3s-agent stuck, helper unavailable escalates to reboot.
• Non-blocking restart recovers node.
• Non-blocking restart moves service to deactivating/stop-sigkill; reboot escalation occurs.
• Reboot command accepted but SSH does not return within timeout.
• Reboot returns and all readiness conditions clear.
• False positive prevention: node Ready=True, kubelet open, events old/stale only, no action.
• Idempotency: repeated Ananke runs do not issue repeated restarts/reboots while an incident is active.

3. Longhorn And Kubernetes Readiness Reconciliation

Ananke must not treat Kubernetes node Ready as sufficient for storage workloads.

Implement a Longhorn readiness reconciler:

• Read Kubernetes Nodes, Longhorn Node CRs, Longhorn manager DaemonSet, Longhorn manager pods, CSI pods, engine-image pods, Volume CRs, VolumeAttachments, and relevant PVC-bound pods.
• Compare:
  • Kubernetes Ready/SchedulingDisabled/taints
  • Node labels required by Longhorn manager DaemonSet, especially longhorn-host=true
  • Longhorn Node Ready and Schedulable conditions
  • Longhorn Node spec.allowScheduling
  • Longhorn manager pod presence on nodes where Longhorn Node exists or where PVC workloads are allowed
  • CSI plugin readiness
  • VolumeAttachment errors that reference stale Longhorn node readiness
• Detect label drift: Longhorn node exists and CSI/engine pods exist, but manager pod is missing because the node no longer matches the DaemonSet selector.
• Repair label drift only if it is safe:
  • There is a known desired label source in repo/config/state, or
  • Longhorn Node CR exists for that Kubernetes node and existing Longhorn metadata indicates the node is intended to be a Longhorn node.
  • Emit exactly what label will be restored and why.
• If safe repair cannot be proven, cordon or avoid scheduling PVC workloads to the node and report a blocker.
• After restoring labels or manager pods, wait for Longhorn Node Ready=True.
• For VolumeAttachments that failed because Longhorn thought a now-ready node was not ready, trigger a bounded retry:
  • Prefer deleting/recreating the controller-owned Pending pod if that is the established pattern.
  • Delete stale failed VolumeAttachment only if Kubernetes controller semantics and repo policy allow it.
  • Never delete PV/PVC/Longhorn Volume data.

Specific expectations from incident:

• titan-22 class: Kubernetes Ready, missing longhorn-host=true, no manager pod, Longhorn Node Ready=False ManagerPodMissing. Ananke should restore label or report exact unsafe reason, then wait for manager pod and Longhorn Ready.
• titan-15/titan-17 class: Kubernetes cordon makes Longhorn Schedulable=False despite Longhorn allowScheduling=true and ready disks. Ananke should not leave them cordoned after the blocking repair condition is resolved or after operator-approved override.

Tests:

• Longhorn manager selector changes are handled generically.
• Missing Longhorn label restored from desired state.
• Missing Longhorn label not restored if node lacks Longhorn Node CR and no desired-state evidence exists.
• Longhorn Node Ready=False ManagerPodMissing becomes Ready after label restoration and manager pod startup.
• VolumeAttachment error due stale Longhorn node readiness is retried after Longhorn Ready.
• Kubernetes cordoned node causes Longhorn Schedulable=False and is uncordoned after repair.
• Kubernetes cordoned node remains cordoned if required package/kernel preflight still fails and no override is configured.
• allowScheduling=false is respected; Ananke should not enable Longhorn disk scheduling unless explicitly configured, but it may still ensure manager/CSI readiness for attachment if workloads are allowed there.

4. Cordon/Uncordon And Host Package Repair Policy

Ananke cordoned titan-15 and titan-17 because encrypted Longhorn mounts exposed missing cryptsetup; it failed to install because sudo required a password. This must become a closed-loop repair.

Requirements:

• Distinguish:
  • Kubernetes scheduling cordon
  • Longhorn disk scheduling (allowScheduling)
  • Longhorn node readiness/schedulability
  • ordinary PreferNoSchedule taints used as soft placement hints
• If Ananke cordons a node, it must record why, when, what remediation is required, and what condition will uncordon it.
• If the remediation is package/module installation, use Vault-backed sudo and verify the result.
• If package repair succeeds, uncordon automatically and verify Longhorn Schedulable=True.
• If package repair fails due missing sudo secret, wrong sudo password, package manager lock, apt/dpkg failure, or network outage, report the exact class and do not leave the node silently disabled.
• Support an operator override policy for "uncordon for Longhorn availability even if encrypted workload preflight is not perfect", but this must be explicit/configured and logged.
• Do not remove existing soft taints such as atlas.bstein.dev/spillover=true:PreferNoSchedule or longhorn=true:PreferNoSchedule unless the repo’s desired-state policy says to.

Tests:

• Missing cryptsetup with valid sudo installs package, loads module if needed, uncordons node.
• Missing cryptsetup with missing sudo secret reports privilege blocker and leaves cordon with reason.
• Node already has package but missing kernel module: module load path tested.
• Node has Longhorn ready disks but is Kubernetes cordoned: Longhorn schedulability blocker is reported.
• Uncordon preserves unrelated soft taints.
• Re-running recovery is idempotent.

5. Stale RWO PVC Owner Recovery

Implement a generic stale-owner recovery flow for controller-owned pods and RWO PVCs.

Detection:

• Old pod has metadata.deletionTimestamp older than threshold.
• Replacement pod from same controller exists and is pending/initializing.
• Replacement events include Multi-Attach, FailedAttachVolume, Volume is already used by pod(s), or Volume is already exclusively attached to one node.
• PVC is ReadWriteOnce or equivalent single-writer mode.
• Old and new pods are on different nodes.
• Longhorn Volume/VolumeAttachment shows old node still owns attachment or new attach blocked.
• Ordinary pod deletes have been attempted repeatedly without changing ownership.

Recovery grouping:

• Group old pod, replacement pod, controller, PVC, PV, VolumeAttachment(s), Longhorn Volume, old node, new node, and relevant events into one incident.
• Emit one concise operator record.
• Stop repeated blind pod deletes while the incident is active.

Safety decisions:

• Inspect old pod container statuses and volume mounts.
• Classify containers:
  • PVC-writing application containers: mount the blocked PVC.
  • Sidecars that do not mount the blocked PVC: Vault agent, projected service account, emptyDir-only, etc.
  • Unknown containers: treat as unsafe.
• If only sidecars remain and no container mounts the PVC, force-delete the stale pod after a bounded wait.
• If an application container still runs and mounts the PVC, do not blindly force-delete. Instead:
  • Wait for kubelet termination if it is making progress.
  • If stalled and host runtime inspection is available through Vault-backed sudo, inspect and optionally stop the specific container through a controlled allowlisted path.
  • If Ananke cannot prove the PVC-writing container is stopped, report unsafe-stale-owner and do not clear the API object.
• If old node kubelet/runtime is unhealthy, route through managed node recovery before stale-owner cleanup.

Tests:

• Sidecar-only stale pod force-clears and replacement attaches.
• Live app container with blocked PVC waits and does not force-delete.
• Live app container eventually exits; replacement attaches without force-delete.
• Host runtime inspection unavailable due sudo privilege reports unsafe blocker.
• Host runtime inspection available and controlled container stop succeeds.
• Longhorn detach/attach follows cleanup.
• Replacement readiness is rechecked after attach.
• Multiple PVCs in one pod are handled.
• Non-RWO PVC is not processed by this recovery path.

6. Vault Health Robustness

Startup should not fail because one Vault probe path is flaky if other authoritative health signals prove Vault is usable.

Requirements:

• Prefer multiple health signals:
  • pod Running/Ready
  • service endpoints populated
  • HTTP /v1/sys/health through service or pod proxy
  • vault status through exec as one signal, not the only signal
• If kubectl exec vault status is killed/transiently fails but HTTP health reports initialized/unsealed/active or standby-acceptable per config, classify as transient exec failure and continue with warning.
• If HTTP health and exec disagree, report clearly and retry with bounded backoff.
• Never log tokens.

Tests:

• exec killed but HTTP health unsealed => startup continues with warning.
• exec says sealed and HTTP says sealed => startup blocks/unseals.
• endpoints missing => startup waits.
• HTTP unavailable but exec succeeds => startup may proceed if policy allows.

7. Image Pull And DNS Failure Classification

Ananke should not repeatedly recycle pods whose only issue is unchanged image pull or DNS failure.

Requirements:

• Detect ImagePullBackOff/ErrImagePull events caused by DNS/registry lookup.
• Run separate DNS checks:
  • CoreDNS pods ready
  • in-cluster DNS query from a known diagnostic pod if available
  • node host DNS or kubelet image pull DNS failures grouped by node
  • registry reachability by hostname class, not one app
• If the image pull failure is unchanged and recent, do not repeatedly delete the pod.
• Emit a compact blocker: image, registry host, node, last error, and whether DNS is cluster-wide or node-specific.
• Retry only after DNS/registry health changes or after bounded interval.

Tests:

• Docker Hub auth DNS failure is classified.
• Private registry DNS failure is classified.
• One-off pull failure that later succeeds is not over-escalated.
• Repeated pod recycle is suppressed.

8. Startup Completion And Post-Success Monitoring

Ananke declared startup_status=success while late runtime/storage issues still surfaced. Make completion smarter.

Requirements:

• Startup can reach "cluster core recovered" while still monitoring app convergence.
• Do not mark total recovery complete until:
  • critical services pass
  • all managed nodes are either healthy, intentionally excluded, or explicitly blocked
  • Longhorn node readiness/schedulability matches desired policy
  • no active node-runtime incidents exist
  • no unsafe stale-owner incidents exist
• Add a post-success watch window for late events: sandbox reservation, FailedKillPod, kubelet metrics scrape failures, Longhorn manager missing, VolumeAttachment errors.
• Distinguish statuses:
  • core_recovered
  • converging
  • success
  • success_with_noncritical_blockers
  • blocked_requires_operator
• Report app-level blockers separately from cluster-level blockers.

Tests:

• Critical services pass but node runtime wedge appears during post-success window => not final success until resolved.
• Noncritical app image pull blocker can be success_with_noncritical_blockers.
• Unsafe stale RWO owner blocks full success.
• All clear => success.

Status And Operator Reporting

Improve operator output. The user should not need to read journals to know what is happening.

Status should include:

• Whether a startup/recovery workflow is already running.
• Current phase and elapsed time.
• Current incidents grouped by type:
  • node runtime recovery
  • Longhorn/Kubernetes drift
  • stale RWO owner
  • Vault health
  • image/DNS pull
  • host privilege/preflight
• For each incident: resource, trigger signals, action taken, next action, elapsed time, and whether operator input is needed.
• Counts should be accurate and stable.
• Repeated noisy details should be summarized with "same blocker still present for X minutes" instead of repeated lines.

Security Constraints

• Do not log sudo passwords, Vault tokens, unsealed keys, or rendered secret contents.
• Redact secrets in command arguments and environment.
• Use command allowlists for privileged host actions.
• Make destructive operations explicit and guarded:
  • reboot only after escalation criteria
  • force-delete pod only after stale-owner safety checks
  • deleting VolumeAttachments only if repo policy allows and after safeguards
• Keep audit logs of actions: node/resource, reason, command class, result, elapsed time, but not secrets.

Implementation Guidance

Before editing:

• Read Ananke’s existing startup/recovery code, Kubernetes client abstractions, Vault access layer, SSH/host command layer, Longhorn helpers if any, status/progress writer, and tests.
• Reuse existing abstractions. Do not invent a parallel framework unless necessary.
• Use fake clients and table-driven tests where the repo already uses them.
• Keep changes modular:
  • host privilege provider
  • node runtime incident detector/reconciler
  • Longhorn readiness reconciler
  • stale RWO owner reconciler
  • image/DNS classifier
  • status/progress summarizer
• Ensure all reconcilers are idempotent and bounded.

Suggested model:

• Define incident structs with stable keys, timestamps, observed signals, action history, next action, and terminal status.
• Every recovery action should require:
  • detection evidence
  • policy permission
  • safety gate
  • timeout
  • postcondition check
• Avoid "delete pod until healthy" loops. Replace with evidence-driven remediation.

Validation Plan

Run the repository’s standard checks first, then add targeted tests.

Required validation:

• Unit tests for each new classifier/reconciler.
• Tests for redaction/no secret leakage.
• Tests for idempotency on repeated reconcile calls.
• Tests for progress/status output.
• Integration-style tests with fake Kubernetes/Longhorn objects:
  • nodes, pods, events, PVCs, PVs, VolumeAttachments, Longhorn Node CRs, Longhorn Volume CRs, DaemonSets, and pods.
• Host command fake tests:
  • SSH unavailable
  • sudo unavailable
  • sudo succeeds
  • command timeout
  • k3s-agent service states
  • reboot accepted/not accepted
• Vault fake tests:
  • secret found
  • secret missing
  • Vault sealed/unavailable
  • HTTP health fallback
• End-to-end scenario tests composed from the incident:
  • titan-05 hard runtime wedge -> reboot -> Ready
  • titan-04 post-success sandbox wedge -> restart fails -> reboot -> Ready
  • titan-15/titan-17 cryptsetup/cordon flow -> sudo repair or explicit blocker -> uncordon when safe
  • titan-22 missing longhorn-host label -> restore label -> manager pod appears -> Longhorn Ready
  • Firefly sidecar-only stale owner -> force delete -> attach replacement
  • Firefly live app container stale owner -> wait/report unsafe, no force delete
  • Vault exec killed but HTTP healthy -> continue
  • DNS image pull errors -> classify without repeated recycle

Run commands appropriate for the repo, likely including:

go test ./...
go test -race ./...
go vet ./...

If the repo uses other tooling, discover and run it. If any expensive/integration tests require cluster credentials, document how to run them and ensure fake-client tests cover the logic by default.

Acceptance Criteria

The work is complete only when:

• Ananke can retrieve and use Vault-backed sudo credentials securely or clearly reports why it cannot.
• Node runtime wedges are detected and recovered by a bounded escalation ladder.
• Ananke can handle Ready nodes that cannot create sandboxes.
• Ananke can recover from k3s-agent stop-sigkill/timeout by controlled reboot when configured.
• Ananke does not leave Longhorn nodes cordoned indefinitely after repair/override conditions are satisfied.
• Ananke detects and fixes or reports Longhorn/Kubernetes drift, including missing manager-pod labels.
• Ananke verifies Longhorn node readiness before trusting Kubernetes node readiness for PVC workloads.
• Ananke handles stale RWO ownership safely, distinguishing sidecar-only stale pods from live PVC-writing app containers.
• Ananke avoids noisy repeated deletion of unchanged image-pull failures.
• Vault health checks tolerate transient exec failures when HTTP health is authoritative.
• Startup success reflects real cluster recovery, with explicit success_with_noncritical_blockers or blocked_requires_operator states where appropriate.
• All changes are covered by rigorous tests and pass the repo’s standard test suite.

What Not To Do

• Do not hard-code Titan node names, app names, PVC names, or namespaces.
• Do not log secrets.
• Do not delete PVs, PVCs, or Longhorn Volume data.
• Do not force-delete pods with live app containers mounting RWO PVCs unless a controlled, audited stop path has proven storage safety.
• Do not repeatedly restart/reboot nodes without incident state and cooldowns.
• Do not treat Kubernetes Ready as equivalent to Longhorn Ready.
• Do not declare full success while node-runtime or unsafe stale-owner incidents are active.

Useful Terminology From The Incident

• "Node runtime wedge": Kubernetes or host state where kubelet/containerd cannot create/kill sandboxes even if the node appears partially Ready.
• "Stale RWO owner": a terminating pod or runtime survivor that still holds an exclusive PVC and blocks replacement attach.
• "Sidecar-only stale owner": old pod still exists, but only non-PVC sidecars remain; force deletion may be safe after checks.
• "Unsafe stale owner": app container still running and mounting the PVC; wait or controlled stop required.
• "Longhorn/Kubernetes drift": Kubernetes node readiness/labels and Longhorn node readiness/manager state disagree.
• "Host privilege unavailable": Ananke cannot run necessary sudo actions despite needing host-level repair.

Use this prompt to implement production-grade, categorical recovery logic in Ananke, not one-off incident handling.