ananke/docs/ananke-hardening-prompt-2026-07-07.md

# Prompt For Ananke Recovery Hardening

You are a fresh, stateless Codex session working in the Ananke repository. Your goal is to implement, test, and validate generalized recovery improvements in Ananke based on a real Titan cluster power-loss incident on 2026-07-07. Treat this as production reliability work: read the repo first, follow existing code style and architecture, keep changes cohesive, and solve failure classes categorically rather than hard-coding node names, namespaces, or one-off incidents.

## Mission

Make Ananke resilient and automatic during post-power-loss cluster recovery. After your changes, one Ananke startup/recovery process should be able to:

- Bring Vault and critical dependencies back safely.
- Repair or escalate wedged k3s/containerd nodes using existing Ananke SSH/host access.
- Use sudo credentials from Vault, securely and noninteractively, for host repair and diagnostics.
- Avoid leaving Longhorn storage nodes cordoned indefinitely.
- Detect Kubernetes/Longhorn readiness mismatches and repair label/manager/VolumeAttachment drift.
- Resolve stale pod/PVC ownership safely, with different behavior for sidecar-only stale owners versus still-running application containers that mount the PVC.
- Continue post-success convergence monitoring long enough to catch late node/runtime/storage wedges.
- Report concise, actionable blockers without noisy repeated pod recycling.

Do not implement special cases for `titan-04`, `titan-05`, `titan-15`, `titan-17`, `titan-22`, Firefly, Jellyfin, or any specific app. Use those names only as incident fixtures/test names. The code should operate on generic managed nodes, generic Longhorn nodes, generic Kubernetes workloads, and generic RWO PVCs.

## Incident Context

The Titan cluster lost power for a few minutes. Ariadne may have started shutdown behavior. The operator asked for a single Ananke process to bring the cluster back up and learn from any manual help required.

Ananke did a lot correctly:

- It recovered Vault and unsealed it.
- It recovered critical workloads, Vault injector, Postgres, Keycloak, OpenSearch, oauth2 logs, CoreDNS, metrics-server, and many app pods.
- It ran a single startup workflow and eventually completed successfully.
- It recycled many stuck pods and ran several checks.

But Ananke needed manual assistance in multiple reusable failure classes:

- Node runtime wedges where Kubernetes readiness did not fully reflect kubelet/containerd health.
- Host repairs blocked by sudo privilege gaps.
- Longhorn nodes cordoned by Kubernetes remained unavailable for storage scheduling.
- Longhorn readiness diverged from Kubernetes readiness.
- Missing node labels caused Longhorn manager pods to disappear.
- Stale terminating pods held RWO volumes and blocked replacement pods.
- Repeated pod deletes did not resolve storage ownership or runtime reservation issues.
- Ananke declared startup success before all late runtime/storage fallout was resolved.

## Live Incident Facts To Encode As Tests

Use these as fixtures for tests and documentation, not as hard-coded behavior:

- `titan-05`: k3s-agent restart hung. Host SSH worked. `k3s-agent.service` entered `deactivating`, `final-sigkill`, `activating/start`, and `Result=timeout`. Kubelet port `10250` stayed closed. Kubernetes heartbeat was stale with `node.kubernetes.io/unreachable` taints. In-cluster helper pod could not start. Manual controlled reboot recovered it.
- `titan-04`: after Ananke reported startup success, the node was Kubernetes `Ready` and kubelet `10250` initially open, but new pods failed with `FailedCreatePodSandBox`, `failed to reserve sandbox name`, and `context deadline exceeded`. A non-blocking `k3s-agent` restart then stuck in `deactivating/stop-sigkill`, kubelet `10250` closed, Kubernetes moved to `Ready=Unknown`, and a controlled reboot recovered it.
- `titan-07`: a pod looked running in the API, but `kubectl exec` failed with `cannot exec in a deleted state`; restarting k3s-agent and recreating the pod fixed it.
- `titan-14`: a replacement pod hit `failed to reserve container name` / `CreateContainerError`; k3s-agent restart and rerunning the in-cluster helper fixed it. Later another pod on the same node showed the same sandbox reservation pattern, but then progressed after enough time. Tests should distinguish transient progress from a stall.
- `titan-15` and `titan-17`: Ananke cordoned them after encrypted Longhorn mounts exposed missing `cryptsetup`. Longhorn itself had `allowScheduling=true` and ready disks, but Longhorn `Schedulable=False` because Kubernetes had cordoned the nodes. Manual `kubectl uncordon titan-15 titan-17` restored Kubernetes scheduling and Longhorn schedulability. Ananke had attempted `cryptsetup-bin` repair but failed because `atlas` sudo required a password.
- `titan-22`: Kubernetes `Ready=True`, CSI plugin and engine images were running, but Longhorn node `Ready=False` because `longhorn-manager` pod was missing. The root cause was missing node label `longhorn-host=true`, while the `longhorn-manager` DaemonSet selected `longhorn-host=true`. Manual `kubectl label node titan-22 longhorn-host=true --overwrite` started `longhorn-manager` and Longhorn node Ready became true. A pending wallet pod still carried an old failed VolumeAttachment until the pod was deleted/recreated.
- Firefly stale owner, safe-to-force variant: `finance/firefly-...-q8rtq` was terminating for more than 25 minutes, had no finalizers, and only a Vault sidecar remained running. Replacement pod was blocked by `Multi-Attach` for an RWO PVC. Force-deleting the stale pod cleared ownership and allowed Longhorn attach to the replacement.
- Firefly stale owner, unsafe-to-force variant: a later terminating Firefly pod still had the main application container running and mounting the RWO PVC. It was not safe to blindly force-delete. Waiting allowed kubelet to finish termination, then Longhorn attached the volume to the replacement.
- Manual runtime inspection attempt on `titan-06`: `sudo -n crictl ...` failed because `atlas` required a sudo password. Ananke is expected to have access to sudo passwords in Vault, so this must be designed as a first-class Ananke capability rather than manual operator shell access.
- Vault health check robustness: at one point `kubectl exec vault-0 -- vault status` was killed, but HTTP health via Kubernetes pod proxy showed `initialized=true`, `sealed=false`, `standby=false`. Startup should not fail solely on transient `kubectl exec` failure when endpoint and HTTP health prove Vault is available.
- Image/DNS issues: several image pulls failed with `lookup registry-1.docker.io`, `lookup auth.docker.io`, or `lookup production.cloudfront.docker.com: Try again`, then later succeeded. Ananke repeatedly recycling unchanged ImagePullBackOff pods created noise.

## Required Implementation Areas

### 1. Secure Sudo Capability From Vault

Ananke should be able to run bounded host diagnostics and repairs that require sudo.

Requirements:

- Discover and use the existing Vault integration and secret naming conventions already present in the repo/config. Do not invent a new hard-coded secret path if the repo has one.
- Support noninteractive sudo for configured managed nodes.
- Never log sudo passwords, Vault tokens, rendered secrets, command input containing secrets, or full environment dumps.
- Provide a preflight check: for each managed node, can Ananke retrieve host sudo material and can it run a harmless command such as `sudo -S -p '' true` with a timeout?
- If credentials are missing, wrong, or Vault is unavailable, Ananke should report `host-privilege-unavailable` with node, intended action, and secret lookup class, not spin.
- Use strict command allowlists for privileged commands. The recovery path should not become arbitrary remote root execution.
- Use bounded timeouts for all host commands and classify timeout versus auth failure versus command failure.
- Prefer safe wrappers for:
  - `systemctl show/is-active/restart --no-block k3s-agent`
  - `systemctl reboot` when configured and escalated
  - `crictl ps/pods/inspect/stop` for recovery diagnostics and controlled container stop
  - package checks/install for `cryptsetup-bin`, `open-iscsi`, `nfs-common`, `dmsetup`
  - `modprobe dm_crypt` or equivalent kernel-module validation where appropriate

Tests:

- Vault lookup succeeds and sudo command succeeds.
- Vault lookup succeeds but sudo password is wrong.
- Vault lookup missing.
- Vault unavailable while Vault HTTP health is also degraded.
- Vault unavailable but action does not require sudo.
- Command times out.
- Command returns nonzero.
- Logs are scanned to ensure no password appears.

### 2. Managed Node Runtime Recovery Primitive

Implement a generic state machine for node runtime recovery.

Detection signals:

- Kubernetes node `Ready=Unknown` or stale heartbeat.
- Kubernetes node `Ready=True` but pods on that node show repeated:
  - `FailedCreatePodSandBox`
  - `failed to reserve sandbox name`
  - `failed to reserve container name`
  - `CreateContainerError`
  - `FailedKillPod`
  - long-lived `ContainerCreating` or `PodInitializing` with no progress
  - `kubectl exec`/logs failures indicating deleted runtime state
- Host SSH works but kubelet port `127.0.0.1:10250` is closed or slow.
- `systemctl show k3s-agent` reports `deactivating`, `stop-sigterm`, `stop-sigkill`, `final-sigterm`, `final-sigkill`, `activating/start`, or `Result=timeout`.
- Existing in-cluster maintenance helper pod for the node cannot start, stays `ContainerCreating`, or cannot be scheduled because the node runtime is already unhealthy.

Escalation ladder:

- Open a single node recovery incident and suppress noisy repeated pod recycling for pods on that node while host repair is in progress.
- Cordon the node before host-level intervention unless it is already unreachable/tainted.
- Try in-cluster maintenance helper if kubelet/runtime can create pods.
- If helper cannot run, use Ananke SSH route and sudo from Vault.
- Prefer `systemctl --no-block restart k3s-agent`; never use a blocking restart over SSH as the primary recovery action.
- Wait a bounded interval and require both:
  - Kubernetes Ready=True with fresh heartbeat
  - local kubelet `10250` open
- If k3s-agent remains stuck in stop/kill/activating states or heartbeats remain stale, escalate to configured controlled reboot.
- After reboot, wait for SSH, k3s-agent active/running, kubelet port open, Kubernetes Ready=True, and unreachable taints cleared.
- Uncordon only after the node is healthy and after storage-specific checks pass.
- Continue this monitoring after startup success for a configurable post-success convergence window.

Tests:

- Ready node with repeated sandbox reservation events triggers recovery.
- Ready node with one transient sandbox event and then progress does not trigger restart.
- Unknown node with SSH unavailable reports external blocker without repeated action.
- Unknown node with SSH available, kubelet closed, k3s-agent stuck, helper unavailable escalates to reboot.
- Non-blocking restart recovers node.
- Non-blocking restart moves service to `deactivating/stop-sigkill`; reboot escalation occurs.
- Reboot command accepted but SSH does not return within timeout.
- Reboot returns and all readiness conditions clear.
- False positive prevention: node Ready=True, kubelet open, events old/stale only, no action.
- Idempotency: repeated Ananke runs do not issue repeated restarts/reboots while an incident is active.

### 3. Longhorn And Kubernetes Readiness Reconciliation

Ananke must not treat Kubernetes node Ready as sufficient for storage workloads.

Implement a Longhorn readiness reconciler:

- Read Kubernetes Nodes, Longhorn Node CRs, Longhorn manager DaemonSet, Longhorn manager pods, CSI pods, engine-image pods, Volume CRs, VolumeAttachments, and relevant PVC-bound pods.
- Compare:
  - Kubernetes Ready/SchedulingDisabled/taints
  - Node labels required by Longhorn manager DaemonSet, especially `longhorn-host=true`
  - Longhorn Node `Ready` and `Schedulable` conditions
  - Longhorn Node `spec.allowScheduling`
  - Longhorn manager pod presence on nodes where Longhorn Node exists or where PVC workloads are allowed
  - CSI plugin readiness
  - VolumeAttachment errors that reference stale Longhorn node readiness
- Detect label drift: Longhorn node exists and CSI/engine pods exist, but manager pod is missing because the node no longer matches the DaemonSet selector.
- Repair label drift only if it is safe:
  - There is a known desired label source in repo/config/state, or
  - Longhorn Node CR exists for that Kubernetes node and existing Longhorn metadata indicates the node is intended to be a Longhorn node.
  - Emit exactly what label will be restored and why.
- If safe repair cannot be proven, cordon or avoid scheduling PVC workloads to the node and report a blocker.
- After restoring labels or manager pods, wait for Longhorn Node Ready=True.
- For VolumeAttachments that failed because Longhorn thought a now-ready node was not ready, trigger a bounded retry:
  - Prefer deleting/recreating the controller-owned Pending pod if that is the established pattern.
  - Delete stale failed VolumeAttachment only if Kubernetes controller semantics and repo policy allow it.
  - Never delete PV/PVC/Longhorn Volume data.

Specific expectations from incident:

- `titan-22` class: Kubernetes Ready, missing `longhorn-host=true`, no manager pod, Longhorn Node Ready=False ManagerPodMissing. Ananke should restore label or report exact unsafe reason, then wait for manager pod and Longhorn Ready.
- `titan-15`/`titan-17` class: Kubernetes cordon makes Longhorn Schedulable=False despite Longhorn `allowScheduling=true` and ready disks. Ananke should not leave them cordoned after the blocking repair condition is resolved or after operator-approved override.

Tests:

- Longhorn manager selector changes are handled generically.
- Missing Longhorn label restored from desired state.
- Missing Longhorn label not restored if node lacks Longhorn Node CR and no desired-state evidence exists.
- Longhorn Node Ready=False ManagerPodMissing becomes Ready after label restoration and manager pod startup.
- VolumeAttachment error due stale Longhorn node readiness is retried after Longhorn Ready.
- Kubernetes cordoned node causes Longhorn Schedulable=False and is uncordoned after repair.
- Kubernetes cordoned node remains cordoned if required package/kernel preflight still fails and no override is configured.
- `allowScheduling=false` is respected; Ananke should not enable Longhorn disk scheduling unless explicitly configured, but it may still ensure manager/CSI readiness for attachment if workloads are allowed there.

### 4. Cordon/Uncordon And Host Package Repair Policy

Ananke cordoned `titan-15` and `titan-17` because encrypted Longhorn mounts exposed missing `cryptsetup`; it failed to install because sudo required a password. This must become a closed-loop repair.

Requirements:

- Distinguish:
  - Kubernetes scheduling cordon
  - Longhorn disk scheduling (`allowScheduling`)
  - Longhorn node readiness/schedulability
  - ordinary `PreferNoSchedule` taints used as soft placement hints
- If Ananke cordons a node, it must record why, when, what remediation is required, and what condition will uncordon it.
- If the remediation is package/module installation, use Vault-backed sudo and verify the result.
- If package repair succeeds, uncordon automatically and verify Longhorn Schedulable=True.
- If package repair fails due missing sudo secret, wrong sudo password, package manager lock, apt/dpkg failure, or network outage, report the exact class and do not leave the node silently disabled.
- Support an operator override policy for "uncordon for Longhorn availability even if encrypted workload preflight is not perfect", but this must be explicit/configured and logged.
- Do not remove existing soft taints such as `atlas.bstein.dev/spillover=true:PreferNoSchedule` or `longhorn=true:PreferNoSchedule` unless the repo’s desired-state policy says to.

Tests:

- Missing cryptsetup with valid sudo installs package, loads module if needed, uncordons node.
- Missing cryptsetup with missing sudo secret reports privilege blocker and leaves cordon with reason.
- Node already has package but missing kernel module: module load path tested.
- Node has Longhorn ready disks but is Kubernetes cordoned: Longhorn schedulability blocker is reported.
- Uncordon preserves unrelated soft taints.
- Re-running recovery is idempotent.

### 5. Stale RWO PVC Owner Recovery

Implement a generic stale-owner recovery flow for controller-owned pods and RWO PVCs.

Detection:

- Old pod has `metadata.deletionTimestamp` older than threshold.
- Replacement pod from same controller exists and is pending/initializing.
- Replacement events include `Multi-Attach`, `FailedAttachVolume`, `Volume is already used by pod(s)`, or `Volume is already exclusively attached to one node`.
- PVC is `ReadWriteOnce` or equivalent single-writer mode.
- Old and new pods are on different nodes.
- Longhorn Volume/VolumeAttachment shows old node still owns attachment or new attach blocked.
- Ordinary pod deletes have been attempted repeatedly without changing ownership.

Recovery grouping:

- Group old pod, replacement pod, controller, PVC, PV, VolumeAttachment(s), Longhorn Volume, old node, new node, and relevant events into one incident.
- Emit one concise operator record.
- Stop repeated blind pod deletes while the incident is active.

Safety decisions:

- Inspect old pod container statuses and volume mounts.
- Classify containers:
  - PVC-writing application containers: mount the blocked PVC.
  - Sidecars that do not mount the blocked PVC: Vault agent, projected service account, emptyDir-only, etc.
  - Unknown containers: treat as unsafe.
- If only sidecars remain and no container mounts the PVC, force-delete the stale pod after a bounded wait.
- If an application container still runs and mounts the PVC, do not blindly force-delete. Instead:
  - Wait for kubelet termination if it is making progress.
  - If stalled and host runtime inspection is available through Vault-backed sudo, inspect and optionally stop the specific container through a controlled allowlisted path.
  - If Ananke cannot prove the PVC-writing container is stopped, report `unsafe-stale-owner` and do not clear the API object.
- If old node kubelet/runtime is unhealthy, route through managed node recovery before stale-owner cleanup.

Tests:

- Sidecar-only stale pod force-clears and replacement attaches.
- Live app container with blocked PVC waits and does not force-delete.
- Live app container eventually exits; replacement attaches without force-delete.
- Host runtime inspection unavailable due sudo privilege reports unsafe blocker.
- Host runtime inspection available and controlled container stop succeeds.
- Longhorn detach/attach follows cleanup.
- Replacement readiness is rechecked after attach.
- Multiple PVCs in one pod are handled.
- Non-RWO PVC is not processed by this recovery path.

### 6. Vault Health Robustness

Startup should not fail because one Vault probe path is flaky if other authoritative health signals prove Vault is usable.

Requirements:

- Prefer multiple health signals:
  - pod Running/Ready
  - service endpoints populated
  - HTTP `/v1/sys/health` through service or pod proxy
  - `vault status` through exec as one signal, not the only signal
- If `kubectl exec vault status` is killed/transiently fails but HTTP health reports initialized/unsealed/active or standby-acceptable per config, classify as transient exec failure and continue with warning.
- If HTTP health and exec disagree, report clearly and retry with bounded backoff.
- Never log tokens.

Tests:

- exec killed but HTTP health unsealed => startup continues with warning.
- exec says sealed and HTTP says sealed => startup blocks/unseals.
- endpoints missing => startup waits.
- HTTP unavailable but exec succeeds => startup may proceed if policy allows.

### 7. Image Pull And DNS Failure Classification

Ananke should not repeatedly recycle pods whose only issue is unchanged image pull or DNS failure.

Requirements:

- Detect ImagePullBackOff/ErrImagePull events caused by DNS/registry lookup.
- Run separate DNS checks:
  - CoreDNS pods ready
  - in-cluster DNS query from a known diagnostic pod if available
  - node host DNS or kubelet image pull DNS failures grouped by node
  - registry reachability by hostname class, not one app
- If the image pull failure is unchanged and recent, do not repeatedly delete the pod.
- Emit a compact blocker: image, registry host, node, last error, and whether DNS is cluster-wide or node-specific.
- Retry only after DNS/registry health changes or after bounded interval.

Tests:

- Docker Hub auth DNS failure is classified.
- Private registry DNS failure is classified.
- One-off pull failure that later succeeds is not over-escalated.
- Repeated pod recycle is suppressed.

### 8. Startup Completion And Post-Success Monitoring

Ananke declared `startup_status=success` while late runtime/storage issues still surfaced. Make completion smarter.

Requirements:

- Startup can reach "cluster core recovered" while still monitoring app convergence.
- Do not mark total recovery complete until:
  - critical services pass
  - all managed nodes are either healthy, intentionally excluded, or explicitly blocked
  - Longhorn node readiness/schedulability matches desired policy
  - no active node-runtime incidents exist
  - no unsafe stale-owner incidents exist
- Add a post-success watch window for late events: sandbox reservation, FailedKillPod, kubelet metrics scrape failures, Longhorn manager missing, VolumeAttachment errors.
- Distinguish statuses:
  - `core_recovered`
  - `converging`
  - `success`
  - `success_with_noncritical_blockers`
  - `blocked_requires_operator`
- Report app-level blockers separately from cluster-level blockers.

Tests:

- Critical services pass but node runtime wedge appears during post-success window => not final success until resolved.
- Noncritical app image pull blocker can be `success_with_noncritical_blockers`.
- Unsafe stale RWO owner blocks full success.
- All clear => success.

## Status And Operator Reporting

Improve operator output. The user should not need to read journals to know what is happening.

Status should include:

- Whether a startup/recovery workflow is already running.
- Current phase and elapsed time.
- Current incidents grouped by type:
  - node runtime recovery
  - Longhorn/Kubernetes drift
  - stale RWO owner
  - Vault health
  - image/DNS pull
  - host privilege/preflight
- For each incident: resource, trigger signals, action taken, next action, elapsed time, and whether operator input is needed.
- Counts should be accurate and stable.
- Repeated noisy details should be summarized with "same blocker still present for X minutes" instead of repeated lines.

## Security Constraints

- Do not log sudo passwords, Vault tokens, unsealed keys, or rendered secret contents.
- Redact secrets in command arguments and environment.
- Use command allowlists for privileged host actions.
- Make destructive operations explicit and guarded:
  - reboot only after escalation criteria
  - force-delete pod only after stale-owner safety checks
  - deleting VolumeAttachments only if repo policy allows and after safeguards
- Keep audit logs of actions: node/resource, reason, command class, result, elapsed time, but not secrets.

## Implementation Guidance

Before editing:

- Read Ananke’s existing startup/recovery code, Kubernetes client abstractions, Vault access layer, SSH/host command layer, Longhorn helpers if any, status/progress writer, and tests.
- Reuse existing abstractions. Do not invent a parallel framework unless necessary.
- Use fake clients and table-driven tests where the repo already uses them.
- Keep changes modular:
  - host privilege provider
  - node runtime incident detector/reconciler
  - Longhorn readiness reconciler
  - stale RWO owner reconciler
  - image/DNS classifier
  - status/progress summarizer
- Ensure all reconcilers are idempotent and bounded.

Suggested model:

- Define incident structs with stable keys, timestamps, observed signals, action history, next action, and terminal status.
- Every recovery action should require:
  - detection evidence
  - policy permission
  - safety gate
  - timeout
  - postcondition check
- Avoid "delete pod until healthy" loops. Replace with evidence-driven remediation.

## Validation Plan

Run the repository’s standard checks first, then add targeted tests.

Required validation:

- Unit tests for each new classifier/reconciler.
- Tests for redaction/no secret leakage.
- Tests for idempotency on repeated reconcile calls.
- Tests for progress/status output.
- Integration-style tests with fake Kubernetes/Longhorn objects:
  - nodes, pods, events, PVCs, PVs, VolumeAttachments, Longhorn Node CRs, Longhorn Volume CRs, DaemonSets, and pods.
- Host command fake tests:
  - SSH unavailable
  - sudo unavailable
  - sudo succeeds
  - command timeout
  - k3s-agent service states
  - reboot accepted/not accepted
- Vault fake tests:
  - secret found
  - secret missing
  - Vault sealed/unavailable
  - HTTP health fallback
- End-to-end scenario tests composed from the incident:
  - `titan-05` hard runtime wedge -> reboot -> Ready
  - `titan-04` post-success sandbox wedge -> restart fails -> reboot -> Ready
  - `titan-15`/`titan-17` cryptsetup/cordon flow -> sudo repair or explicit blocker -> uncordon when safe
  - `titan-22` missing `longhorn-host` label -> restore label -> manager pod appears -> Longhorn Ready
  - Firefly sidecar-only stale owner -> force delete -> attach replacement
  - Firefly live app container stale owner -> wait/report unsafe, no force delete
  - Vault exec killed but HTTP healthy -> continue
  - DNS image pull errors -> classify without repeated recycle

Run commands appropriate for the repo, likely including:

```bash
go test ./...
go test -race ./...
go vet ./...
```

If the repo uses other tooling, discover and run it. If any expensive/integration tests require cluster credentials, document how to run them and ensure fake-client tests cover the logic by default.

## Acceptance Criteria

The work is complete only when:

- Ananke can retrieve and use Vault-backed sudo credentials securely or clearly reports why it cannot.
- Node runtime wedges are detected and recovered by a bounded escalation ladder.
- Ananke can handle Ready nodes that cannot create sandboxes.
- Ananke can recover from k3s-agent stop-sigkill/timeout by controlled reboot when configured.
- Ananke does not leave Longhorn nodes cordoned indefinitely after repair/override conditions are satisfied.
- Ananke detects and fixes or reports Longhorn/Kubernetes drift, including missing manager-pod labels.
- Ananke verifies Longhorn node readiness before trusting Kubernetes node readiness for PVC workloads.
- Ananke handles stale RWO ownership safely, distinguishing sidecar-only stale pods from live PVC-writing app containers.
- Ananke avoids noisy repeated deletion of unchanged image-pull failures.
- Vault health checks tolerate transient exec failures when HTTP health is authoritative.
- Startup success reflects real cluster recovery, with explicit `success_with_noncritical_blockers` or `blocked_requires_operator` states where appropriate.
- All changes are covered by rigorous tests and pass the repo’s standard test suite.

## What Not To Do

- Do not hard-code Titan node names, app names, PVC names, or namespaces.
- Do not log secrets.
- Do not delete PVs, PVCs, or Longhorn Volume data.
- Do not force-delete pods with live app containers mounting RWO PVCs unless a controlled, audited stop path has proven storage safety.
- Do not repeatedly restart/reboot nodes without incident state and cooldowns.
- Do not treat Kubernetes Ready as equivalent to Longhorn Ready.
- Do not declare full success while node-runtime or unsafe stale-owner incidents are active.

## Useful Terminology From The Incident

- "Node runtime wedge": Kubernetes or host state where kubelet/containerd cannot create/kill sandboxes even if the node appears partially Ready.
- "Stale RWO owner": a terminating pod or runtime survivor that still holds an exclusive PVC and blocks replacement attach.
- "Sidecar-only stale owner": old pod still exists, but only non-PVC sidecars remain; force deletion may be safe after checks.
- "Unsafe stale owner": app container still running and mounting the PVC; wait or controlled stop required.
- "Longhorn/Kubernetes drift": Kubernetes node readiness/labels and Longhorn node readiness/manager state disagree.
- "Host privilege unavailable": Ananke cannot run necessary sudo actions despite needing host-level repair.

Use this prompt to implement production-grade, categorical recovery logic in Ananke, not one-off incident handling.