diff --git a/docs/CLUSTER_STABILIZATION.md b/docs/CLUSTER_STABILIZATION.md index 3ba5abee..76cc92ce 100644 --- a/docs/CLUSTER_STABILIZATION.md +++ b/docs/CLUSTER_STABILIZATION.md @@ -5,6 +5,32 @@ This record begins on 2026-10-03 UTC, following authorization to implement the The October 2 audit remains a historical baseline, not a current health claim. Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path. +## Six-priority progress tracker + +Started against the user's approved list on 2026-10-03. Updated: 20:45 UTC. +"Verified" means the stated check passed; it does not imply a completed soak or +failure drill. Physical repairs and disruptive recovery drills remain separate +from routine software work. + +| Priority | Status | Completed evidence | Next action / completion gate | +| --- | --- | --- | --- | +| 1. Immediate software repairs | In progress | Soteria test builds 1293/1294 passed; release 1295 failed its supply-chain gate. Monerod's LMDB open fails with I/O error despite healthy Longhorn replicas | Correct the release findings, publish and validate a backup; repair Monerod's storage path without deleting its database | +| 2. Reliable nodes | Partial; physical checks needed | Titan-04/14/18 excluded from new placement; Titan-05/06 remain offline. Fresh Titan-04/11 undervoltage was observed | Power/media checks, preservation or replacement of suspect media, then networking/storage/load validation before return. Establish the actual condition of 09/10/16 | +| 3. Recovery coverage | In progress | External datastore restore and second LAN copy verified; 19 application DB dumps completed; scheduled application backup completed at 08:37 UTC; representative Gitea restore passed | Complete eligible PVC coverage, verify Soteria restore, resolve Hermes tenant-2 safely, and provide approved local-only recovery copies | +| 4. Capacity and placement | Partial | CI agent cap reduced to two; shared PostgreSQL/OpenSearch resource corrections deployed; Vault injectors spread | Measure peak demand and I/O, correct remaining requests/placement, and calculate compatible spare capacity for one worker loss | +| 5. Supported software | Planned; backups established | Installed K3s cohorts identified as 1.31.5 and 1.33.3; titan-db Ubuntu 24.10 is beyond support | Select a compatible supported baseline; stage OS/K3s/storage upgrades with rollback. Plan collation/index repair | +| 6. Maintainability and sustained health | In progress | Operator guide and rollback record published; Ananke stale-event loop fixed; updater failure reporting corrected; overlapping restart/eviction behavior removed | Complete the service recovery checklist, validate remaining automation boundaries, arrange controlled recovery checks, then observe 7-14 days | + +### Current work and acceptance checks + +- [ ] Resolve Soteria release gate findings without disabling the gate. +- [ ] Publish the corrected Soteria image through the normal pipeline and Flux. +- [ ] Verify an eligible backup and an isolated restore; retain Hermes exclusions. +- [ ] Diagnose Monerod's host/attachment/filesystem error before selecting recovery. +- [ ] Recover Monerod with its existing data and verify application RPC health. +- [x] Verify the first scheduled native application PostgreSQL backup completed. +- [ ] Recheck health and update this tracker after each deployed repair. + ## Verified changes | Change | Evidence | Rollback | @@ -39,13 +65,15 @@ replay or starting K3s. A complete control-plane disaster drill remains outstand and a representative backup verification remain required. * Native application backups now run daily from Titan-0b. The timer is enabled, the verified initial bundle timestamp is scraped, and the 36-hour freshness - alert is provisioned. The first scheduled run remains to be observed. + alert is provisioned. The first scheduled run completed at 08:37 UTC with + service result success and exit status zero. * Soteria CI: the coverage report was generated after Sonar analysis, causing a false zero-coverage gate despite 96.4% measured test coverage. Commit f167d7d orders tests before analysis and awaits the Sonar gate; Jenkins declarative validation passed. Release build 1292 failed during Git checkout with a 404; current credential/repository checks and build 1293's source checkout succeed. - The release retry remains queued; do not claim the runtime + Builds 1293/1294 subsequently passed. Release retry 1295 failed the supply-chain + gate; its findings are under investigation. Do not claim the runtime backup defect is repaired until the new image and a real eligible backup pass.