From feca76763196b5d091ec9dba830c1081dfc9abd0 Mon Sep 17 00:00:00 2001 From: jenkins Date: Sat, 3 Oct 2026 23:40:43 -0500 Subject: [PATCH] docs: update remaining cluster stabilization work --- docs/CLUSTER_STABILIZATION.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/CLUSTER_STABILIZATION.md b/docs/CLUSTER_STABILIZATION.md index 2c225b62..ce8f4202 100644 --- a/docs/CLUSTER_STABILIZATION.md +++ b/docs/CLUSTER_STABILIZATION.md @@ -7,14 +7,14 @@ Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path. ## Six-priority progress tracker -Started against the user's approved list on 2026-10-03. Updated: 21:32 UTC. +Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 04:38 UTC. "Verified" means the stated check passed; it does not imply a completed soak or failure drill. Physical repairs and disruptive recovery drills remain separate from routine software work. | Priority | Status | Completed evidence | Next action / completion gate | | --- | --- | --- | --- | -| 1. Immediate software repairs | In progress | Monerod remounted its existing database on Titan-19, became 2/2 Ready with zero restarts, returned RPC OK and advanced 20 blocks. Soteria's release permissions/build defects are corrected; release 1297 is running | Observe Monerod latency/sync; publish Soteria and validate an eligible backup and restore | +| 1. Immediate software repairs | In progress; Soteria publication blocked | Monerod remains 2/2 Ready; its daemon has zero restarts, status sidecar one. Soteria release 1297 passed enforced gates but failed Go VCS stamping inside the image build; runtime remains 0.1.0-120 | Repair image build metadata, publish Soteria and validate an eligible backup and restore; continue Monerod latency/sync checks | | 2. Reliable nodes | Partial; physical checks needed | Titan-04/14/18 excluded from new placement; Titan-05/06 remain offline. Fresh Titan-04/11 undervoltage was observed | Power/media checks, preservation or replacement of suspect media, then networking/storage/load validation before return. Establish the actual condition of 09/10/16 | | 3. Recovery coverage | In progress; cloud uploads blocked | External datastore restore and second LAN copy verified; 19 application DB dumps completed; scheduled application backup completed at 08:37 UTC; representative Gitea restore passed. Backblaze storage-cap failure established | Resolve the account storage cap within an agreed budget, complete eligible PVC coverage and Soteria restore, resolve Hermes tenant-2 safely, and provide approved local-only recovery copies | | 4. Capacity and placement | Partial | CI agent cap reduced to two; shared PostgreSQL/OpenSearch resource corrections deployed; Vault injectors spread. Longhorn now uses least-effort balancing and one rebuild per node | Account for the measured 2-3.5 GiB storage-process footprint, correct remaining requests/placement, and calculate compatible spare capacity for one worker loss | @@ -33,7 +33,7 @@ from routine software work. - [x] Recover Monerod with its existing data: get_version returned OK; height advanced from 3775922 to 3775942. Broader get_info requests remain slow. - [x] Deploy and verify bounded Longhorn balancing/rebuild settings. - [x] Verify the first scheduled native application PostgreSQL backup completed. -- [x] Recheck controller health: 57/57 active Flux Kustomizations Ready at 21:18 UTC. +- [x] Recheck controller health: 57/57 active Flux Kustomizations and 19/21 nodes Ready at 2026-10-04 04:38 UTC; Titan-05/06 remain offline. - [ ] Complete application checks and the 7-14 day observation period. - [ ] Resolve Backblaze's storage cap before accepting cloud backup coverage; preserve existing copies. @@ -93,6 +93,15 @@ replay or starting K3s. A complete control-plane disaster drill remains outstand 1297 runs the complete gate with PUBLISH_IMAGES=true. Do not claim the runtime backup defect is repaired until the new image and a real eligible backup pass. +At the October 4 04:38 UTC follow-up, release 1297 had failed after 4369 seconds. +Its enforced quality stages passed, but `go build` in the Kaniko image build +failed with `error obtaining VCS status: exit status 128`. This is now the +publication blocker. Build 1298 subsequently succeeded with PUBLISH_IMAGES=false, +so it did not deploy the runtime fix; the live image remains 0.1.0-120. Sonar's +analysis also emitted a missing Node executable error and a Go parser error; +passing the current gate is not proof that every analysis completed. Repair +those toolchain/evidence checks without weakening enforcement. + ### October 3 storage recovery and capacity evidence Titan-08's kernel recorded I/O failure, an aborted ext4 journal and a read-only