titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs: update remaining cluster stabilization work

Browse Source
This commit is contained in:
jenkins 2026-10-03 23:40:43 -05:00
parent c0d128e67b
commit feca767631
1 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/CLUSTER_STABILIZATION.md
View File

@ -7,14 +7,14 @@ Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path.
## Six-priority progress tracker
Started against the user's approved list on 2026-10-03. Updated: 21:32 UTC.
Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 04:38 UTC.
"Verified" means the stated check passed; it does not imply a completed soak or
failure drill. Physical repairs and disruptive recovery drills remain separate
from routine software work.
| Priority | Status | Completed evidence | Next action / completion gate |
| --- | --- | --- | --- |
| 1. Immediate software repairs | In progress | Monerod remounted its existing database on Titan-19, became 2/2 Ready with zero restarts, returned RPC OK and advanced 20 blocks. Soteria's release permissions/build defects are corrected; release 1297 is running | Observe Monerod latency/sync; publish Soteria and validate an eligible backup and restore |
| 1. Immediate software repairs | In progress; Soteria publication blocked | Monerod remains 2/2 Ready; its daemon has zero restarts, status sidecar one. Soteria release 1297 passed enforced gates but failed Go VCS stamping inside the image build; runtime remains 0.1.0-120 | Repair image build metadata, publish Soteria and validate an eligible backup and restore; continue Monerod latency/sync checks |
| 2. Reliable nodes | Partial; physical checks needed | Titan-04/14/18 excluded from new placement; Titan-05/06 remain offline. Fresh Titan-04/11 undervoltage was observed | Power/media checks, preservation or replacement of suspect media, then networking/storage/load validation before return. Establish the actual condition of 09/10/16 |
| 3. Recovery coverage | In progress; cloud uploads blocked | External datastore restore and second LAN copy verified; 19 application DB dumps completed; scheduled application backup completed at 08:37 UTC; representative Gitea restore passed. Backblaze storage-cap failure established | Resolve the account storage cap within an agreed budget, complete eligible PVC coverage and Soteria restore, resolve Hermes tenant-2 safely, and provide approved local-only recovery copies |
| 4. Capacity and placement | Partial | CI agent cap reduced to two; shared PostgreSQL/OpenSearch resource corrections deployed; Vault injectors spread. Longhorn now uses least-effort balancing and one rebuild per node | Account for the measured 2-3.5 GiB storage-process footprint, correct remaining requests/placement, and calculate compatible spare capacity for one worker loss |
@ -33,7 +33,7 @@ from routine software work.
- [x] Recover Monerod with its existing data: get_version returned OK; height advanced from 3775922 to 3775942. Broader get_info requests remain slow.
- [x] Deploy and verify bounded Longhorn balancing/rebuild settings.
- [x] Verify the first scheduled native application PostgreSQL backup completed.
- [x] Recheck controller health: 57/57 active Flux Kustomizations Ready at 21:18 UTC.
- [x] Recheck controller health: 57/57 active Flux Kustomizations and 19/21 nodes Ready at 2026-10-04 04:38 UTC; Titan-05/06 remain offline.
- [ ] Complete application checks and the 7-14 day observation period.
- [ ] Resolve Backblaze's storage cap before accepting cloud backup coverage; preserve existing copies.
@ -93,6 +93,15 @@ replay or starting K3s. A complete control-plane disaster drill remains outstand
1297 runs the complete gate with PUBLISH_IMAGES=true. Do not claim the runtime
backup defect is repaired until the new image and a real eligible backup pass.
At the October 4 04:38 UTC follow-up, release 1297 had failed after 4369 seconds.
Its enforced quality stages passed, but `go build` in the Kaniko image build
failed with `error obtaining VCS status: exit status 128`. This is now the
publication blocker. Build 1298 subsequently succeeded with PUBLISH_IMAGES=false,
so it did not deploy the runtime fix; the live image remains 0.1.0-120. Sonar's
analysis also emitted a missing Node executable error and a Go parser error;
passing the current gate is not proof that every analysis completed. Repair
those toolchain/evidence checks without weakening enforcement.
### October 3 storage recovery and capacity evidence
Titan-08's kernel recorded I/O failure, an aborted ext4 journal and a read-only