titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs: update remaining cluster stabilization work

Browse Source
This commit is contained in:
jenkins 2026-10-03 23:40:43 -05:00
parent c0d128e67b
commit feca767631
1 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/CLUSTER_STABILIZATION.md
View File

@ -7,14 +7,14 @@ Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path.
## Six-priority progress tracker ## Six-priority progress tracker
Started against the user's approved list on 2026-10-03. Updated: 21:32 UTC. Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 04:38 UTC.
"Verified" means the stated check passed; it does not imply a completed soak or "Verified" means the stated check passed; it does not imply a completed soak or
failure drill. Physical repairs and disruptive recovery drills remain separate failure drill. Physical repairs and disruptive recovery drills remain separate
from routine software work. from routine software work.
| Priority | Status | Completed evidence | Next action / completion gate | | Priority | Status | Completed evidence | Next action / completion gate |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| 1. Immediate software repairs | In progress | Monerod remounted its existing database on Titan-19, became 2/2 Ready with zero restarts, returned RPC OK and advanced 20 blocks. Soteria's release permissions/build defects are corrected; release 1297 is running | Observe Monerod latency/sync; publish Soteria and validate an eligible backup and restore | | 1. Immediate software repairs | In progress; Soteria publication blocked | Monerod remains 2/2 Ready; its daemon has zero restarts, status sidecar one. Soteria release 1297 passed enforced gates but failed Go VCS stamping inside the image build; runtime remains 0.1.0-120 | Repair image build metadata, publish Soteria and validate an eligible backup and restore; continue Monerod latency/sync checks |
| 2. Reliable nodes | Partial; physical checks needed | Titan-04/14/18 excluded from new placement; Titan-05/06 remain offline. Fresh Titan-04/11 undervoltage was observed | Power/media checks, preservation or replacement of suspect media, then networking/storage/load validation before return. Establish the actual condition of 09/10/16 | | 2. Reliable nodes | Partial; physical checks needed | Titan-04/14/18 excluded from new placement; Titan-05/06 remain offline. Fresh Titan-04/11 undervoltage was observed | Power/media checks, preservation or replacement of suspect media, then networking/storage/load validation before return. Establish the actual condition of 09/10/16 |
| 3. Recovery coverage | In progress; cloud uploads blocked | External datastore restore and second LAN copy verified; 19 application DB dumps completed; scheduled application backup completed at 08:37 UTC; representative Gitea restore passed. Backblaze storage-cap failure established | Resolve the account storage cap within an agreed budget, complete eligible PVC coverage and Soteria restore, resolve Hermes tenant-2 safely, and provide approved local-only recovery copies | | 3. Recovery coverage | In progress; cloud uploads blocked | External datastore restore and second LAN copy verified; 19 application DB dumps completed; scheduled application backup completed at 08:37 UTC; representative Gitea restore passed. Backblaze storage-cap failure established | Resolve the account storage cap within an agreed budget, complete eligible PVC coverage and Soteria restore, resolve Hermes tenant-2 safely, and provide approved local-only recovery copies |
| 4. Capacity and placement | Partial | CI agent cap reduced to two; shared PostgreSQL/OpenSearch resource corrections deployed; Vault injectors spread. Longhorn now uses least-effort balancing and one rebuild per node | Account for the measured 2-3.5 GiB storage-process footprint, correct remaining requests/placement, and calculate compatible spare capacity for one worker loss | | 4. Capacity and placement | Partial | CI agent cap reduced to two; shared PostgreSQL/OpenSearch resource corrections deployed; Vault injectors spread. Longhorn now uses least-effort balancing and one rebuild per node | Account for the measured 2-3.5 GiB storage-process footprint, correct remaining requests/placement, and calculate compatible spare capacity for one worker loss |
@ -33,7 +33,7 @@ from routine software work.
- [x] Recover Monerod with its existing data: get_version returned OK; height advanced from 3775922 to 3775942. Broader get_info requests remain slow. - [x] Recover Monerod with its existing data: get_version returned OK; height advanced from 3775922 to 3775942. Broader get_info requests remain slow.
- [x] Deploy and verify bounded Longhorn balancing/rebuild settings. - [x] Deploy and verify bounded Longhorn balancing/rebuild settings.
- [x] Verify the first scheduled native application PostgreSQL backup completed. - [x] Verify the first scheduled native application PostgreSQL backup completed.
- [x] Recheck controller health: 57/57 active Flux Kustomizations Ready at 21:18 UTC. - [x] Recheck controller health: 57/57 active Flux Kustomizations and 19/21 nodes Ready at 2026-10-04 04:38 UTC; Titan-05/06 remain offline.
- [ ] Complete application checks and the 7-14 day observation period. - [ ] Complete application checks and the 7-14 day observation period.
- [ ] Resolve Backblaze's storage cap before accepting cloud backup coverage; preserve existing copies. - [ ] Resolve Backblaze's storage cap before accepting cloud backup coverage; preserve existing copies.
@ -93,6 +93,15 @@ replay or starting K3s. A complete control-plane disaster drill remains outstand
1297 runs the complete gate with PUBLISH_IMAGES=true. Do not claim the runtime 1297 runs the complete gate with PUBLISH_IMAGES=true. Do not claim the runtime
backup defect is repaired until the new image and a real eligible backup pass. backup defect is repaired until the new image and a real eligible backup pass.
At the October 4 04:38 UTC follow-up, release 1297 had failed after 4369 seconds.
Its enforced quality stages passed, but `go build` in the Kaniko image build
failed with `error obtaining VCS status: exit status 128`. This is now the
publication blocker. Build 1298 subsequently succeeded with PUBLISH_IMAGES=false,
so it did not deploy the runtime fix; the live image remains 0.1.0-120. Sonar's
analysis also emitted a missing Node executable error and a Go parser error;
passing the current gate is not proof that every analysis completed. Repair
those toolchain/evidence checks without weakening enforcement.
### October 3 storage recovery and capacity evidence ### October 3 storage recovery and capacity evidence
Titan-08's kernel recorded I/O failure, an aborted ext4 journal and a read-only Titan-08's kernel recorded I/O failure, an aborted ext4 journal and a read-only