titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs: track six cluster health priorities

Browse Source
This commit is contained in:
jenkins 2026-10-03 15:46:13 -05:00
parent c345e962c0
commit b48f1dbd1d
1 changed files with 30 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
docs/CLUSTER_STABILIZATION.md
View File

@ -5,6 +5,32 @@ This record begins on 2026-10-03 UTC, following authorization to implement the
The October 2 audit remains a historical baseline, not a current health claim.
Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path.
## Six-priority progress tracker
Started against the user's approved list on 2026-10-03. Updated: 20:45 UTC.
"Verified" means the stated check passed; it does not imply a completed soak or
failure drill. Physical repairs and disruptive recovery drills remain separate
from routine software work.
| Priority | Status | Completed evidence | Next action / completion gate |
| --- | --- | --- | --- |
| 1. Immediate software repairs | In progress | Soteria test builds 1293/1294 passed; release 1295 failed its supply-chain gate. Monerod's LMDB open fails with I/O error despite healthy Longhorn replicas | Correct the release findings, publish and validate a backup; repair Monerod's storage path without deleting its database |
| 2. Reliable nodes | Partial; physical checks needed | Titan-04/14/18 excluded from new placement; Titan-05/06 remain offline. Fresh Titan-04/11 undervoltage was observed | Power/media checks, preservation or replacement of suspect media, then networking/storage/load validation before return. Establish the actual condition of 09/10/16 |
| 3. Recovery coverage | In progress | External datastore restore and second LAN copy verified; 19 application DB dumps completed; scheduled application backup completed at 08:37 UTC; representative Gitea restore passed | Complete eligible PVC coverage, verify Soteria restore, resolve Hermes tenant-2 safely, and provide approved local-only recovery copies |
| 4. Capacity and placement | Partial | CI agent cap reduced to two; shared PostgreSQL/OpenSearch resource corrections deployed; Vault injectors spread | Measure peak demand and I/O, correct remaining requests/placement, and calculate compatible spare capacity for one worker loss |
| 5. Supported software | Planned; backups established | Installed K3s cohorts identified as 1.31.5 and 1.33.3; titan-db Ubuntu 24.10 is beyond support | Select a compatible supported baseline; stage OS/K3s/storage upgrades with rollback. Plan collation/index repair |
| 6. Maintainability and sustained health | In progress | Operator guide and rollback record published; Ananke stale-event loop fixed; updater failure reporting corrected; overlapping restart/eviction behavior removed | Complete the service recovery checklist, validate remaining automation boundaries, arrange controlled recovery checks, then observe 7-14 days |
### Current work and acceptance checks
- [ ] Resolve Soteria release gate findings without disabling the gate.
- [ ] Publish the corrected Soteria image through the normal pipeline and Flux.
- [ ] Verify an eligible backup and an isolated restore; retain Hermes exclusions.
- [ ] Diagnose Monerod's host/attachment/filesystem error before selecting recovery.
- [ ] Recover Monerod with its existing data and verify application RPC health.
- [x] Verify the first scheduled native application PostgreSQL backup completed.
- [ ] Recheck health and update this tracker after each deployed repair.
## Verified changes
| Change | Evidence | Rollback |
@ -39,13 +65,15 @@ replay or starting K3s. A complete control-plane disaster drill remains outstand
and a representative backup verification remain required.
* Native application backups now run daily from Titan-0b. The timer is enabled,
the verified initial bundle timestamp is scraped, and the 36-hour freshness
alert is provisioned. The first scheduled run remains to be observed.
alert is provisioned. The first scheduled run completed at 08:37 UTC with
service result success and exit status zero.
* Soteria CI: the coverage report was generated after Sonar analysis, causing a
false zero-coverage gate despite 96.4% measured test coverage. Commit f167d7d
orders tests before analysis and awaits the Sonar gate; Jenkins declarative
validation passed. Release build 1292 failed during Git checkout with a 404;
current credential/repository checks and build 1293's source checkout succeed.
The release retry remains queued; do not claim the runtime
Builds 1293/1294 subsequently passed. Release retry 1295 failed the supply-chain
gate; its findings are under investigation. Do not claim the runtime
backup defect is repaired until the new image and a real eligible backup pass.