titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs: record log repair and active RAM-log migration boundary

Browse Source
This commit is contained in:
jenkins 2026-10-04 01:13:54 -05:00
parent 9522bcc5d8
commit 52dffae88c
1 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/CLUSTER_STABILIZATION.md
View File

@ -7,7 +7,7 @@ Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path.
## Six-priority progress tracker
Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 06:10 UTC.
Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 06:20 UTC.
"Verified" means the stated check passed; it does not imply a completed soak or
failure drill. Physical repairs and disruptive recovery drills remain separate
from routine software work.
@ -62,6 +62,15 @@ replacement. No node was reflashed or rebooted during this work.
but ENABLED=true, copying between obsolete RAM-log locations although /var/log
already points to external ext4 storage. The guarded versioned script disabled
those hooks; the native logrotate service completed with result success/status 0.
- [x] Apply the same guarded repair on Titan-14 after confirming its RAM-log unit
was already masked/inactive and logs use external ext4 storage. Logrotate then
completed with result success/status 0. The later short I/O sample was quiet;
this does not resolve its earlier intermittent storage stalls.
- [ ] Migrate the active 50 MiB RAM-log filesystems on Titan-12/18 in a controlled
node maintenance cycle. The obsolete-hook script intentionally refuses them.
Their native stop action lazily unmounts /var/log while processes may still
hold files open; do not treat changing ENABLED as a complete live migration.
Preserve logs and arrange workload evacuation/restart before removing the mount.
- [x] Correct Ariadne's memory reservation/limit from 128/512 MiB to 512/1024 MiB
after repeated cgroup OOM kills. The replacement is 2/2 Ready with zero restarts.
This supplies headroom; it does not establish that its memory growth is bounded.
@ -99,7 +108,7 @@ replacement. No node was reflashed or rebooted during this work.
| 12/13/19 | atlas/root passwords restored; root SSH keys preserved; password-backed sudo works after legacy grant retirement | Observe stability; use the corrected recovery image for future rebuilds |
| 20/21 | atlas sudo already worked; root passwords now match Vault | Continue workload memory diagnosis; Jetson-21 recorded a Data Prepper cgroup OOM |
| 04/11 | Fresh undervoltage messages continued during this audit, despite sufficient disk space | Onboard EXT5V samples were 4.82132 V (04) and 4.75968 V (11). Check delivered power/cables/peripherals; do not clear quarantine based on one sample |
| 14 | Runtime USB flash, substantial I/O pressure; 39-45% iowait in the sample, no new kernel I/O error in the two-hour sample | Preserve data; repair or replace runtime media and validate before return |
| 14 | Runtime USB flash; 39-45% iowait in the first sample, quiet later, no new kernel I/O error in the two-hour sample; stale RAM-log hooks corrected | Preserve data; investigate intermittent storage stalls and validate sustained operation before return; hardware failure is not proven by I/O pressure alone |
| 18 | Quarantine has reduced current I/O pressure; its 50 MiB RAM-log filesystem is still active | Keep quarantined until sustained storage checks pass; do not apply the inactive-RAM-log repair script to it |
| 05 | Answers LAN ping at .31; expected SSH, kubelet and common recovery ports refuse connections | Console/user-space service check; this is not evidence that the machine is powered off |
| 09/10 | Answer LAN ping and SSH on port 22; recorded 2277 port is closed; host keys differ from saved keys | Confirm reinstall/image identity before supplying passwords; no host-key checks bypassed |
@ -126,7 +135,7 @@ existing path; use the dated manual SSH/Vault access record for intervention
outside the automated recovery flow. Do not claim a full power-loss drill from
read-only connection tests.
Rollback: Kubernetes changes use their focused Git commit and Flux. Titan-19's
Rollback: Kubernetes changes use their focused Git commit and Flux. Titan-14/19's
previous RAM-log config is root-only under
`/var/lib/atlas-maintenance/ramlog-before-20261004/`; restoring it reintroduces the
obsolete hooks. Retired sudo grants are root-only under