titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs: record log repair and active RAM-log migration boundary

Browse Source
This commit is contained in:
jenkins 2026-10-04 01:13:54 -05:00
parent 9522bcc5d8
commit 52dffae88c
1 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/CLUSTER_STABILIZATION.md
View File

@ -7,7 +7,7 @@ Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path.
## Six-priority progress tracker ## Six-priority progress tracker
Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 06:10 UTC. Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 06:20 UTC.
"Verified" means the stated check passed; it does not imply a completed soak or "Verified" means the stated check passed; it does not imply a completed soak or
failure drill. Physical repairs and disruptive recovery drills remain separate failure drill. Physical repairs and disruptive recovery drills remain separate
from routine software work. from routine software work.
@ -62,6 +62,15 @@ replacement. No node was reflashed or rebooted during this work.
but ENABLED=true, copying between obsolete RAM-log locations although /var/log but ENABLED=true, copying between obsolete RAM-log locations although /var/log
already points to external ext4 storage. The guarded versioned script disabled already points to external ext4 storage. The guarded versioned script disabled
those hooks; the native logrotate service completed with result success/status 0. those hooks; the native logrotate service completed with result success/status 0.
- [x] Apply the same guarded repair on Titan-14 after confirming its RAM-log unit
was already masked/inactive and logs use external ext4 storage. Logrotate then
completed with result success/status 0. The later short I/O sample was quiet;
this does not resolve its earlier intermittent storage stalls.
- [ ] Migrate the active 50 MiB RAM-log filesystems on Titan-12/18 in a controlled
node maintenance cycle. The obsolete-hook script intentionally refuses them.
Their native stop action lazily unmounts /var/log while processes may still
hold files open; do not treat changing ENABLED as a complete live migration.
Preserve logs and arrange workload evacuation/restart before removing the mount.
- [x] Correct Ariadne's memory reservation/limit from 128/512 MiB to 512/1024 MiB - [x] Correct Ariadne's memory reservation/limit from 128/512 MiB to 512/1024 MiB
after repeated cgroup OOM kills. The replacement is 2/2 Ready with zero restarts. after repeated cgroup OOM kills. The replacement is 2/2 Ready with zero restarts.
This supplies headroom; it does not establish that its memory growth is bounded. This supplies headroom; it does not establish that its memory growth is bounded.
@ -99,7 +108,7 @@ replacement. No node was reflashed or rebooted during this work.
| 12/13/19 | atlas/root passwords restored; root SSH keys preserved; password-backed sudo works after legacy grant retirement | Observe stability; use the corrected recovery image for future rebuilds | | 12/13/19 | atlas/root passwords restored; root SSH keys preserved; password-backed sudo works after legacy grant retirement | Observe stability; use the corrected recovery image for future rebuilds |
| 20/21 | atlas sudo already worked; root passwords now match Vault | Continue workload memory diagnosis; Jetson-21 recorded a Data Prepper cgroup OOM | | 20/21 | atlas sudo already worked; root passwords now match Vault | Continue workload memory diagnosis; Jetson-21 recorded a Data Prepper cgroup OOM |
| 04/11 | Fresh undervoltage messages continued during this audit, despite sufficient disk space | Onboard EXT5V samples were 4.82132 V (04) and 4.75968 V (11). Check delivered power/cables/peripherals; do not clear quarantine based on one sample | | 04/11 | Fresh undervoltage messages continued during this audit, despite sufficient disk space | Onboard EXT5V samples were 4.82132 V (04) and 4.75968 V (11). Check delivered power/cables/peripherals; do not clear quarantine based on one sample |
| 14 | Runtime USB flash, substantial I/O pressure; 39-45% iowait in the sample, no new kernel I/O error in the two-hour sample | Preserve data; repair or replace runtime media and validate before return | | 14 | Runtime USB flash; 39-45% iowait in the first sample, quiet later, no new kernel I/O error in the two-hour sample; stale RAM-log hooks corrected | Preserve data; investigate intermittent storage stalls and validate sustained operation before return; hardware failure is not proven by I/O pressure alone |
| 18 | Quarantine has reduced current I/O pressure; its 50 MiB RAM-log filesystem is still active | Keep quarantined until sustained storage checks pass; do not apply the inactive-RAM-log repair script to it | | 18 | Quarantine has reduced current I/O pressure; its 50 MiB RAM-log filesystem is still active | Keep quarantined until sustained storage checks pass; do not apply the inactive-RAM-log repair script to it |
| 05 | Answers LAN ping at .31; expected SSH, kubelet and common recovery ports refuse connections | Console/user-space service check; this is not evidence that the machine is powered off | | 05 | Answers LAN ping at .31; expected SSH, kubelet and common recovery ports refuse connections | Console/user-space service check; this is not evidence that the machine is powered off |
| 09/10 | Answer LAN ping and SSH on port 22; recorded 2277 port is closed; host keys differ from saved keys | Confirm reinstall/image identity before supplying passwords; no host-key checks bypassed | | 09/10 | Answer LAN ping and SSH on port 22; recorded 2277 port is closed; host keys differ from saved keys | Confirm reinstall/image identity before supplying passwords; no host-key checks bypassed |
@ -126,7 +135,7 @@ existing path; use the dated manual SSH/Vault access record for intervention
outside the automated recovery flow. Do not claim a full power-loss drill from outside the automated recovery flow. Do not claim a full power-loss drill from
read-only connection tests. read-only connection tests.
Rollback: Kubernetes changes use their focused Git commit and Flux. Titan-19's Rollback: Kubernetes changes use their focused Git commit and Flux. Titan-14/19's
previous RAM-log config is root-only under previous RAM-log config is root-only under
`/var/lib/atlas-maintenance/ramlog-before-20261004/`; restoring it reintroduces the `/var/lib/atlas-maintenance/ramlog-before-20261004/`; restoring it reintroduces the
obsolete hooks. Retired sudo grants are root-only under obsolete hooks. Retired sudo grants are root-only under