From 52dffae88c00e7d74c42c6b2f72460b1f255a13a Mon Sep 17 00:00:00 2001 From: jenkins Date: Sun, 4 Oct 2026 01:13:54 -0500 Subject: [PATCH] docs: record log repair and active RAM-log migration boundary --- docs/CLUSTER_STABILIZATION.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/CLUSTER_STABILIZATION.md b/docs/CLUSTER_STABILIZATION.md index 9ec6dd40..18e63370 100644 --- a/docs/CLUSTER_STABILIZATION.md +++ b/docs/CLUSTER_STABILIZATION.md @@ -7,7 +7,7 @@ Use [Cluster operations](CLUSTER_OPERATIONS.md) for the ordinary operator path. ## Six-priority progress tracker -Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 06:10 UTC. +Started against the user's approved list on 2026-10-03. Updated: 2026-10-04 06:20 UTC. "Verified" means the stated check passed; it does not imply a completed soak or failure drill. Physical repairs and disruptive recovery drills remain separate from routine software work. @@ -62,6 +62,15 @@ replacement. No node was reflashed or rebooted during this work. but ENABLED=true, copying between obsolete RAM-log locations although /var/log already points to external ext4 storage. The guarded versioned script disabled those hooks; the native logrotate service completed with result success/status 0. +- [x] Apply the same guarded repair on Titan-14 after confirming its RAM-log unit + was already masked/inactive and logs use external ext4 storage. Logrotate then + completed with result success/status 0. The later short I/O sample was quiet; + this does not resolve its earlier intermittent storage stalls. +- [ ] Migrate the active 50 MiB RAM-log filesystems on Titan-12/18 in a controlled + node maintenance cycle. The obsolete-hook script intentionally refuses them. + Their native stop action lazily unmounts /var/log while processes may still + hold files open; do not treat changing ENABLED as a complete live migration. + Preserve logs and arrange workload evacuation/restart before removing the mount. - [x] Correct Ariadne's memory reservation/limit from 128/512 MiB to 512/1024 MiB after repeated cgroup OOM kills. The replacement is 2/2 Ready with zero restarts. This supplies headroom; it does not establish that its memory growth is bounded. @@ -99,7 +108,7 @@ replacement. No node was reflashed or rebooted during this work. | 12/13/19 | atlas/root passwords restored; root SSH keys preserved; password-backed sudo works after legacy grant retirement | Observe stability; use the corrected recovery image for future rebuilds | | 20/21 | atlas sudo already worked; root passwords now match Vault | Continue workload memory diagnosis; Jetson-21 recorded a Data Prepper cgroup OOM | | 04/11 | Fresh undervoltage messages continued during this audit, despite sufficient disk space | Onboard EXT5V samples were 4.82132 V (04) and 4.75968 V (11). Check delivered power/cables/peripherals; do not clear quarantine based on one sample | -| 14 | Runtime USB flash, substantial I/O pressure; 39-45% iowait in the sample, no new kernel I/O error in the two-hour sample | Preserve data; repair or replace runtime media and validate before return | +| 14 | Runtime USB flash; 39-45% iowait in the first sample, quiet later, no new kernel I/O error in the two-hour sample; stale RAM-log hooks corrected | Preserve data; investigate intermittent storage stalls and validate sustained operation before return; hardware failure is not proven by I/O pressure alone | | 18 | Quarantine has reduced current I/O pressure; its 50 MiB RAM-log filesystem is still active | Keep quarantined until sustained storage checks pass; do not apply the inactive-RAM-log repair script to it | | 05 | Answers LAN ping at .31; expected SSH, kubelet and common recovery ports refuse connections | Console/user-space service check; this is not evidence that the machine is powered off | | 09/10 | Answer LAN ping and SSH on port 22; recorded 2277 port is closed; host keys differ from saved keys | Confirm reinstall/image identity before supplying passwords; no host-key checks bypassed | @@ -126,7 +135,7 @@ existing path; use the dated manual SSH/Vault access record for intervention outside the automated recovery flow. Do not claim a full power-loss drill from read-only connection tests. -Rollback: Kubernetes changes use their focused Git commit and Flux. Titan-19's +Rollback: Kubernetes changes use their focused Git commit and Flux. Titan-14/19's previous RAM-log config is root-only under `/var/lib/atlas-maintenance/ramlog-before-20261004/`; restoring it reintroduces the obsolete hooks. Retired sudo grants are root-only under