Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/
classifier), titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary,
restored) for t_26da4c88. titan-20/21 are CPU-committed with no safe
headroom (titan-20 at 227 MiB free memory at its 24h worst point);
titan-22 has real idle CPU/RAM but its shared-GPU time-slicing has no
VRAM/engine isolation, so no workload is relocated. Adds alerting for
the sharpest gaps found (titan-20 memory exhaustion, titan-22 CPU/RAM/
GPU-VRAM pressure, Jellyfin CPU throttling, titan-21 CPU pressure) and
two Atlas GPU dashboard panels (VRAM, NVENC/NVDEC utilization) so a
future opportunistic-workload PR or a live transcode incident is
visible without a promql session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.
The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.
The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.
The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.
Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.
Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
