titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: monitoring(titan): capacity guardrails for titan-20/21/22 audit (final) #53

Draft
hermes-automation wants to merge 5 commits from feature/t_26da4c88-titan-capacity-guardrails-v4 into main
pull from: feature/t_26da4c88-titan-capacity-guardrails-v4
merge into: titan:main
Conversation 0 Commits 5 Files Changed 5 +781
hermes-automation commented 2026-08-23 15:04:45 +00:00
Member
Copy Link

Supersedes #50, #51, and #52 (broker is create-only; each prior PR
was missing a follow-up fix made while main kept moving mid-audit —
do not merge #50, #51, or #52; close all three in favor of this one,
which is final).

Summary

Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/classifier),
titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary) for Kanban card
t_26da4c88. Full report: docs/titan_20_21_22_capacity_optimization.md.

  • titan-20/21 have no safe headroom and it's trending worse: titan-20
    CPU p95 88.6% over the trailing 3h, memory available down to 1.7% at
    its floor; titan-21 CPU p95 65%, memory 38-39% available. No workload was
    added to either node.
  • titan-22 has real, durable idle CPU/RAM: 22-26% CPU p95, 77%+ memory
    available — even after four workloads (Jenkins, VictoriaMetrics,
    Wger, SonarQube) were placed there live by four separate direct main
    commits during this audit (271f3e8c3, 0ebab9d41, 17c5f5093),
    outside this PR and outside any PR — see report §4a. That placement
    is not authored by this PR; it is documented and evidence-validated by
    it. Jellyfin shows 0 restarts / 0 CPU-throttled periods throughout.
  • No active hardware transcode occurred during that co-residency window
    (DCGM encoder/decoder/SM all read 0% for the full 3h), so
    "Jellyfin-under-transcode survives these neighbors" is still unproven —
    flagged as the concrete open validation item, not claimed as done.
  • Shared-GPU time-slicing on titan-22 has no VRAM/engine isolation
    (plain NVIDIA time-slicing, no MPS/MIG) — conditional GPU sharing
    (Option C) is evaluated and rejected as unsafe until the cluster has a
    GPU-sharing mode with real per-tenant isolation.
  • New risk surfaced, not previously flagged: victoria-metrics-single-server
    (the cluster's only metrics backend) is now hard-pinned to the same node
    as Jellyfin/Jenkins/Wger/SonarQube — a titan-22 incident would blind the
    dashboards needed to diagnose that same incident. Flagged for Brad as a
    follow-up decision, not changed here.
  • No workload is relocated by this PR itself. Per the card's own acceptance
    criteria, this ships guardrails/dashboard/report only.

Changes

  • docs/titan_20_21_22_capacity_optimization.md — durable capacity model,
    workload map, ranked distribution options (A-D), live-placement addendum
    (§4a), and validation plan.
  • services/monitoring/vmalert-atlas-availability.yaml — new
    atlas.titan-media-capacity alert group: titan-20 node memory
    exhaustion, titan-22 CPU/RAM/GPU-VRAM pressure, Jellyfin CPU throttling,
    titan-21 CPU pressure.
  • services/monitoring/grafana-dashboard-gpu.yaml +
    services/monitoring/dashboards/atlas-gpu.json — two new Atlas GPU
    panels (VRAM used/free, NVENC/NVDEC utilization by node), generated via
    scripts/render/dashboards_render_atlas.py (re-ran the generator;
    committed output is byte-identical to what it regenerates).

Rollback plan

Purely additive: one new doc, one new vmalert rule group (new group name,
does not modify or remove any existing group/rule), and two new dashboard
panels appended to the existing Atlas GPU dashboard (does not modify any
existing panel). No workload manifests, taints, priority classes, or node
labels are touched by this PR. To roll back, revert this PR's merge commit;
nothing else depends on the new alerts/panels/doc. No live cluster mutation
is performed by merging this PR beyond Flux reconciling the new
vmalert/dashboard ConfigMaps — if the new alert group is noisy, delete
the atlas.titan-media-capacity group block from
vmalert-atlas-availability.yaml in a follow-up PR.

Validation

  • kustomize build services/monitoring succeeds.
  • python3 scripts/render/dashboards_render_atlas.py regenerates
    grafana-dashboard-gpu.yaml/atlas-gpu.json byte-identical to what's
    committed.
  • New/changed YAML and JSON parse cleanly.
  • Full pytest testing/tests scripts/tests (2814 passed) shows only 14
    pre-existing failures on origin/main, none in files this PR
    touches and none related to titan-20/21/22 placement/monitoring except
    two caused by the same direct main commits documented in report §4a
    (test_victoria_metrics_has_dashboard_burst_headroom,
    test_metrics_storage_holds_a_full_retention_period — from
    271f3e8c3/1e6eda863, not from this PR).
  • No live transcode-under-load validation was performed or is claimed —
    this read-only session (get/list/watch only, no mutate verbs) cannot
    trigger one; report §8 documents the plan to run once merged.

Board comments with these conclusions were also posted to t_39cf1905
(Jenkins/WebUI release — Jenkins node-placement blocker looks cleared) and
t_60d2deb6 (Chat latency instrumentation — titan-20/21 node pressure
context + VictoriaMetrics SPOF heads-up).

Leaving as draft for Brad's review per policy; not merging.

**Supersedes #50, #51, and #52** (broker is create-only; each prior PR was missing a follow-up fix made while main kept moving mid-audit — do not merge #50, #51, or #52; close all three in favor of this one, which is final). ## Summary Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/classifier), titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary) for Kanban card `t_26da4c88`. Full report: `docs/titan_20_21_22_capacity_optimization.md`. - **titan-20/21 have no safe headroom** and it's trending worse: titan-20 CPU p95 88.6% over the trailing 3h, memory available down to **1.7%** at its floor; titan-21 CPU p95 65%, memory 38-39% available. No workload was added to either node. - **titan-22 has real, durable idle CPU/RAM**: 22-26% CPU p95, 77%+ memory available — even after **four** workloads (Jenkins, VictoriaMetrics, Wger, SonarQube) were placed there live by four separate direct `main` commits during this audit (`271f3e8c3`, `0ebab9d41`, `17c5f5093`), **outside this PR and outside any PR** — see report §4a. That placement is not authored by this PR; it is documented and evidence-validated by it. Jellyfin shows 0 restarts / 0 CPU-throttled periods throughout. - **No active hardware transcode occurred during that co-residency window** (DCGM encoder/decoder/SM all read 0% for the full 3h), so "Jellyfin-under-transcode survives these neighbors" is still unproven — flagged as the concrete open validation item, not claimed as done. - **Shared-GPU time-slicing on titan-22 has no VRAM/engine isolation** (plain NVIDIA time-slicing, no MPS/MIG) — conditional GPU sharing (Option C) is evaluated and rejected as unsafe until the cluster has a GPU-sharing mode with real per-tenant isolation. - **New risk surfaced, not previously flagged**: `victoria-metrics-single-server` (the cluster's only metrics backend) is now hard-pinned to the same node as Jellyfin/Jenkins/Wger/SonarQube — a titan-22 incident would blind the dashboards needed to diagnose that same incident. Flagged for Brad as a follow-up decision, not changed here. - No workload is relocated by this PR itself. Per the card's own acceptance criteria, this ships guardrails/dashboard/report only. ## Changes - `docs/titan_20_21_22_capacity_optimization.md` — durable capacity model, workload map, ranked distribution options (A-D), live-placement addendum (§4a), and validation plan. - `services/monitoring/vmalert-atlas-availability.yaml` — new `atlas.titan-media-capacity` alert group: titan-20 node memory exhaustion, titan-22 CPU/RAM/GPU-VRAM pressure, Jellyfin CPU throttling, titan-21 CPU pressure. - `services/monitoring/grafana-dashboard-gpu.yaml` + `services/monitoring/dashboards/atlas-gpu.json` — two new `Atlas GPU` panels (VRAM used/free, NVENC/NVDEC utilization by node), generated via `scripts/render/dashboards_render_atlas.py` (re-ran the generator; committed output is byte-identical to what it regenerates). ## Rollback plan Purely additive: one new doc, one new `vmalert` rule group (new group name, does not modify or remove any existing group/rule), and two new dashboard panels appended to the existing `Atlas GPU` dashboard (does not modify any existing panel). No workload manifests, taints, priority classes, or node labels are touched by this PR. To roll back, revert this PR's merge commit; nothing else depends on the new alerts/panels/doc. No live cluster mutation is performed by merging this PR beyond Flux reconciling the new `vmalert`/dashboard ConfigMaps — if the new alert group is noisy, delete the `atlas.titan-media-capacity` group block from `vmalert-atlas-availability.yaml` in a follow-up PR. ## Validation - `kustomize build services/monitoring` succeeds. - `python3 scripts/render/dashboards_render_atlas.py` regenerates `grafana-dashboard-gpu.yaml`/`atlas-gpu.json` byte-identical to what's committed. - New/changed YAML and JSON parse cleanly. - Full `pytest testing/tests scripts/tests` (2814 passed) shows only 14 **pre-existing** failures on `origin/main`, none in files this PR touches and none related to titan-20/21/22 placement/monitoring except two caused by the same direct `main` commits documented in report §4a (`test_victoria_metrics_has_dashboard_burst_headroom`, `test_metrics_storage_holds_a_full_retention_period` — from `271f3e8c3`/`1e6eda863`, not from this PR). - No live transcode-under-load validation was performed or is claimed — this read-only session (`get/list/watch` only, no mutate verbs) cannot trigger one; report §8 documents the plan to run once merged. Board comments with these conclusions were also posted to `t_39cf1905` (Jenkins/WebUI release — Jenkins node-placement blocker looks cleared) and `t_60d2deb6` (Chat latency instrumentation — titan-20/21 node pressure context + VictoriaMetrics SPOF heads-up). Leaving as **draft** for Brad's review per policy; not merging.
hermes-automation added 5 commits 2026-08-23 15:04:45 +00:00 
Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/
classifier), titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary,
restored) for t_26da4c88. titan-20/21 are CPU-committed with no safe
headroom (titan-20 at 227 MiB free memory at its 24h worst point);
titan-22 has real idle CPU/RAM but its shared-GPU time-slicing has no
VRAM/engine isolation, so no workload is relocated. Adds alerting for
the sharpest gaps found (titan-20 memory exhaustion, titan-22 CPU/RAM/
GPU-VRAM pressure, Jellyfin CPU throttling, titan-21 CPU pressure) and
two Atlas GPU dashboard panels (VRAM, NVENC/NVDEC utilization) so a
future opportunistic-workload PR or a live transcode incident is
visible without a promql session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Brad placed Jenkins, VictoriaMetrics, and Wger onto titan-22 directly
(main commits 271f3e8c3, 0ebab9d41) between this audit's first pass and
PR #50 going up. Revalidate with fresh 3h evidence instead of shipping a
report that describes a since-superseded snapshot: titan-22 headroom is
materially unchanged (22-26% CPU p95, mem 77%+ available) with the three
new residents, Jellyfin shows 0 restarts/throttling throughout, but no
GPU/transcode activity occurred in the window so the transcode-under-load
validation is still outstanding. Also flags VictoriaMetrics now being a
single point of failure for titan-22 observability (hard-pinned, no
fallback), and updates the board-comment record (§7) to match what was
actually posted to t_39cf1905/t_60d2deb6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
17c5f5093 landed on main mid-revision, moving sonarqube onto titan-22
with the same scavenger/CPU-only/no-GPU pattern as Jenkins/VictoriaMetrics/
Wger. Too recent (~4min runtime) for trend data, but healthy so far.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trailing wording fix: the header still said "three workloads" after
sonarqube (the 4th) was folded into §2/§4a in the prior commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Strengthens report §8: beyond the RBAC can-i output already cited, checked
(1) create verbs for jobs/pods and secrets read access (all no), (2) VM
for any jellyfin/session/transcode-named metric series (none exist), and
(3) Jellyfin's own API directly (public info reachable, /Sessions
correctly 401s without a key this session doesn't have). Confirms the
live transcode-under-load validation gap is a hard, verified constraint,
not an unexplored option.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/t_26da4c88-titan-capacity-guardrails-v4:feature/t_26da4c88-titan-capacity-guardrails-v4
git checkout feature/t_26da4c88-titan-capacity-guardrails-v4
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#53
No description provided.
Delete Branch "feature/t_26da4c88-titan-capacity-guardrails-v4"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?