titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: monitoring(titan): capacity guardrails for titan-20/21/22 audit #50

Draft
hermes-automation wants to merge 1 commits from feature/t_26da4c88-titan-capacity-guardrails into main
pull from: feature/t_26da4c88-titan-capacity-guardrails
merge into: titan:main
Conversation 0 Commits 1 Files Changed 5 +657
hermes-automation commented 2026-08-23 14:54:02 +00:00
Member
Copy Link

Summary

Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/classifier),
titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary) for Kanban card
t_26da4c88. Full report: docs/titan_20_21_22_capacity_optimization.md.

  • titan-20/21 have no safe headroom (titan-20: 88.6% CPU p95 over the
    last 3h, memory available as low as 1.7% at its floor; titan-21: 65% CPU
    p95). No workload was added to either node.
  • titan-22 has real, durable idle CPU/RAM (22-26% CPU p95, 77%+ memory
    available) even after three workloads (Jenkins, VictoriaMetrics, Wger)
    were placed there live by direct main commits during this audit,
    outside this PR — see report §4a. That placement is not authored by this
    PR; it is documented and evidence-validated by it.
  • Shared-GPU time-slicing on titan-22 has no VRAM/engine isolation
    (plain NVIDIA time-slicing, no MPS/MIG) — conditional GPU sharing
    (Option C) is evaluated and rejected as unsafe until the cluster has a
    GPU-sharing mode with real per-tenant isolation.
  • No workload is relocated by this PR. Per the card's own acceptance
    criteria, this ships guardrails/dashboard/report only.

Changes

  • docs/titan_20_21_22_capacity_optimization.md — durable capacity model,
    workload map, ranked distribution options (A-D), and validation plan.
  • services/monitoring/vmalert-atlas-availability.yaml — new
    atlas.titan-media-capacity alert group: titan-20 node memory
    exhaustion, titan-22 CPU/RAM/GPU-VRAM pressure, Jellyfin CPU throttling,
    titan-21 CPU pressure.
  • services/monitoring/grafana-dashboard-gpu.yaml +
    services/monitoring/dashboards/atlas-gpu.json — two new Atlas GPU
    panels (VRAM used/free, NVENC/NVDEC utilization by node), generated via
    scripts/render/dashboards_render_atlas.py (re-ran the generator;
    committed output is byte-identical to what it regenerates).

Rollback plan

Purely additive: one new doc, one new vmalert rule group (new group name,
does not modify or remove any existing group/rule), and two new dashboard
panels appended to the existing Atlas GPU dashboard (does not modify any
existing panel). No workload manifests, taints, priority classes, or node
labels are touched. To roll back, revert this PR's merge commit; nothing
else depends on the new alerts/panels/doc. No live cluster mutation is
performed by merging this PR beyond Flux reconciling the new
vmalert/dashboard ConfigMaps — if the new alert group is noisy, delete
the atlas.titan-media-capacity group block from
vmalert-atlas-availability.yaml in a follow-up PR.

Validation

  • kustomize build services/monitoring succeeds.
  • python3 scripts/render/dashboards_render_atlas.py regenerates
    grafana-dashboard-gpu.yaml/atlas-gpu.json byte-identical to what's
    committed.
  • New/changed YAML and JSON parse cleanly.
  • pytest scripts/tests scripts/tests/test_dashboards_render_atlas_drilldowns.py testing/tests/test_atlas_ai_dashboard.py and the monitoring capacity
    suites pass, except two pre-existing failures on origin/main
    unrelated to this PR's diff (test_victoria_metrics_has_dashboard_burst_headroom,
    test_metrics_storage_holds_a_full_retention_period) — both caused by
    the same direct main commits documented in report §4a
    (271f3e8c3, 1e6eda863), not by this PR. Neither test's file is
    touched here.
  • No live transcode-under-load validation was performed or is claimed —
    this read-only session (get/list/watch only, no mutate verbs) cannot
    trigger one; report §8 documents the plan to run once merged.

Leaving as draft for Brad's review per policy; not merging.

## Summary Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/classifier), titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary) for Kanban card `t_26da4c88`. Full report: `docs/titan_20_21_22_capacity_optimization.md`. - **titan-20/21 have no safe headroom** (titan-20: 88.6% CPU p95 over the last 3h, memory available as low as 1.7% at its floor; titan-21: 65% CPU p95). No workload was added to either node. - **titan-22 has real, durable idle CPU/RAM** (22-26% CPU p95, 77%+ memory available) even after three workloads (Jenkins, VictoriaMetrics, Wger) were placed there live by direct `main` commits during this audit, outside this PR — see report §4a. That placement is not authored by this PR; it is documented and evidence-validated by it. - **Shared-GPU time-slicing on titan-22 has no VRAM/engine isolation** (plain NVIDIA time-slicing, no MPS/MIG) — conditional GPU sharing (Option C) is evaluated and rejected as unsafe until the cluster has a GPU-sharing mode with real per-tenant isolation. - No workload is relocated by this PR. Per the card's own acceptance criteria, this ships guardrails/dashboard/report only. ## Changes - `docs/titan_20_21_22_capacity_optimization.md` — durable capacity model, workload map, ranked distribution options (A-D), and validation plan. - `services/monitoring/vmalert-atlas-availability.yaml` — new `atlas.titan-media-capacity` alert group: titan-20 node memory exhaustion, titan-22 CPU/RAM/GPU-VRAM pressure, Jellyfin CPU throttling, titan-21 CPU pressure. - `services/monitoring/grafana-dashboard-gpu.yaml` + `services/monitoring/dashboards/atlas-gpu.json` — two new `Atlas GPU` panels (VRAM used/free, NVENC/NVDEC utilization by node), generated via `scripts/render/dashboards_render_atlas.py` (re-ran the generator; committed output is byte-identical to what it regenerates). ## Rollback plan Purely additive: one new doc, one new `vmalert` rule group (new group name, does not modify or remove any existing group/rule), and two new dashboard panels appended to the existing `Atlas GPU` dashboard (does not modify any existing panel). No workload manifests, taints, priority classes, or node labels are touched. To roll back, revert this PR's merge commit; nothing else depends on the new alerts/panels/doc. No live cluster mutation is performed by merging this PR beyond Flux reconciling the new `vmalert`/dashboard ConfigMaps — if the new alert group is noisy, delete the `atlas.titan-media-capacity` group block from `vmalert-atlas-availability.yaml` in a follow-up PR. ## Validation - `kustomize build services/monitoring` succeeds. - `python3 scripts/render/dashboards_render_atlas.py` regenerates `grafana-dashboard-gpu.yaml`/`atlas-gpu.json` byte-identical to what's committed. - New/changed YAML and JSON parse cleanly. - `pytest scripts/tests scripts/tests/test_dashboards_render_atlas_drilldowns.py testing/tests/test_atlas_ai_dashboard.py` and the monitoring capacity suites pass, except two **pre-existing** failures on `origin/main` unrelated to this PR's diff (`test_victoria_metrics_has_dashboard_burst_headroom`, `test_metrics_storage_holds_a_full_retention_period`) — both caused by the same direct `main` commits documented in report §4a (`271f3e8c3`, `1e6eda863`), not by this PR. Neither test's file is touched here. - No live transcode-under-load validation was performed or is claimed — this read-only session (`get/list/watch` only, no mutate verbs) cannot trigger one; report §8 documents the plan to run once merged. Leaving as **draft** for Brad's review per policy; not merging.
hermes-automation added 1 commit 2026-08-23 14:54:04 +00:00 
Evidence-led capacity/QoS audit of titan-20 (Hermes LLM fallback/
classifier), titan-21 (STT/TTS), and titan-22 (Jellyfin media-primary,
restored) for t_26da4c88. titan-20/21 are CPU-committed with no safe
headroom (titan-20 at 227 MiB free memory at its 24h worst point);
titan-22 has real idle CPU/RAM but its shared-GPU time-slicing has no
VRAM/engine isolation, so no workload is relocated. Adds alerting for
the sharpest gaps found (titan-20 memory exhaustion, titan-22 CPU/RAM/
GPU-VRAM pressure, Jellyfin CPU throttling, titan-21 CPU pressure) and
two Atlas GPU dashboard panels (VRAM, NVENC/NVDEC utilization) so a
future opportunistic-workload PR or a live transcode incident is
visible without a promql session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/t_26da4c88-titan-capacity-guardrails:feature/t_26da4c88-titan-capacity-guardrails
git checkout feature/t_26da4c88-titan-capacity-guardrails
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#50
No description provided.
Delete Branch "feature/t_26da4c88-titan-capacity-guardrails"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?