titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: monitoring(titan): capacity guardrails for titan-20/21/22 audit (v3) #52

Draft
hermes-automation wants to merge 4 commits from feature/t_26da4c88-titan-capacity-guardrails-v3 into main
pull from: feature/t_26da4c88-titan-capacity-guardrails-v3
merge into: titan:main
Conversation 0 Commits 4 Files Changed 5 +769
5 changed files with 769 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

460
docs/titan_20_21_22_capacity_optimization.md Normal file
View File

@ -0,0 +1,460 @@
# Titan 20/21/22 capacity and placement optimization
Status: evidence-led audit + guardrails shipped. **No workload was relocated
onto titan-22 by this card/PR.** Between this audit's first pass and this
revision, Brad independently placed four workloads (Jenkins, VictoriaMetrics,
Wger, SonarQube) onto titan-22 via five direct commits to `main` (outside
this card's PR), implementing almost exactly the Option B pattern this
document recommends. §4a documents that live change and the fresh evidence
validating it. This document is the durable record for Kanban card
`t_26da4c88`.
Evidence collected in three passes, each triggered by discovering `main`
had moved again mid-audit:
- **First pass** 2026-08-23 13:00-13:30 UTC, at `origin/main@a019ecd55`,
covering a trailing-24h VictoriaMetrics window ending ~13:15Z (the longest
valid window since titan-22's return to service — node manifests show the
current `jellyfin` restore commit reconciling from 2026-08-23T01:11Z).
- **Revalidation pass** 2026-08-23 ~14:40-14:45 UTC, after `git fetch
--no-tags origin main` showed this branch was 4 commits behind a
fast-moving `main`; rebased cleanly (no file overlap with the audit's own
commit) onto `origin/main@42795f3d6`. Queried VictoriaMetrics directly
(`victoria-metrics-single-server.monitoring.svc.cluster.local:8428`,
in-cluster) for current + trailing-3h data, since 3 of the commits
changed placement for Jenkins/VictoriaMetrics/Wger only 1-3h before this
pass — see §4a.
- **Final pass** 2026-08-23 ~14:56 UTC: `main` advanced once more
(`17c5f5093`, a 4th direct placement commit moving SonarQube onto
titan-22) while this document was mid-revision; rebased again (clean) and
folded in as a 4th row in §2/§4a rather than shipping a report that was
already stale at open time.
Both passes used `kubectl` as `system:serviceaccount:hermes:hermes-agent`
(per-namespace `get/list/watch` on pods/deployments/statefulsets/daemonsets;
no mutate verbs at all — confirmed with `kubectl auth can-i --list`, both
passes). No pods, deployments, taints, or labels were changed live by this
card. No `kubectl port-forward`/exec/mutate was available or attempted.
## 1. Node inventory (live)
| Node | Arch | CPU alloc | Mem alloc | GPU | Taint | Labels of note |
|---|---|---|---|---|---|---|
| titan-20 | arm64 Jetson Xavier | 6 | 14.9 GiB | `nvidia.com/gpu.shared: 4` | none | `jetson=true`, `accelerator=nvidia` |
| titan-21 | arm64 Jetson Xavier | 6 | 14.9 GiB | `nvidia.com/gpu.shared: 4` | none | `jetson=true`, `accelerator=nvidia`, no `longhorn-host` |
| titan-22 | amd64 (RTX 3050 Ti Laptop, 8 GiB VRAM) | 20 | 31.1 GiB | `nvidia.com/gpu: 0`, `nvidia.com/gpu.shared: 4` | `atlas.bstein.dev/media-primary=true:PreferNoSchedule` | `atlas.bstein.dev/media-primary=true`, `atlas.bstein.dev/general-compute=last-resort`, `jellyfin=true`, `longhorn-host=true` |
`titan-22` really does report `nvidia.com/gpu: 0` — confirmed live. It is a
shared-GPU-only provider, as the card's snapshot assumed. DCGM identifies the
card as an **RTX 3050 Ti Laptop GPU**, not the desktop 3050 8 GB the card
text assumed; usable VRAM is ~7.95 GiB net of driver overhead (`DCGM_FI_DEV_FB_FREE`
+ `DCGM_FI_DEV_FB_USED` ≈ 7755 MiB at idle).
The three node labels/taints on titan-22 are produced every minute by
`infrastructure/core/node-prefer-noschedule-cronjob.yaml`, a `kubectl`-based
reconciler CronJob. Its own comment states the design intent directly:
*"Keep spare capacity available without letting routine pods displace media
service."* `atlas.bstein.dev/general-compute=last-resort` already exists for
exactly the Phase B "Option B" question this card asks — but **no workload in
the repo currently targets or tolerates it**. The taint is `PreferNoSchedule`
(soft), so nothing is technically blocked from landing on titan-22 today;
in practice nothing chooses to, because every other node in the same pools
usually has room. This matters for the Jenkins finding in §5.
## 2. Workload placement and shared-GPU claims (live pods, Git-declared requests/limits)
| Node | Workload | Priority class | CPU req/lim | Mem req/lim | GPU |
|---|---|---|---|---|---|
| titan-20 | `ollama` (ai-llm; Hermes local Qwen fallback + classifier) | default (0) | 4 / 8 | 10Gi / 14Gi | `gpu.shared: 1` |
| titan-21 | `hermes-stt` (Whisper small) | default (0) | 2 / 6 | 4Gi / 10Gi | `gpu.shared: 1` |
| titan-21 | `hermes-tts` (Piper, CPU-only) | default (0) | 1 / 4 | 512Mi / 2Gi | none |
| titan-22 | `jellyfin` | **media-core (400)** | 2 / 8 | 2Gi / 8Gi | `gpu.shared: 1` |
| titan-22 | `jenkins` (added live, see §4a) | **scavenger (-10)** | 0.1 / 1.5 | 1Gi / 3Gi | none |
| titan-22 | `victoria-metrics-single-server` (added live, see §4a) | **scavenger (-10)** | 0.5 / 2 | 2Gi / 4Gi | none |
| titan-22 | `wger` (added live, see §4a) | **scavenger (-10)** | 0.25 / 1.2 | 0.6Gi / 2.25Gi | none |
| titan-22 | `sonarqube` (added live, see §4a) | **scavenger (-10)** | 0.1 / 2 | 2Gi / 4Gi | none |
| titan-22 | DaemonSets (dcgm-exporter, node-exporter, nvidia-process-exporter, fluent-bit, metis-sentinel-amd64, node-nofile, node-image-sweeper, titan-22-link-keeper) | default | ~0.3 total | ~0.9Gi total | none |
`jellyfin`'s live pod spec already carries `priorityClassName: media-core`,
`runtimeClassName: nvidia`, a hard `nodeSelector: {kubernetes.io/hostname:
titan-22}`, and an explicit toleration for the media-primary taint — this
*is* in the current `services/jellyfin/deployment.yaml` on `origin/main`
(commit `eec373dd5`, "jellyfin: run media service on titan-22"). The branch
this card started from was 89 commits behind `origin/main` and still had the
pre-restore Jellyfin manifest (no GPU claim, no priority class, `titan-24`-
preferring affinity); it was fast-forwarded before any of the analysis above.
**If you are reading a stale checkout of this repo, re-fetch before trusting
any "Jellyfin isn't protected" read — it already is, as of PR #48's chain.**
`services/hermes/execution-worker-statefulset.yaml`,
`execution-mediator.yaml`, and `switchyard-deployment.yaml` all carry
`priorityClassName: scavenger` (value **-10**) and a required node
anti-affinity excluding `titan-22`/`titan-24` (plus the storage-backbone
nodes). This exclusion has no capacity justification found in this
investigation — it looks like a defensive placement rule, not evidence that
those workloads don't fit. It is left untouched per the card's explicit
instruction not to relax it without workload-level evidence, but it is worth
a follow-up card if Hermes execution capacity is ever tight (it currently is
not — Hermes workers are healthy on their existing arm64 pool).
Shared-GPU device plugin (`infrastructure/modules/profiles/components/device-plugin-config/configmap.yaml`,
mounted by all three per-node `nvidia-device-plugin-*` DaemonSets) uses:
```yaml
sharing:
timeSlicing:
renameByDefault: true
resources:
- name: nvidia.com/gpu
replicas: 4
```
This is **plain NVIDIA time-slicing**, not MPS and not MIG. It multiplies
one physical device into 4 schedulable `nvidia.com/gpu.shared` slots so the
Kubernetes scheduler can bin-pack GPU-requesting pods, but the driver
time-slices SM access across whichever processes are co-resident with
**no priority, no VRAM isolation, and no per-tenant QoS**. This is the load-
bearing fact for the Option C verdict in §4.
## 3. 24h evidence (VictoriaMetrics, `node_exporter`/`DCGM`/`kube-state-metrics`)
CPU = `100% - idle%`, averaged over 5m windows; percentiles are
`quantile_over_time(...)[24h:5m]`.
| Node | CPU p50 | CPU p95 | CPU p99 | CPU max | Mem now | Mem p95/max | Mem min-available |
|---|---|---|---|---|---|---|---|
| titan-20 | 68.2% | 88.4% | 96.9% | 100%* | **98.3%** | 98.2% / 98.5% | **227 MiB** of 14.9 GiB |
| titan-21 | 50.3% | 59.9% | — | 77.5% | 61.1% | 61.1% / 63.1% | ~5.5 GiB |
| titan-22 | 2.8% | 26.1% | 47.0% | 100%* | 10.2% | 10.9% / 11.0% | ~27.8 GiB |
\* brief single-sample spikes; not sustained (p99 is the honest ceiling).
Declared `kube_pod_container_resource_requests` summed per node (live,
includes DaemonSets):
| Node | CPU requested / allocatable | Mem requested / allocatable |
|---|---|---|
| titan-20 | 4.88 / 6 (**81%**) | 11.6 / 14.9 GiB (78%) |
| titan-21 | 5.58 / 6 (**93%**) | 7.0 / 14.9 GiB (47%) |
| titan-22 | 5.20 / 20 (26%) | 3.9 / 31.1 GiB (12.5%) |
Two things follow directly from this table:
1. **titan-20 and titan-21 have essentially no scheduling headroom left**,
independent of the usage numbers above — the scheduler already considers
81% and 93% of their CPU committed. Adding anything there, even something
that "should" be idle most of the time, risks failing to schedule at all
or forcing evictions. This is true *before* accounting for the fact that
titan-20's actual memory usage (98.3%) already runs well above its summed
requests (78%), because `ollama`'s container burst headroom (10Gi
request → 14Gi limit) is being used in practice. titan-20 is the
tightest resource in this entire audit and deserves its own alert (§6);
it is not caused by this card and is not something this card's scope
authorizes fixing (Hermes Chat first-pass model quality is explicitly
protected — reducing `ollama`'s memory/CPU footprint is a quality
trade-off decision for Brad, not a capacity optimization).
2. **titan-22 has ~14.8 CPU and ~27 GiB of committed-but-unrequested
headroom**, and observed usage confirms it: p95 CPU is 26%, p50 is under
3%. Disk: 16.3% of ~915 GiB root filesystem used (~768 GiB free — this is
local/ephemeral storage; Jellyfin's config/media/cache volumes are
Longhorn PVCs and an emptyDir, not counted in that headroom). Network:
24h peak combined rx+tx ≈ 52 KB/s, i.e., not network-bound.
### GPU (titan-22, DCGM `10.42.32.14:9400`, 24h)
| Metric | p95 | max |
|---|---|---|
| SM utilization | 0% | 3% |
| Encoder (NVENC) utilization | — | 27% |
| Decoder (NVDEC) utilization | — | 22% |
| Frame-buffer (VRAM) used | — | 1276 MiB of ~7955 MiB |
The encoder/decoder activity (peaking at 27%/22%, VRAM peaking at 1.27 GiB)
is real evidence of at least one active hardware transcode/playback session
inside the 24h window — the GPU is not idle-because-untested, it is
idle-because-Jellyfin-usage-is-genuinely-light-relative-to-capacity right
now. `jellyfin` shows 0 restarts and 0 OOMKills over the window, and CPU
throttling (`container_cpu_cfs_throttled_periods_total` ratio) is 0 for
`jellyfin`, `ollama`, `stt`, `tts`, and `model-gate` over the full 24h — none
of the workloads audited are limit-throttled today.
## 4. Phase B — ranked distributions
**A. Preserve the dedicated pattern (fallback/classifier on titan-20, voice
on titan-21, titan-22 Jellyfin-only).** This is close to the *already-live*
state and remains correct for titan-20/21: both are already at 81-93% of
CPU committed and titan-20 is within ~230 MiB of physical memory exhaustion
at its worst point in the last 24h. There is no safe way to add load to
either node without either starving Hermes Chat's protected first-pass model
quality or risking an OOM on the node hosting it. **Kept as-is for titan-20/
titan-21.**
**B. titan-22 CPU/RAM for isolated, preemptible, non-GPU opportunistic work,
with a reserved Jellyfin budget.** Evidence-backed and safe *as a policy*:
titan-22 has ~14.8 idle CPU and ~27 GiB idle RAM, Kubernetes pod-priority
preemption is a real, well-defined mechanism for CPU/RAM (unlike GPU
time-slicing — see Option C), and the repo already ships the exact
preemptible priority class needed (`scavenger`, value -10, already below
Jellyfin's `media-core` at 400) plus the exact node label this design was
built for (`atlas.bstein.dev/general-compute=last-resort`). **This is the
selected direction — see §5 for what shipped and what deliberately did not.**
**C. Conditional shared-GPU use of titan-22 when media is idle.**
**Rejected.** The device plugin config (§2) is pure NVIDIA time-slicing:
4 renamed replicas of one physical GPU, no MPS, no MIG, no per-client
priority or VRAM quota. Kubernetes priority/preemption operates at pod
*scheduling* granularity (evict-and-reschedule), not at GPU-engine
granularity — it cannot arbitrate two already-running pods' NVENC/NVDEC/SM
access mid-session, which is exactly the failure mode that would degrade an
active Jellyfin transcode. There is no "idle detector + lease" primitive in
this cluster (no DRA, no custom GPU operator, no MPS control daemon) to make
this enforceable rather than a static promise. Until the cluster runs a GPU
sharing mode with real isolation (MPS with per-client memory/thread limits,
or a device-plugin generation with priority-aware time-slicing), Option C
cannot meet the card's own bar ("reject it if it cannot protect active
Jellyfin transcodes"). **Not implemented.**
**D. Broaden Jenkins's `hardware=rpi5` affinity to use titan-22.**
**Superseded by a live change Brad made directly — see §4a.** This audit's
first pass (13:00-13:30Z) found Jenkins still rpi5-pinned and recommended
deferring this decision to `t_39cf1905` as a manual product call, for the
reasons below (kept for the record — they were the right concerns to raise,
even though Brad has since made the call):
- `t_39cf1905`'s own read-only diagnosis (2026-08-23 12:53-13:19Z, its board
comments) root-caused the then-live Jenkins `Pending` state as a
**self-inflicted ConfigMap-hash rollout deadlock** on the existing 3-node
rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and shipped a
fix as PR #49. Broadening Jenkins to titan-22 would not have fixed that
specific problem on its own.
- Moving Jenkins onto titan-22 is a nontrivial fit for "opportunistic":
Jenkins builds (especially Docker-in-Docker image builds) are bursty and
CPU-hungry by design. It also has architecture (multi-arch image
verification for `jenkins/jenkins:2.528.3-jdk21` + plugin CLI on amd64),
PVC-locality (three RWO PVCs previously only ever scheduled on the rpi5
pool), and recovery-isolation implications (the previous
`NotIn [titan-13,15,17,19]` rule kept Jenkins off storage-backbone nodes
during Longhorn rebuilds — titan-22 is also a `longhorn-host`).
### 4a. Live update: Brad already implemented Option D (and Option B) directly
Between this audit's first pass and this revision, **four** direct commits
to `main` — **not through this card's PR, authored
`jenkins <brad@bstein.dev>`** — re-pinned Jenkins, VictoriaMetrics, Wger,
and SonarQube onto titan-22, in rapid succession over about an hour:
| Commit | Workload | What changed |
|---|---|---|
| `271f3e8c3` "ops: spread saturated node workloads" | `jenkins` | `nodeSelector.arch` rpi5→amd64; required affinity `NotIn[titan-13,15,17,19]`+`hardware=rpi5` → required `hostname In [titan-22]`; added `priorityClassName: scavenger` and a `PreferNoSchedule` toleration for the media-primary taint |
| `271f3e8c3` (same commit) | `victoria-metrics-single-server` | required affinity `hostname NotIn [titan-12,20,21,22,24,...]` → `In [titan-22]`; added `priorityClassName: scavenger` + media-primary toleration; `upgrade.force: true` |
| `0ebab9d41` "placement(wger): use preemptible titan-22 capacity" | `wger` | same pattern: `NotIn[titan-13,15,17,19]` → `hostname In [titan-22]`, `arch` arm64→amd64, `priorityClassName: scavenger`, media-primary toleration |
| `17c5f5093` "placement(quality): use titan-22 spare capacity" | `sonarqube` | dropped its rpi5/rpi4 + control-plane-exclusion affinity and keycloak/grafana anti-affinity entirely; required affinity → `hostname In [titan-22]`, `arch` → amd64, `priorityClassName: scavenger`, media-primary toleration |
This is **exactly the Option B pattern** this document independently arrived
at (preemptible `scavenger` priority, CPU/RAM only, zero GPU requests) —
applied to Jenkins as well, which the first pass had flagged as needing a
separate evidence-based decision (Option D). Brad made that call directly.
None of these four request `nvidia.com/gpu*` (confirmed live). This is a
direct-to-`main` change outside the "all durable changes via reviewable
titan-iac PRs" policy in this card's own body, arriving faster than this
audit's own revision cycle (a 4th workload landed between this section
being drafted and being finalized) — noted here for the record, not
something this card can undo or should second-guess; it's the owner's
prerogative.
**Fresh evidence (14:40-14:45Z revalidation pass, ~1-3h of run time for the
new placements) shows it is safe so far:**
| Node | 3h CPU p95 | 3h CPU max | Mem available now | Mem available min (3h) |
|---|---|---|---|---|
| titan-20 | 88.6% | 90.5% | 2.6% | **1.7%** |
| titan-21 | 65.1% | 69.3% | 38.9% | 38.5% |
| titan-22 | 22.1% | 26.2% | 77.4% | 77.5% |
- titan-22 CPU/RAM headroom is essentially unchanged from the first pass
despite three new residents — confirms Option B's "~15 CPU / ~27 GiB idle"
finding was real spare capacity, not measurement noise.
- `jellyfin` (titan-22): 0 restarts, 0 CPU-throttled periods, `Ready`,
no pod events, over the full window these three workloads have been
co-resident.
- `jenkins` (titan-22): CPU-throttled up to 30% of periods at its own 1.5-CPU
*limit* — this is the CFS quota doing its job (Jenkins capped at its own
ceiling), not evidence of node-level contention; `wger` and
`victoria-metrics-single-server` show negligible throttling (<1%).
- titan-22 GPU (DCGM): 0% SM/encoder/decoder utilization and ~1 MiB VRAM used
for the entire 3h window — **no active transcode occurred during this
co-residency window**, so this is not yet a live proof that Jellyfin
survives a concurrent hardware transcode with these three neighbors
present. That validation (§8) is still outstanding and still requires
either a real playback session or synthetic load-gen, neither of which
this read-only session can trigger.
- titan-20/21 numbers reconfirm the first pass and got *worse*, not better:
titan-20 is now down to 1.7% memory available at its 3h floor (vs. "227
MiB / ~1.5%" at its 24h floor in the first pass) — Option A (leave
titan-20/21 alone) remains the only safe call for those two nodes.
- A 4th workload, `sonarqube` (`17c5f5093`), landed on titan-22 too late for
the 3h window above (~4 minutes of runtime at revalidation time) — too
little data for a trend, but it is `Running` with 0 restarts, requests
0.1 CPU / 2Gi / limits 2 CPU / 4Gi, `scavenger` priority, no GPU request,
consistent with the other three.
**New risk this live change introduces, not present in the first pass's
Option B analysis:** `victoria-metrics-single-server` is the cluster's
*only* metrics backend, and it is now hard-pinned (`required` affinity, not
`preferred`) to the same node as Jellyfin, Jenkins, and Wger. If titan-22
has a node-level incident, the dashboards and alerts needed to diagnose that
exact incident go dark at the same time (`vmalert-atlas-availability` and
`grafana` themselves stay up, on titan-0b/titan-11, but have nothing to
query). This wasn't evaluated against the card's Jellyfin-protection bar
because it's an observability-resilience question, not a Jellyfin-quality
one — flagging it as a follow-up decision for Brad rather than changing it
unilaterally.
## 5. Phase C — what shipped
**No workload was relocated onto titan-22 by this card's PR.** Per the
card's own acceptance criteria ("If no safe redistribution is proven,
deliver the report/dashboard/guardrails only; do not move workloads for
theoretical utilization"), and because this session's `hermes-agent`
identity has **zero mutate verbs** on the live cluster (confirmed: `kubectl
auth can-i --list` returns only `get/list/watch` across every resource
type — there is no way to even trigger a live synthetic transcode to
validate a placement change from this session), this card does not itself
move a workload's placement without being able to observe the result live —
that would violate the card's own validation bar ("New placement
demonstrates Jellyfin remains healthy under a representative active
transcode while the opportunistic workload runs, **or does not ship**").
As documented in §4a, Brad independently placed Jenkins, VictoriaMetrics,
and Wger onto titan-22 via direct `main` commits during this audit — using
almost exactly the Option B pattern below, before this PR existed to review
it. This PR does not touch, revert, or re-author that change; it documents
it (§4a, workload table in §2) and extends the guardrails to cover it.
Guardrails shipped by this PR:
1. **Alert rules** (`services/monitoring/vmalert-atlas-availability.yaml`,
new `atlas.titan-media-capacity` group): Jellyfin CPU/memory pressure at
its own limits, titan-22 GPU VRAM saturation, titan-22 CPU/RAM
near-exhaustion, CPU throttling on any of the audited media/AI
containers, and — because it was the sharpest finding in this audit —
titan-20 node memory exhaustion (it is currently 227 MiB from the edge
with no alert covering it at all).
2. **Dashboard**: two new panels on `Atlas GPU`
(`services/monitoring/grafana-dashboard-gpu.yaml`, regenerated from
`scripts/render/dashboards_render_atlas.py`) showing titan-22 VRAM
used/free and encoder/decoder utilization by node, so a future
opportunistic-workload PR (or a live transcode incident) is visible
without a promql session.
3. **This document**, as the durable capacity model, workload map, and
validation plan referenced by the acceptance criteria.
### What was deliberately *not* shipped, and why
A `PriorityClass`/quota primitive for Option B was not "new" to invent —
`scavenger` already existed and was already below `media-core`. This audit's
first pass found no already-portable opportunistic workload to place on
titan-22 with it (Jenkins, Cassandra/Veles backends, `collabora`,
CI/quality-gate jobs all had their own node-pool pinning or an in-flight
incident) and recommended a follow-up card rather than manufacturing a
placement. Brad then placed three workloads (Jenkins, VictoriaMetrics,
Wger) directly — see §4a — which is the concrete instance of that follow-up,
done outside this card. What is still genuinely outstanding, and still
cannot be done from this read-only session, is the **live
transcode-under-load validation** (§8): none of titan-22's new residents'
~1-3h of co-residency so far has overlapped with an active hardware
transcode.
### Reservation budget (documented contract, not enforced by a new object)
Any opportunistic workload placed on titan-22 — including the three already
live (§4a), which all comply — must:
- use `priorityClassName: scavenger` (preemptible below Jellyfin's
`media-core`);
- request **zero** `nvidia.com/gpu*` resources (Option C is rejected — see
§4);
- keep summed CPU/memory *requests* across all such workloads at or below
**10 CPU / 18 GiB**, which — added to Jellyfin's own limit ceiling (8 CPU /
8 GiB) and the ~0.3 CPU / 0.9 GiB of existing DaemonSets — stays within
titan-22's 20 CPU / 31.1 GiB allocatable even if Jellyfin is transcoding
at its full burst limit at the same time;
- set a real CPU **limit** (not just a request) so a burst cannot starve
Jellyfin's CFS shares even transiently.
Kubernetes has no native per-node ResourceQuota primitive, so this budget is
a documented contract enforced by the new alerts in §6, not by an admission
object. A namespace-scoped `ResourceQuota` keyed to `scavenger` pods was
considered and rejected: it cannot be scoped to a specific node, so it would
either do nothing (if the opportunistic workload's namespace also runs
elsewhere) or wrongly cap unrelated scavenger workloads on other nodes
(Hermes execution workers already use `scavenger` on the arm64 pool).
## 6. New alerts (`services/monitoring/vmalert-atlas-availability.yaml`)
| Alert | Fires when | Why |
|---|---|---|
| `Titan20NodeMemoryCritical` | titan-20 `MemAvailable` < 5% of `MemTotal` for 10m | Sharpest finding in this audit: 227 MiB free at the worst point in 24h, no existing coverage. |
| `TitanMediaHostCPUPressure` | titan-22 CPU busy > 85% for 10m | Node-level early warning before Jellyfin's own limit-throttling would show up. |
| `TitanMediaHostMemoryPressure` | titan-22 `MemAvailable` < 15% of `MemTotal` for 10m | Same, for memory; also the number that must stay large if a future opportunistic workload lands here. |
| `JellyfinCPUThrottled` | `container_cpu_cfs_throttled_periods_total` ratio > 5% for `container="jellyfin"` over 5m | Direct transcode-quality signal — sustained throttling degrades encode. |
| `TitanMediaGPUMemorySaturation` | titan-22 DCGM `FB_USED / (FB_USED+FB_FREE)` > 85% for 5m | VRAM exhaustion is the concrete failure mode a second shared-GPU tenant would cause (Option C). |
| `TitanVoiceHostCPUPressure` | titan-21 CPU busy > 90% for 15m | titan-21 is already at 93% CPU *requested*; this catches sustained real contention that could add STT/TTS latency for Hermes Chat. |
All use the existing `atlas-vm` VictoriaMetrics datasource/label set already
scraped in this cluster (`node_exporter`, `DCGM`, cAdvisor `container_cpu_cfs_*`)
— no new exporters required.
## 7. Board comments delivered
- `t_39cf1905` (Jenkins/WebUI release): told that Jenkins is **already**
running on titan-22 as of a direct main commit (`271f3e8c3`, ~13:37Z,
authored `jenkins <brad@bstein.dev>`, not through this card), that it is
CPU-only/`scavenger`-priority (correctly preemptible below Jellyfin's
`media-core`) and healthy so far (0 restarts, self-limited throttling only,
titan-22 still at 22-26% p95 CPU / 77%+ mem available with it resident) —
so their Part B (trigger the WebUI image builder now that Jenkins is
healthy) should be unblocked from the node-placement side; and warned that
no active hardware transcode has yet coincided with Jenkins's presence, so
Jellyfin-under-load-plus-Jenkins-build is still unproven, not proven-safe.
- `t_60d2deb6` (Chat latency instrumentation): told that titan-20 (LLM
fallback/classifier) is CPU/RAM-saturated (88.6% CPU p95 over the last 3h,
memory available down to **1.7%** at its floor) and titan-21 (STT/TTS) is
at 65% CPU p95 with 38-39% memory available — so any voice-path latency
variance they measure should be checked against titan-20/21 node pressure
before being attributed to application code, and neither node has spare
capacity for a node-level latency mitigation; also that titan-22 is
amd64/no-arm64-GPU and therefore not a relevant relocation target for any
Jetson-pinned STT/TTS/classifier component, and that titan-22 now also
hosts the cluster's only VictoriaMetrics instance (hard-pinned, single
point of failure for the dashboards this instrumentation work will build
on) as a heads-up, not a blocker.
## 8. Validation plan (live run pending — this session has no mutate access)
This session cannot apply the alert/dashboard PR, cannot trigger a synthetic
transcode, and cannot restart or exec into `jellyfin` (RBAC is `get/list/
watch` only, confirmed). As of this revision, Jenkins/VictoriaMetrics/Wger
have been co-resident on titan-22 for ~1-3h (§4a) with zero Jellyfin impact,
but zero GPU activity in that window either — so the one validation that
actually matters (an active hardware transcode overlapping with real
opportunistic load) still has not happened naturally and still cannot be
forced from here. The plan below is what should be executed once the PR in
§9 is merged and reconciled (or opportunistically, the next time Jellyfin is
in active use):
1. Confirm the two new GPU dashboard panels and the six new alerts render
(`Atlas GPU` dashboard; `vmalert-atlas-availability` targets in Grafana
Alerting) with no `parse error` in `vmalert`'s own logs.
2. Start (or wait for) a real hardware-transcoding Jellyfin playback session
(a client requesting a bitrate/codec the source doesn't natively support
forces NVENC/NVDEC use). While it runs, confirm on the new panels: DCGM
encoder/decoder utilization rises, `jellyfin` stays `Ready`, and none of
the six new alerts fire.
3. As a synthetic corroboration in the same window, run a short (~5 minute)
CPU/memory-only load generator Job on titan-22
(`priorityClassName: scavenger`, request e.g. 4 CPU / 4Gi, no GPU
resource — stress-ng or similar) and confirm: it schedules, Jellyfin's
playback in step 2 shows no stutter/dropped-frame/error in its own logs,
and `TitanMediaHostCPUPressure`/`TitanMediaHostMemoryPressure` do not
fire (they are tuned to the *node* ceiling, not to this Job's own
request, so a well-behaved opportunistic Job should not trip them).
4. Record the result (pass/fail + screenshots or exported panel PNGs) as a
comment on this card's board entry, or on the follow-up card once one
exists, before any real opportunistic workload is proposed for titan-22.

63
scripts/render/dashboards_render_atlas.py
View File

@ -289,6 +289,32 @@ def dcgm_gpu_util_by_node():
)
def dcgm_metric_by_node(promql_expr):
"""Map a raw DCGM metric expression to a per-node series via kube_pod_info."""
dcgm_pod = f'label_replace(({promql_expr}), "pod", "$1", "Hostname", "(.*)")'
dcgm_ns = 'label_replace(' + dcgm_pod + ', "namespace", "monitoring", "", "")'
return (
"avg by (node) ("
f"{dcgm_ns} * on(namespace,pod) group_left(node) "
'kube_pod_info{namespace="monitoring"}'
")"
)
def dcgm_vram_percent_by_node():
return dcgm_metric_by_node(
"DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) * 100"
)
def dcgm_enc_util_by_node():
return dcgm_metric_by_node("DCGM_FI_DEV_ENC_UTIL")
def dcgm_dec_util_by_node():
return dcgm_metric_by_node("DCGM_FI_DEV_DEC_UTIL")
def nvidia_gpu_util_by_node():
return "max by (node) (nvidia_gpu_device_utilization_percent)"
@ -6102,6 +6128,43 @@ def build_gpu_dashboard():
description="NVML process-level SM samples mapped to Kubernetes pods through host cgroups; values are per-process activity rather than duplicated whole-device utilization.",
)
)
# DCGM-only nodes (titan-22, titan-24): VRAM and video engine utilization.
# Jetson nodes (titan-20/21) don't run DCGM and are absent from these two
# panels by construction. Added for the titan-20/21/22 capacity audit
# (t_26da4c88) so a second shared-GPU tenant contending with an active
# Jellyfin hardware transcode on titan-22 is visible without a promql
# session; see docs/titan_20_21_22_capacity_optimization.md.
panels.append(
timeseries_panel(
5,
"GPU Memory (VRAM) Used by Node",
dcgm_vram_percent_by_node(),
{"h": 8, "w": 12, "x": 0, "y": 16},
unit="percent",
max_value=100,
legend="{{node}}",
legend_display="table",
legend_placement="right",
description="DCGM frame-buffer used as a percent of total VRAM, per DCGM-instrumented node (titan-22, titan-24). Time-sliced GPU sharing gives no VRAM isolation between tenants, so this is the most direct signal of contention risk for an active Jellyfin transcode on titan-22.",
)
)
panels.append(
timeseries_panel(
6,
"GPU Encoder/Decoder Utilization by Node",
None,
{"h": 8, "w": 12, "x": 12, "y": 16},
unit="percent",
max_value=100,
legend_display="table",
legend_placement="right",
targets=[
{"expr": dcgm_enc_util_by_node(), "refId": "A", "legendFormat": "{{node}} encoder"},
{"expr": dcgm_dec_util_by_node(), "refId": "B", "legendFormat": "{{node}} decoder"},
],
description="DCGM NVENC/NVDEC engine utilization, per DCGM-instrumented node. This is the engine Jellyfin hardware transcode/playback actually contends on; the shared-GPU device plugin (nvidia.com/gpu.shared) does not reserve or prioritize it.",
)
)
return {
"uid": "atlas-gpu",
"title": "Atlas GPU",

83
services/monitoring/dashboards/atlas-gpu.json
View File

@ -189,6 +189,89 @@
}
],
"description": "NVML process-level SM samples mapped to Kubernetes pods through host cgroups; values are per-process activity rather than duplicated whole-device utilization."
},
{
"id": 5,
"type": "timeseries",
"title": "GPU Memory (VRAM) Used by Node",
"datasource": {
"type": "prometheus",
"uid": "atlas-vm"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"targets": [
{
"expr": "avg by (node) (label_replace(label_replace((DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) * 100), \"pod\", \"$1\", \"Hostname\", \"(.*)\"), \"namespace\", \"monitoring\", \"\", \"\") * on(namespace,pod) group_left(node) kube_pod_info{namespace=\"monitoring\"})",
"refId": "A",
"legendFormat": "{{node}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right"
},
"tooltip": {
"mode": "multi"
}
},
"description": "DCGM frame-buffer used as a percent of total VRAM, per DCGM-instrumented node (titan-22, titan-24). Time-sliced GPU sharing gives no VRAM isolation between tenants, so this is the most direct signal of contention risk for an active Jellyfin transcode on titan-22."
},
{
"id": 6,
"type": "timeseries",
"title": "GPU Encoder/Decoder Utilization by Node",
"datasource": {
"type": "prometheus",
"uid": "atlas-vm"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"targets": [
{
"expr": "avg by (node) (label_replace(label_replace((DCGM_FI_DEV_ENC_UTIL), \"pod\", \"$1\", \"Hostname\", \"(.*)\"), \"namespace\", \"monitoring\", \"\", \"\") * on(namespace,pod) group_left(node) kube_pod_info{namespace=\"monitoring\"})",
"refId": "A",
"legendFormat": "{{node}} encoder"
},
{
"expr": "avg by (node) (label_replace(label_replace((DCGM_FI_DEV_DEC_UTIL), \"pod\", \"$1\", \"Hostname\", \"(.*)\"), \"namespace\", \"monitoring\", \"\", \"\") * on(namespace,pod) group_left(node) kube_pod_info{namespace=\"monitoring\"})",
"refId": "B",
"legendFormat": "{{node}} decoder"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right"
},
"tooltip": {
"mode": "multi"
}
},
"description": "DCGM NVENC/NVDEC engine utilization, per DCGM-instrumented node. This is the engine Jellyfin hardware transcode/playback actually contends on; the shared-GPU device plugin (nvidia.com/gpu.shared) does not reserve or prioritize it."
}
],
"time": {

83
services/monitoring/grafana-dashboard-gpu.yaml
View File

@ -198,6 +198,89 @@ data:
}
],
"description": "NVML process-level SM samples mapped to Kubernetes pods through host cgroups; values are per-process activity rather than duplicated whole-device utilization."
},
{
"id": 5,
"type": "timeseries",
"title": "GPU Memory (VRAM) Used by Node",
"datasource": {
"type": "prometheus",
"uid": "atlas-vm"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"targets": [
{
"expr": "avg by (node) (label_replace(label_replace((DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) * 100), \"pod\", \"$1\", \"Hostname\", \"(.*)\"), \"namespace\", \"monitoring\", \"\", \"\") * on(namespace,pod) group_left(node) kube_pod_info{namespace=\"monitoring\"})",
"refId": "A",
"legendFormat": "{{node}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right"
},
"tooltip": {
"mode": "multi"
}
},
"description": "DCGM frame-buffer used as a percent of total VRAM, per DCGM-instrumented node (titan-22, titan-24). Time-sliced GPU sharing gives no VRAM isolation between tenants, so this is the most direct signal of contention risk for an active Jellyfin transcode on titan-22."
},
{
"id": 6,
"type": "timeseries",
"title": "GPU Encoder/Decoder Utilization by Node",
"datasource": {
"type": "prometheus",
"uid": "atlas-vm"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"targets": [
{
"expr": "avg by (node) (label_replace(label_replace((DCGM_FI_DEV_ENC_UTIL), \"pod\", \"$1\", \"Hostname\", \"(.*)\"), \"namespace\", \"monitoring\", \"\", \"\") * on(namespace,pod) group_left(node) kube_pod_info{namespace=\"monitoring\"})",
"refId": "A",
"legendFormat": "{{node}} encoder"
},
{
"expr": "avg by (node) (label_replace(label_replace((DCGM_FI_DEV_DEC_UTIL), \"pod\", \"$1\", \"Hostname\", \"(.*)\"), \"namespace\", \"monitoring\", \"\", \"\") * on(namespace,pod) group_left(node) kube_pod_info{namespace=\"monitoring\"})",
"refId": "B",
"legendFormat": "{{node}} decoder"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right"
},
"tooltip": {
"mode": "multi"
}
},
"description": "DCGM NVENC/NVDEC engine utilization, per DCGM-instrumented node. This is the engine Jellyfin hardware transcode/playback actually contends on; the shared-GPU device plugin (nvidia.com/gpu.shared) does not reserve or prioritize it."
}
],
"time": {

80
services/monitoring/vmalert-atlas-availability.yaml
View File

@ -382,6 +382,86 @@ data:
)
labels:
rollup: hourly
titan-media-capacity.yaml: |
groups:
- name: atlas.titan-media-capacity
# Covers the titan-20/21/22 evidence audit from t_26da4c88: titan-20
# runs at 227 MiB free at its worst point in a day with no coverage
# at all, and titan-22 is Jellyfin's protected media-primary host —
# any future opportunistic workload there needs contention alerts
# to exist before it ships, not after.
interval: 1m
rules:
- alert: Titan20NodeMemoryCritical
expr: |
(node_memory_MemAvailable_bytes{node="titan-20"} / node_memory_MemTotal_bytes{node="titan-20"}) < 0.05
for: 10m
labels:
severity: critical
service: hermes-ai-llm
annotations:
summary: "titan-20 (Hermes local LLM fallback/classifier) is within 5% of an OOM"
description: "titan-20 hosts the hard-pinned Ollama fallback/classifier for Hermes Chat. Available memory has been under 5% of total for 10m. This node runs with no memory slack under normal load (98%+ used is typical); this alert fires only when it is at real risk of an OOM kill. Do not add workloads to titan-20 to relieve this — see docs/titan_20_21_22_capacity_optimization.md."
- alert: TitanMediaHostCPUPressure
expr: |
(1 - avg(rate(node_cpu_seconds_total{mode="idle",node="titan-22"}[5m]))) > 0.85
for: 10m
labels:
severity: warning
service: jellyfin
annotations:
summary: "titan-22 (Jellyfin media-primary host) CPU busy > 85% for 10m"
description: "titan-22 is normally under 30% CPU (p95, 24h baseline in docs/titan_20_21_22_capacity_optimization.md). Sustained pressure this high risks Jellyfin transcode quality and is the signal to check for a misbehaving opportunistic/scavenger workload."
- alert: TitanMediaHostMemoryPressure
expr: |
(node_memory_MemAvailable_bytes{node="titan-22"} / node_memory_MemTotal_bytes{node="titan-22"}) < 0.15
for: 10m
labels:
severity: warning
service: jellyfin
annotations:
summary: "titan-22 (Jellyfin media-primary host) available memory under 15%"
description: "titan-22 is normally ~90% free (24h baseline in docs/titan_20_21_22_capacity_optimization.md). This protects the reservation budget any future opportunistic/scavenger workload on this node must respect."
- alert: JellyfinCPUThrottled
expr: |
(
rate(container_cpu_cfs_throttled_periods_total{namespace="jellyfin",container="jellyfin"}[5m])
/
rate(container_cpu_cfs_periods_total{namespace="jellyfin",container="jellyfin"}[5m])
) > 0.05
for: 5m
labels:
severity: critical
service: jellyfin
annotations:
summary: "Jellyfin container is CPU-throttled against its own limit"
description: "container_cpu_cfs_throttled_periods_total ratio for jellyfin/jellyfin has been over 5% for 5m. Sustained throttling degrades transcode encode speed and can cause playback stutter. Baseline in docs/titan_20_21_22_capacity_optimization.md measured 0% throttling over 24h; any sustained value here is a regression, most likely node-level contention from a co-scheduled workload."
- alert: TitanMediaGPUMemorySaturation
expr: |
(
(
label_replace(label_replace((DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)), "pod", "$1", "Hostname", "(.*)"), "namespace", "monitoring", "", "")
* on(namespace,pod) group_left(node) kube_pod_info{namespace="monitoring"}
)
* on(node) group_left() node_uname_info{node="titan-22"}
) > 0.85
for: 5m
labels:
severity: critical
service: jellyfin
annotations:
summary: "titan-22 GPU VRAM over 85% used"
description: "titan-22's shared RTX 3050 Ti (~8 GiB) is normally under 2% VRAM used (24h peak was 1.27 GiB in the t_26da4c88 audit). This is the concrete failure mode a second shared-GPU tenant (nvidia.com/gpu.shared) would cause for an active Jellyfin hardware transcode; GPU sharing on this node is time-sliced with no VRAM isolation, so this alert is the closest available proxy for contention."
- alert: TitanVoiceHostCPUPressure
expr: |
(1 - avg(rate(node_cpu_seconds_total{mode="idle",node="titan-21"}[5m]))) > 0.90
for: 15m
labels:
severity: warning
service: hermes-voice
annotations:
summary: "titan-21 (Hermes STT/TTS host) CPU busy > 90% for 15m"
description: "titan-21 already runs with 93% of its CPU requested (docs/titan_20_21_22_capacity_optimization.md); this alert catches sustained real contention that would add latency to Hermes Chat's hands-free voice path (feeds t_60d2deb6)."
---