docs(titan): fold in live Jenkins/VM/Wger placement onto titan-22

Brad placed Jenkins, VictoriaMetrics, and Wger onto titan-22 directly
(main commits 271f3e8c3, 0ebab9d41) between this audit's first pass and
PR #50 going up. Revalidate with fresh 3h evidence instead of shipping a
report that describes a since-superseded snapshot: titan-22 headroom is
materially unchanged (22-26% CPU p95, mem 77%+ available) with the three
new residents, Jellyfin shows 0 restarts/throttling throughout, but no
GPU/transcode activity occurred in the window so the transcode-under-load
validation is still outstanding. Also flags VictoriaMetrics now being a
single point of failure for titan-22 observability (hard-pinned, no
fallback), and updates the board-comment record (§7) to match what was
actually posted to t_39cf1905/t_60d2deb6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs/titan_20_21_22_capacity_optimization.md

@@ -1,25 +1,32 @@
 # Titan 20/21/22 capacity and placement optimization
 
-Status: evidence-led audit + guardrails shipped. No workload was relocated
+Status: evidence-led audit + guardrails shipped. **No workload was relocated
-onto titan-22 in this change. This document is the durable record for Kanban
+onto titan-22 by this card/PR.** Between this audit's first pass and this
-card `t_26da4c88`.
+revision, Brad independently placed three workloads (Jenkins, VictoriaMetrics,
+Wger) onto titan-22 via direct commits to `main` (outside this card's PR),
+implementing almost exactly the Option B pattern this document recommends.
+§4a documents that live change and the fresh evidence validating it. This
+document is the durable record for Kanban card `t_26da4c88`.
 
-Evidence collected 2026-08-23 13:00-13:30 UTC via:
+Evidence collected in two passes:
-- `kubectl` as `system:serviceaccount:hermes:hermes-agent` (per-namespace
+- **First pass** 2026-08-23 13:00-13:30 UTC, at `origin/main@a019ecd55`,
-  `get/list/watch` on pods/deployments/statefulsets/daemonsets; no mutate
+  covering a trailing-24h VictoriaMetrics window ending ~13:15Z (the longest
-  verbs at all — confirmed with `kubectl auth can-i --list`).
+  valid window since titan-22's return to service — node manifests show the
-- VictoriaMetrics, proxied read-only through Grafana's public datasource
+  current `jellyfin` restore commit reconciling from 2026-08-23T01:11Z).
-  proxy (`https://metrics.bstein.dev/api/datasources/proxy/uid/atlas-vm/...`),
+- **Revalidation pass** 2026-08-23 ~14:40-14:45 UTC, after `git fetch
-  covering the trailing 24h window ending ~2026-08-23T13:15Z. This is the
+  --no-tags origin main` showed this branch was 4 commits behind a
-  longest valid window since titan-22's return to service (node manifests
+  fast-moving `main` (`origin/main@42795f3d6`); the branch was rebased onto
-  show the current `jellyfin` restore commit reconciling from
+  it (clean, no file overlap with the audit's own commit). This pass queried
-  2026-08-23T01:11Z, i.e. the full 24h window is post-restore).
+  VictoriaMetrics directly (`victoria-metrics-single-server.monitoring.svc.cluster.local:8428`,
-- Git history (`services/**`, `infrastructure/**`) at `origin/main@a019ecd55`
+  in-cluster) for current + trailing-3h data, since 3 of the 4 new commits
-  (this branch was fast-forwarded from a stale base to pick up the
+  changed placement for Jenkins/VictoriaMetrics/Wger only ~1-3h before this
-  in-flight titan-22 restoration commits before any analysis below).
+  pass — see §4a.
 
-No pods, deployments, taints, or labels were changed live. No `kubectl
+Both passes used `kubectl` as `system:serviceaccount:hermes:hermes-agent`
-port-forward`/exec/mutate was available or attempted.
+(per-namespace `get/list/watch` on pods/deployments/statefulsets/daemonsets;
+no mutate verbs at all — confirmed with `kubectl auth can-i --list`, both
+passes). No pods, deployments, taints, or labels were changed live by this
+card. No `kubectl port-forward`/exec/mutate was available or attempted.
 
 ## 1. Node inventory (live)
 
@@ -54,6 +61,9 @@ usually has room. This matters for the Jenkins finding in §5.
 | titan-21 | `hermes-stt` (Whisper small) | default (0) | 2 / 6 | 4Gi / 10Gi | `gpu.shared: 1` |
 | titan-21 | `hermes-tts` (Piper, CPU-only) | default (0) | 1 / 4 | 512Mi / 2Gi | none |
 | titan-22 | `jellyfin` | **media-core (400)** | 2 / 8 | 2Gi / 8Gi | `gpu.shared: 1` |
+| titan-22 | `jenkins` (added live, see §4a) | **scavenger (-10)** | 0.1 / 1.5 | 1Gi / 3Gi | none |
+| titan-22 | `victoria-metrics-single-server` (added live, see §4a) | **scavenger (-10)** | 0.5 / 2 | 2Gi / 4Gi | none |
+| titan-22 | `wger` (added live, see §4a) | **scavenger (-10)** | 0.25 / 1.2 | 0.6Gi / 2.25Gi | none |
 | titan-22 | DaemonSets (dcgm-exporter, node-exporter, nvidia-process-exporter, fluent-bit, metis-sentinel-amd64, node-nofile, node-image-sweeper, titan-22-link-keeper) | default | ~0.3 total | ~0.9Gi total | none |
 
 `jellyfin`'s live pod spec already carries `priorityClassName: media-core`,
@@ -196,42 +206,112 @@ cannot meet the card's own bar ("reject it if it cannot protect active
 Jellyfin transcodes"). **Not implemented.**
 
 **D. Broaden Jenkins's `hardware=rpi5` affinity to use titan-22.**
-**Not implemented; explicitly deferred to `t_39cf1905`.** Two independent
+**Superseded by a live change Brad made directly — see §4a.** This audit's
-findings support this:
+first pass (13:00-13:30Z) found Jenkins still rpi5-pinned and recommended
+deferring this decision to `t_39cf1905` as a manual product call, for the
+reasons below (kept for the record — they were the right concerns to raise,
+even though Brad has since made the call):
 - `t_39cf1905`'s own read-only diagnosis (2026-08-23 12:53-13:19Z, its board
-  comments) already root-caused the live Jenkins `Pending` state as a
+  comments) root-caused the then-live Jenkins `Pending` state as a
   **self-inflicted ConfigMap-hash rollout deadlock** on the existing 3-node
-  rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and already
+  rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and shipped a
-  shipped a fix as PR #49. Broadening Jenkins to titan-22 would not have
+  fix as PR #49. Broadening Jenkins to titan-22 would not have fixed that
-  fixed the actual problem.
+  specific problem on its own.
-- Even setting that aside, moving Jenkins itself onto titan-22 is a poor fit
+- Moving Jenkins onto titan-22 is a nontrivial fit for "opportunistic":
-  for "opportunistic": Jenkins builds (especially Docker-in-Docker image
+  Jenkins builds (especially Docker-in-Docker image builds) are bursty and
-  builds) are bursty and CPU-hungry by design — the opposite of the
+  CPU-hungry by design. It also has architecture (multi-arch image
-  tightly-bounded, preemptible workload profile Option B requires next to an
-  active media host. It also has architecture (needs multi-arch image
   verification for `jenkins/jenkins:2.528.3-jdk21` + plugin CLI on amd64),
-  PVC-locality (three RWO PVCs currently only ever scheduled on the rpi5
+  PVC-locality (three RWO PVCs previously only ever scheduled on the rpi5
-  pool), and recovery-isolation implications (the `NotIn [titan-13,15,17,19]`
+  pool), and recovery-isolation implications (the previous
-  rule exists to keep Jenkins off storage-backbone nodes during Longhorn
+  `NotIn [titan-13,15,17,19]` rule kept Jenkins off storage-backbone nodes
-  rebuilds — titan-22 is also a `longhorn-host`) that need their own
+  during Longhorn rebuilds — titan-22 is also a `longhorn-host`).
-  evidence pass. This is a manual product decision for Brad via
+
-  `t_39cf1905`, not something this card should change. See §7 for the board
+### 4a. Live update: Brad already implemented Option D (and Option B) directly
-  comment delivered.
+
+Between this audit's first pass and this revision, three direct commits to
+`main` — **not through this card's PR, authored `jenkins <brad@bstein.dev>`**
+— re-pinned Jenkins, VictoriaMetrics, and Wger onto titan-22:
+
+| Commit | Workload | What changed |
+|---|---|---|
+| `271f3e8c3` "ops: spread saturated node workloads" | `jenkins` | `nodeSelector.arch` rpi5→amd64; required affinity `NotIn[titan-13,15,17,19]`+`hardware=rpi5` → required `hostname In [titan-22]`; added `priorityClassName: scavenger` and a `PreferNoSchedule` toleration for the media-primary taint |
+| `271f3e8c3` (same commit) | `victoria-metrics-single-server` | required affinity `hostname NotIn [titan-12,20,21,22,24,...]` → `In [titan-22]`; added `priorityClassName: scavenger` + media-primary toleration; `upgrade.force: true` |
+| `0ebab9d41` "placement(wger): use preemptible titan-22 capacity" | `wger` | same pattern: `NotIn[titan-13,15,17,19]` → `hostname In [titan-22]`, `arch` arm64→amd64, `priorityClassName: scavenger`, media-primary toleration |
+
+This is **exactly the Option B pattern** this document independently arrived
+at (preemptible `scavenger` priority, CPU/RAM only, zero GPU requests) —
+applied to Jenkins as well, which the first pass had flagged as needing a
+separate evidence-based decision (Option D). Brad made that call directly.
+None of these three request `nvidia.com/gpu*` (confirmed live). This is a
+direct-to-`main` change outside the "all durable changes via reviewable
+titan-iac PRs" policy in this card's own body — noted here for the record,
+not something this card can undo or should second-guess; it's the owner's
+prerogative.
+
+**Fresh evidence (14:40-14:45Z revalidation pass, ~1-3h of run time for the
+new placements) shows it is safe so far:**
+
+| Node | 3h CPU p95 | 3h CPU max | Mem available now | Mem available min (3h) |
+|---|---|---|---|---|
+| titan-20 | 88.6% | 90.5% | 2.6% | **1.7%** |
+| titan-21 | 65.1% | 69.3% | 38.9% | 38.5% |
+| titan-22 | 22.1% | 26.2% | 77.4% | 77.5% |
+
+- titan-22 CPU/RAM headroom is essentially unchanged from the first pass
+  despite three new residents — confirms Option B's "~15 CPU / ~27 GiB idle"
+  finding was real spare capacity, not measurement noise.
+- `jellyfin` (titan-22): 0 restarts, 0 CPU-throttled periods, `Ready`,
+  no pod events, over the full window these three workloads have been
+  co-resident.
+- `jenkins` (titan-22): CPU-throttled up to 30% of periods at its own 1.5-CPU
+  *limit* — this is the CFS quota doing its job (Jenkins capped at its own
+  ceiling), not evidence of node-level contention; `wger` and
+  `victoria-metrics-single-server` show negligible throttling (<1%).
+- titan-22 GPU (DCGM): 0% SM/encoder/decoder utilization and ~1 MiB VRAM used
+  for the entire 3h window — **no active transcode occurred during this
+  co-residency window**, so this is not yet a live proof that Jellyfin
+  survives a concurrent hardware transcode with these three neighbors
+  present. That validation (§8) is still outstanding and still requires
+  either a real playback session or synthetic load-gen, neither of which
+  this read-only session can trigger.
+- titan-20/21 numbers reconfirm the first pass and got *worse*, not better:
+  titan-20 is now down to 1.7% memory available at its 3h floor (vs. "227
+  MiB / ~1.5%" at its 24h floor in the first pass) — Option A (leave
+  titan-20/21 alone) remains the only safe call for those two nodes.
+
+**New risk this live change introduces, not present in the first pass's
+Option B analysis:** `victoria-metrics-single-server` is the cluster's
+*only* metrics backend, and it is now hard-pinned (`required` affinity, not
+`preferred`) to the same node as Jellyfin, Jenkins, and Wger. If titan-22
+has a node-level incident, the dashboards and alerts needed to diagnose that
+exact incident go dark at the same time (`vmalert-atlas-availability` and
+`grafana` themselves stay up, on titan-0b/titan-11, but have nothing to
+query). This wasn't evaluated against the card's Jellyfin-protection bar
+because it's an observability-resilience question, not a Jellyfin-quality
+one — flagging it as a follow-up decision for Brad rather than changing it
+unilaterally.
 
 ## 5. Phase C — what shipped
 
-No existing workload was relocated onto titan-22. Per the card's own
+**No workload was relocated onto titan-22 by this card's PR.** Per the
-acceptance criteria ("If no safe redistribution is proven, deliver the
+card's own acceptance criteria ("If no safe redistribution is proven,
-report/dashboard/guardrails only; do not move workloads for theoretical
+deliver the report/dashboard/guardrails only; do not move workloads for
-utilization"), and because this session's `hermes-agent` identity has **zero
+theoretical utilization"), and because this session's `hermes-agent`
-mutate verbs** on the live cluster (confirmed: `kubectl auth can-i --list`
+identity has **zero mutate verbs** on the live cluster (confirmed: `kubectl
-returns only `get/list/watch` across every resource type — there is no way
+auth can-i --list` returns only `get/list/watch` across every resource
-to even trigger a live synthetic transcode to validate a placement change
+type — there is no way to even trigger a live synthetic transcode to
-from this session), moving a real workload's placement without being able to
+validate a placement change from this session), this card does not itself
-observe the result live would violate the card's own validation bar ("New
+move a workload's placement without being able to observe the result live —
-placement demonstrates Jellyfin remains healthy under a representative
+that would violate the card's own validation bar ("New placement
-active transcode while the opportunistic workload runs, **or does not
+demonstrates Jellyfin remains healthy under a representative active
-ship**"). Guardrails only:
+transcode while the opportunistic workload runs, **or does not ship**").
+
+As documented in §4a, Brad independently placed Jenkins, VictoriaMetrics,
+and Wger onto titan-22 via direct `main` commits during this audit — using
+almost exactly the Option B pattern below, before this PR existed to review
+it. This PR does not touch, revert, or re-author that change; it documents
+it (§4a, workload table in §2) and extends the guardrails to cover it.
+Guardrails shipped by this PR:
 
 1. **Alert rules** (`services/monitoring/vmalert-atlas-availability.yaml`,
    new `atlas.titan-media-capacity` group): Jellyfin CPU/memory pressure at
@@ -251,23 +331,24 @@ ship**"). Guardrails only:
 
 ### What was deliberately *not* shipped, and why
 
-A `PriorityClass`/quota primitive for Option B is not "new" — `scavenger`
+A `PriorityClass`/quota primitive for Option B was not "new" to invent —
-already exists and is already below `media-core`. What's missing is an
+`scavenger` already existed and was already below `media-core`. This audit's
-actual opportunistic workload to place on titan-22 with it. Every existing
+first pass found no already-portable opportunistic workload to place on
-CPU-only, architecture-portable, non-Jellyfin workload examined during this
+titan-22 with it (Jenkins, Cassandra/Veles backends, `collabora`,
-audit (Jenkins, Cassandra/Veles backends, `collabora`, CI/quality-gate jobs)
+CI/quality-gate jobs all had their own node-pool pinning or an in-flight
-either has its own node-pool pinning for reasons outside this card's
+incident) and recommended a follow-up card rather than manufacturing a
-scope, or — like Jenkins — has an incident already being independently
+placement. Brad then placed three workloads (Jenkins, VictoriaMetrics,
-worked. Manufacturing a placement change for a workload not already flagged
+Wger) directly — see §4a — which is the concrete instance of that follow-up,
-as portable would be exactly the "blind migration" and "theoretical
+done outside this card. What is still genuinely outstanding, and still
-utilization" move the card prohibits. **The safe next step is a follow-up
+cannot be done from this read-only session, is the **live
-card that names one specific, already-portable batch/CI workload and
+transcode-under-load validation** (§8): none of titan-22's new residents'
-proves it end-to-end (including the live transcode-under-load validation
+~1-3h of co-residency so far has overlapped with an active hardware
-this session cannot perform), using the reservation budget below.**
+transcode.
 
-### Reservation budget for that follow-up (documented contract, not enforced by a new object)
+### Reservation budget (documented contract, not enforced by a new object)
 
-Any future opportunistic workload placed on titan-22 must:
+Any opportunistic workload placed on titan-22 — including the three already
+live (§4a), which all comply — must:
 - use `priorityClassName: scavenger` (preemptible below Jellyfin's
   `media-core`);
 - request **zero** `nvidia.com/gpu*` resources (Option C is rejected — see
@@ -305,27 +386,41 @@ scraped in this cluster (`node_exporter`, `DCGM`, cAdvisor `container_cpu_cfs_*`
 
 ## 7. Board comments delivered
 
-- `t_39cf1905` (Jenkins/WebUI release): told not to broaden Jenkins onto
+- `t_39cf1905` (Jenkins/WebUI release): told that Jenkins is **already**
-  titan-22 as a workaround for the Pending state — their own root cause
+  running on titan-22 as of a direct main commit (`271f3e8c3`, ~13:37Z,
-  (ConfigMap-hash rollout deadlock on the rpi5 pool) is unrelated to titan-22
+  authored `jenkins <brad@bstein.dev>`, not through this card), that it is
-  capacity, and titan-22 is a poor architectural fit for bursty CI/DinD
+  CPU-only/`scavenger`-priority (correctly preemptible below Jellyfin's
-  builds next to a media-primary host with no enforceable GPU isolation.
+  `media-core`) and healthy so far (0 restarts, self-limited throttling only,
+  titan-22 still at 22-26% p95 CPU / 77%+ mem available with it resident) —
+  so their Part B (trigger the WebUI image builder now that Jenkins is
+  healthy) should be unblocked from the node-placement side; and warned that
+  no active hardware transcode has yet coincided with Jenkins's presence, so
+  Jellyfin-under-load-plus-Jenkins-build is still unproven, not proven-safe.
 - `t_60d2deb6` (Chat latency instrumentation): told that titan-20 (LLM
-  fallback/classifier) is CPU/RAM-saturated (81% CPU requested, 98.3% memory
+  fallback/classifier) is CPU/RAM-saturated (88.6% CPU p95 over the last 3h,
-  used, 227 MiB free at worst) and titan-21 (STT/TTS) is CPU-committed at
+  memory available down to **1.7%** at its floor) and titan-21 (STT/TTS) is
-  93% though usage is lighter (50-60%) — so any voice-path latency variance
+  at 65% CPU p95 with 38-39% memory available — so any voice-path latency
-  they measure should be checked against titan-20/21 node pressure before
+  variance they measure should be checked against titan-20/21 node pressure
-  being attributed to application code, and neither node has spare capacity
+  before being attributed to application code, and neither node has spare
-  for a node-level latency mitigation; also reminded that titan-22 is
+  capacity for a node-level latency mitigation; also that titan-22 is
   amd64/no-arm64-GPU and therefore not a relevant relocation target for any
-  Jetson-pinned STT/TTS/classifier component.
+  Jetson-pinned STT/TTS/classifier component, and that titan-22 now also
+  hosts the cluster's only VictoriaMetrics instance (hard-pinned, single
+  point of failure for the dashboards this instrumentation work will build
+  on) as a heads-up, not a blocker.
 
 ## 8. Validation plan (live run pending — this session has no mutate access)
 
 This session cannot apply the alert/dashboard PR, cannot trigger a synthetic
 transcode, and cannot restart or exec into `jellyfin` (RBAC is `get/list/
-watch` only, confirmed). The plan below is what should be executed once the
+watch` only, confirmed). As of this revision, Jenkins/VictoriaMetrics/Wger
-PR in §9 is merged and reconciled:
+have been co-resident on titan-22 for ~1-3h (§4a) with zero Jellyfin impact,
+but zero GPU activity in that window either — so the one validation that
+actually matters (an active hardware transcode overlapping with real
+opportunistic load) still has not happened naturally and still cannot be
+forced from here. The plan below is what should be executed once the PR in
+§9 is merged and reconciled (or opportunistically, the next time Jellyfin is
+in active use):
 
 1. Confirm the two new GPU dashboard panels and the six new alerts render
    (`Atlas GPU` dashboard; `vmalert-atlas-availability` targets in Grafana