From 4f48a1d3def699d54d245a8a13d214eeb225a956 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Sun, 23 Aug 2026 14:55:34 +0000 Subject: [PATCH] docs(titan): fold in live Jenkins/VM/Wger placement onto titan-22 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brad placed Jenkins, VictoriaMetrics, and Wger onto titan-22 directly (main commits 271f3e8c3, 0ebab9d41) between this audit's first pass and PR #50 going up. Revalidate with fresh 3h evidence instead of shipping a report that describes a since-superseded snapshot: titan-22 headroom is materially unchanged (22-26% CPU p95, mem 77%+ available) with the three new residents, Jellyfin shows 0 restarts/throttling throughout, but no GPU/transcode activity occurred in the window so the transcode-under-load validation is still outstanding. Also flags VictoriaMetrics now being a single point of failure for titan-22 observability (hard-pinned, no fallback), and updates the board-comment record (§7) to match what was actually posted to t_39cf1905/t_60d2deb6. Co-Authored-By: Claude Sonnet 5 --- docs/titan_20_21_22_capacity_optimization.md | 249 +++++++++++++------ 1 file changed, 172 insertions(+), 77 deletions(-) diff --git a/docs/titan_20_21_22_capacity_optimization.md b/docs/titan_20_21_22_capacity_optimization.md index b1b2db79..244136f7 100644 --- a/docs/titan_20_21_22_capacity_optimization.md +++ b/docs/titan_20_21_22_capacity_optimization.md @@ -1,25 +1,32 @@ # Titan 20/21/22 capacity and placement optimization -Status: evidence-led audit + guardrails shipped. No workload was relocated -onto titan-22 in this change. This document is the durable record for Kanban -card `t_26da4c88`. +Status: evidence-led audit + guardrails shipped. **No workload was relocated +onto titan-22 by this card/PR.** Between this audit's first pass and this +revision, Brad independently placed three workloads (Jenkins, VictoriaMetrics, +Wger) onto titan-22 via direct commits to `main` (outside this card's PR), +implementing almost exactly the Option B pattern this document recommends. +§4a documents that live change and the fresh evidence validating it. This +document is the durable record for Kanban card `t_26da4c88`. -Evidence collected 2026-08-23 13:00-13:30 UTC via: -- `kubectl` as `system:serviceaccount:hermes:hermes-agent` (per-namespace - `get/list/watch` on pods/deployments/statefulsets/daemonsets; no mutate - verbs at all — confirmed with `kubectl auth can-i --list`). -- VictoriaMetrics, proxied read-only through Grafana's public datasource - proxy (`https://metrics.bstein.dev/api/datasources/proxy/uid/atlas-vm/...`), - covering the trailing 24h window ending ~2026-08-23T13:15Z. This is the - longest valid window since titan-22's return to service (node manifests - show the current `jellyfin` restore commit reconciling from - 2026-08-23T01:11Z, i.e. the full 24h window is post-restore). -- Git history (`services/**`, `infrastructure/**`) at `origin/main@a019ecd55` - (this branch was fast-forwarded from a stale base to pick up the - in-flight titan-22 restoration commits before any analysis below). +Evidence collected in two passes: +- **First pass** 2026-08-23 13:00-13:30 UTC, at `origin/main@a019ecd55`, + covering a trailing-24h VictoriaMetrics window ending ~13:15Z (the longest + valid window since titan-22's return to service — node manifests show the + current `jellyfin` restore commit reconciling from 2026-08-23T01:11Z). +- **Revalidation pass** 2026-08-23 ~14:40-14:45 UTC, after `git fetch + --no-tags origin main` showed this branch was 4 commits behind a + fast-moving `main` (`origin/main@42795f3d6`); the branch was rebased onto + it (clean, no file overlap with the audit's own commit). This pass queried + VictoriaMetrics directly (`victoria-metrics-single-server.monitoring.svc.cluster.local:8428`, + in-cluster) for current + trailing-3h data, since 3 of the 4 new commits + changed placement for Jenkins/VictoriaMetrics/Wger only ~1-3h before this + pass — see §4a. -No pods, deployments, taints, or labels were changed live. No `kubectl -port-forward`/exec/mutate was available or attempted. +Both passes used `kubectl` as `system:serviceaccount:hermes:hermes-agent` +(per-namespace `get/list/watch` on pods/deployments/statefulsets/daemonsets; +no mutate verbs at all — confirmed with `kubectl auth can-i --list`, both +passes). No pods, deployments, taints, or labels were changed live by this +card. No `kubectl port-forward`/exec/mutate was available or attempted. ## 1. Node inventory (live) @@ -54,6 +61,9 @@ usually has room. This matters for the Jenkins finding in §5. | titan-21 | `hermes-stt` (Whisper small) | default (0) | 2 / 6 | 4Gi / 10Gi | `gpu.shared: 1` | | titan-21 | `hermes-tts` (Piper, CPU-only) | default (0) | 1 / 4 | 512Mi / 2Gi | none | | titan-22 | `jellyfin` | **media-core (400)** | 2 / 8 | 2Gi / 8Gi | `gpu.shared: 1` | +| titan-22 | `jenkins` (added live, see §4a) | **scavenger (-10)** | 0.1 / 1.5 | 1Gi / 3Gi | none | +| titan-22 | `victoria-metrics-single-server` (added live, see §4a) | **scavenger (-10)** | 0.5 / 2 | 2Gi / 4Gi | none | +| titan-22 | `wger` (added live, see §4a) | **scavenger (-10)** | 0.25 / 1.2 | 0.6Gi / 2.25Gi | none | | titan-22 | DaemonSets (dcgm-exporter, node-exporter, nvidia-process-exporter, fluent-bit, metis-sentinel-amd64, node-nofile, node-image-sweeper, titan-22-link-keeper) | default | ~0.3 total | ~0.9Gi total | none | `jellyfin`'s live pod spec already carries `priorityClassName: media-core`, @@ -196,42 +206,112 @@ cannot meet the card's own bar ("reject it if it cannot protect active Jellyfin transcodes"). **Not implemented.** **D. Broaden Jenkins's `hardware=rpi5` affinity to use titan-22.** -**Not implemented; explicitly deferred to `t_39cf1905`.** Two independent -findings support this: +**Superseded by a live change Brad made directly — see §4a.** This audit's +first pass (13:00-13:30Z) found Jenkins still rpi5-pinned and recommended +deferring this decision to `t_39cf1905` as a manual product call, for the +reasons below (kept for the record — they were the right concerns to raise, +even though Brad has since made the call): - `t_39cf1905`'s own read-only diagnosis (2026-08-23 12:53-13:19Z, its board - comments) already root-caused the live Jenkins `Pending` state as a + comments) root-caused the then-live Jenkins `Pending` state as a **self-inflicted ConfigMap-hash rollout deadlock** on the existing 3-node - rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and already - shipped a fix as PR #49. Broadening Jenkins to titan-22 would not have - fixed the actual problem. -- Even setting that aside, moving Jenkins itself onto titan-22 is a poor fit - for "opportunistic": Jenkins builds (especially Docker-in-Docker image - builds) are bursty and CPU-hungry by design — the opposite of the - tightly-bounded, preemptible workload profile Option B requires next to an - active media host. It also has architecture (needs multi-arch image + rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and shipped a + fix as PR #49. Broadening Jenkins to titan-22 would not have fixed that + specific problem on its own. +- Moving Jenkins onto titan-22 is a nontrivial fit for "opportunistic": + Jenkins builds (especially Docker-in-Docker image builds) are bursty and + CPU-hungry by design. It also has architecture (multi-arch image verification for `jenkins/jenkins:2.528.3-jdk21` + plugin CLI on amd64), - PVC-locality (three RWO PVCs currently only ever scheduled on the rpi5 - pool), and recovery-isolation implications (the `NotIn [titan-13,15,17,19]` - rule exists to keep Jenkins off storage-backbone nodes during Longhorn - rebuilds — titan-22 is also a `longhorn-host`) that need their own - evidence pass. This is a manual product decision for Brad via - `t_39cf1905`, not something this card should change. See §7 for the board - comment delivered. + PVC-locality (three RWO PVCs previously only ever scheduled on the rpi5 + pool), and recovery-isolation implications (the previous + `NotIn [titan-13,15,17,19]` rule kept Jenkins off storage-backbone nodes + during Longhorn rebuilds — titan-22 is also a `longhorn-host`). + +### 4a. Live update: Brad already implemented Option D (and Option B) directly + +Between this audit's first pass and this revision, three direct commits to +`main` — **not through this card's PR, authored `jenkins `** +— re-pinned Jenkins, VictoriaMetrics, and Wger onto titan-22: + +| Commit | Workload | What changed | +|---|---|---| +| `271f3e8c3` "ops: spread saturated node workloads" | `jenkins` | `nodeSelector.arch` rpi5→amd64; required affinity `NotIn[titan-13,15,17,19]`+`hardware=rpi5` → required `hostname In [titan-22]`; added `priorityClassName: scavenger` and a `PreferNoSchedule` toleration for the media-primary taint | +| `271f3e8c3` (same commit) | `victoria-metrics-single-server` | required affinity `hostname NotIn [titan-12,20,21,22,24,...]` → `In [titan-22]`; added `priorityClassName: scavenger` + media-primary toleration; `upgrade.force: true` | +| `0ebab9d41` "placement(wger): use preemptible titan-22 capacity" | `wger` | same pattern: `NotIn[titan-13,15,17,19]` → `hostname In [titan-22]`, `arch` arm64→amd64, `priorityClassName: scavenger`, media-primary toleration | + +This is **exactly the Option B pattern** this document independently arrived +at (preemptible `scavenger` priority, CPU/RAM only, zero GPU requests) — +applied to Jenkins as well, which the first pass had flagged as needing a +separate evidence-based decision (Option D). Brad made that call directly. +None of these three request `nvidia.com/gpu*` (confirmed live). This is a +direct-to-`main` change outside the "all durable changes via reviewable +titan-iac PRs" policy in this card's own body — noted here for the record, +not something this card can undo or should second-guess; it's the owner's +prerogative. + +**Fresh evidence (14:40-14:45Z revalidation pass, ~1-3h of run time for the +new placements) shows it is safe so far:** + +| Node | 3h CPU p95 | 3h CPU max | Mem available now | Mem available min (3h) | +|---|---|---|---|---| +| titan-20 | 88.6% | 90.5% | 2.6% | **1.7%** | +| titan-21 | 65.1% | 69.3% | 38.9% | 38.5% | +| titan-22 | 22.1% | 26.2% | 77.4% | 77.5% | + +- titan-22 CPU/RAM headroom is essentially unchanged from the first pass + despite three new residents — confirms Option B's "~15 CPU / ~27 GiB idle" + finding was real spare capacity, not measurement noise. +- `jellyfin` (titan-22): 0 restarts, 0 CPU-throttled periods, `Ready`, + no pod events, over the full window these three workloads have been + co-resident. +- `jenkins` (titan-22): CPU-throttled up to 30% of periods at its own 1.5-CPU + *limit* — this is the CFS quota doing its job (Jenkins capped at its own + ceiling), not evidence of node-level contention; `wger` and + `victoria-metrics-single-server` show negligible throttling (<1%). +- titan-22 GPU (DCGM): 0% SM/encoder/decoder utilization and ~1 MiB VRAM used + for the entire 3h window — **no active transcode occurred during this + co-residency window**, so this is not yet a live proof that Jellyfin + survives a concurrent hardware transcode with these three neighbors + present. That validation (§8) is still outstanding and still requires + either a real playback session or synthetic load-gen, neither of which + this read-only session can trigger. +- titan-20/21 numbers reconfirm the first pass and got *worse*, not better: + titan-20 is now down to 1.7% memory available at its 3h floor (vs. "227 + MiB / ~1.5%" at its 24h floor in the first pass) — Option A (leave + titan-20/21 alone) remains the only safe call for those two nodes. + +**New risk this live change introduces, not present in the first pass's +Option B analysis:** `victoria-metrics-single-server` is the cluster's +*only* metrics backend, and it is now hard-pinned (`required` affinity, not +`preferred`) to the same node as Jellyfin, Jenkins, and Wger. If titan-22 +has a node-level incident, the dashboards and alerts needed to diagnose that +exact incident go dark at the same time (`vmalert-atlas-availability` and +`grafana` themselves stay up, on titan-0b/titan-11, but have nothing to +query). This wasn't evaluated against the card's Jellyfin-protection bar +because it's an observability-resilience question, not a Jellyfin-quality +one — flagging it as a follow-up decision for Brad rather than changing it +unilaterally. ## 5. Phase C — what shipped -No existing workload was relocated onto titan-22. Per the card's own -acceptance criteria ("If no safe redistribution is proven, deliver the -report/dashboard/guardrails only; do not move workloads for theoretical -utilization"), and because this session's `hermes-agent` identity has **zero -mutate verbs** on the live cluster (confirmed: `kubectl auth can-i --list` -returns only `get/list/watch` across every resource type — there is no way -to even trigger a live synthetic transcode to validate a placement change -from this session), moving a real workload's placement without being able to -observe the result live would violate the card's own validation bar ("New -placement demonstrates Jellyfin remains healthy under a representative -active transcode while the opportunistic workload runs, **or does not -ship**"). Guardrails only: +**No workload was relocated onto titan-22 by this card's PR.** Per the +card's own acceptance criteria ("If no safe redistribution is proven, +deliver the report/dashboard/guardrails only; do not move workloads for +theoretical utilization"), and because this session's `hermes-agent` +identity has **zero mutate verbs** on the live cluster (confirmed: `kubectl +auth can-i --list` returns only `get/list/watch` across every resource +type — there is no way to even trigger a live synthetic transcode to +validate a placement change from this session), this card does not itself +move a workload's placement without being able to observe the result live — +that would violate the card's own validation bar ("New placement +demonstrates Jellyfin remains healthy under a representative active +transcode while the opportunistic workload runs, **or does not ship**"). + +As documented in §4a, Brad independently placed Jenkins, VictoriaMetrics, +and Wger onto titan-22 via direct `main` commits during this audit — using +almost exactly the Option B pattern below, before this PR existed to review +it. This PR does not touch, revert, or re-author that change; it documents +it (§4a, workload table in §2) and extends the guardrails to cover it. +Guardrails shipped by this PR: 1. **Alert rules** (`services/monitoring/vmalert-atlas-availability.yaml`, new `atlas.titan-media-capacity` group): Jellyfin CPU/memory pressure at @@ -251,23 +331,24 @@ ship**"). Guardrails only: ### What was deliberately *not* shipped, and why -A `PriorityClass`/quota primitive for Option B is not "new" — `scavenger` -already exists and is already below `media-core`. What's missing is an -actual opportunistic workload to place on titan-22 with it. Every existing -CPU-only, architecture-portable, non-Jellyfin workload examined during this -audit (Jenkins, Cassandra/Veles backends, `collabora`, CI/quality-gate jobs) -either has its own node-pool pinning for reasons outside this card's -scope, or — like Jenkins — has an incident already being independently -worked. Manufacturing a placement change for a workload not already flagged -as portable would be exactly the "blind migration" and "theoretical -utilization" move the card prohibits. **The safe next step is a follow-up -card that names one specific, already-portable batch/CI workload and -proves it end-to-end (including the live transcode-under-load validation -this session cannot perform), using the reservation budget below.** +A `PriorityClass`/quota primitive for Option B was not "new" to invent — +`scavenger` already existed and was already below `media-core`. This audit's +first pass found no already-portable opportunistic workload to place on +titan-22 with it (Jenkins, Cassandra/Veles backends, `collabora`, +CI/quality-gate jobs all had their own node-pool pinning or an in-flight +incident) and recommended a follow-up card rather than manufacturing a +placement. Brad then placed three workloads (Jenkins, VictoriaMetrics, +Wger) directly — see §4a — which is the concrete instance of that follow-up, +done outside this card. What is still genuinely outstanding, and still +cannot be done from this read-only session, is the **live +transcode-under-load validation** (§8): none of titan-22's new residents' +~1-3h of co-residency so far has overlapped with an active hardware +transcode. -### Reservation budget for that follow-up (documented contract, not enforced by a new object) +### Reservation budget (documented contract, not enforced by a new object) -Any future opportunistic workload placed on titan-22 must: +Any opportunistic workload placed on titan-22 — including the three already +live (§4a), which all comply — must: - use `priorityClassName: scavenger` (preemptible below Jellyfin's `media-core`); - request **zero** `nvidia.com/gpu*` resources (Option C is rejected — see @@ -305,27 +386,41 @@ scraped in this cluster (`node_exporter`, `DCGM`, cAdvisor `container_cpu_cfs_*` ## 7. Board comments delivered -- `t_39cf1905` (Jenkins/WebUI release): told not to broaden Jenkins onto - titan-22 as a workaround for the Pending state — their own root cause - (ConfigMap-hash rollout deadlock on the rpi5 pool) is unrelated to titan-22 - capacity, and titan-22 is a poor architectural fit for bursty CI/DinD - builds next to a media-primary host with no enforceable GPU isolation. +- `t_39cf1905` (Jenkins/WebUI release): told that Jenkins is **already** + running on titan-22 as of a direct main commit (`271f3e8c3`, ~13:37Z, + authored `jenkins `, not through this card), that it is + CPU-only/`scavenger`-priority (correctly preemptible below Jellyfin's + `media-core`) and healthy so far (0 restarts, self-limited throttling only, + titan-22 still at 22-26% p95 CPU / 77%+ mem available with it resident) — + so their Part B (trigger the WebUI image builder now that Jenkins is + healthy) should be unblocked from the node-placement side; and warned that + no active hardware transcode has yet coincided with Jenkins's presence, so + Jellyfin-under-load-plus-Jenkins-build is still unproven, not proven-safe. - `t_60d2deb6` (Chat latency instrumentation): told that titan-20 (LLM - fallback/classifier) is CPU/RAM-saturated (81% CPU requested, 98.3% memory - used, 227 MiB free at worst) and titan-21 (STT/TTS) is CPU-committed at - 93% though usage is lighter (50-60%) — so any voice-path latency variance - they measure should be checked against titan-20/21 node pressure before - being attributed to application code, and neither node has spare capacity - for a node-level latency mitigation; also reminded that titan-22 is + fallback/classifier) is CPU/RAM-saturated (88.6% CPU p95 over the last 3h, + memory available down to **1.7%** at its floor) and titan-21 (STT/TTS) is + at 65% CPU p95 with 38-39% memory available — so any voice-path latency + variance they measure should be checked against titan-20/21 node pressure + before being attributed to application code, and neither node has spare + capacity for a node-level latency mitigation; also that titan-22 is amd64/no-arm64-GPU and therefore not a relevant relocation target for any - Jetson-pinned STT/TTS/classifier component. + Jetson-pinned STT/TTS/classifier component, and that titan-22 now also + hosts the cluster's only VictoriaMetrics instance (hard-pinned, single + point of failure for the dashboards this instrumentation work will build + on) as a heads-up, not a blocker. ## 8. Validation plan (live run pending — this session has no mutate access) This session cannot apply the alert/dashboard PR, cannot trigger a synthetic transcode, and cannot restart or exec into `jellyfin` (RBAC is `get/list/ -watch` only, confirmed). The plan below is what should be executed once the -PR in §9 is merged and reconciled: +watch` only, confirmed). As of this revision, Jenkins/VictoriaMetrics/Wger +have been co-resident on titan-22 for ~1-3h (§4a) with zero Jellyfin impact, +but zero GPU activity in that window either — so the one validation that +actually matters (an active hardware transcode overlapping with real +opportunistic load) still has not happened naturally and still cannot be +forced from here. The plan below is what should be executed once the PR in +§9 is merged and reconciled (or opportunistically, the next time Jellyfin is +in active use): 1. Confirm the two new GPU dashboard panels and the six new alerts render (`Atlas GPU` dashboard; `vmalert-atlas-availability` targets in Grafana