docs(titan): fold in live Jenkins/VM/Wger placement onto titan-22
Brad placed Jenkins, VictoriaMetrics, and Wger onto titan-22 directly (main commits 271f3e8c3, 0ebab9d41) between this audit's first pass and PR #50 going up. Revalidate with fresh 3h evidence instead of shipping a report that describes a since-superseded snapshot: titan-22 headroom is materially unchanged (22-26% CPU p95, mem 77%+ available) with the three new residents, Jellyfin shows 0 restarts/throttling throughout, but no GPU/transcode activity occurred in the window so the transcode-under-load validation is still outstanding. Also flags VictoriaMetrics now being a single point of failure for titan-22 observability (hard-pinned, no fallback), and updates the board-comment record (§7) to match what was actually posted to t_39cf1905/t_60d2deb6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
parent
2185a90825
commit
4f48a1d3de
@ -1,25 +1,32 @@
|
||||
# Titan 20/21/22 capacity and placement optimization
|
||||
|
||||
Status: evidence-led audit + guardrails shipped. No workload was relocated
|
||||
onto titan-22 in this change. This document is the durable record for Kanban
|
||||
card `t_26da4c88`.
|
||||
Status: evidence-led audit + guardrails shipped. **No workload was relocated
|
||||
onto titan-22 by this card/PR.** Between this audit's first pass and this
|
||||
revision, Brad independently placed three workloads (Jenkins, VictoriaMetrics,
|
||||
Wger) onto titan-22 via direct commits to `main` (outside this card's PR),
|
||||
implementing almost exactly the Option B pattern this document recommends.
|
||||
§4a documents that live change and the fresh evidence validating it. This
|
||||
document is the durable record for Kanban card `t_26da4c88`.
|
||||
|
||||
Evidence collected 2026-08-23 13:00-13:30 UTC via:
|
||||
- `kubectl` as `system:serviceaccount:hermes:hermes-agent` (per-namespace
|
||||
`get/list/watch` on pods/deployments/statefulsets/daemonsets; no mutate
|
||||
verbs at all — confirmed with `kubectl auth can-i --list`).
|
||||
- VictoriaMetrics, proxied read-only through Grafana's public datasource
|
||||
proxy (`https://metrics.bstein.dev/api/datasources/proxy/uid/atlas-vm/...`),
|
||||
covering the trailing 24h window ending ~2026-08-23T13:15Z. This is the
|
||||
longest valid window since titan-22's return to service (node manifests
|
||||
show the current `jellyfin` restore commit reconciling from
|
||||
2026-08-23T01:11Z, i.e. the full 24h window is post-restore).
|
||||
- Git history (`services/**`, `infrastructure/**`) at `origin/main@a019ecd55`
|
||||
(this branch was fast-forwarded from a stale base to pick up the
|
||||
in-flight titan-22 restoration commits before any analysis below).
|
||||
Evidence collected in two passes:
|
||||
- **First pass** 2026-08-23 13:00-13:30 UTC, at `origin/main@a019ecd55`,
|
||||
covering a trailing-24h VictoriaMetrics window ending ~13:15Z (the longest
|
||||
valid window since titan-22's return to service — node manifests show the
|
||||
current `jellyfin` restore commit reconciling from 2026-08-23T01:11Z).
|
||||
- **Revalidation pass** 2026-08-23 ~14:40-14:45 UTC, after `git fetch
|
||||
--no-tags origin main` showed this branch was 4 commits behind a
|
||||
fast-moving `main` (`origin/main@42795f3d6`); the branch was rebased onto
|
||||
it (clean, no file overlap with the audit's own commit). This pass queried
|
||||
VictoriaMetrics directly (`victoria-metrics-single-server.monitoring.svc.cluster.local:8428`,
|
||||
in-cluster) for current + trailing-3h data, since 3 of the 4 new commits
|
||||
changed placement for Jenkins/VictoriaMetrics/Wger only ~1-3h before this
|
||||
pass — see §4a.
|
||||
|
||||
No pods, deployments, taints, or labels were changed live. No `kubectl
|
||||
port-forward`/exec/mutate was available or attempted.
|
||||
Both passes used `kubectl` as `system:serviceaccount:hermes:hermes-agent`
|
||||
(per-namespace `get/list/watch` on pods/deployments/statefulsets/daemonsets;
|
||||
no mutate verbs at all — confirmed with `kubectl auth can-i --list`, both
|
||||
passes). No pods, deployments, taints, or labels were changed live by this
|
||||
card. No `kubectl port-forward`/exec/mutate was available or attempted.
|
||||
|
||||
## 1. Node inventory (live)
|
||||
|
||||
@ -54,6 +61,9 @@ usually has room. This matters for the Jenkins finding in §5.
|
||||
| titan-21 | `hermes-stt` (Whisper small) | default (0) | 2 / 6 | 4Gi / 10Gi | `gpu.shared: 1` |
|
||||
| titan-21 | `hermes-tts` (Piper, CPU-only) | default (0) | 1 / 4 | 512Mi / 2Gi | none |
|
||||
| titan-22 | `jellyfin` | **media-core (400)** | 2 / 8 | 2Gi / 8Gi | `gpu.shared: 1` |
|
||||
| titan-22 | `jenkins` (added live, see §4a) | **scavenger (-10)** | 0.1 / 1.5 | 1Gi / 3Gi | none |
|
||||
| titan-22 | `victoria-metrics-single-server` (added live, see §4a) | **scavenger (-10)** | 0.5 / 2 | 2Gi / 4Gi | none |
|
||||
| titan-22 | `wger` (added live, see §4a) | **scavenger (-10)** | 0.25 / 1.2 | 0.6Gi / 2.25Gi | none |
|
||||
| titan-22 | DaemonSets (dcgm-exporter, node-exporter, nvidia-process-exporter, fluent-bit, metis-sentinel-amd64, node-nofile, node-image-sweeper, titan-22-link-keeper) | default | ~0.3 total | ~0.9Gi total | none |
|
||||
|
||||
`jellyfin`'s live pod spec already carries `priorityClassName: media-core`,
|
||||
@ -196,42 +206,112 @@ cannot meet the card's own bar ("reject it if it cannot protect active
|
||||
Jellyfin transcodes"). **Not implemented.**
|
||||
|
||||
**D. Broaden Jenkins's `hardware=rpi5` affinity to use titan-22.**
|
||||
**Not implemented; explicitly deferred to `t_39cf1905`.** Two independent
|
||||
findings support this:
|
||||
**Superseded by a live change Brad made directly — see §4a.** This audit's
|
||||
first pass (13:00-13:30Z) found Jenkins still rpi5-pinned and recommended
|
||||
deferring this decision to `t_39cf1905` as a manual product call, for the
|
||||
reasons below (kept for the record — they were the right concerns to raise,
|
||||
even though Brad has since made the call):
|
||||
- `t_39cf1905`'s own read-only diagnosis (2026-08-23 12:53-13:19Z, its board
|
||||
comments) already root-caused the live Jenkins `Pending` state as a
|
||||
comments) root-caused the then-live Jenkins `Pending` state as a
|
||||
**self-inflicted ConfigMap-hash rollout deadlock** on the existing 3-node
|
||||
rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and already
|
||||
shipped a fix as PR #49. Broadening Jenkins to titan-22 would not have
|
||||
fixed the actual problem.
|
||||
- Even setting that aside, moving Jenkins itself onto titan-22 is a poor fit
|
||||
for "opportunistic": Jenkins builds (especially Docker-in-Docker image
|
||||
builds) are bursty and CPU-hungry by design — the opposite of the
|
||||
tightly-bounded, preemptible workload profile Option B requires next to an
|
||||
active media host. It also has architecture (needs multi-arch image
|
||||
rpi5 pool (titan-07/08/11), unrelated to titan-22 capacity, and shipped a
|
||||
fix as PR #49. Broadening Jenkins to titan-22 would not have fixed that
|
||||
specific problem on its own.
|
||||
- Moving Jenkins onto titan-22 is a nontrivial fit for "opportunistic":
|
||||
Jenkins builds (especially Docker-in-Docker image builds) are bursty and
|
||||
CPU-hungry by design. It also has architecture (multi-arch image
|
||||
verification for `jenkins/jenkins:2.528.3-jdk21` + plugin CLI on amd64),
|
||||
PVC-locality (three RWO PVCs currently only ever scheduled on the rpi5
|
||||
pool), and recovery-isolation implications (the `NotIn [titan-13,15,17,19]`
|
||||
rule exists to keep Jenkins off storage-backbone nodes during Longhorn
|
||||
rebuilds — titan-22 is also a `longhorn-host`) that need their own
|
||||
evidence pass. This is a manual product decision for Brad via
|
||||
`t_39cf1905`, not something this card should change. See §7 for the board
|
||||
comment delivered.
|
||||
PVC-locality (three RWO PVCs previously only ever scheduled on the rpi5
|
||||
pool), and recovery-isolation implications (the previous
|
||||
`NotIn [titan-13,15,17,19]` rule kept Jenkins off storage-backbone nodes
|
||||
during Longhorn rebuilds — titan-22 is also a `longhorn-host`).
|
||||
|
||||
### 4a. Live update: Brad already implemented Option D (and Option B) directly
|
||||
|
||||
Between this audit's first pass and this revision, three direct commits to
|
||||
`main` — **not through this card's PR, authored `jenkins <brad@bstein.dev>`**
|
||||
— re-pinned Jenkins, VictoriaMetrics, and Wger onto titan-22:
|
||||
|
||||
| Commit | Workload | What changed |
|
||||
|---|---|---|
|
||||
| `271f3e8c3` "ops: spread saturated node workloads" | `jenkins` | `nodeSelector.arch` rpi5→amd64; required affinity `NotIn[titan-13,15,17,19]`+`hardware=rpi5` → required `hostname In [titan-22]`; added `priorityClassName: scavenger` and a `PreferNoSchedule` toleration for the media-primary taint |
|
||||
| `271f3e8c3` (same commit) | `victoria-metrics-single-server` | required affinity `hostname NotIn [titan-12,20,21,22,24,...]` → `In [titan-22]`; added `priorityClassName: scavenger` + media-primary toleration; `upgrade.force: true` |
|
||||
| `0ebab9d41` "placement(wger): use preemptible titan-22 capacity" | `wger` | same pattern: `NotIn[titan-13,15,17,19]` → `hostname In [titan-22]`, `arch` arm64→amd64, `priorityClassName: scavenger`, media-primary toleration |
|
||||
|
||||
This is **exactly the Option B pattern** this document independently arrived
|
||||
at (preemptible `scavenger` priority, CPU/RAM only, zero GPU requests) —
|
||||
applied to Jenkins as well, which the first pass had flagged as needing a
|
||||
separate evidence-based decision (Option D). Brad made that call directly.
|
||||
None of these three request `nvidia.com/gpu*` (confirmed live). This is a
|
||||
direct-to-`main` change outside the "all durable changes via reviewable
|
||||
titan-iac PRs" policy in this card's own body — noted here for the record,
|
||||
not something this card can undo or should second-guess; it's the owner's
|
||||
prerogative.
|
||||
|
||||
**Fresh evidence (14:40-14:45Z revalidation pass, ~1-3h of run time for the
|
||||
new placements) shows it is safe so far:**
|
||||
|
||||
| Node | 3h CPU p95 | 3h CPU max | Mem available now | Mem available min (3h) |
|
||||
|---|---|---|---|---|
|
||||
| titan-20 | 88.6% | 90.5% | 2.6% | **1.7%** |
|
||||
| titan-21 | 65.1% | 69.3% | 38.9% | 38.5% |
|
||||
| titan-22 | 22.1% | 26.2% | 77.4% | 77.5% |
|
||||
|
||||
- titan-22 CPU/RAM headroom is essentially unchanged from the first pass
|
||||
despite three new residents — confirms Option B's "~15 CPU / ~27 GiB idle"
|
||||
finding was real spare capacity, not measurement noise.
|
||||
- `jellyfin` (titan-22): 0 restarts, 0 CPU-throttled periods, `Ready`,
|
||||
no pod events, over the full window these three workloads have been
|
||||
co-resident.
|
||||
- `jenkins` (titan-22): CPU-throttled up to 30% of periods at its own 1.5-CPU
|
||||
*limit* — this is the CFS quota doing its job (Jenkins capped at its own
|
||||
ceiling), not evidence of node-level contention; `wger` and
|
||||
`victoria-metrics-single-server` show negligible throttling (<1%).
|
||||
- titan-22 GPU (DCGM): 0% SM/encoder/decoder utilization and ~1 MiB VRAM used
|
||||
for the entire 3h window — **no active transcode occurred during this
|
||||
co-residency window**, so this is not yet a live proof that Jellyfin
|
||||
survives a concurrent hardware transcode with these three neighbors
|
||||
present. That validation (§8) is still outstanding and still requires
|
||||
either a real playback session or synthetic load-gen, neither of which
|
||||
this read-only session can trigger.
|
||||
- titan-20/21 numbers reconfirm the first pass and got *worse*, not better:
|
||||
titan-20 is now down to 1.7% memory available at its 3h floor (vs. "227
|
||||
MiB / ~1.5%" at its 24h floor in the first pass) — Option A (leave
|
||||
titan-20/21 alone) remains the only safe call for those two nodes.
|
||||
|
||||
**New risk this live change introduces, not present in the first pass's
|
||||
Option B analysis:** `victoria-metrics-single-server` is the cluster's
|
||||
*only* metrics backend, and it is now hard-pinned (`required` affinity, not
|
||||
`preferred`) to the same node as Jellyfin, Jenkins, and Wger. If titan-22
|
||||
has a node-level incident, the dashboards and alerts needed to diagnose that
|
||||
exact incident go dark at the same time (`vmalert-atlas-availability` and
|
||||
`grafana` themselves stay up, on titan-0b/titan-11, but have nothing to
|
||||
query). This wasn't evaluated against the card's Jellyfin-protection bar
|
||||
because it's an observability-resilience question, not a Jellyfin-quality
|
||||
one — flagging it as a follow-up decision for Brad rather than changing it
|
||||
unilaterally.
|
||||
|
||||
## 5. Phase C — what shipped
|
||||
|
||||
No existing workload was relocated onto titan-22. Per the card's own
|
||||
acceptance criteria ("If no safe redistribution is proven, deliver the
|
||||
report/dashboard/guardrails only; do not move workloads for theoretical
|
||||
utilization"), and because this session's `hermes-agent` identity has **zero
|
||||
mutate verbs** on the live cluster (confirmed: `kubectl auth can-i --list`
|
||||
returns only `get/list/watch` across every resource type — there is no way
|
||||
to even trigger a live synthetic transcode to validate a placement change
|
||||
from this session), moving a real workload's placement without being able to
|
||||
observe the result live would violate the card's own validation bar ("New
|
||||
placement demonstrates Jellyfin remains healthy under a representative
|
||||
active transcode while the opportunistic workload runs, **or does not
|
||||
ship**"). Guardrails only:
|
||||
**No workload was relocated onto titan-22 by this card's PR.** Per the
|
||||
card's own acceptance criteria ("If no safe redistribution is proven,
|
||||
deliver the report/dashboard/guardrails only; do not move workloads for
|
||||
theoretical utilization"), and because this session's `hermes-agent`
|
||||
identity has **zero mutate verbs** on the live cluster (confirmed: `kubectl
|
||||
auth can-i --list` returns only `get/list/watch` across every resource
|
||||
type — there is no way to even trigger a live synthetic transcode to
|
||||
validate a placement change from this session), this card does not itself
|
||||
move a workload's placement without being able to observe the result live —
|
||||
that would violate the card's own validation bar ("New placement
|
||||
demonstrates Jellyfin remains healthy under a representative active
|
||||
transcode while the opportunistic workload runs, **or does not ship**").
|
||||
|
||||
As documented in §4a, Brad independently placed Jenkins, VictoriaMetrics,
|
||||
and Wger onto titan-22 via direct `main` commits during this audit — using
|
||||
almost exactly the Option B pattern below, before this PR existed to review
|
||||
it. This PR does not touch, revert, or re-author that change; it documents
|
||||
it (§4a, workload table in §2) and extends the guardrails to cover it.
|
||||
Guardrails shipped by this PR:
|
||||
|
||||
1. **Alert rules** (`services/monitoring/vmalert-atlas-availability.yaml`,
|
||||
new `atlas.titan-media-capacity` group): Jellyfin CPU/memory pressure at
|
||||
@ -251,23 +331,24 @@ ship**"). Guardrails only:
|
||||
|
||||
### What was deliberately *not* shipped, and why
|
||||
|
||||
A `PriorityClass`/quota primitive for Option B is not "new" — `scavenger`
|
||||
already exists and is already below `media-core`. What's missing is an
|
||||
actual opportunistic workload to place on titan-22 with it. Every existing
|
||||
CPU-only, architecture-portable, non-Jellyfin workload examined during this
|
||||
audit (Jenkins, Cassandra/Veles backends, `collabora`, CI/quality-gate jobs)
|
||||
either has its own node-pool pinning for reasons outside this card's
|
||||
scope, or — like Jenkins — has an incident already being independently
|
||||
worked. Manufacturing a placement change for a workload not already flagged
|
||||
as portable would be exactly the "blind migration" and "theoretical
|
||||
utilization" move the card prohibits. **The safe next step is a follow-up
|
||||
card that names one specific, already-portable batch/CI workload and
|
||||
proves it end-to-end (including the live transcode-under-load validation
|
||||
this session cannot perform), using the reservation budget below.**
|
||||
A `PriorityClass`/quota primitive for Option B was not "new" to invent —
|
||||
`scavenger` already existed and was already below `media-core`. This audit's
|
||||
first pass found no already-portable opportunistic workload to place on
|
||||
titan-22 with it (Jenkins, Cassandra/Veles backends, `collabora`,
|
||||
CI/quality-gate jobs all had their own node-pool pinning or an in-flight
|
||||
incident) and recommended a follow-up card rather than manufacturing a
|
||||
placement. Brad then placed three workloads (Jenkins, VictoriaMetrics,
|
||||
Wger) directly — see §4a — which is the concrete instance of that follow-up,
|
||||
done outside this card. What is still genuinely outstanding, and still
|
||||
cannot be done from this read-only session, is the **live
|
||||
transcode-under-load validation** (§8): none of titan-22's new residents'
|
||||
~1-3h of co-residency so far has overlapped with an active hardware
|
||||
transcode.
|
||||
|
||||
### Reservation budget for that follow-up (documented contract, not enforced by a new object)
|
||||
### Reservation budget (documented contract, not enforced by a new object)
|
||||
|
||||
Any future opportunistic workload placed on titan-22 must:
|
||||
Any opportunistic workload placed on titan-22 — including the three already
|
||||
live (§4a), which all comply — must:
|
||||
- use `priorityClassName: scavenger` (preemptible below Jellyfin's
|
||||
`media-core`);
|
||||
- request **zero** `nvidia.com/gpu*` resources (Option C is rejected — see
|
||||
@ -305,27 +386,41 @@ scraped in this cluster (`node_exporter`, `DCGM`, cAdvisor `container_cpu_cfs_*`
|
||||
|
||||
## 7. Board comments delivered
|
||||
|
||||
- `t_39cf1905` (Jenkins/WebUI release): told not to broaden Jenkins onto
|
||||
titan-22 as a workaround for the Pending state — their own root cause
|
||||
(ConfigMap-hash rollout deadlock on the rpi5 pool) is unrelated to titan-22
|
||||
capacity, and titan-22 is a poor architectural fit for bursty CI/DinD
|
||||
builds next to a media-primary host with no enforceable GPU isolation.
|
||||
- `t_39cf1905` (Jenkins/WebUI release): told that Jenkins is **already**
|
||||
running on titan-22 as of a direct main commit (`271f3e8c3`, ~13:37Z,
|
||||
authored `jenkins <brad@bstein.dev>`, not through this card), that it is
|
||||
CPU-only/`scavenger`-priority (correctly preemptible below Jellyfin's
|
||||
`media-core`) and healthy so far (0 restarts, self-limited throttling only,
|
||||
titan-22 still at 22-26% p95 CPU / 77%+ mem available with it resident) —
|
||||
so their Part B (trigger the WebUI image builder now that Jenkins is
|
||||
healthy) should be unblocked from the node-placement side; and warned that
|
||||
no active hardware transcode has yet coincided with Jenkins's presence, so
|
||||
Jellyfin-under-load-plus-Jenkins-build is still unproven, not proven-safe.
|
||||
- `t_60d2deb6` (Chat latency instrumentation): told that titan-20 (LLM
|
||||
fallback/classifier) is CPU/RAM-saturated (81% CPU requested, 98.3% memory
|
||||
used, 227 MiB free at worst) and titan-21 (STT/TTS) is CPU-committed at
|
||||
93% though usage is lighter (50-60%) — so any voice-path latency variance
|
||||
they measure should be checked against titan-20/21 node pressure before
|
||||
being attributed to application code, and neither node has spare capacity
|
||||
for a node-level latency mitigation; also reminded that titan-22 is
|
||||
fallback/classifier) is CPU/RAM-saturated (88.6% CPU p95 over the last 3h,
|
||||
memory available down to **1.7%** at its floor) and titan-21 (STT/TTS) is
|
||||
at 65% CPU p95 with 38-39% memory available — so any voice-path latency
|
||||
variance they measure should be checked against titan-20/21 node pressure
|
||||
before being attributed to application code, and neither node has spare
|
||||
capacity for a node-level latency mitigation; also that titan-22 is
|
||||
amd64/no-arm64-GPU and therefore not a relevant relocation target for any
|
||||
Jetson-pinned STT/TTS/classifier component.
|
||||
Jetson-pinned STT/TTS/classifier component, and that titan-22 now also
|
||||
hosts the cluster's only VictoriaMetrics instance (hard-pinned, single
|
||||
point of failure for the dashboards this instrumentation work will build
|
||||
on) as a heads-up, not a blocker.
|
||||
|
||||
## 8. Validation plan (live run pending — this session has no mutate access)
|
||||
|
||||
This session cannot apply the alert/dashboard PR, cannot trigger a synthetic
|
||||
transcode, and cannot restart or exec into `jellyfin` (RBAC is `get/list/
|
||||
watch` only, confirmed). The plan below is what should be executed once the
|
||||
PR in §9 is merged and reconciled:
|
||||
watch` only, confirmed). As of this revision, Jenkins/VictoriaMetrics/Wger
|
||||
have been co-resident on titan-22 for ~1-3h (§4a) with zero Jellyfin impact,
|
||||
but zero GPU activity in that window either — so the one validation that
|
||||
actually matters (an active hardware transcode overlapping with real
|
||||
opportunistic load) still has not happened naturally and still cannot be
|
||||
forced from here. The plan below is what should be executed once the PR in
|
||||
§9 is merged and reconciled (or opportunistically, the next time Jellyfin is
|
||||
in active use):
|
||||
|
||||
1. Confirm the two new GPU dashboard panels and the six new alerts render
|
||||
(`Atlas GPU` dashboard; `vmalert-atlas-availability` targets in Grafana
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user