Now that both the agent image (a68d1c4d, via the kustomize images: override)
and the hux sidecar (build-39) are multi-arch with amd64 leaves, move the worker
onto the amd64 accelerator titan-22:
- Add an OR'd nodeSelectorTerm for amd64 + node-role.kubernetes.io/accelerator +
hostname titan-22, with NO worker=true requirement. Keep the arm64 pi-fleet
term as an OR'd fallback so the worker is never stranded.
- Strong primary preference: hostname=titan-22 at weight 100 (scheduler max),
pi-fleet rpi5 nudge lowered to 50, so hermes actually lives on titan-22.
- Tolerate node-role.kubernetes.io/accelerator=true:NoSchedule (harmless where
absent) and the soft atlas.bstein.dev/media-primary:PreferNoSchedule that
titan-22 currently carries, so the weight-100 preference is not offset and
placement is deterministic.
Completes the titan-22 effort the flip branches missed; the earlier branches
never repointed to multi-arch images, which is why the worker never landed here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The arch-specific CLI stamp assumed the shared node_modules keeps both arches'
codex native deps, but a sibling-arch npm install removes this arch's binary from
the shared volume. So after running on the other arch, the stamp exists yet the
native codex dep is gone -> configure-agent-clients fails -> churn. Gate the
install on the current arch's native codex package being present, so it self-heals.
The hermes-agent installs its CLI toolchain at runtime into the shared
/opt/data/tools Longhorn volume, but every download hardcoded arm64. On
the amd64 node titan-22 that left configure-agent-clients failing with
"Missing optional dependency @openai/codex-linux-x64" and the operator
toolchain fetching arm64 binaries, so the pod churned.
Detect the running node's arch (uname -m; fail closed on anything but
aarch64/x86_64) and resolve every asset per-arch:
- install-agent-tools init script (agent-deployment.yaml): ttyd and
kubectl download the arch-correct asset with the arch-correct sha256
(real ttyd 1.7.7 x86_64 and kubectl v1.33.3 amd64 checksums added; the
arm64 ones kept). The npm CLI stamp is now arch-specific
(.cli-versions-<vers>-${arch}) so a fresh arch re-runs npm install and
pulls its own native optional deps; npm keeps both arches' packages.
- install_agent_tools.sh: flux/helm/kustomize/jq/yq/gh/vault/sops/age/
k9s/terraform/go URLs, tarball subdirs (helm linux-${arch}, gh dir),
and checksums are all arch-resolved with both arches pinned. Stamps
and the Go tree are arch-specific, and an active-arch marker forces a
republish of the single-arch ${bin} binaries when the pod moves
between arches on the shared volume. Single fetch/verify helper kept.
Tests updated to assert the arch-aware form (both arches' Go checksums,
${dl_arch} templating) instead of the arm64-only literal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Applying the titan-22 flip ahead of the full accelerator classification, so
titan-22 still carries its soft media-primary taint. Tolerate it too so the
strong titan-22 preference isn't penalised. Harmless once media-primary is gone.
APPLY ONLY AFTER the multi-arch hermes-agent image is built + validated
(both arch leaves + promoted index). Supersedes the earlier titan-22
flip on feature/hermes-agent-multiarch (dfa50b75), which required
worker=true and tolerated the old media-primary taint.
Rewrites the runtime node affinity so hermes-agent runs on titan-22:
- Adds a second, OR'd nodeSelectorTerm matching amd64 + hostname
titan-22 + node-role.kubernetes.io/accelerator=true. It does NOT
require node-role.kubernetes.io/worker (titan-22 is no longer a
generic worker).
- Keeps the arm64 pi-fleet term untouched as an OR'd fallback so the
worker is never stranded if titan-22 is unavailable.
- Makes titan-22 the STRONG/primary preference: a hostname=titan-22
preference at weight 100 (the scheduler maximum) outranks the pi-fleet
rpi5 nudge, lowered to weight 50, so hermes actually lives on titan-22.
- Tolerates node-role.kubernetes.io/accelerator=true:NoSchedule so it
can consider titan-22; this does not change jellyfin's media-core
priority or preemption.
Updates test_hermes_agent_layout.py to the two-term topology, the
[100, 50] preference weights, the titan-22 primary preference, the
absence of a worker requirement on the titan-22 term, and the toleration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Requiring rpi5 while the historical hostname exclusion list removes the very rpi5
nodes that currently have headroom (titan-04/06) left only loaded rpi5s
(titan-05/07/11), so the agent could not schedule and worker went down. Revert to
rpi5-PREFERRED (soft) so it schedules again; proper rpi5 placement needs the
exclusion list refreshed against current node health/capacity, tracked separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Un-pinning let the scheduler land worker on titan-12 (rpi4). The agent is heavy
enough that an rpi4 risks the /api/status slowness that trips its liveness probe
- the exact flap we are avoiding. Make hardware=rpi5 a hard requirement so it
runs only on rpi5 storage workers (excluding the saturated/known-flaky ones);
the scheduler places it on a roomy rpi5 (titan-05).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Worker (hermes-agent) was hard-pinned to titan-08 (a workaround after an earlier
attempt to place it on the amd64 titan-22 failed on architecture). That single-
node pin is exactly what makes it fragile: a titan-08 blip (as just happened when
the node's Longhorn CSI went down) strands worker, and the Recreate strategy then
deadlocks because the replacement can't schedule on the one tight node.
Restore the intended multi-node design: run on any arm64 storage worker except the
known-bad/weak ones (matching the repo's own affinity test, which was red). Its
Longhorn volumes have data-locality disabled with replicas on titan-15/17/19, so
there is no locality penalty to running on another node; the scheduler now places
it on a roomier Pi (e.g. titan-05) and a node blip simply reschedules it.
Also relax the gateway /api/status liveness probe (timeout 10s->15s,
failureThreshold 3->5) so a transient slowness (e.g. a brief storage hiccup) no
longer trips a kill-and-restart cascade.
Follow-up (separate): multi-arch agent image to enable the amd64 titan-22 target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Beam-2 with a temperature-fallback ladder made the final-model warmup
run all three temperature retries under beam search before the server
bound its port, so /health was refused for ~8 min and STT was down that
whole time on every roll (and hinted at slow per-utterance decodes).
Production now runs the accurate large-v3-turbo model greedily at a
single temperature, keeping the proper-noun priming prompt that fixes
names like Amy/Córdoba - fast startup, fast decodes, accuracy intact.
Beam stays env-tunable (HERMES_STT_FINAL_BEAM_SIZE) for a future pass;
serve-before-warmup is a recommended follow-up so cold start never
blocks readiness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
A transcription that landed while the private Whisper service was
restarting (an image roll) crashed hermes_stt_client.py with a raw
urllib ConnectionRefused traceback that got dumped into the
conversation. The client now retries the request with backoff (up to 5
attempts, ~10s - long enough to ride an STT pod restart) and, on a
persistent outage, exits with one concise line ('speech transcription
unavailable...') instead of a stack trace. Delivered via the coordinator
ConfigMap; the next reconcile picks it up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
