Now that both the agent image (a68d1c4d, via the kustomize images: override)
and the hux sidecar (build-39) are multi-arch with amd64 leaves, move the worker
onto the amd64 accelerator titan-22:
- Add an OR'd nodeSelectorTerm for amd64 + node-role.kubernetes.io/accelerator +
hostname titan-22, with NO worker=true requirement. Keep the arm64 pi-fleet
term as an OR'd fallback so the worker is never stranded.
- Strong primary preference: hostname=titan-22 at weight 100 (scheduler max),
pi-fleet rpi5 nudge lowered to 50, so hermes actually lives on titan-22.
- Tolerate node-role.kubernetes.io/accelerator=true:NoSchedule (harmless where
absent) and the soft atlas.bstein.dev/media-primary:PreferNoSchedule that
titan-22 currently carries, so the weight-100 preference is not offset and
placement is deterministic.
Completes the titan-22 effort the flip branches missed; the earlier branches
never repointed to multi-arch images, which is why the worker never landed here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The webui release handoff verified org.opencontainers.image.revision on the
Harbor artifact's own extra_attrs.config.Labels. That works for a single-arch
image, but a multi-arch manifest list has no top-level config, so Harbor reports
the label on each per-arch child. build-38 built + published the index fine, then
failed post-publish with 'Harbor artifact omitted OCI image labels'.
verify_registry_digest now checks the top-level config labels when present
(single-arch, unchanged) and otherwise walks the index references, fetching each
child artifact by digest and asserting its revision label. Mirrors how the agent
image lane already tolerates a multi-arch index, without dropping the supply-chain
label check. Adds multi-arch pass/reject tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
test_gateway_image_honors_ui_model_and_caps_reasoning still pinned the old
arm64-only docker.io base (nousresearch/hermes-agent@sha256:47d4bd4c...). The
agent image moved to the multi-arch mirror base
(harbor-core.harbor.svc.cluster.local/mirror/hermes-agent@sha256:9c841866...)
in 8a710845 for the two-leg build; the test wasn't updated, so it was a latent
red only the webui/quality lane runs. Point the assertion at the current base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Repoints both Dockerfile.hermes-webui FROM bases to in-cluster Harbor mirrors
(webui base OCI index + the a68d1c4d multi-arch hermes-agent manifest list),
adds the suspended webui base-mirror Job, and gives the WebUI image build an
amd64 leg on titan-24 plus a manifest-list combine — so the hux sidecar can
schedule onto the amd64 accelerator node titan-22.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The arch-specific CLI stamp assumed the shared node_modules keeps both arches'
codex native deps, but a sibling-arch npm install removes this arch's binary from
the shared volume. So after running on the other arch, the stamp exists yet the
native codex dep is gone -> configure-agent-clients fails -> churn. Gate the
install on the current arch's native codex package being present, so it self-heals.
Make registry.bstein.dev/bstein/hermes-webui a linux/amd64 + linux/arm64
manifest list so the agent pod's `hux` sidecar (which runs the webui image)
can schedule onto the amd64 node titan-22. Reuses the hermes-agent multi-arch
pattern already on main.
- Dockerfile.hermes-webui: repoint both FROMs to multi-arch, internal sources.
The upstream WebUI base (ghcr sha256:a83a3893..., already a multi-arch OCI
index) is now pulled from the in-cluster Harbor mirror; the agent base moves
from the retired arm64-only leaf (81970563) to the multi-arch agent index
(a68d1c4d). Kaniko selects the matching arch leaf per build node.
- services/harbor/hermes-webui-base-mirror-job.yaml: new suspended, operator-run
skopeo `copy --all` Job mirroring the upstream WebUI base index into Harbor's
`mirror` project (modeled on hermes-agent-base-mirror-job.yaml; reuses the
generic ensure-project helper). Wired into the harbor kustomization.
- Jenkinsfile.hermes-webui-image: arm64 leg (titan-20) + amd64 leg (titan-24,
hostname+arch pin, toleration Exists, resource-capped, own checkout scm) +
Combine multi-arch index stage; per-arch evidence archived alongside the index.
- hermes_multiarch_combine.py: generalize the destination pattern/component to
serve both hermes-agent and hermes-webui (fail-closed to just those two).
- Tests updated to the two-arch topology (two legs, combine, both FROM bases,
the mirror Job, twelve archived evidence files).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The hermes-agent installs its CLI toolchain at runtime into the shared
/opt/data/tools Longhorn volume, but every download hardcoded arm64. On
the amd64 node titan-22 that left configure-agent-clients failing with
"Missing optional dependency @openai/codex-linux-x64" and the operator
toolchain fetching arm64 binaries, so the pod churned.
Detect the running node's arch (uname -m; fail closed on anything but
aarch64/x86_64) and resolve every asset per-arch:
- install-agent-tools init script (agent-deployment.yaml): ttyd and
kubectl download the arch-correct asset with the arch-correct sha256
(real ttyd 1.7.7 x86_64 and kubectl v1.33.3 amd64 checksums added; the
arm64 ones kept). The npm CLI stamp is now arch-specific
(.cli-versions-<vers>-${arch}) so a fresh arch re-runs npm install and
pulls its own native optional deps; npm keeps both arches' packages.
- install_agent_tools.sh: flux/helm/kustomize/jq/yq/gh/vault/sops/age/
k9s/terraform/go URLs, tarball subdirs (helm linux-${arch}, gh dir),
and checksums are all arch-resolved with both arches pinned. Stamps
and the Go tree are arch-specific, and an active-arch marker forces a
republish of the single-arch ${bin} binaries when the pod moves
between arches on the shared volume. Single fetch/verify helper kept.
Tests updated to assert the arch-aware form (both arches' Go checksums,
${dl_arch} templating) instead of the arm64-only literal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Applying the titan-22 flip ahead of the full accelerator classification, so
titan-22 still carries its soft media-primary taint. Tolerate it too so the
strong titan-22 preference isn't penalised. Harmless once media-primary is gone.
APPLY ONLY AFTER the multi-arch hermes-agent image is built + validated
(both arch leaves + promoted index). Supersedes the earlier titan-22
flip on feature/hermes-agent-multiarch (dfa50b75), which required
worker=true and tolerated the old media-primary taint.
Rewrites the runtime node affinity so hermes-agent runs on titan-22:
- Adds a second, OR'd nodeSelectorTerm matching amd64 + hostname
titan-22 + node-role.kubernetes.io/accelerator=true. It does NOT
require node-role.kubernetes.io/worker (titan-22 is no longer a
generic worker).
- Keeps the arm64 pi-fleet term untouched as an OR'd fallback so the
worker is never stranded if titan-22 is unavailable.
- Makes titan-22 the STRONG/primary preference: a hostname=titan-22
preference at weight 100 (the scheduler maximum) outranks the pi-fleet
rpi5 nudge, lowered to weight 50, so hermes actually lives on titan-22.
- Tolerates node-role.kubernetes.io/accelerator=true:NoSchedule so it
can consider titan-22; this does not change jellyfin's media-core
priority or preemption.
Updates test_hermes_agent_layout.py to the two-term topology, the
[100, 50] preference weights, the titan-22 primary preference, the
absence of a worker requirement on the titan-22 term, and the toleration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
harbor-core internally advertises an HTTPS token realm, so skopeo could not push
over HTTP. Push to registry.bstein.dev (valid cert, the path kaniko already
uses); the image lands in the same 'mirror' project and stays internally pullable.
skopeo derived the token realm as HTTPS and got 'HTTP response to HTTPS client'.
A registries.conf with insecure=true for harbor-core:80 makes the registry AND
its token request use HTTP.
harbor-core serves http on :80 only; the skopeo dest omitted the port so it
dialed :443 and timed out. Pin the dest to :80 (with --dest-tls-verify=false).
The Job's ensure-project init container reads /vault/secrets/harbor-admin-password,
but Vault appended its init container AFTER ensure-project, so the secret file
was absent and the init failed. Force vault-agent-init to run first.
The hermes-agent-image pipeline failed intermittently on external network:
Kaniko's docker.io fallback for the base image is IPv6-broken from build
pods, and the "Validate reviewed release source" stage pip-installed pytest
from files.pythonhosted.org (DNS failures). Neither should touch the public
internet.
Base image: repoint the Dockerfile FROM from docker.io to the in-cluster
Harbor "mirror" project, keeping the exact content-addressed index digest
(9c841866...) and both arch leaves. A Flux-managed one-shot Job
(services/harbor/hermes-agent-base-mirror-job.yaml, suspend: true like the
cassandra bootstrap job) runs `skopeo copy --all` from docker.io into Harbor
using the same Vault-injected admin credential as the existing Harbor
immutability jobs; a tiny fail-closed helper ensures the public target
project first. Digest pinning and multi-arch are preserved; Kaniko pulls it
over the internal insecure registry with no docker.io fallback.
Test deps: install pytest/PyYAML fully offline (`pip --no-index
--find-links`) from a reviewed in-repo wheelhouse
(ci/vendor/hermes-agent-test-wheels) matching the arm64 python:3.12 build
container, so the validate stage never resolves a public index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The amd64 leg runs on its own fresh titan-24 pod but never checked out the SCM,
so its independent reviewed-revision boundary check hit 'git rev-parse
origin/main -> fatal: not a git repository' and the build failed with exit 128
(the arm64 leg built and pushed fine). Add checkout scm to the amd64 stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not
a general worker. The amd64 build leg was requiring node-role worker=true, which
forced labeling titan-24 as a worker and opened it to unrelated cluster
scheduling. It already pins by hostname+arch, so drop the worker requirement and
remove the titan-24 worker-join from the node-prefer CronJob entirely. The build
targets titan-24 specifically (hostname) and tolerates its taint; nothing else
in the cluster gets scheduled there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Operator steps in order: prepare/uncordon titan-24, merge, run one validation
build (both arch leaves + promoted index), apply the titan-22 affinity flip,
verify placement. Documents the correction that worker membership is reconciled
by the Flux node-prefer-noschedule CronJob, not Ansible, and lists what was
validated locally vs what only a real Jenkins build can prove.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The native amd64 hermes-agent image leg builds on titan-24. Worker membership
in this cluster is reconciled by the node-prefer-noschedule CronJob (kubectl
label), not Ansible, so add titan-24 there:
- clear_worker titan-24 amd64 -> node-role.kubernetes.io/worker=true + hardware=amd64
- a soft PreferNoSchedule guard taint (atlas.bstein.dev/sui-validator=true)
mirroring titan-22's media guard, so routine pods do not crowd the
out-of-cluster Sui validator that co-hosts titan-24. GPU workloads pinned to
titan-24 by hostname are unaffected (PreferNoSchedule never blocks a pinned
pod), and the amd64 build pod tolerates this taint explicitly.
Operator note: this reconciler does not manage cordons (owned by Ananke
recovery). titan-24 is on the recovery uncordon denylist, so the operator must
ensure titan-24 is uncordoned/schedulable before the first amd64 build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
