atlas-iac/docs/hermes_agent_multiarch.md

hermes-agent multi-arch rollout runbook

Goal: build the hermes-agent container image for both linux/arm64 and linux/amd64 so the hermes-agent workload can eventually run on the amd64 node titan-22, while the arm64 pi fleet remains a valid fallback.

This branch (feature/hermes-agent-multiarch) contains everything needed. The steps below are the operator procedure, in order. Do not reorder: the titan-22 runtime flip is deliberately the last step and only after a validated multi-arch image exists.

---

What changed on the branch (for reviewers)

Area	File	Change
Base image	dockerfiles/Dockerfile.hermes-agent	FROM repointed from the arm64-only leaf sha256:47d4bd4c… to the upstream multi-arch OCI index sha256:9c841866021c54c4596849f6135717e8a4d52ba510b7f52c50aef1de1a283973 (tag v2026.7.7.2, revision 9de9c25f…). Same reviewed upstream version; arm64 leaf is byte-identical to the old pin.
Pipeline	ci/Jenkinsfile.hermes-agent-image	arm64 kaniko leg now pushes …-build-<N>-arm64; new amd64 kaniko leg on a titan-24-pinned, resource-capped pod pushes …-build-<N>-amd64; new Combine multi-arch index stage; evidence archive expanded.
Combiner	ci/scripts/hermes_multiarch_combine.py	New fail-closed pure-python step that verifies both leaves and publishes/re-verifies a Docker manifest list at the arch-less …-build-<N> tag, emitting the index digest in the existing digest-file/image-file format.
Promotion	ci/scripts/hermes_oci_promote.py	Unchanged. Docker manifest lists are already inside its MANIFEST_TYPES allow-list, so it promotes the multi-arch INDEX to …-release with no edit.
Release scripts	ci/scripts/hermes_image_release.py	Unchanged. The combiner hands it the arch-less tag + index digest, exactly the shape it already validates/renders.
Worker join	infrastructure/core/node-prefer-noschedule-cronjob.yaml	clear_worker titan-24 amd64 + a soft PreferNoSchedule guard taint protecting the co-hosted Sui validator.
Runtime flip (LAST)	services/hermes/agent-deployment.yaml	Adds titan-22 to the allowed set with a soft preference + media-taint toleration, keeping the arm64 fleet as fallback. Prepared as a separate commit — apply only after validation.

Correction to the original assumption

Worker/arch/taint labels in this cluster are not managed by Ansible. They are reconciled every minute by the Flux-managed node-prefer-noschedule CronJob (kubectl label/kubectl taint). hosts/ Ansible is a thin skeleton and titan-24 is already present there. Therefore the "worker-join" change lives in that CronJob, and it is applied by Flux at merge, not by a hardware playbook run. The only genuine hardware/operator action is ensuring titan-24 is uncordoned (it is on the Ananke recovery uncordon denylist).

---

Step (a) — Prepare titan-24 as a schedulable amd64 worker

The worker label + guard taint ship in this branch's CronJob change and land automatically once Flux reconciles after merge (step b). Before the first amd64 build, the operator must make sure titan-24 can actually accept the pinned pod:

# On a control-plane host / with cluster admin kubeconfig:
kubectl get node titan-24 -o wide            # confirm Ready, amd64
kubectl uncordon titan-24                     # recovery keeps it cordoned by default
# The Sui validator runs as docker-compose at /opt/sui-validator (outside k8s);
# it is protected by the build pod's tight resource caps, not by k8s priority.

After merge (step b), confirm the CronJob applied the labels/taint:

kubectl get node titan-24 --show-labels | tr ',' '\n' | grep -E 'worker|hardware'
# expect: node-role.kubernetes.io/worker=true and hardware=amd64
kubectl describe node titan-24 | grep -i taint
# expect: atlas.bstein.dev/sui-validator=true:PreferNoSchedule

If titan-24 must NOT be labeled yet, hold the merge; the labels are inert until the pod is scheduled and the guard taint only softly repels routine pods.

Step (b) — Merge the branch to main

Merge feature/hermes-agent-multiarch to main without the final titan-22 flip commit if you want to stage the runtime change separately (see step d). The Dockerfile, Jenkinsfile, combiner, tests, and CronJob are safe to merge: the arm64 build path is unchanged and the amd64 leg + combine only add new, fail-closed artifacts. Flux then applies the CronJob (titan-24 becomes a worker).

Step (c) — Trigger ONE validation build

Trigger the existing manual hermes-agent-image job exactly as today (the same PUBLISH_IMAGE=true, EXPECTED_SOURCE_REVISION=<full reviewed sha on main>, CONFIRM_PUBLISH=PUBLISH HERMES AGENT gate). Nothing about the trigger changes.

Success looks like:

1. arm64 leg builds on an rpi5 pod and pushes …-build-<N>-arm64.
2. amd64 leg builds on titan-24 and pushes …-build-<N>-amd64.
3. Combine multi-arch index prints "result": "published" with an index_digest, and the registry resolves …-build-<N> to a Docker manifest list referencing exactly the two leaves.
4. Render / Verify-and-archive / Publish stages behave exactly as before but on the index digest; the …-build-<N>-release tag is the multi-arch index.
5. Archived evidence now includes both per-arch digests/images plus the index digest/image.

Independent confirmation the release is multi-arch:

skopeo inspect --raw docker://registry.bstein.dev/bstein/hermes-agent:git-<sha>-build-<N>-release \
  | jq -r '.mediaType, (.manifests[] | "\(.platform.os)/\(.platform.architecture) \(.digest)")'
# expect: application/vnd.docker.distribution.manifest.list.v2+json
#         linux/amd64 sha256:…
#         linux/arm64 sha256:…

Flux will roll the hermes-agent deployment to the new index digest. It still runs on arm64 (unchanged affinity), and arm64 nodes keep pulling the arm64 leaf — so this step is safe even though the runtime is still arm64-only.

Step (d) — Apply the titan-22 runtime affinity flip

Only after step (c) proves the multi-arch image. Apply the final, separately messaged commit APPLY ONLY AFTER multi-arch image validated: allow hermes-agent on titan-22 (it edits services/hermes/agent-deployment.yaml and the matching layout test). It:

• adds a second nodeSelectorTerm matching amd64 titan-22 (worker) — the arm64 fleet term is untouched, so the pi fleet remains a valid fallback;
• adds a soft preferredDuringScheduling nudge toward titan-22;
• adds a toleration for titan-22's atlas.bstein.dev/media-primary=true:PreferNoSchedule taint. This does not change jellyfin's own scheduling or priority — a toleration only lets hermes-agent also consider titan-22 as spare capacity.

Merge that commit; Flux recreates the pod (deployment strategy is Recreate).

Step (e) — Verify the worker lands correctly

kubectl -n hermes get pod -l app=hermes-agent -o wide
# It may still land on an rpi5 (fallback is intact and pis are preferred-eligible);
# it is now ALSO permitted on titan-22. To confirm titan-22 can host it, cordon
# the pi fleet briefly or scale/drain, or simply confirm the pod tolerates the
# media taint and schedules when pis are unavailable.
kubectl -n hermes describe pod -l app=hermes-agent | grep -iE 'node:|toleration'

To roll back the runtime flip, revert the step-(d) commit; the arm64-only affinity returns and the multi-arch image keeps working on arm64.

---

What is validated locally vs what only a real build proves

Validated locally on this branch:

• The base index digest resolves to both linux/arm64 (the old pin) and linux/amd64, same revision 9de9c25f… (skopeo inspect --raw).
• hermes_multiarch_combine.py end-to-end unit tests (full hash + arch-config verification chain, idempotency, and every fail-closed branch).
• The updated + new image-builder tests, hermes_oci_promote, and the automation tests pass.
• kustomize build services/hermes and kustomize build infrastructure/core succeed.

Only a real Jenkins build can prove:

• Kaniko auto-selects the correct native leaf from the index on each arch, and the amd64 heredoc-compat replay produces a working amd64 image.
• The amd64 pod actually schedules on titan-24 within its resource cap without starving the Sui validator.
• Harbor accepts the manually-assembled manifest list PUT and immutability applies to the index tag.
• Cross-pod stash/unstash of the amd64 leaf evidence behaves as expected.