atlas-iac/docs/hermes_agent_multiarch.md

# hermes-agent multi-arch rollout runbook

Goal: build the `hermes-agent` container image for **both** `linux/arm64` and
`linux/amd64` so the `hermes-agent` workload can eventually run on the amd64
node `titan-22`, while the arm64 pi fleet remains a valid fallback.

This branch (`feature/hermes-agent-multiarch`) contains everything needed. The
steps below are the **operator** procedure, in order. Do not reorder: the
titan-22 runtime flip is deliberately the last step and only after a validated
multi-arch image exists.

---

## What changed on the branch (for reviewers)

| Area | File | Change |
| --- | --- | --- |
| Base image | `dockerfiles/Dockerfile.hermes-agent` | `FROM` repointed from the arm64-only leaf `sha256:47d4bd4c…` to the upstream **multi-arch OCI index** `sha256:9c841866021c54c4596849f6135717e8a4d52ba510b7f52c50aef1de1a283973` (tag `v2026.7.7.2`, revision `9de9c25f…`). Same reviewed upstream version; arm64 leaf is byte-identical to the old pin. |
| Pipeline | `ci/Jenkinsfile.hermes-agent-image` | arm64 kaniko leg now pushes `…-build-<N>-arm64`; new amd64 kaniko leg on a titan-24-pinned, resource-capped pod pushes `…-build-<N>-amd64`; new `Combine multi-arch index` stage; evidence archive expanded. |
| Combiner | `ci/scripts/hermes_multiarch_combine.py` | New fail-closed pure-python step that verifies both leaves and publishes/re-verifies a Docker manifest **list** at the arch-less `…-build-<N>` tag, emitting the index digest in the existing digest-file/image-file format. |
| Promotion | `ci/scripts/hermes_oci_promote.py` | **Unchanged.** Docker manifest lists are already inside its `MANIFEST_TYPES` allow-list, so it promotes the multi-arch INDEX to `…-release` with no edit. |
| Release scripts | `ci/scripts/hermes_image_release.py` | **Unchanged.** The combiner hands it the arch-less tag + index digest, exactly the shape it already validates/renders. |
| Worker join | `infrastructure/core/node-prefer-noschedule-cronjob.yaml` | `clear_worker titan-24 amd64` + a soft `PreferNoSchedule` guard taint protecting the co-hosted Sui validator. |
| Runtime flip (LAST) | `services/hermes/agent-deployment.yaml` | Adds titan-22 to the allowed set with a soft preference + media-taint toleration, keeping the arm64 fleet as fallback. **Prepared as a separate commit — apply only after validation.** |

### Correction to the original assumption

Worker/arch/taint labels in this cluster are **not** managed by Ansible. They are
reconciled every minute by the Flux-managed `node-prefer-noschedule` CronJob
(`kubectl label`/`kubectl taint`). `hosts/` Ansible is a thin skeleton and
titan-24 is already present there. Therefore the "worker-join" change lives in
that CronJob, and it is applied by **Flux at merge**, not by a hardware playbook
run. The only genuine hardware/operator action is ensuring titan-24 is
uncordoned (it is on the Ananke recovery uncordon denylist).

---

## Step (a) — Prepare titan-24 as a schedulable amd64 worker

The worker label + guard taint ship in this branch's CronJob change and land
automatically once Flux reconciles after merge (step b). Before the first amd64
build, the operator must make sure titan-24 can actually accept the pinned pod:

```sh
# On a control-plane host / with cluster admin kubeconfig:
kubectl get node titan-24 -o wide            # confirm Ready, amd64
kubectl uncordon titan-24                     # recovery keeps it cordoned by default
# The Sui validator runs as docker-compose at /opt/sui-validator (outside k8s);
# it is protected by the build pod's tight resource caps, not by k8s priority.
```

After merge (step b), confirm the CronJob applied the labels/taint:

```sh
kubectl get node titan-24 --show-labels | tr ',' '\n' | grep -E 'worker|hardware'
# expect: node-role.kubernetes.io/worker=true and hardware=amd64
kubectl describe node titan-24 | grep -i taint
# expect: atlas.bstein.dev/sui-validator=true:PreferNoSchedule
```

If titan-24 must NOT be labeled yet, hold the merge; the labels are inert until
the pod is scheduled and the guard taint only softly repels routine pods.

## Step (b) — Merge the branch to main

Merge `feature/hermes-agent-multiarch` to `main` **without** the final titan-22
flip commit if you want to stage the runtime change separately (see step d). The
Dockerfile, Jenkinsfile, combiner, tests, and CronJob are safe to merge: the
arm64 build path is unchanged and the amd64 leg + combine only add new,
fail-closed artifacts. Flux then applies the CronJob (titan-24 becomes a
worker).

## Step (c) — Trigger ONE validation build

Trigger the existing manual `hermes-agent-image` job exactly as today (the same
`PUBLISH_IMAGE=true`, `EXPECTED_SOURCE_REVISION=<full reviewed sha on main>`,
`CONFIRM_PUBLISH=PUBLISH HERMES AGENT` gate). Nothing about the trigger changes.

Success looks like:

1. **arm64 leg** builds on an rpi5 pod and pushes `…-build-<N>-arm64`.
2. **amd64 leg** builds on titan-24 and pushes `…-build-<N>-amd64`.
3. **Combine multi-arch index** prints `"result": "published"` with an
   `index_digest`, and the registry resolves `…-build-<N>` to a Docker manifest
   list referencing exactly the two leaves.
4. Render / Verify-and-archive / Publish stages behave exactly as before but on
   the **index** digest; the `…-build-<N>-release` tag is the multi-arch index.
5. Archived evidence now includes both per-arch digests/images plus the index
   digest/image.

Independent confirmation the release is multi-arch:

```sh
skopeo inspect --raw docker://registry.bstein.dev/bstein/hermes-agent:git-<sha>-build-<N>-release \
  | jq -r '.mediaType, (.manifests[] | "\(.platform.os)/\(.platform.architecture) \(.digest)")'
# expect: application/vnd.docker.distribution.manifest.list.v2+json
#         linux/amd64 sha256:…
#         linux/arm64 sha256:…
```

Flux will roll the `hermes-agent` deployment to the new index digest. It still
runs on arm64 (unchanged affinity), and arm64 nodes keep pulling the arm64 leaf
— so this step is safe even though the runtime is still arm64-only.

## Step (d) — Apply the titan-22 runtime affinity flip

**Only after step (c) proves the multi-arch image.** Apply the final, separately
messaged commit `APPLY ONLY AFTER multi-arch image validated: allow hermes-agent
on titan-22` (it edits `services/hermes/agent-deployment.yaml` and the matching
layout test). It:

- adds a second `nodeSelectorTerm` matching amd64 `titan-22` (worker) — the
  arm64 fleet term is untouched, so the pi fleet remains a valid fallback;
- adds a soft `preferredDuringScheduling` nudge toward titan-22;
- adds a toleration for titan-22's `atlas.bstein.dev/media-primary=true:PreferNoSchedule`
  taint. This does **not** change jellyfin's own scheduling or priority — a
  toleration only lets hermes-agent *also* consider titan-22 as spare capacity.

Merge that commit; Flux recreates the pod (deployment strategy is `Recreate`).

## Step (e) — Verify the worker lands correctly

```sh
kubectl -n hermes get pod -l app=hermes-agent -o wide
# It may still land on an rpi5 (fallback is intact and pis are preferred-eligible);
# it is now ALSO permitted on titan-22. To confirm titan-22 can host it, cordon
# the pi fleet briefly or scale/drain, or simply confirm the pod tolerates the
# media taint and schedules when pis are unavailable.
kubectl -n hermes describe pod -l app=hermes-agent | grep -iE 'node:|toleration'
```

To roll back the runtime flip, revert the step-(d) commit; the arm64-only
affinity returns and the multi-arch image keeps working on arm64.

---

## What is validated locally vs what only a real build proves

**Validated locally on this branch:**
- The base index digest resolves to both `linux/arm64` (the old pin) and
  `linux/amd64`, same revision `9de9c25f…` (`skopeo inspect --raw`).
- `hermes_multiarch_combine.py` end-to-end unit tests (full hash + arch-config
  verification chain, idempotency, and every fail-closed branch).
- The updated + new image-builder tests, `hermes_oci_promote`, and the automation
  tests pass.
- `kustomize build services/hermes` and `kustomize build infrastructure/core`
  succeed.

**Only a real Jenkins build can prove:**
- Kaniko auto-selects the correct native leaf from the index on each arch, and
  the amd64 heredoc-compat replay produces a working amd64 image.
- The amd64 pod actually schedules on titan-24 within its resource cap without
  starving the Sui validator.
- Harbor accepts the manually-assembled manifest list PUT and immutability
  applies to the index tag.
- Cross-pod `stash`/`unstash` of the amd64 leaf evidence behaves as expected.