The hermes-agent-image pipeline failed intermittently on external network:
Kaniko's docker.io fallback for the base image is IPv6-broken from build
pods, and the "Validate reviewed release source" stage pip-installed pytest
from files.pythonhosted.org (DNS failures). Neither should touch the public
internet.
Base image: repoint the Dockerfile FROM from docker.io to the in-cluster
Harbor "mirror" project, keeping the exact content-addressed index digest
(9c841866...) and both arch leaves. A Flux-managed one-shot Job
(services/harbor/hermes-agent-base-mirror-job.yaml, suspend: true like the
cassandra bootstrap job) runs `skopeo copy --all` from docker.io into Harbor
using the same Vault-injected admin credential as the existing Harbor
immutability jobs; a tiny fail-closed helper ensures the public target
project first. Digest pinning and multi-arch are preserved; Kaniko pulls it
over the internal insecure registry with no docker.io fallback.
Test deps: install pytest/PyYAML fully offline (`pip --no-index
--find-links`) from a reviewed in-repo wheelhouse
(ci/vendor/hermes-agent-test-wheels) matching the arm64 python:3.12 build
container, so the validate stage never resolves a public index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The amd64 leg runs on its own fresh titan-24 pod but never checked out the SCM,
so its independent reviewed-revision boundary check hit 'git rev-parse
origin/main -> fatal: not a git repository' and the build failed with exit 128
(the arm64 leg built and pushed fine). Add checkout scm to the amd64 stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not
a general worker. The amd64 build leg was requiring node-role worker=true, which
forced labeling titan-24 as a worker and opened it to unrelated cluster
scheduling. It already pins by hostname+arch, so drop the worker requirement and
remove the titan-24 worker-join from the node-prefer CronJob entirely. The build
targets titan-24 specifically (hostname) and tolerates its taint; nothing else
in the cluster gets scheduled there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Repoint the hermes-agent base FROM at the upstream multi-arch OCI INDEX
digest (tag v2026.7.7.2, revision 9de9c25f) whose arm64 leaf is byte-for-byte
the previously pinned single-arch base, so the arm64 build is unchanged while
the same reviewed version now also resolves an amd64 leaf. Kaniko selects the
matching leaf per build platform.
Rework the release pipeline to build both arches natively and promote a
multi-arch image without switching off kaniko or weakening any existing
security assertion:
- Keep the arm64 kaniko leg on the unchanged rpi5 coordinating pod; it now
pushes an arch-suffixed candidate tag (...-build-<N>-arm64).
- Add a second native amd64 kaniko leg on a titan-24-pinned, tolerating,
resource-capped pod (ceiling strictly below the arm64 leg) that
independently re-verifies the reviewed revision and stashes its leaf
evidence (...-build-<N>-amd64).
- Add ci/scripts/hermes_multiarch_combine.py: a pure-python, fail-closed
combiner that re-reads each per-arch leaf from the registry, proves its
digest AND its config architecture, assembles a Docker manifest LIST
(already inside the promote allow-list), refuses to overwrite an existing
final tag, publishes the arch-less ...-build-<N> tag, and re-verifies the
registry resolved the exact index referencing exactly the two leaves. It
emits the index digest in the SAME digest-file/image-file format the
single-arch step produced, so render/verify-evidence/hermes_oci_promote.py
promote the INDEX with no change to those scripts.
Tests: add test_hermes_multiarch_combine.py (full hash/verification chain);
strengthen the image-builder suites for the two-arch topology (both kaniko
legs carry the reviewed heredoc-compat build-arg; amd64 leg pinned+capped+
boundary-checked; combine stage wiring; expanded evidence archive) without
weakening the arm64-leg assertions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
