The hermes-agent-image pipeline failed intermittently on external network:
Kaniko's docker.io fallback for the base image is IPv6-broken from build
pods, and the "Validate reviewed release source" stage pip-installed pytest
from files.pythonhosted.org (DNS failures). Neither should touch the public
internet.
Base image: repoint the Dockerfile FROM from docker.io to the in-cluster
Harbor "mirror" project, keeping the exact content-addressed index digest
(9c841866...) and both arch leaves. A Flux-managed one-shot Job
(services/harbor/hermes-agent-base-mirror-job.yaml, suspend: true like the
cassandra bootstrap job) runs `skopeo copy --all` from docker.io into Harbor
using the same Vault-injected admin credential as the existing Harbor
immutability jobs; a tiny fail-closed helper ensures the public target
project first. Digest pinning and multi-arch are preserved; Kaniko pulls it
over the internal insecure registry with no docker.io fallback.
Test deps: install pytest/PyYAML fully offline (`pip --no-index
--find-links`) from a reviewed in-repo wheelhouse
(ci/vendor/hermes-agent-test-wheels) matching the arm64 python:3.12 build
container, so the validate stage never resolves a public index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The amd64 leg runs on its own fresh titan-24 pod but never checked out the SCM,
so its independent reviewed-revision boundary check hit 'git rev-parse
origin/main -> fatal: not a git repository' and the build failed with exit 128
(the arm64 leg built and pushed fine). Add checkout scm to the amd64 stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not
a general worker. The amd64 build leg was requiring node-role worker=true, which
forced labeling titan-24 as a worker and opened it to unrelated cluster
scheduling. It already pins by hostname+arch, so drop the worker requirement and
remove the titan-24 worker-join from the node-prefer CronJob entirely. The build
targets titan-24 specifically (hostname) and tolerates its taint; nothing else
in the cluster gets scheduled there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Repoint the hermes-agent base FROM at the upstream multi-arch OCI INDEX
digest (tag v2026.7.7.2, revision 9de9c25f) whose arm64 leaf is byte-for-byte
the previously pinned single-arch base, so the arm64 build is unchanged while
the same reviewed version now also resolves an amd64 leaf. Kaniko selects the
matching leaf per build platform.
Rework the release pipeline to build both arches natively and promote a
multi-arch image without switching off kaniko or weakening any existing
security assertion:
- Keep the arm64 kaniko leg on the unchanged rpi5 coordinating pod; it now
pushes an arch-suffixed candidate tag (...-build-<N>-arm64).
- Add a second native amd64 kaniko leg on a titan-24-pinned, tolerating,
resource-capped pod (ceiling strictly below the arm64 leg) that
independently re-verifies the reviewed revision and stashes its leaf
evidence (...-build-<N>-amd64).
- Add ci/scripts/hermes_multiarch_combine.py: a pure-python, fail-closed
combiner that re-reads each per-arch leaf from the registry, proves its
digest AND its config architecture, assembles a Docker manifest LIST
(already inside the promote allow-list), refuses to overwrite an existing
final tag, publishes the arch-less ...-build-<N> tag, and re-verifies the
registry resolved the exact index referencing exactly the two leaves. It
emits the index digest in the SAME digest-file/image-file format the
single-arch step produced, so render/verify-evidence/hermes_oci_promote.py
promote the INDEX with no change to those scripts.
Tests: add test_hermes_multiarch_combine.py (full hash/verification chain);
strengthen the image-builder suites for the two-arch topology (both kaniko
legs carry the reviewed heredoc-compat build-arg; amd64 leg pinned+capped+
boundary-checked; combine stage wiring; expanded evidence archive) without
weakening the arm64-leg assertions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The env setters moved to block style so Flux can rewrite them; the
renderer belt now matches the same shape and the release test asserts
the bound value lines rather than the old flow mapping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Activation-layer staging, fail-closed until enablement: a per-tenant
hux-evidence-producer sidecar on the exact reviewed WebUI image runs
hux_producer.run_once on a 60s loop, inert until the Vault-staged
evidence key (tolerant init, tmpfs, 0400, staged only for the hux
service and producer containers - never hermes or webui), the policy
ConfigMap, and the scope ConfigMap exist. Adds least-privilege
read-only RBAC (pods+statefulset in hermes, the single named Flux
Kustomization), tenant egress to the Kubernetes API ClusterIP and the
traefik edge, the policy allowlist, hux_producer packaging in the WebUI
image, and a third expected WebUI consumer in the Flux release
renderer. Delivery and image-automation gates enforce the boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The Flux release renderer now accepts an expected-consumer set per
workload (chat may carry the HUX sidecar as a second consumer of the
exact same WebUI image) and binds HUX_IMAGE_TAG/HUX_IMAGE_DIGEST env
metadata to the released tag and digest when those fields are present.
Rendering fails when the binding fields are incomplete, keeping the
image identity single-sourced. Tests adapt to both topologies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
Exercise the mailu sync retry, attribute, and skip branches, the
listener non-object JSON path, and the hygiene conftest skip; drop the
unreachable inverted-range clamp in the semgrep report (the line helper
already floors the end line) and pin that behavior with a test. Exclude
the mailu __main__ guards from measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
