titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not
a general worker. The amd64 build leg was requiring node-role worker=true, which
forced labeling titan-24 as a worker and opened it to unrelated cluster
scheduling. It already pins by hostname+arch, so drop the worker requirement and
remove the titan-24 worker-join from the node-prefer CronJob entirely. The build
targets titan-24 specifically (hostname) and tolerates its taint; nothing else
in the cluster gets scheduled there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
The native amd64 hermes-agent image leg builds on titan-24. Worker membership
in this cluster is reconciled by the node-prefer-noschedule CronJob (kubectl
label), not Ansible, so add titan-24 there:
- clear_worker titan-24 amd64 -> node-role.kubernetes.io/worker=true + hardware=amd64
- a soft PreferNoSchedule guard taint (atlas.bstein.dev/sui-validator=true)
mirroring titan-22's media guard, so routine pods do not crowd the
out-of-cluster Sui validator that co-hosts titan-24. GPU workloads pinned to
titan-24 by hostname are unaffected (PreferNoSchedule never blocks a pinned
pod), and the amd64 build pod tolerates this taint explicitly.
Operator note: this reconciler does not manage cordons (owned by Ananke
recovery). titan-24 is on the recovery uncordon denylist, so the operator must
ensure titan-24 is uncordoned/schedulable before the first amd64 build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
