titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

fix(hermes-agent): do NOT make titan-24 a general worker for the amd64 build

Browse Source 
titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not
a general worker. The amd64 build leg was requiring node-role worker=true, which
forced labeling titan-24 as a worker and opened it to unrelated cluster
scheduling. It already pins by hostname+arch, so drop the worker requirement and
remove the titan-24 worker-join from the node-prefer CronJob entirely. The build
targets titan-24 specifically (hostname) and tolerates its taint; nothing else
in the cluster gets scheduled there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
This commit is contained in:
jenkins 2026-08-25 11:50:52 -03:00
parent 2b7d140d9e
commit 2bfdee6169
3 changed files with 10 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
ci/Jenkinsfile.hermes-agent-image
View File

@ -270,10 +270,14 @@ spec:
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
# titan-24 is an accelerator node (not a general worker) that co-hosts the
# out-of-cluster Sui validator. Pin the disposable amd64 build to it by
# hostname + arch ONLY — do NOT require node-role worker, so titan-24 is never
# opened to general cluster scheduling. The toleration + tight caps below keep
# this off the validator's back.
nodeSelector:
kubernetes.io/arch: amd64
kubernetes.io/hostname: titan-24
node-role.kubernetes.io/worker: "true"
tolerations:
# titan-24 co-hosts the out-of-cluster Sui validator; tolerate whatever
# PreferNoSchedule/NoSchedule guard taint the node carries so the pinned

15
infrastructure/core/node-prefer-noschedule-cronjob.yaml
View File

@ -59,7 +59,6 @@ spec:
clear_worker titan-14 rpi4
clear_worker titan-18 rpi4
clear_worker titan-22 amd64
clear_worker titan-24 amd64
if k get node titan-22 >/dev/null 2>&1; then
k label node titan-22 \
@ -70,20 +69,6 @@ spec:
k taint node titan-22 atlas.bstein.dev/media-primary=true:PreferNoSchedule --overwrite=true || true
fi
if k get node titan-24 >/dev/null 2>&1; then
# titan-24 (amd64, RTX 3080) co-hosts the out-of-cluster Sui
# validator (docker-compose at /opt/sui-validator). It joins as a
# worker so the native amd64 hermes-agent image leg can build here,
# but a soft guard taint keeps routine pods from crowding the
# validator. GPU workloads pinned to titan-24 by hostname are
# unaffected (PreferNoSchedule never blocks a pinned pod); the
# amd64 build pod tolerates this taint explicitly.
k label node titan-24 \
atlas.bstein.dev/general-compute=last-resort \
atlas.bstein.dev/sui-validator=true \
--overwrite=true || true
k taint node titan-24 atlas.bstein.dev/sui-validator=true:PreferNoSchedule --overwrite=true || true
fi
if k get node titan-23 >/dev/null 2>&1; then
k label node titan-23 \

7
testing/tests/test_hermes_image_builder_fresh_review.py
View File

@ -364,13 +364,16 @@ def test_pipeline_requires_and_archives_exact_release_evidence() -> None:
def test_amd64_leg_is_pinned_to_titan24_and_resource_capped() -> None:
"""The amd64 build leg lands on titan-24, tolerates its guard taint, and is capped."""
"""The amd64 build leg lands on titan-24 by hostname ONLY (titan-24 is an
accelerator, not a general worker), tolerates its taint, and is capped."""
spec = _amd64_pod_spec()
# Pin by arch + hostname only. Requiring node-role worker would force titan-24
# to be labeled a general worker, opening it to unrelated cluster scheduling.
assert spec["nodeSelector"] == {
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "titan-24",
"node-role.kubernetes.io/worker": "true",
}
assert "node-role.kubernetes.io/worker" not in spec["nodeSelector"]
# titan-24 co-hosts the Sui validator; the disposable build must tolerate
# whatever guard taint the node carries so the pinned pod still schedules.
assert {"operator": "Exists"} in spec["tolerations"]