fix(hermes-agent): do NOT make titan-24 a general worker for the amd64 build
titan-24 is an accelerator node (co-hosts the out-of-cluster Sui validator), not a general worker. The amd64 build leg was requiring node-role worker=true, which forced labeling titan-24 as a worker and opened it to unrelated cluster scheduling. It already pins by hostname+arch, so drop the worker requirement and remove the titan-24 worker-join from the node-prefer CronJob entirely. The build targets titan-24 specifically (hostname) and tolerates its taint; nothing else in the cluster gets scheduled there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
This commit is contained in:
parent
2b7d140d9e
commit
2bfdee6169
@ -270,10 +270,14 @@ spec:
|
|||||||
securityContext:
|
securityContext:
|
||||||
fsGroup: 1000
|
fsGroup: 1000
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
# titan-24 is an accelerator node (not a general worker) that co-hosts the
|
||||||
|
# out-of-cluster Sui validator. Pin the disposable amd64 build to it by
|
||||||
|
# hostname + arch ONLY — do NOT require node-role worker, so titan-24 is never
|
||||||
|
# opened to general cluster scheduling. The toleration + tight caps below keep
|
||||||
|
# this off the validator's back.
|
||||||
nodeSelector:
|
nodeSelector:
|
||||||
kubernetes.io/arch: amd64
|
kubernetes.io/arch: amd64
|
||||||
kubernetes.io/hostname: titan-24
|
kubernetes.io/hostname: titan-24
|
||||||
node-role.kubernetes.io/worker: "true"
|
|
||||||
tolerations:
|
tolerations:
|
||||||
# titan-24 co-hosts the out-of-cluster Sui validator; tolerate whatever
|
# titan-24 co-hosts the out-of-cluster Sui validator; tolerate whatever
|
||||||
# PreferNoSchedule/NoSchedule guard taint the node carries so the pinned
|
# PreferNoSchedule/NoSchedule guard taint the node carries so the pinned
|
||||||
|
|||||||
@ -59,7 +59,6 @@ spec:
|
|||||||
clear_worker titan-14 rpi4
|
clear_worker titan-14 rpi4
|
||||||
clear_worker titan-18 rpi4
|
clear_worker titan-18 rpi4
|
||||||
clear_worker titan-22 amd64
|
clear_worker titan-22 amd64
|
||||||
clear_worker titan-24 amd64
|
|
||||||
|
|
||||||
if k get node titan-22 >/dev/null 2>&1; then
|
if k get node titan-22 >/dev/null 2>&1; then
|
||||||
k label node titan-22 \
|
k label node titan-22 \
|
||||||
@ -70,20 +69,6 @@ spec:
|
|||||||
k taint node titan-22 atlas.bstein.dev/media-primary=true:PreferNoSchedule --overwrite=true || true
|
k taint node titan-22 atlas.bstein.dev/media-primary=true:PreferNoSchedule --overwrite=true || true
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if k get node titan-24 >/dev/null 2>&1; then
|
|
||||||
# titan-24 (amd64, RTX 3080) co-hosts the out-of-cluster Sui
|
|
||||||
# validator (docker-compose at /opt/sui-validator). It joins as a
|
|
||||||
# worker so the native amd64 hermes-agent image leg can build here,
|
|
||||||
# but a soft guard taint keeps routine pods from crowding the
|
|
||||||
# validator. GPU workloads pinned to titan-24 by hostname are
|
|
||||||
# unaffected (PreferNoSchedule never blocks a pinned pod); the
|
|
||||||
# amd64 build pod tolerates this taint explicitly.
|
|
||||||
k label node titan-24 \
|
|
||||||
atlas.bstein.dev/general-compute=last-resort \
|
|
||||||
atlas.bstein.dev/sui-validator=true \
|
|
||||||
--overwrite=true || true
|
|
||||||
k taint node titan-24 atlas.bstein.dev/sui-validator=true:PreferNoSchedule --overwrite=true || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if k get node titan-23 >/dev/null 2>&1; then
|
if k get node titan-23 >/dev/null 2>&1; then
|
||||||
k label node titan-23 \
|
k label node titan-23 \
|
||||||
|
|||||||
@ -364,13 +364,16 @@ def test_pipeline_requires_and_archives_exact_release_evidence() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def test_amd64_leg_is_pinned_to_titan24_and_resource_capped() -> None:
|
def test_amd64_leg_is_pinned_to_titan24_and_resource_capped() -> None:
|
||||||
"""The amd64 build leg lands on titan-24, tolerates its guard taint, and is capped."""
|
"""The amd64 build leg lands on titan-24 by hostname ONLY (titan-24 is an
|
||||||
|
accelerator, not a general worker), tolerates its taint, and is capped."""
|
||||||
spec = _amd64_pod_spec()
|
spec = _amd64_pod_spec()
|
||||||
|
# Pin by arch + hostname only. Requiring node-role worker would force titan-24
|
||||||
|
# to be labeled a general worker, opening it to unrelated cluster scheduling.
|
||||||
assert spec["nodeSelector"] == {
|
assert spec["nodeSelector"] == {
|
||||||
"kubernetes.io/arch": "amd64",
|
"kubernetes.io/arch": "amd64",
|
||||||
"kubernetes.io/hostname": "titan-24",
|
"kubernetes.io/hostname": "titan-24",
|
||||||
"node-role.kubernetes.io/worker": "true",
|
|
||||||
}
|
}
|
||||||
|
assert "node-role.kubernetes.io/worker" not in spec["nodeSelector"]
|
||||||
# titan-24 co-hosts the Sui validator; the disposable build must tolerate
|
# titan-24 co-hosts the Sui validator; the disposable build must tolerate
|
||||||
# whatever guard taint the node carries so the pinned pod still schedules.
|
# whatever guard taint the node carries so the pinned pod still schedules.
|
||||||
assert {"operator": "Exists"} in spec["tolerations"]
|
assert {"operator": "Exists"} in spec["tolerations"]
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user