WIP: jenkins: rank the controller above its own build agents #49

Draft

hermes-automation wants to merge 1 commits from fix/t_39cf1905-jenkins-controller-priority into main

Author	SHA1	Message	Date
Hermes Agent	f997171b55	jenkins: rank the controller above its own build agents The Jenkins controller and the ephemeral build agents it schedules both ran at priority 0. Because the controller Deployment uses the Recreate strategy, any JCasC change tears the controller down before its replacement is scheduled, and while the eligible node pool is saturated that freed slot can be taken by one of the controller's own pending agents. That is what happened on 2026-08-23. Merging #48 changed services/jenkins/configmap-jcasc.yaml, which rolled the hashed jenkins-jcasc-revision ConfigMap and so rolled the Deployment. At 12:19:57Z the old controller pod was deleted; at 12:19:59Z the build agent typhon-133-1pmgb, pending since 11:42:15Z, took the freed capacity on titan-11, and the new controller pod has been Pending ever since. The scheduler cannot resolve it because every candidate victim is also priority 0: "No preemption victims found for incoming pod". The agent cannot exit on its own either, since its jnlp container just loops on "Connection refused" against the controller that cannot start. It is a self-deadlock, not a transient capacity dip. Give the controller a jenkins-core class (500) so it outranks its agents and can preempt one when it has nowhere else to go, and mark the JCasC default agent template jenkins-agent (50, preemptionPolicy: Never) so agents never preempt anything themselves. Values follow the existing -core/-sim classes. The controller-side class is what breaks the deadlock: agents declared inline by a Jenkinsfile podTemplate, like typhon-133, do not inherit the JCasC default template, so only the controller's own priority covers every agent. PriorityClasses ship in infrastructure/modules/base via the core Kustomization, so jenkins now dependsOn core, matching cassandra and veles. Verified no dependency cycle: core itself has no dependsOn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-23 13:12:40 +00:00

Author

SHA1

Message

Date

Hermes Agent

f997171b55

jenkins: rank the controller above its own build agents

The Jenkins controller and the ephemeral build agents it schedules both ran
at priority 0. Because the controller Deployment uses the Recreate strategy,
any JCasC change tears the controller down before its replacement is
scheduled, and while the eligible node pool is saturated that freed slot can
be taken by one of the controller's own pending agents.

That is what happened on 2026-08-23. Merging #48 changed
services/jenkins/configmap-jcasc.yaml, which rolled the hashed
jenkins-jcasc-revision ConfigMap and so rolled the Deployment. At 12:19:57Z
the old controller pod was deleted; at 12:19:59Z the build agent
typhon-133-1pmgb, pending since 11:42:15Z, took the freed capacity on
titan-11, and the new controller pod has been Pending ever since. The
scheduler cannot resolve it because every candidate victim is also priority
0: "No preemption victims found for incoming pod". The agent cannot exit on
its own either, since its jnlp container just loops on "Connection refused"
against the controller that cannot start. It is a self-deadlock, not a
transient capacity dip.

Give the controller a jenkins-core class (500) so it outranks its agents and
can preempt one when it has nowhere else to go, and mark the JCasC default
agent template jenkins-agent (50, preemptionPolicy: Never) so agents never
preempt anything themselves. Values follow the existing *-core/*-sim classes.

The controller-side class is what breaks the deadlock: agents declared inline
by a Jenkinsfile podTemplate, like typhon-133, do not inherit the JCasC
default template, so only the controller's own priority covers every agent.

PriorityClasses ship in infrastructure/modules/base via the core
Kustomization, so jenkins now dependsOn core, matching cassandra and veles.
Verified no dependency cycle: core itself has no dependsOn.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-23 13:12:40 +00:00

WIP: jenkins: rank the controller above its own build agents #49

1 Commits