titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: jenkins: rank the controller above its own build agents #49

Draft
hermes-automation wants to merge 1 commits from fix/t_39cf1905-jenkins-controller-priority into main
pull from: fix/t_39cf1905-jenkins-controller-priority
merge into: titan:main
Conversation 0 Commits 1 Files Changed 5 +26
hermes-automation commented 2026-08-23 13:17:21 +00:00
Member
Copy Link

Summary

The Jenkins controller has been Pending since 12:19:59Z today, which keeps Flux
jenkins at Ready=False and blocks hermes and hermes-chat behind it. This is a
self-deadlock in scheduling priority, not a transient capacity dip, so it does not
clear on its own.

What happened

The controller and the ephemeral build agents it schedules all run at priority 0. The
controller Deployment uses the Recreate strategy, so a JCasC change tears the old pod
down before the replacement is scheduled.

Merging #48 changed services/jenkins/configmap-jcasc.yaml, which rolled the hashed
jenkins-jcasc-revision ConfigMap (d6m9hfkg68 -> 45ddd9bt2f) and so rolled the
Deployment:

  • 12:19:57Z old controller pod jenkins-547476b857-gxkds deleted, freeing titan-11
  • 12:19:59Z build agent typhon-133-1pmgb (pending since 11:42:15Z) takes that slot
  • 12:19:59Z new controller pod jenkins-7cfc78d57-fjrbp created, immediately unschedulable

It cannot recover:

  • The scheduler has no lever, because every candidate victim is also priority 0 —
    "No preemption victims found for incoming pod".
  • The agent cannot exit, because its jnlp container only loops on
    "Connection refused" against the controller that cannot start (restarts=0).
  • The controller's required affinity (hardware=rpi5, minus titan-13/15/17/19) leaves
    exactly three eligible nodes, and all three are at or near CPU saturation:
    titan-07 3597/3600m, titan-08 3599/3600m, titan-11 3284/3600m.

Change

  • Add jenkins-core (500) and jenkins-agent (50, preemptionPolicy: Never),
    following the existing *-core / *-sim classes.
  • Set priorityClassName: jenkins-core on the controller so it outranks its agents and
    can preempt one when it has nowhere else to go.
  • Set priorityClassName: jenkins-agent on the JCasC default agent template so agents
    never preempt anything themselves.
  • Add dependsOn: core to the jenkins Flux Kustomization, since PriorityClasses ship
    in infrastructure/modules/base via core (matches cassandra and veles).

The controller-side class is the part that actually breaks the deadlock. Agents declared
inline by a Jenkinsfile podTemplate — like typhon-133 — do not inherit the JCasC
default template, so only the controller's own priority covers every agent.

Verification

  • kubectl kustomize renders clean for services/jenkins, infrastructure/core, and
    the priorityclass module.
  • Rendered diff against main is exactly: the two priorityClassName additions, the
    expected jenkins-jcasc-revision hash roll (45ddd9bt2f -> 9k4gg827b8), and the
    controller class. No image, resource, or affinity drift.
  • The nested JCasC agent YAML still parses; inner spec keys are
    affinity, nodeSelector, priorityClassName, securityContext, topologySpreadConstraints.
  • No Flux dependency cycle: core itself has no dependsOn.

Repo test suites were not run here — this environment has no pytest, and the repo's
gate runs on the Jenkins controller this PR is meant to restore.

Note

Merging this rolls the controller again (the JCasC hash changes). That is intended: the
new pod carries jenkins-core and can preempt the stuck agent, which is what clears the
deadlock through the Git/Flux path rather than by hand.

This PR is Part A of t_39cf1905 only. No image has been published and no digest pin is
included; the WebUI release lane still needs a healthy controller before it can run.

## Summary The Jenkins controller has been `Pending` since 12:19:59Z today, which keeps Flux `jenkins` at `Ready=False` and blocks `hermes` and `hermes-chat` behind it. This is a self-deadlock in scheduling priority, not a transient capacity dip, so it does not clear on its own. ## What happened The controller and the ephemeral build agents it schedules all run at priority 0. The controller Deployment uses the `Recreate` strategy, so a JCasC change tears the old pod down *before* the replacement is scheduled. Merging #48 changed `services/jenkins/configmap-jcasc.yaml`, which rolled the hashed `jenkins-jcasc-revision` ConfigMap (`d6m9hfkg68` -> `45ddd9bt2f`) and so rolled the Deployment: - `12:19:57Z` old controller pod `jenkins-547476b857-gxkds` deleted, freeing titan-11 - `12:19:59Z` build agent `typhon-133-1pmgb` (pending since `11:42:15Z`) takes that slot - `12:19:59Z` new controller pod `jenkins-7cfc78d57-fjrbp` created, immediately unschedulable It cannot recover: - The scheduler has no lever, because every candidate victim is also priority 0 — `"No preemption victims found for incoming pod"`. - The agent cannot exit, because its `jnlp` container only loops on `"Connection refused"` against the controller that cannot start (`restarts=0`). - The controller's required affinity (`hardware=rpi5`, minus titan-13/15/17/19) leaves exactly three eligible nodes, and all three are at or near CPU saturation: titan-07 3597/3600m, titan-08 3599/3600m, titan-11 3284/3600m. ## Change - Add `jenkins-core` (500) and `jenkins-agent` (50, `preemptionPolicy: Never`), following the existing `*-core` / `*-sim` classes. - Set `priorityClassName: jenkins-core` on the controller so it outranks its agents and can preempt one when it has nowhere else to go. - Set `priorityClassName: jenkins-agent` on the JCasC default agent template so agents never preempt anything themselves. - Add `dependsOn: core` to the `jenkins` Flux Kustomization, since PriorityClasses ship in `infrastructure/modules/base` via `core` (matches cassandra and veles). The controller-side class is the part that actually breaks the deadlock. Agents declared inline by a Jenkinsfile `podTemplate` — like `typhon-133` — do not inherit the JCasC default template, so only the controller's own priority covers every agent. ## Verification - `kubectl kustomize` renders clean for `services/jenkins`, `infrastructure/core`, and the priorityclass module. - Rendered diff against `main` is exactly: the two `priorityClassName` additions, the expected `jenkins-jcasc-revision` hash roll (`45ddd9bt2f` -> `9k4gg827b8`), and the controller class. No image, resource, or affinity drift. - The nested JCasC agent YAML still parses; inner `spec` keys are `affinity, nodeSelector, priorityClassName, securityContext, topologySpreadConstraints`. - No Flux dependency cycle: `core` itself has no `dependsOn`. Repo test suites were not run here — this environment has no `pytest`, and the repo's gate runs on the Jenkins controller this PR is meant to restore. ## Note Merging this rolls the controller again (the JCasC hash changes). That is intended: the new pod carries `jenkins-core` and can preempt the stuck agent, which is what clears the deadlock through the Git/Flux path rather than by hand. This PR is Part A of t_39cf1905 only. No image has been published and no digest pin is included; the WebUI release lane still needs a healthy controller before it can run.
hermes-automation added 1 commit 2026-08-23 13:17:22 +00:00 
The Jenkins controller and the ephemeral build agents it schedules both ran
at priority 0. Because the controller Deployment uses the Recreate strategy,
any JCasC change tears the controller down before its replacement is
scheduled, and while the eligible node pool is saturated that freed slot can
be taken by one of the controller's own pending agents.

That is what happened on 2026-08-23. Merging #48 changed
services/jenkins/configmap-jcasc.yaml, which rolled the hashed
jenkins-jcasc-revision ConfigMap and so rolled the Deployment. At 12:19:57Z
the old controller pod was deleted; at 12:19:59Z the build agent
typhon-133-1pmgb, pending since 11:42:15Z, took the freed capacity on
titan-11, and the new controller pod has been Pending ever since. The
scheduler cannot resolve it because every candidate victim is also priority
0: "No preemption victims found for incoming pod". The agent cannot exit on
its own either, since its jnlp container just loops on "Connection refused"
against the controller that cannot start. It is a self-deadlock, not a
transient capacity dip.

Give the controller a jenkins-core class (500) so it outranks its agents and
can preempt one when it has nowhere else to go, and mark the JCasC default
agent template jenkins-agent (50, preemptionPolicy: Never) so agents never
preempt anything themselves. Values follow the existing *-core/*-sim classes.

The controller-side class is what breaks the deadlock: agents declared inline
by a Jenkinsfile podTemplate, like typhon-133, do not inherit the JCasC
default template, so only the controller's own priority covers every agent.

PriorityClasses ship in infrastructure/modules/base via the core
Kustomization, so jenkins now dependsOn core, matching cassandra and veles.
Verified no dependency cycle: core itself has no dependsOn.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This pull request has changes conflicting with the target branch.
  • services/jenkins/deployment.yaml
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/t_39cf1905-jenkins-controller-priority:fix/t_39cf1905-jenkins-controller-priority
git checkout fix/t_39cf1905-jenkins-controller-priority
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#49
No description provided.
Delete Branch "fix/t_39cf1905-jenkins-controller-priority"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?