titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity
atlas-iac/services/jenkins/deployment.yaml
Hermes Agent f997171b55 jenkins: rank the controller above its own build agents 
The Jenkins controller and the ephemeral build agents it schedules both ran
at priority 0. Because the controller Deployment uses the Recreate strategy,
any JCasC change tears the controller down before its replacement is
scheduled, and while the eligible node pool is saturated that freed slot can
be taken by one of the controller's own pending agents.

That is what happened on 2026-08-23. Merging #48 changed
services/jenkins/configmap-jcasc.yaml, which rolled the hashed
jenkins-jcasc-revision ConfigMap and so rolled the Deployment. At 12:19:57Z
the old controller pod was deleted; at 12:19:59Z the build agent
typhon-133-1pmgb, pending since 11:42:15Z, took the freed capacity on
titan-11, and the new controller pod has been Pending ever since. The
scheduler cannot resolve it because every candidate victim is also priority
0: "No preemption victims found for incoming pod". The agent cannot exit on
its own either, since its jnlp container just loops on "Connection refused"
against the controller that cannot start. It is a self-deadlock, not a
transient capacity dip.

Give the controller a jenkins-core class (500) so it outranks its agents and
can preempt one when it has nowhere else to go, and mark the JCasC default
agent template jenkins-agent (50, preemptionPolicy: Never) so agents never
preempt anything themselves. Values follow the existing *-core/*-sim classes.

The controller-side class is what breaks the deadlock: agents declared inline
by a Jenkinsfile podTemplate, like typhon-133, do not inherit the JCasC
default template, so only the controller's own priority covers every agent.

PriorityClasses ship in infrastructure/modules/base via the core
Kustomization, so jenkins now dependsOn core, matching cassandra and veles.
Verified no dependency cycle: core itself has no dependsOn.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 13:12:40 +00:00

251 lines
9.0 KiB
YAML
Raw Blame History

# services/jenkins/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: jenkins
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
strategy:
type: Recreate
template:
metadata:
labels:
app: jenkins
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-requests-cpu: "25m"
vault.hashicorp.com/agent-requests-mem: "64Mi"
vault.hashicorp.com/agent-limits-cpu: "250m"
vault.hashicorp.com/agent-limits-mem: "128Mi"
vault.hashicorp.com/role: "jenkins"
vault.hashicorp.com/agent-inject-secret-jenkins-env: "kv/data/atlas/jenkins/jenkins-oidc"
vault.hashicorp.com/agent-inject-template-jenkins-env: |
{{ with secret "kv/data/atlas/jenkins/jenkins-oidc" }}
OIDC_CLIENT_ID={{ .Data.data.clientId }}
OIDC_CLIENT_SECRET={{ .Data.data.clientSecret }}
OIDC_AUTH_URL={{ .Data.data.authorizationUrl }}
OIDC_TOKEN_URL={{ .Data.data.tokenUrl }}
OIDC_USERINFO_URL={{ .Data.data.userInfoUrl }}
OIDC_LOGOUT_URL={{ .Data.data.logoutUrl }}
{{ end }}
{{ with secret "kv/data/atlas/jenkins/harbor-robot-creds" }}
HARBOR_ROBOT_USERNAME={{ .Data.data.username }}
HARBOR_ROBOT_PASSWORD={{ .Data.data.password }}
HARBOR_STREAMING_ROBOT_USERNAME={{ .Data.data.username }}
HARBOR_STREAMING_ROBOT_PASSWORD={{ .Data.data.password }}
{{ end }}
{{ with secret "kv/data/atlas/jenkins/harbor-streaming-robot-creds" }}
HARBOR_STREAMING_ROBOT_USERNAME={{ .Data.data.username }}
HARBOR_STREAMING_ROBOT_PASSWORD={{ .Data.data.password }}
{{ end }}
{{ with secret "kv/data/atlas/shared/harbor-pull" }}
{{- if and .Data.data.username .Data.data.password }}
HARBOR_PULL_USERNAME={{ .Data.data.username }}
HARBOR_PULL_PASSWORD={{ .Data.data.password }}
{{- end }}
{{ end }}
{{ with secret "kv/data/atlas/jenkins/gitea-pat" }}
GITEA_PAT_USERNAME={{ .Data.data.username }}
GITEA_PAT_TOKEN={{ .Data.data.token }}
{{ end }}
{{ with secret "kv/data/atlas/quality/sonarqube-oidc" }}
SONARQUBE_TOKEN={{ .Data.data.sonarqube_exporter_token }}
{{ end }}
{{ with secret "kv/data/atlas/jenkins/webhook-tokens" }}
TITAN_IAC_WEBHOOK_TOKEN={{ .Data.data.titan_iac_quality_gate }}
GIT_NOTIFY_TOKEN_BSTEIN_DEV_HOME={{ .Data.data.git_notify_bstein_dev_home }}
{{ end }}
{{ with secret "kv/data/atlas/jenkins/ariadne-api" }}
ARIADNE_JENKINS_API_USER={{ .Data.data.username }}
ARIADNE_JENKINS_API_TOKEN={{ .Data.data.token }}
{{ end }}
{{ with secret "kv/data/atlas/hermes/developer-jenkins" }}
HERMES_AGENT_IMAGE_BUILD_TOKEN={{ .Data.data.build_token }}
{{ end }}
bstein.dev/restarted-at: "2026-05-20T09:40:31Z"
spec:
serviceAccountName: jenkins
# The controller must outrank the build agents it schedules. Without this
# the controller and its agents are both priority 0, so a Recreate
# rollout can free the controller's slot only for one of its own agents
# to take it, leaving the controller permanently Pending with the
# scheduler reporting "No preemption victims found for incoming pod".
priorityClassName: jenkins-core
nodeSelector:
kubernetes.io/arch: arm64
node-role.kubernetes.io/worker: "true"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- titan-13
- titan-15
- titan-17
- titan-19
- key: hardware
operator: In
values: ["rpi5"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: atlas.bstein.dev/spillover
operator: DoesNotExist
- weight: 95
preference:
matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- titan-13
- titan-15
- titan-17
- titan-19
- weight: 90
preference:
matchExpressions:
- key: hardware
operator: In
values: ["rpi5"]
- weight: 50
preference:
matchExpressions:
- key: hardware
operator: In
values: ["rpi4"]
hostAliases:
- ip: 38.28.125.112
hostnames:
- sso.bstein.dev
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: install-plugins
image: jenkins/jenkins:2.528.3-jdk21
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
set -euo pipefail
rm -rf /usr/share/jenkins/ref/plugins/*
jenkins-plugin-cli --plugin-file /plugins/plugins.txt
volumeMounts:
- name: plugins
mountPath: /plugins/plugins.txt
subPath: plugins.txt
- name: plugin-dir
mountPath: /usr/share/jenkins/ref/plugins
containers:
- name: jenkins
image: jenkins/jenkins:2.528.3-jdk21
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- |
set -e
exec env $(cat /vault/secrets/jenkins-env) /usr/bin/tini -- /usr/local/bin/jenkins.sh
ports:
- name: http
containerPort: 8080
- name: agent-listener
containerPort: 50000
env:
- name: JAVA_OPTS
value: "-Xms512m -Xmx2048m -Duser.timezone=America/Chicago -Dkubernetes.websocket.timeout=60000 -Dorg.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator.websocketConnectionTimeout=120"
- name: TZ
value: "America/Chicago"
- name: JENKINS_OPTS
value: "--webroot=/var/jenkins_cache/war"
- name: JENKINS_SLAVE_AGENT_PORT
value: "50000"
- name: CASC_JENKINS_CONFIG
value: /config/jcasc
- name: ENABLE_OIDC
value: "true"
- name: OIDC_ISSUER
value: "https://sso.bstein.dev/realms/atlas"
resources:
requests:
cpu: 100m
memory: 1024Mi
limits:
cpu: 1500m
memory: 3Gi
livenessProbe:
httpGet:
path: /login
port: http
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: /login
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
startupProbe:
httpGet:
path: /login
port: http
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 60
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
- name: jenkins-cache
mountPath: /var/jenkins_cache
- name: jcasc
mountPath: /config/jcasc
- name: init-scripts
mountPath: /usr/share/jenkins/ref/init.groovy.d
- name: init-scripts
mountPath: /var/jenkins_home/init.groovy.d
- name: plugin-dir
mountPath: /usr/share/jenkins/ref/plugins
- name: tmp
mountPath: /tmp
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins
- name: jenkins-cache
persistentVolumeClaim:
claimName: jenkins-cache-v2
- name: plugin-dir
persistentVolumeClaim:
claimName: jenkins-plugins-v2
- name: plugins
configMap:
name: jenkins-plugins
- name: jcasc
configMap:
name: jenkins-jcasc
- name: jcasc-revision
configMap:
name: jenkins-jcasc-revision
- name: init-scripts
configMap:
name: jenkins-init-scripts
- name: tmp
emptyDir:
medium: Memory
Reference in New Issue View Git Blame Copy Permalink