The Jenkins controller and the ephemeral build agents it schedules both ran
at priority 0. Because the controller Deployment uses the Recreate strategy,
any JCasC change tears the controller down before its replacement is
scheduled, and while the eligible node pool is saturated that freed slot can
be taken by one of the controller's own pending agents.
That is what happened on 2026-08-23. Merging #48 changed
services/jenkins/configmap-jcasc.yaml, which rolled the hashed
jenkins-jcasc-revision ConfigMap and so rolled the Deployment. At 12:19:57Z
the old controller pod was deleted; at 12:19:59Z the build agent
typhon-133-1pmgb, pending since 11:42:15Z, took the freed capacity on
titan-11, and the new controller pod has been Pending ever since. The
scheduler cannot resolve it because every candidate victim is also priority
0: "No preemption victims found for incoming pod". The agent cannot exit on
its own either, since its jnlp container just loops on "Connection refused"
against the controller that cannot start. It is a self-deadlock, not a
transient capacity dip.
Give the controller a jenkins-core class (500) so it outranks its agents and
can preempt one when it has nowhere else to go, and mark the JCasC default
agent template jenkins-agent (50, preemptionPolicy: Never) so agents never
preempt anything themselves. Values follow the existing *-core/*-sim classes.
The controller-side class is what breaks the deadlock: agents declared inline
by a Jenkinsfile podTemplate, like typhon-133, do not inherit the JCasC
default template, so only the controller's own priority covers every agent.
PriorityClasses ship in infrastructure/modules/base via the core
Kustomization, so jenkins now dependsOn core, matching cassandra and veles.
Verified no dependency cycle: core itself has no dependsOn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The titan-iac-quality-gate multibranch job's git branch source was
defined without a discovery trait, so JCasC created it with empty
traits and it indexed zero branches — no PR ever built, no status
check was ever reported, and auto-merge could never fire. Add
traits { gitBranchDiscovery() } to match every working multibranch
job (e.g. hermes-code-demo-branches).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cassandra pipelineJob had no trigger, so Gitea's notifyCommit
webhook found no matching job and pushes only built when started
manually. Poll every 5 minutes like lesavka/typhon so pushes build
within one poll cycle and the webhook race disappears.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fixture check waited on the Job's .status.succeeded/.status.failed. The
kubelet records a container's exit code the instant it stops, but the Job
controller reconciles those fields on its own schedule: observed at 2m24s,
4m27s and 34m on this cluster for identical work. During that window the pod
had already exited 1 and the build sat printing 'Will try again after 13 sec',
looking hung long after the test had finished, which is unusable in a demo.
Polling the pod's terminated exitCode removes the controller from the path
entirely. The Job is still used, so the evidence shape is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without the junit plugin the pipeline's junit step throws NoSuchMethodError,
jenkins.failed_tests is always empty, and auto-triage has only raw console
text to reason from. That was the root of two separate diagnosis failures:
the enforced failure being crowded out of the evidence budget, and the
patcher being unable to locate the defective source file.
Pinned to junit 1369.v15da_00283f06, the newest release that runs on core
2.528.3 - the current 1418 requires 2.533. scm-api moves 724 -> 728 because
the workflow-cps these pull in requires it; verified the whole 30-plugin
dependency closure needs no core newer than 2.528.3, and rehearsed the exact
jenkins-plugin-cli install in a throwaway pod before committing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Indexing succeeded but discovered no branches without an explicit
gitBranchDiscovery trait, so hermes-repair/* never got a validation build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds demo driver script and branch-level test gate so a Hermes-proposed
pull request carries a green build before human merge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Pipeline ensures the fixture ConfigMap and seeds unhealthy via patch-file
- Test-runner mounts the ConfigMap read-only; no storage attach involved
- New hermes-demo-repair SA + Role scoped to patching only the fixture CM
- Removes the fixture PVC (Flux prunes it)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New hermes-triage-demo namespace with fixture PVC and RBAC scoped to
Jenkins agent Job creation only
- JCasC pipelineJob hermes-triage-demo: SEED_FAILURE-parameterized fixture
check running as a Kubernetes Job in the demo namespace, emitting the
incident ID to pod stdout (Fluent Bit -> OpenSearch kube-*) and JUnit to
Jenkins
- Hermes init container can seed API_SERVER_KEY in the persistent .env
from an optional hermes-api-server-key Secret (no-op until it exists)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
