From e54d581ef582614e206668909388526c2a5ceea5 Mon Sep 17 00:00:00 2001 From: jenkins Date: Tue, 25 Aug 2026 09:29:21 -0300 Subject: [PATCH] fix(hermes): un-pin worker from titan-08; spread across arm64 storage workers Worker (hermes-agent) was hard-pinned to titan-08 (a workaround after an earlier attempt to place it on the amd64 titan-22 failed on architecture). That single- node pin is exactly what makes it fragile: a titan-08 blip (as just happened when the node's Longhorn CSI went down) strands worker, and the Recreate strategy then deadlocks because the replacement can't schedule on the one tight node. Restore the intended multi-node design: run on any arm64 storage worker except the known-bad/weak ones (matching the repo's own affinity test, which was red). Its Longhorn volumes have data-locality disabled with replicas on titan-15/17/19, so there is no locality penalty to running on another node; the scheduler now places it on a roomier Pi (e.g. titan-05) and a node blip simply reschedules it. Also relax the gateway /api/status liveness probe (timeout 10s->15s, failureThreshold 3->5) so a transient slowness (e.g. a brief storage hiccup) no longer trips a kill-and-restart cascade. Follow-up (separate): multi-arch agent image to enable the amd64 titan-22 target. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf --- services/hermes/agent-deployment.yaml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/services/hermes/agent-deployment.yaml b/services/hermes/agent-deployment.yaml index 2726da23..9fce2b2c 100644 --- a/services/hermes/agent-deployment.yaml +++ b/services/hermes/agent-deployment.yaml @@ -109,8 +109,8 @@ spec: operator: In values: ["true"] - key: kubernetes.io/hostname - operator: In - values: [titan-08] + operator: NotIn + values: [titan-04, titan-06, titan-08, titan-13, titan-14, titan-17, titan-18, titan-19] preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: @@ -824,7 +824,8 @@ spec: command: [curl, -fsS, 'http://127.0.0.1:9119/api/status'] initialDelaySeconds: 90 periodSeconds: 30 - timeoutSeconds: 10 + timeoutSeconds: 15 + failureThreshold: 5 securityContext: allowPrivilegeEscalation: false runAsUser: 0