atlas-iac/docs/hermes_multiuser_capacity_assessment.md

Hermes multi-user capacity assessment

Status: read-only investigation, no production changes made. This document answers "can Hermes safely handle more simultaneous users" with evidence from the current titan-iac manifests/code, plus non-invasive live cluster reads. It does not increase any replica count or concurrency cap.

0. Method and access boundary

• Static evidence: services/hermes/** manifests and services/hermes/router/*.go, services/hermes/scripts/*.py on branch wt/t_65356568, git log/git blame for recent drift.
• Live evidence: read via the in-cluster atlas-observer kubeconfig context. This identity (system:serviceaccount:hermes:hermes-agent) is observer-only: kubectl -n hermes get pods/deploy/sts/hpa/pdb all returned Forbidden. kubectl -n hermes auth can-i --list confirms the only granted verbs are cluster-scoped get/list/watch on namespaces, nodes, persistentvolumes, storage APIs, and nodes.metrics.k8s.io, plus API discovery/health endpoints. There is no namespace-scoped read access to pods, deployments, replica counts, or pod-level resource usage in hermes (or any namespace) from this session. This matches the standing finding that hermes-agent RBAC is observer-only.
• What this means for this report: all replica counts, resource requests/limits, probes, and affinity rules below are read from the Git manifests (which is what Flux actually reconciles, so it is a trustworthy source for desired state) rather than confirmed against live pod status. Node-level capacity and current node CPU/memory utilization are live, real-time cluster evidence (nodes.metrics.k8s.io + nodes are both readable). One non-invasive HTTP probe against https://chat.bstein.dev/ confirmed the public endpoint is live and redirecting to auth (302) at the time of writing.
• No pods, deployments, config, or replica counts were changed. No load test was executed against any environment.

1. Architecture and request path

browser
  -> Traefik Ingress (chat.hermes.bstein.dev / chat.bstein.dev)
  -> oauth2-proxy-hermes-chat  (Deployment, replicas: 1, oauth2-proxy.yaml:153)
       session state: Redis, externalized  (--session-store-type=redis,
       redis://hermes-oauth-sessions...:6379/1, oauth2-proxy.yaml:203-204)
  -> hermes-chat-router  (Deployment, replicas: 1, chat-router.yaml:10)
       Go binary, services/hermes/router/main.go
       in-process, PVC-persisted identity -> slot map (tenants.json)
  -> hermes-chat-tenant-<slot>.hermes-chat-tenant  (StatefulSet, replicas: 4,
       chat-statefulset.yaml:13) — one pod per Keycloak subject, pinned by ordinal
       webui (8787) -> gateway (8642) -> per-tenant sandbox (chat-sandbox-<ordinal>:9080)
  -> hermes-switchyard  (Deployment, replicas: 1, switchyard-deployment.yaml:10)
       shared model router for ALL tenants + triage + Kanban
  -> hermes-claude-broker / hermes-codex-broker (containers in hermes-agent pod,
       replicas: 1, agent-deployment.yaml:10) / hermes-model-gate (Deployment,
       replicas: 1, model-gate-deployment.yaml:10, proxies to local Ollama)
  -> hermes-stt / hermes-tts  (Deployments, replicas: 1 each, voice-deployment.yaml,
       pinned to node titan-21, shared GPU)

Every component on this path is replicas: 1 except hermes-chat-tenant (4). The StatefulSet is not a load-balanced pool: each pod is a fully isolated, permanently-assigned per-user backend, not interchangeable capacity.

2. Known limits (confirmed from manifests/code)

2.1 Hard ceiling: 4 usable chat users today, and a live configuration bug

services/hermes/router/main.go (tenantRouter.slotFor, lines 153-176) permanently assigns each Keycloak subject a slot 0..TENANT_SLOTS-1 on first login, persists it to a PVC-backed JSON file, and never releases it. There is no unassign/deprovision path in the router. Unit tests (router/main_test.go, "expected the fixed private tenant pool to report capacity") assert that once all slots are taken, slotFor returns an error.

Confirmed drift: chat-router.yaml:70 sets TENANT_SLOTS=8, but chat-statefulset.yaml:13 sets replicas: 4. git log -p on both files shows they were changed together at 18965a2f (4->8), then chat-statefulset.yaml was scaled down twice — 79baa7ec ("return two chat slots to fit the agent runtime", 8->6) and e28b32bd (6->4) — without a matching change to TENANT_SLOTS, and chat-sandbox.yaml still defines 8 per-ordinal sandbox Deployments (hermes-chat-sandbox-0 through -7). Net effect: the 5th–8th distinct Keycloak login is durably assigned a slot that has no backing StatefulSet pod. main.go's proxy error path (around line 382) returns a "starting" error to that user permanently, not just during a cold start, because the pod will never exist at the current replica count. This is a pre-existing configuration bug, independent of any future scale-up decision, and it silently masks the real behavior of the capacity ceiling (a 5th user does not get a clean "at capacity" rejection matching the tested behavior — they get a StatefulSet-pod-not-found style failure). This assessment does not fix it (per scope: no production changes), but it should be the first fix considered before any scale-out, since it affects correctness at the current replica count too.

2.2 The real shared bottleneck is not the 4 tenant pods, it's the brokers behind them

All 4 tenant pods, the triage/operator instance, and Kanban/CLI-lane traffic funnel through the same single-replica hermes-switchyard, which routes to the same single-replica hermes-claude-broker / hermes-codex-broker (containers inside the one hermes-agent pod) or hermes-model-gate.

• hermes-claude-broker enforces a hard, code-level concurrency cap: HERMES_CLAUDE_BROKER_CONCURRENCY=2 (agent-deployment.yaml:1160) sizes a threading.BoundedSemaphore(2) (scripts/claude_oauth_broker.py:63) wrapped around a blocking subprocess.run call to the native Claude Code CLI. Requests over the cap block silently (no 429/503 to the caller) until a slot frees. This cap is shared cluster-wide across every chat tenant, the triage instance, and indirectly the Kanban/CLI-lane workers (same provider credentials, so heavy CLI-lane usage can throttle chat responses via the provider's own rate limits even without touching this semaphore directly).
• hermes-codex-broker and hermes-model-gate have no equivalent concurrency cap in code — they are bounded only by container CPU/memory limits (1 CPU / 1Gi for the codex-broker container, 250m/128Mi for model-gate) and by the single upstream Ollama/Jetson instance's real throughput, which is unknown (no load test exists).

Implication: scaling hermes-chat-tenant replicas alone (e.g. 4 -> 8) does not proportionally increase real AI-response throughput. The Claude-routed share of traffic is capped at 2 concurrent completions regardless of how many tenant pods exist, and Codex/local-model throughput under concurrent load is untested.

2.3 Single points of failure and deploy-time outage windows

• No HorizontalPodAutoscaler exists anywhere in this repository (confirmed by a repo-wide grep, not just services/hermes/).
• No PodDisruptionBudget exists for any hermes component (the one PDB in the repo is infrastructure/traefik/pdb.yaml, unrelated).
• oauth2-proxy, hermes-chat-router, hermes-switchyard, hermes-model-gate, hermes-agent (brokers), and voice STT/TTS are all replicas: 1. A pod restart/crash/node drain on any of them takes down chat, voice, or model access for all users simultaneously, even though the tenant StatefulSet gives each user an isolated conversation backend.
• hermes (main/triage), hermes-chat-router, hermes-local-image, and hermes-agent all use strategy: Recreate — every rollout is a full teardown-then-recreate, not a rolling update, adding a real availability gap bounded by readiness/liveness initialDelaySeconds (up to 90s+ for the main hermes deployment). The only genuinely zero-downtime rollout path in this whole service is hermes-switchyard (RollingUpdate, maxSurge:1, maxUnavailable:0) and the tenant StatefulSet itself (RollingUpdate, podManagementPolicy: Parallel).
• OAuth session storage (hermes-oauth-sessions, a single Redis Deployment, replicas: 1, oauth-session-store.yaml) has no replication; its loss logs out every user cluster-wide (sessions are externalized from oauth2-proxy but not made highly available).

2.4 The cluster is a heterogeneous home-lab/edge fleet with real, uneven headroom

Live nodes.metrics.k8s.io reads (non-invasive) show 21 nodes: mostly Raspberry Pi 4/5 workers (~3.6-4 vCPU / ~6.5-8GB allocatable each), plus a few larger amd64/Jetson-class nodes with shared GPUs (titan-20/21, titan-22, titan-24) and one large amd64 node (titan-23, 48 vCPU/256GB, no GPU, not labeled worker). This is not elastic cloud capacity — headroom is finite and already uneven:

Node	CPU used/alloc	Mem used/alloc	Note
titan-12	3.95 / 3.60 (110%)	5.65 / 6.50 (87%)	rpi4, already over its CPU allocatable at snapshot time
titan-13	3.45 / 3.60 (96%)	5.85 / 6.50 (90%)	rpi4, hot
titan-17	2.71 / 3.60 (75%)	5.54 / 6.50 (85%)	rpi4, hot on memory
titan-07	1.23 / 3.60 (34%)	5.24 / 6.63 (79%)	rpi5, memory-constrained
titan-04	0.35 / 3.60 (10%)	1.69 / 6.63 (26%)	rpi5, most headroom in the pool
titan-21 (voice, GPU)	0.91 / 6.00 (15%)	8.02 / 14.56 (55%)	shared GPU node, moderate headroom
titan-23 (48 vCPU/256GB, no GPU label)	0.74 / 48 (1.5%)	22.9 / 251.6 (9%)	large headroom, not currently used by hermes affinity rules (arm64-only selectors exclude it)
titan-24 (GPU, Wolf/Hermes shared)	1.16 / 24 (4.8%)	22.6 / 62.7 (36%)	model-gate/Ollama contention with Wolf, per NOTES.md

chat-statefulset.yaml's node affinity (lines 54-74) requires arm64 + node-role.kubernetes.io/worker=true and already excludes 7 specific hostnames (titan-05,08,13,14,17,18,19) — but two of the nodes it currently can still land on, titan-12 and titan-15, are already running at 75-110% of allocatable CPU. Any additional tenant/broker/router replica scheduled into this same arm64 worker pool inherits that contention; it is not safe to assume "add a replica" has free capacity behind it without checking node headroom at scale-out time. The large amd64 node (titan-23) has abundant headroom but is excluded by the current arm64-only affinity rules on every hermes component — worth a design question, not something this assessment changes.

2.5 No staging/canary environment exists

services/hermes/kustomization.yaml is a single flat kustomization with no overlay split. services/hermes-chat/ contains only a namespace and one PVC (vestigial, no workload). services/hermes-triage-demo/ is unrelated (Ariadne's automated-repair demo loop, no chat workload). Any load test that exercises the real chat path runs against the production namespace and the production hermes-chat-tenant StatefulSet — there is no isolated copy to test against.

2.6 No load-test tooling and thin production metrics exist today

• No k6/locust/vegeta/wrk/artillery/ab scripts exist anywhere in the repo for hermes. A load test has to be built from scratch.
• No ServiceMonitor/PodMonitor/PrometheusRule objects exist for hermes (annotation-based scraping only). /metrics exists for the hermes-agent pod (CLI/provider quota usage, port 9010), hermes-switchyard (port 9005, the richest signal — switchyard_requests_total, switchyard_model_call_latency_ms_bucket, switchyard_routing_overhead_ms_bucket, switchyard_errors_total, switchyard_client_responses_total), and the CLI-lane metrics service (port 9011, agent/Kanban-lane specific). No /metrics endpoint exists on hermes-chat-tenant, hermes-model-gate, oauth2-proxy, hermes-chat-router, or voice — there is no per-request latency/error signal from the actual multi-user chat pods themselves.
• services/monitoring/grafana-dashboard-ai.yaml already has a P95 model-call-latency panel, a success/error-rate panel, and a per-pod CPU panel filtered to hermes-(agent|chat-tenant|switchyard|model-gate).*, all sourced from Switchyard metrics + cAdvisor — a usable baseline to reuse, but nothing is wired to an alert threshold, and there is no queue-depth or replica-health panel.

3. Unknowns (require an actual test to answer)

• Real max throughput of hermes-switchyard, hermes-model-gate -> Ollama, and the STT/TTS GPU path under concurrent load — no data exists today.
• Real behavior/latency of the Claude broker's semaphore-2 queue under contention (does it degrade gracefully to slow responses, or eventually time out and error?) — no data exists today.
• Whether oauth2-proxy + Redis session store can handle concurrent logins/refreshes at a higher user count (currently untested; the Deployment is stateless-safe to scale, but has never been scaled). Also whether Redis itself (hermes-oauth-sessions, 1Gi PVC, no HA) becomes a bottleneck or SPOF under more simultaneous sessions.
• Actual current pod-level CPU/memory usage and restart history for any hermes component — not observable with the current RBAC grant. Only node-level aggregate usage is visible, not which pods are consuming it.
• Whether scaling hermes-chat-tenant toward its already-configured TENANT_SLOTS=8 (after fixing the mismatch) is actually safe given current node headroom (titan-12/titan-15 pressure, see 2.4) — needs a scheduling dry-run / headroom check at decision time, not assumed from this snapshot.

4. SLO and load-test proposal (not yet executed — needs approval, see §6)

4.1 Target SLOs to validate (proposed, for review)

Metric	Proposed target	Source
P95 end-to-end chat response latency	< 8s for local/Codex-routed, < 20s for Claude-routed (queued behind semaphore-2)	new synthetic client timer + existing switchyard_model_call_latency_ms_bucket
Error rate (5xx / proxy failures)	< 1% sustained	new synthetic client + switchyard_errors_total / switchyard_client_responses_total
Login/session success rate	100% (oauth2-proxy + Redis)	synthetic client HTTP status
Node CPU/mem headroom during test	stay below 90% allocatable on any node hosting hermes pods	nodes.metrics.k8s.io (already provably readable)
Claude-broker queue depth / wait time	document actual behavior, no target yet (unknown baseline)	needs new instrumentation — none exists today

4.2 Concurrency / ramp / mix proposal

• Concurrency levels: 1 (baseline), 2 (at the Claude-broker cap), 4 (at today's real tenant-pool ceiling), and — only if §2.1's mismatch is resolved and node headroom is re-checked — 8 (at the configured TENANT_SLOTS). Do not exceed the number of actually-backed tenant slots.
• Ramp: hold each level for at least 10 minutes after reaching steady state before increasing, to separate cold-start effects (webui/gateway startup, model warm-up) from steady-state capacity.
• Traffic mix: proportion requests across the three model routes (Claude / Codex / local Ollama) matching real usage if known, otherwise test each route in isolation first (to characterize each bottleneck independently — semaphore-2 vs. no-cap-but-untested), then a blended mix.
• Session behavior: each synthetic user must be a distinct, dedicated test Keycloak identity (slot assignment is permanent — do not burn real user slots or reuse production accounts for load generation).
• Failure injection (optional, staged): kill the single hermes-switchyard or hermes-model-gate pod mid-test to confirm the Recreate/no-PDB SPOF behavior matches expectations and measure recovery time, only after the throughput characterization above is complete.

5. Safe staged scaling plan (for future execution, after human approval)

This is a plan to evaluate, not a plan already executed.

1. Fix the TENANT_SLOTS/replica-count mismatch first (§2.1). This is a correctness fix at the current capacity, not a capacity increase, and should land before any load test so results aren't confounded by the existing 502-on-phantom-slot bug.
2. Instrument before scaling: add /metrics (or at minimum structured access logs) to hermes-chat-tenant, hermes-chat-router, and hermes-model-gate so a load test has a real per-request signal instead of relying solely on Switchyard's aggregate view.
3. Baseline test at today's real ceiling (4 users), off-hours, using dedicated test Keycloak identities, per §4.
4. Re-check node headroom (§2.4) at decision time, not from this snapshot, before considering any StatefulSet scale-out — titan-12 and titan-15 were already at/above CPU allocatable during this assessment.
5. If and only if the baseline is healthy and headroom allows, propose scaling hermes-chat-tenant toward the already-configured 8 slots one or two ordinals at a time (StatefulSet podManagementPolicy: Parallel supports this), re-measuring node headroom and Switchyard/broker latency after each step.
6. Address the shared-broker ceiling separately from tenant-pod count: raising HERMES_CLAUDE_BROKER_CONCURRENCY above 2 is a distinct decision gated by the underlying Claude Code provider's real rate limits, not by Kubernetes resources — do not conflate it with StatefulSet scaling.
7. Only after tenant/broker capacity is validated, consider whether the single-replica shared components (hermes-switchyard, hermes-model-gate, oauth2-proxy) need a second replica or a PDB — note oauth2-proxy is the only one of these that is currently stateless-safe to scale (Redis-backed sessions); the others would need code changes first (in-process locks/ state noted in §2).

6. Rollback and observability gates

• Every step above is a Git-reviewed manifest change applied via Flux, so rollback is git revert + Flux reconcile, consistent with this repo's existing GitOps model — no manual kubectl edit in any step.
• Gate each scaling step on: node CPU/mem headroom (§2.4 thresholds), the new /metrics signal from step 2 above staying within the SLOs in §4.1, and zero increase in switchyard_errors_total / switchyard_classifier_fail_open_total rate versus the pre-change baseline.
• Because there is no staging environment (§2.5) and no PDB, treat every step as a production change: schedule off-hours, announce before/after, and keep the prior manifest revision ready to revert immediately if error rate or node pressure crosses the gate.
• Because RBAC here is observer-only (§0), whoever executes a future load test needs either elevated read access to confirm live pod/replica status during the test, or must rely entirely on the node-metrics + Switchyard signals already proven readable in this assessment.

7. Recommendation

A load test against the current 4-user ceiling is reasonably safe to run off-hours with dedicated test accounts, once §5 step 1 (the TENANT_SLOTS mismatch) and step 2 (baseline metrics) are addressed — it does not require any replica/cap increase, and the blast radius is bounded to resources already provisioned.

A test at 8 concurrent users, or any StatefulSet/broker scale-up, is not yet safe to schedule. Two of the nodes in the current chat-tenant affinity pool are already at or above their allocatable CPU/memory (§2.4), there is no PDB/HPA anywhere in this service, there is no staging environment to absorb the risk, and the real throughput ceiling for Codex/local-model routes and the actual behavior of the Claude-broker queue under load are both unmeasured. Recommend running the 4-user baseline test first and using its results (plus a fresh node-headroom check) to decide whether 8-user scaling is warranted — this assessment does not make that call.

8. Requested input / blockers to proceed further

This assessment can be delivered without further input. To actually execute the load test proposed in §4, a human owner needs to provide or approve:

1. Sign-off to fix the TENANT_SLOTS/replica mismatch (§2.1) as a correctness PR, separate from any capacity increase.
2. Dedicated test Keycloak identities (slot assignment is permanent — production user accounts must not be used for load generation).
3. An approved off-hours maintenance window, since there is no staging environment and every step in §5 is a production change.
4. A decision on whether to add /metrics instrumentation (§5 step 2) before or in parallel with the first baseline test, since today's Switchyard-only signal cannot attribute latency/errors to a specific tenant pod.
5. If deeper live verification is wanted beyond node-level metrics (e.g., live pod status/restart counts during a test), temporary elevated read RBAC for the executing identity, since the current atlas-observer grant cannot list pods/deployments in hermes (§0).