atlas-iac/docs/hermes_multiuser_capacity_assessment.md

# Hermes multi-user capacity assessment

Status: read-only investigation, no production changes made. This document
answers "can Hermes safely handle more simultaneous users" with evidence from
the current `titan-iac` manifests/code, plus non-invasive live cluster reads.
It does not increase any replica count or concurrency cap.

## 0. Method and access boundary

- Static evidence: `services/hermes/**` manifests and `services/hermes/router/*.go`,
  `services/hermes/scripts/*.py` on branch `wt/t_65356568`, `git log`/`git blame`
  for recent drift.
- Live evidence: read via the in-cluster `atlas-observer` kubeconfig context.
  This identity (`system:serviceaccount:hermes:hermes-agent`) is **observer-only**:
  `kubectl -n hermes get pods/deploy/sts/hpa/pdb` all returned `Forbidden`.
  `kubectl -n hermes auth can-i --list` confirms the only granted verbs are
  cluster-scoped `get/list/watch` on `namespaces`, `nodes`, `persistentvolumes`,
  storage APIs, and `nodes.metrics.k8s.io`, plus API discovery/health endpoints.
  There is no namespace-scoped read access to pods, deployments, replica counts,
  or pod-level resource usage in `hermes` (or any namespace) from this session.
  This matches the standing finding that hermes-agent RBAC is observer-only.
- What this means for this report: **all replica counts, resource
  requests/limits, probes, and affinity rules below are read from the Git
  manifests** (which is what Flux actually reconciles, so it is a trustworthy
  source for desired state) rather than confirmed against live pod status.
  **Node-level capacity and current node CPU/memory utilization are live,
  real-time cluster evidence** (`nodes.metrics.k8s.io` + `nodes` are both
  readable). One non-invasive HTTP probe against `https://chat.bstein.dev/`
  confirmed the public endpoint is live and redirecting to auth (`302`) at
  the time of writing.
- No pods, deployments, config, or replica counts were changed. No load test
  was executed against any environment.

## 1. Architecture and request path

```
browser
  -> Traefik Ingress (chat.hermes.bstein.dev / chat.bstein.dev)
  -> oauth2-proxy-hermes-chat  (Deployment, replicas: 1, oauth2-proxy.yaml:153)
       session state: Redis, externalized  (--session-store-type=redis,
       redis://hermes-oauth-sessions...:6379/1, oauth2-proxy.yaml:203-204)
  -> hermes-chat-router  (Deployment, replicas: 1, chat-router.yaml:10)
       Go binary, services/hermes/router/main.go
       in-process, PVC-persisted identity -> slot map (tenants.json)
  -> hermes-chat-tenant-<slot>.hermes-chat-tenant  (StatefulSet, replicas: 4,
       chat-statefulset.yaml:13) — one pod per Keycloak subject, pinned by ordinal
       webui (8787) -> gateway (8642) -> per-tenant sandbox (chat-sandbox-<ordinal>:9080)
  -> hermes-switchyard  (Deployment, replicas: 1, switchyard-deployment.yaml:10)
       shared model router for ALL tenants + triage + Kanban
  -> hermes-claude-broker / hermes-codex-broker (containers in hermes-agent pod,
       replicas: 1, agent-deployment.yaml:10) / hermes-model-gate (Deployment,
       replicas: 1, model-gate-deployment.yaml:10, proxies to local Ollama)
  -> hermes-stt / hermes-tts  (Deployments, replicas: 1 each, voice-deployment.yaml,
       pinned to node titan-21, shared GPU)
```

Every component on this path is `replicas: 1` **except** `hermes-chat-tenant`
(4). The StatefulSet is not a load-balanced pool: each pod is a fully isolated,
permanently-assigned per-user backend, not interchangeable capacity.

## 2. Known limits (confirmed from manifests/code)

### 2.1 Hard ceiling: 4 usable chat users today, and a live configuration bug

`services/hermes/router/main.go` (`tenantRouter.slotFor`, lines 153-176)
permanently assigns each Keycloak subject a slot `0..TENANT_SLOTS-1` on first
login, persists it to a PVC-backed JSON file, and never releases it. There is
no unassign/deprovision path in the router. Unit tests
(`router/main_test.go`, "expected the fixed private tenant pool to report
capacity") assert that once all slots are taken, `slotFor` returns an error.

**Confirmed drift**: `chat-router.yaml:70` sets `TENANT_SLOTS=8`, but
`chat-statefulset.yaml:13` sets `replicas: 4`. `git log -p` on both files
shows they were changed together at `18965a2f` (4->8), then
`chat-statefulset.yaml` was scaled down twice — `79baa7ec` ("return two chat
slots to fit the agent runtime", 8->6) and `e28b32bd` (6->4) — **without a
matching change to `TENANT_SLOTS`**, and `chat-sandbox.yaml` still defines 8
per-ordinal sandbox Deployments (`hermes-chat-sandbox-0` through `-7`).
Net effect: the 5th–8th distinct Keycloak login is durably assigned a slot
that has no backing StatefulSet pod. `main.go`'s proxy error path (around
line 382) returns a "starting" error to that user **permanently**, not just
during a cold start, because the pod will never exist at the current replica
count. This is a pre-existing configuration bug, independent of any future
scale-up decision, and it silently masks the real behavior of the capacity
ceiling (a 5th user does not get a clean "at capacity" rejection matching the
tested behavior — they get a StatefulSet-pod-not-found style failure).
**This assessment does not fix it** (per scope: no production changes), but
it should be the first fix considered before any scale-out, since it affects
correctness at the current replica count too.

### 2.2 The real shared bottleneck is not the 4 tenant pods, it's the brokers behind them

All 4 tenant pods, the triage/operator instance, and Kanban/CLI-lane traffic
funnel through the same single-replica `hermes-switchyard`, which routes to
the same single-replica `hermes-claude-broker` / `hermes-codex-broker`
(containers inside the one `hermes-agent` pod) or `hermes-model-gate`.

- `hermes-claude-broker` enforces a hard, code-level concurrency cap:
  `HERMES_CLAUDE_BROKER_CONCURRENCY=2` (`agent-deployment.yaml:1160`) sizes a
  `threading.BoundedSemaphore(2)` (`scripts/claude_oauth_broker.py:63`)
  wrapped around a blocking `subprocess.run` call to the native Claude Code
  CLI. Requests over the cap **block silently** (no 429/503 to the caller)
  until a slot frees. This cap is shared cluster-wide across every chat
  tenant, the triage instance, and indirectly the Kanban/CLI-lane workers
  (same provider credentials, so heavy CLI-lane usage can throttle chat
  responses via the provider's own rate limits even without touching this
  semaphore directly).
- `hermes-codex-broker` and `hermes-model-gate` have **no equivalent
  concurrency cap in code** — they are bounded only by container CPU/memory
  `limits` (1 CPU / 1Gi for the codex-broker container, 250m/128Mi for
  model-gate) and by the single upstream Ollama/Jetson instance's real
  throughput, which is unknown (no load test exists).

**Implication**: scaling `hermes-chat-tenant` replicas alone (e.g. 4 -> 8)
does not proportionally increase real AI-response throughput. The
Claude-routed share of traffic is capped at 2 concurrent completions
regardless of how many tenant pods exist, and Codex/local-model throughput
under concurrent load is untested.

### 2.3 Single points of failure and deploy-time outage windows

- No `HorizontalPodAutoscaler` exists anywhere in this repository (confirmed
  by a repo-wide grep, not just `services/hermes/`).
- No `PodDisruptionBudget` exists for any hermes component (the one PDB in
  the repo is `infrastructure/traefik/pdb.yaml`, unrelated).
- `oauth2-proxy`, `hermes-chat-router`, `hermes-switchyard`,
  `hermes-model-gate`, `hermes-agent` (brokers), and voice STT/TTS are all
  `replicas: 1`. A pod restart/crash/node drain on any of them takes down
  chat, voice, or model access for **all** users simultaneously, even though
  the tenant StatefulSet gives each user an isolated conversation backend.
- `hermes` (main/triage), `hermes-chat-router`, `hermes-local-image`, and
  `hermes-agent` all use `strategy: Recreate` — every rollout is a full
  teardown-then-recreate, not a rolling update, adding a real availability
  gap bounded by readiness/liveness `initialDelaySeconds` (up to 90s+ for the
  main `hermes` deployment). The only genuinely zero-downtime rollout path
  in this whole service is `hermes-switchyard`
  (`RollingUpdate`, `maxSurge:1`, `maxUnavailable:0`) and the tenant
  StatefulSet itself (`RollingUpdate`, `podManagementPolicy: Parallel`).
- OAuth session storage (`hermes-oauth-sessions`, a single Redis
  `Deployment`, `replicas: 1`, `oauth-session-store.yaml`) has no
  replication; its loss logs out every user cluster-wide (sessions are
  externalized from oauth2-proxy but not made highly available).

### 2.4 The cluster is a heterogeneous home-lab/edge fleet with real, uneven headroom

Live `nodes.metrics.k8s.io` reads (non-invasive) show 21 nodes: mostly
Raspberry Pi 4/5 workers (~3.6-4 vCPU / ~6.5-8GB allocatable each), plus a
few larger amd64/Jetson-class nodes with shared GPUs (titan-20/21, titan-22,
titan-24) and one large amd64 node (titan-23, 48 vCPU/256GB, no GPU, not
labeled `worker`). This is **not** elastic cloud capacity — headroom is
finite and already uneven:

| Node | CPU used/alloc | Mem used/alloc | Note |
|---|---|---|---|
| titan-12 | 3.95 / 3.60 (**110%**) | 5.65 / 6.50 (87%) | rpi4, already over its CPU allocatable at snapshot time |
| titan-13 | 3.45 / 3.60 (96%) | 5.85 / 6.50 (90%) | rpi4, hot |
| titan-17 | 2.71 / 3.60 (75%) | 5.54 / 6.50 (85%) | rpi4, hot on memory |
| titan-07 | 1.23 / 3.60 (34%) | 5.24 / 6.63 (79%) | rpi5, memory-constrained |
| titan-04 | 0.35 / 3.60 (10%) | 1.69 / 6.63 (26%) | rpi5, most headroom in the pool |
| titan-21 (voice, GPU) | 0.91 / 6.00 (15%) | 8.02 / 14.56 (55%) | shared GPU node, moderate headroom |
| titan-23 (48 vCPU/256GB, no GPU label) | 0.74 / 48 (1.5%) | 22.9 / 251.6 (9%) | large headroom, not currently used by hermes affinity rules (arm64-only selectors exclude it) |
| titan-24 (GPU, Wolf/Hermes shared) | 1.16 / 24 (4.8%) | 22.6 / 62.7 (36%) | model-gate/Ollama contention with Wolf, per NOTES.md |

`chat-statefulset.yaml`'s node affinity (lines 54-74) requires `arm64` +
`node-role.kubernetes.io/worker=true` and already excludes 7 specific
hostnames (`titan-05,08,13,14,17,18,19`) — but two of the nodes it currently
*can* still land on, **titan-12 and titan-15, are already running at 75-110%
of allocatable CPU**. Any additional tenant/broker/router replica scheduled
into this same arm64 worker pool inherits that contention; it is not safe to
assume "add a replica" has free capacity behind it without checking node
headroom at scale-out time. The large amd64 node (titan-23) has abundant
headroom but is excluded by the current `arm64`-only affinity rules on every
hermes component — worth a design question, not something this assessment
changes.

### 2.5 No staging/canary environment exists

`services/hermes/kustomization.yaml` is a single flat kustomization with no
overlay split. `services/hermes-chat/` contains only a namespace and one PVC
(vestigial, no workload). `services/hermes-triage-demo/` is unrelated
(Ariadne's automated-repair demo loop, no chat workload). **Any load test
that exercises the real chat path runs against the production namespace and
the production `hermes-chat-tenant` StatefulSet** — there is no isolated
copy to test against.

### 2.6 No load-test tooling and thin production metrics exist today

- No k6/locust/vegeta/wrk/artillery/ab scripts exist anywhere in the repo
  for hermes. A load test has to be built from scratch.
- No `ServiceMonitor`/`PodMonitor`/`PrometheusRule` objects exist for hermes
  (annotation-based scraping only). `/metrics` exists for the `hermes-agent`
  pod (CLI/provider quota usage, port 9010), `hermes-switchyard` (port 9005,
  the richest signal — `switchyard_requests_total`,
  `switchyard_model_call_latency_ms_bucket`,
  `switchyard_routing_overhead_ms_bucket`, `switchyard_errors_total`,
  `switchyard_client_responses_total`), and the CLI-lane metrics service
  (port 9011, agent/Kanban-lane specific). **No `/metrics` endpoint exists
  on `hermes-chat-tenant`, `hermes-model-gate`, `oauth2-proxy`,
  `hermes-chat-router`, or voice** — there is no per-request latency/error
  signal from the actual multi-user chat pods themselves.
- `services/monitoring/grafana-dashboard-ai.yaml` already has a P95
  model-call-latency panel, a success/error-rate panel, and a per-pod CPU
  panel filtered to `hermes-(agent|chat-tenant|switchyard|model-gate).*`,
  all sourced from Switchyard metrics + cAdvisor — a usable baseline to
  reuse, but nothing is wired to an alert threshold, and there is no
  queue-depth or replica-health panel.

## 3. Unknowns (require an actual test to answer)

- Real max throughput of `hermes-switchyard`, `hermes-model-gate` -> Ollama,
  and the STT/TTS GPU path under concurrent load — no data exists today.
- Real behavior/latency of the Claude broker's semaphore-2 queue under
  contention (does it degrade gracefully to slow responses, or eventually
  time out and error?) — no data exists today.
- Whether oauth2-proxy + Redis session store can handle concurrent
  logins/refreshes at a higher user count (currently untested; the
  Deployment is stateless-safe to scale, but has never been scaled).
  Also whether Redis itself (`hermes-oauth-sessions`, 1Gi PVC, no HA)
  becomes a bottleneck or SPOF under more simultaneous sessions.
- Actual current pod-level CPU/memory usage and restart history for any
  hermes component — not observable with the current RBAC grant. Only
  node-level aggregate usage is visible, not which pods are consuming it.
- Whether scaling `hermes-chat-tenant` toward its already-configured
  `TENANT_SLOTS=8` (after fixing the mismatch) is actually safe given
  current node headroom (titan-12/titan-15 pressure, see 2.4) — needs a
  scheduling dry-run / headroom check at decision time, not assumed from
  this snapshot.

## 4. SLO and load-test proposal (not yet executed — needs approval, see §6)

### 4.1 Target SLOs to validate (proposed, for review)

| Metric | Proposed target | Source |
|---|---|---|
| P95 end-to-end chat response latency | < 8s for local/Codex-routed, < 20s for Claude-routed (queued behind semaphore-2) | new synthetic client timer + existing `switchyard_model_call_latency_ms_bucket` |
| Error rate (5xx / proxy failures) | < 1% sustained | new synthetic client + `switchyard_errors_total` / `switchyard_client_responses_total` |
| Login/session success rate | 100% (oauth2-proxy + Redis) | synthetic client HTTP status |
| Node CPU/mem headroom during test | stay below 90% allocatable on any node hosting hermes pods | `nodes.metrics.k8s.io` (already provably readable) |
| Claude-broker queue depth / wait time | document actual behavior, no target yet (unknown baseline) | needs new instrumentation — none exists today |

### 4.2 Concurrency / ramp / mix proposal

- **Concurrency levels**: 1 (baseline), 2 (at the Claude-broker cap), 4
  (at today's real tenant-pool ceiling), and — only if §2.1's mismatch is
  resolved and node headroom is re-checked — 8 (at the configured
  `TENANT_SLOTS`). Do not exceed the number of actually-backed tenant slots.
- **Ramp**: hold each level for at least 10 minutes after reaching steady
  state before increasing, to separate cold-start effects (webui/gateway
  startup, model warm-up) from steady-state capacity.
- **Traffic mix**: proportion requests across the three model routes
  (Claude / Codex / local Ollama) matching real usage if known, otherwise
  test each route in isolation first (to characterize each bottleneck
  independently — semaphore-2 vs. no-cap-but-untested), then a blended mix.
- **Session behavior**: each synthetic user must be a distinct, dedicated
  test Keycloak identity (slot assignment is permanent — do not burn real
  user slots or reuse production accounts for load generation).
- **Failure injection (optional, staged)**: kill the single
  `hermes-switchyard` or `hermes-model-gate` pod mid-test to confirm the
  `Recreate`/no-PDB SPOF behavior matches expectations and measure recovery
  time, only after the throughput characterization above is complete.

## 5. Safe staged scaling plan (for future execution, after human approval)

This is a plan to evaluate, not a plan already executed.

1. **Fix the `TENANT_SLOTS`/replica-count mismatch first** (§2.1). This is a
   correctness fix at the *current* capacity, not a capacity increase, and
   should land before any load test so results aren't confounded by the
   existing 502-on-phantom-slot bug.
2. **Instrument before scaling**: add `/metrics` (or at minimum structured
   access logs) to `hermes-chat-tenant`, `hermes-chat-router`, and
   `hermes-model-gate` so a load test has a real per-request signal instead
   of relying solely on Switchyard's aggregate view.
3. **Baseline test at today's real ceiling (4 users)**, off-hours, using
   dedicated test Keycloak identities, per §4.
4. **Re-check node headroom** (§2.4) at decision time, not from this
   snapshot, before considering any StatefulSet scale-out — titan-12 and
   titan-15 were already at/above CPU allocatable during this assessment.
5. **If and only if** the baseline is healthy and headroom allows, propose
   scaling `hermes-chat-tenant` toward the already-configured 8 slots one
   or two ordinals at a time (StatefulSet `podManagementPolicy: Parallel`
   supports this), re-measuring node headroom and Switchyard/broker latency
   after each step.
6. **Address the shared-broker ceiling separately from tenant-pod count**:
   raising `HERMES_CLAUDE_BROKER_CONCURRENCY` above 2 is a distinct decision
   gated by the underlying Claude Code provider's real rate limits, not by
   Kubernetes resources — do not conflate it with StatefulSet scaling.
7. **Only after tenant/broker capacity is validated**, consider whether the
   single-replica shared components (`hermes-switchyard`, `hermes-model-gate`,
   `oauth2-proxy`) need a second replica or a PDB — note oauth2-proxy is the
   only one of these that is currently stateless-safe to scale (Redis-backed
   sessions); the others would need code changes first (in-process locks/
   state noted in §2).

## 6. Rollback and observability gates

- Every step above is a Git-reviewed manifest change applied via Flux, so
  rollback is `git revert` + Flux reconcile, consistent with this repo's
  existing GitOps model — no manual `kubectl edit` in any step.
- Gate each scaling step on: node CPU/mem headroom (§2.4 thresholds), the
  new `/metrics` signal from step 2 above staying within the SLOs in §4.1,
  and zero increase in `switchyard_errors_total` / `switchyard_classifier_fail_open_total`
  rate versus the pre-change baseline.
- Because there is no staging environment (§2.5) and no PDB, treat every
  step as a production change: schedule off-hours, announce before/after,
  and keep the prior manifest revision ready to revert immediately if error
  rate or node pressure crosses the gate.
- Because RBAC here is observer-only (§0), whoever executes a future load
  test needs either elevated read access to confirm live pod/replica status
  during the test, or must rely entirely on the node-metrics + Switchyard
  signals already proven readable in this assessment.

## 7. Recommendation

**A load test against the current 4-user ceiling is reasonably safe to run
off-hours with dedicated test accounts, once §5 step 1 (the `TENANT_SLOTS`
mismatch) and step 2 (baseline metrics) are addressed** — it does not
require any replica/cap increase, and the blast radius is bounded to
resources already provisioned.

**A test at 8 concurrent users, or any StatefulSet/broker scale-up, is not
yet safe to schedule.** Two of the nodes in the current chat-tenant affinity
pool are already at or above their allocatable CPU/memory (§2.4), there is
no PDB/HPA anywhere in this service, there is no staging environment to
absorb the risk, and the real throughput ceiling for Codex/local-model
routes and the actual behavior of the Claude-broker queue under load are
both unmeasured. Recommend running the 4-user baseline test first and using
its results (plus a fresh node-headroom check) to decide whether 8-user
scaling is warranted — this assessment does not make that call.

## 8. Requested input / blockers to proceed further

This assessment can be delivered without further input. To actually execute
the load test proposed in §4, a human owner needs to provide or approve:

1. Sign-off to fix the `TENANT_SLOTS`/replica mismatch (§2.1) as a
   correctness PR, separate from any capacity increase.
2. Dedicated test Keycloak identities (slot assignment is permanent —
   production user accounts must not be used for load generation).
3. An approved off-hours maintenance window, since there is no staging
   environment and every step in §5 is a production change.
4. A decision on whether to add `/metrics` instrumentation (§5 step 2)
   before or in parallel with the first baseline test, since today's
   Switchyard-only signal cannot attribute latency/errors to a specific
   tenant pod.
5. If deeper live verification is wanted beyond node-level metrics
   (e.g., live pod status/restart counts during a test), temporary
   elevated read RBAC for the executing identity, since the current
   `atlas-observer` grant cannot list pods/deployments in `hermes` (§0).