diff --git a/docs/hermes_multiuser_capacity_assessment.md b/docs/hermes_multiuser_capacity_assessment.md new file mode 100644 index 00000000..cefc527c --- /dev/null +++ b/docs/hermes_multiuser_capacity_assessment.md @@ -0,0 +1,349 @@ +# Hermes multi-user capacity assessment + +Status: read-only investigation, no production changes made. This document +answers "can Hermes safely handle more simultaneous users" with evidence from +the current `titan-iac` manifests/code, plus non-invasive live cluster reads. +It does not increase any replica count or concurrency cap. + +## 0. Method and access boundary + +- Static evidence: `services/hermes/**` manifests and `services/hermes/router/*.go`, + `services/hermes/scripts/*.py` on branch `wt/t_65356568`, `git log`/`git blame` + for recent drift. +- Live evidence: read via the in-cluster `atlas-observer` kubeconfig context. + This identity (`system:serviceaccount:hermes:hermes-agent`) is **observer-only**: + `kubectl -n hermes get pods/deploy/sts/hpa/pdb` all returned `Forbidden`. + `kubectl -n hermes auth can-i --list` confirms the only granted verbs are + cluster-scoped `get/list/watch` on `namespaces`, `nodes`, `persistentvolumes`, + storage APIs, and `nodes.metrics.k8s.io`, plus API discovery/health endpoints. + There is no namespace-scoped read access to pods, deployments, replica counts, + or pod-level resource usage in `hermes` (or any namespace) from this session. + This matches the standing finding that hermes-agent RBAC is observer-only. +- What this means for this report: **all replica counts, resource + requests/limits, probes, and affinity rules below are read from the Git + manifests** (which is what Flux actually reconciles, so it is a trustworthy + source for desired state) rather than confirmed against live pod status. + **Node-level capacity and current node CPU/memory utilization are live, + real-time cluster evidence** (`nodes.metrics.k8s.io` + `nodes` are both + readable). One non-invasive HTTP probe against `https://chat.bstein.dev/` + confirmed the public endpoint is live and redirecting to auth (`302`) at + the time of writing. +- No pods, deployments, config, or replica counts were changed. No load test + was executed against any environment. + +## 1. Architecture and request path + +``` +browser + -> Traefik Ingress (chat.hermes.bstein.dev / chat.bstein.dev) + -> oauth2-proxy-hermes-chat (Deployment, replicas: 1, oauth2-proxy.yaml:153) + session state: Redis, externalized (--session-store-type=redis, + redis://hermes-oauth-sessions...:6379/1, oauth2-proxy.yaml:203-204) + -> hermes-chat-router (Deployment, replicas: 1, chat-router.yaml:10) + Go binary, services/hermes/router/main.go + in-process, PVC-persisted identity -> slot map (tenants.json) + -> hermes-chat-tenant-.hermes-chat-tenant (StatefulSet, replicas: 4, + chat-statefulset.yaml:13) — one pod per Keycloak subject, pinned by ordinal + webui (8787) -> gateway (8642) -> per-tenant sandbox (chat-sandbox-:9080) + -> hermes-switchyard (Deployment, replicas: 1, switchyard-deployment.yaml:10) + shared model router for ALL tenants + triage + Kanban + -> hermes-claude-broker / hermes-codex-broker (containers in hermes-agent pod, + replicas: 1, agent-deployment.yaml:10) / hermes-model-gate (Deployment, + replicas: 1, model-gate-deployment.yaml:10, proxies to local Ollama) + -> hermes-stt / hermes-tts (Deployments, replicas: 1 each, voice-deployment.yaml, + pinned to node titan-21, shared GPU) +``` + +Every component on this path is `replicas: 1` **except** `hermes-chat-tenant` +(4). The StatefulSet is not a load-balanced pool: each pod is a fully isolated, +permanently-assigned per-user backend, not interchangeable capacity. + +## 2. Known limits (confirmed from manifests/code) + +### 2.1 Hard ceiling: 4 usable chat users today, and a live configuration bug + +`services/hermes/router/main.go` (`tenantRouter.slotFor`, lines 153-176) +permanently assigns each Keycloak subject a slot `0..TENANT_SLOTS-1` on first +login, persists it to a PVC-backed JSON file, and never releases it. There is +no unassign/deprovision path in the router. Unit tests +(`router/main_test.go`, "expected the fixed private tenant pool to report +capacity") assert that once all slots are taken, `slotFor` returns an error. + +**Confirmed drift**: `chat-router.yaml:70` sets `TENANT_SLOTS=8`, but +`chat-statefulset.yaml:13` sets `replicas: 4`. `git log -p` on both files +shows they were changed together at `18965a2f` (4->8), then +`chat-statefulset.yaml` was scaled down twice — `79baa7ec` ("return two chat +slots to fit the agent runtime", 8->6) and `e28b32bd` (6->4) — **without a +matching change to `TENANT_SLOTS`**, and `chat-sandbox.yaml` still defines 8 +per-ordinal sandbox Deployments (`hermes-chat-sandbox-0` through `-7`). +Net effect: the 5th–8th distinct Keycloak login is durably assigned a slot +that has no backing StatefulSet pod. `main.go`'s proxy error path (around +line 382) returns a "starting" error to that user **permanently**, not just +during a cold start, because the pod will never exist at the current replica +count. This is a pre-existing configuration bug, independent of any future +scale-up decision, and it silently masks the real behavior of the capacity +ceiling (a 5th user does not get a clean "at capacity" rejection matching the +tested behavior — they get a StatefulSet-pod-not-found style failure). +**This assessment does not fix it** (per scope: no production changes), but +it should be the first fix considered before any scale-out, since it affects +correctness at the current replica count too. + +### 2.2 The real shared bottleneck is not the 4 tenant pods, it's the brokers behind them + +All 4 tenant pods, the triage/operator instance, and Kanban/CLI-lane traffic +funnel through the same single-replica `hermes-switchyard`, which routes to +the same single-replica `hermes-claude-broker` / `hermes-codex-broker` +(containers inside the one `hermes-agent` pod) or `hermes-model-gate`. + +- `hermes-claude-broker` enforces a hard, code-level concurrency cap: + `HERMES_CLAUDE_BROKER_CONCURRENCY=2` (`agent-deployment.yaml:1160`) sizes a + `threading.BoundedSemaphore(2)` (`scripts/claude_oauth_broker.py:63`) + wrapped around a blocking `subprocess.run` call to the native Claude Code + CLI. Requests over the cap **block silently** (no 429/503 to the caller) + until a slot frees. This cap is shared cluster-wide across every chat + tenant, the triage instance, and indirectly the Kanban/CLI-lane workers + (same provider credentials, so heavy CLI-lane usage can throttle chat + responses via the provider's own rate limits even without touching this + semaphore directly). +- `hermes-codex-broker` and `hermes-model-gate` have **no equivalent + concurrency cap in code** — they are bounded only by container CPU/memory + `limits` (1 CPU / 1Gi for the codex-broker container, 250m/128Mi for + model-gate) and by the single upstream Ollama/Jetson instance's real + throughput, which is unknown (no load test exists). + +**Implication**: scaling `hermes-chat-tenant` replicas alone (e.g. 4 -> 8) +does not proportionally increase real AI-response throughput. The +Claude-routed share of traffic is capped at 2 concurrent completions +regardless of how many tenant pods exist, and Codex/local-model throughput +under concurrent load is untested. + +### 2.3 Single points of failure and deploy-time outage windows + +- No `HorizontalPodAutoscaler` exists anywhere in this repository (confirmed + by a repo-wide grep, not just `services/hermes/`). +- No `PodDisruptionBudget` exists for any hermes component (the one PDB in + the repo is `infrastructure/traefik/pdb.yaml`, unrelated). +- `oauth2-proxy`, `hermes-chat-router`, `hermes-switchyard`, + `hermes-model-gate`, `hermes-agent` (brokers), and voice STT/TTS are all + `replicas: 1`. A pod restart/crash/node drain on any of them takes down + chat, voice, or model access for **all** users simultaneously, even though + the tenant StatefulSet gives each user an isolated conversation backend. +- `hermes` (main/triage), `hermes-chat-router`, `hermes-local-image`, and + `hermes-agent` all use `strategy: Recreate` — every rollout is a full + teardown-then-recreate, not a rolling update, adding a real availability + gap bounded by readiness/liveness `initialDelaySeconds` (up to 90s+ for the + main `hermes` deployment). The only genuinely zero-downtime rollout path + in this whole service is `hermes-switchyard` + (`RollingUpdate`, `maxSurge:1`, `maxUnavailable:0`) and the tenant + StatefulSet itself (`RollingUpdate`, `podManagementPolicy: Parallel`). +- OAuth session storage (`hermes-oauth-sessions`, a single Redis + `Deployment`, `replicas: 1`, `oauth-session-store.yaml`) has no + replication; its loss logs out every user cluster-wide (sessions are + externalized from oauth2-proxy but not made highly available). + +### 2.4 The cluster is a heterogeneous home-lab/edge fleet with real, uneven headroom + +Live `nodes.metrics.k8s.io` reads (non-invasive) show 21 nodes: mostly +Raspberry Pi 4/5 workers (~3.6-4 vCPU / ~6.5-8GB allocatable each), plus a +few larger amd64/Jetson-class nodes with shared GPUs (titan-20/21, titan-22, +titan-24) and one large amd64 node (titan-23, 48 vCPU/256GB, no GPU, not +labeled `worker`). This is **not** elastic cloud capacity — headroom is +finite and already uneven: + +| Node | CPU used/alloc | Mem used/alloc | Note | +|---|---|---|---| +| titan-12 | 3.95 / 3.60 (**110%**) | 5.65 / 6.50 (87%) | rpi4, already over its CPU allocatable at snapshot time | +| titan-13 | 3.45 / 3.60 (96%) | 5.85 / 6.50 (90%) | rpi4, hot | +| titan-17 | 2.71 / 3.60 (75%) | 5.54 / 6.50 (85%) | rpi4, hot on memory | +| titan-07 | 1.23 / 3.60 (34%) | 5.24 / 6.63 (79%) | rpi5, memory-constrained | +| titan-04 | 0.35 / 3.60 (10%) | 1.69 / 6.63 (26%) | rpi5, most headroom in the pool | +| titan-21 (voice, GPU) | 0.91 / 6.00 (15%) | 8.02 / 14.56 (55%) | shared GPU node, moderate headroom | +| titan-23 (48 vCPU/256GB, no GPU label) | 0.74 / 48 (1.5%) | 22.9 / 251.6 (9%) | large headroom, not currently used by hermes affinity rules (arm64-only selectors exclude it) | +| titan-24 (GPU, Wolf/Hermes shared) | 1.16 / 24 (4.8%) | 22.6 / 62.7 (36%) | model-gate/Ollama contention with Wolf, per NOTES.md | + +`chat-statefulset.yaml`'s node affinity (lines 54-74) requires `arm64` + +`node-role.kubernetes.io/worker=true` and already excludes 7 specific +hostnames (`titan-05,08,13,14,17,18,19`) — but two of the nodes it currently +*can* still land on, **titan-12 and titan-15, are already running at 75-110% +of allocatable CPU**. Any additional tenant/broker/router replica scheduled +into this same arm64 worker pool inherits that contention; it is not safe to +assume "add a replica" has free capacity behind it without checking node +headroom at scale-out time. The large amd64 node (titan-23) has abundant +headroom but is excluded by the current `arm64`-only affinity rules on every +hermes component — worth a design question, not something this assessment +changes. + +### 2.5 No staging/canary environment exists + +`services/hermes/kustomization.yaml` is a single flat kustomization with no +overlay split. `services/hermes-chat/` contains only a namespace and one PVC +(vestigial, no workload). `services/hermes-triage-demo/` is unrelated +(Ariadne's automated-repair demo loop, no chat workload). **Any load test +that exercises the real chat path runs against the production namespace and +the production `hermes-chat-tenant` StatefulSet** — there is no isolated +copy to test against. + +### 2.6 No load-test tooling and thin production metrics exist today + +- No k6/locust/vegeta/wrk/artillery/ab scripts exist anywhere in the repo + for hermes. A load test has to be built from scratch. +- No `ServiceMonitor`/`PodMonitor`/`PrometheusRule` objects exist for hermes + (annotation-based scraping only). `/metrics` exists for the `hermes-agent` + pod (CLI/provider quota usage, port 9010), `hermes-switchyard` (port 9005, + the richest signal — `switchyard_requests_total`, + `switchyard_model_call_latency_ms_bucket`, + `switchyard_routing_overhead_ms_bucket`, `switchyard_errors_total`, + `switchyard_client_responses_total`), and the CLI-lane metrics service + (port 9011, agent/Kanban-lane specific). **No `/metrics` endpoint exists + on `hermes-chat-tenant`, `hermes-model-gate`, `oauth2-proxy`, + `hermes-chat-router`, or voice** — there is no per-request latency/error + signal from the actual multi-user chat pods themselves. +- `services/monitoring/grafana-dashboard-ai.yaml` already has a P95 + model-call-latency panel, a success/error-rate panel, and a per-pod CPU + panel filtered to `hermes-(agent|chat-tenant|switchyard|model-gate).*`, + all sourced from Switchyard metrics + cAdvisor — a usable baseline to + reuse, but nothing is wired to an alert threshold, and there is no + queue-depth or replica-health panel. + +## 3. Unknowns (require an actual test to answer) + +- Real max throughput of `hermes-switchyard`, `hermes-model-gate` -> Ollama, + and the STT/TTS GPU path under concurrent load — no data exists today. +- Real behavior/latency of the Claude broker's semaphore-2 queue under + contention (does it degrade gracefully to slow responses, or eventually + time out and error?) — no data exists today. +- Whether oauth2-proxy + Redis session store can handle concurrent + logins/refreshes at a higher user count (currently untested; the + Deployment is stateless-safe to scale, but has never been scaled). + Also whether Redis itself (`hermes-oauth-sessions`, 1Gi PVC, no HA) + becomes a bottleneck or SPOF under more simultaneous sessions. +- Actual current pod-level CPU/memory usage and restart history for any + hermes component — not observable with the current RBAC grant. Only + node-level aggregate usage is visible, not which pods are consuming it. +- Whether scaling `hermes-chat-tenant` toward its already-configured + `TENANT_SLOTS=8` (after fixing the mismatch) is actually safe given + current node headroom (titan-12/titan-15 pressure, see 2.4) — needs a + scheduling dry-run / headroom check at decision time, not assumed from + this snapshot. + +## 4. SLO and load-test proposal (not yet executed — needs approval, see §6) + +### 4.1 Target SLOs to validate (proposed, for review) + +| Metric | Proposed target | Source | +|---|---|---| +| P95 end-to-end chat response latency | < 8s for local/Codex-routed, < 20s for Claude-routed (queued behind semaphore-2) | new synthetic client timer + existing `switchyard_model_call_latency_ms_bucket` | +| Error rate (5xx / proxy failures) | < 1% sustained | new synthetic client + `switchyard_errors_total` / `switchyard_client_responses_total` | +| Login/session success rate | 100% (oauth2-proxy + Redis) | synthetic client HTTP status | +| Node CPU/mem headroom during test | stay below 90% allocatable on any node hosting hermes pods | `nodes.metrics.k8s.io` (already provably readable) | +| Claude-broker queue depth / wait time | document actual behavior, no target yet (unknown baseline) | needs new instrumentation — none exists today | + +### 4.2 Concurrency / ramp / mix proposal + +- **Concurrency levels**: 1 (baseline), 2 (at the Claude-broker cap), 4 + (at today's real tenant-pool ceiling), and — only if §2.1's mismatch is + resolved and node headroom is re-checked — 8 (at the configured + `TENANT_SLOTS`). Do not exceed the number of actually-backed tenant slots. +- **Ramp**: hold each level for at least 10 minutes after reaching steady + state before increasing, to separate cold-start effects (webui/gateway + startup, model warm-up) from steady-state capacity. +- **Traffic mix**: proportion requests across the three model routes + (Claude / Codex / local Ollama) matching real usage if known, otherwise + test each route in isolation first (to characterize each bottleneck + independently — semaphore-2 vs. no-cap-but-untested), then a blended mix. +- **Session behavior**: each synthetic user must be a distinct, dedicated + test Keycloak identity (slot assignment is permanent — do not burn real + user slots or reuse production accounts for load generation). +- **Failure injection (optional, staged)**: kill the single + `hermes-switchyard` or `hermes-model-gate` pod mid-test to confirm the + `Recreate`/no-PDB SPOF behavior matches expectations and measure recovery + time, only after the throughput characterization above is complete. + +## 5. Safe staged scaling plan (for future execution, after human approval) + +This is a plan to evaluate, not a plan already executed. + +1. **Fix the `TENANT_SLOTS`/replica-count mismatch first** (§2.1). This is a + correctness fix at the *current* capacity, not a capacity increase, and + should land before any load test so results aren't confounded by the + existing 502-on-phantom-slot bug. +2. **Instrument before scaling**: add `/metrics` (or at minimum structured + access logs) to `hermes-chat-tenant`, `hermes-chat-router`, and + `hermes-model-gate` so a load test has a real per-request signal instead + of relying solely on Switchyard's aggregate view. +3. **Baseline test at today's real ceiling (4 users)**, off-hours, using + dedicated test Keycloak identities, per §4. +4. **Re-check node headroom** (§2.4) at decision time, not from this + snapshot, before considering any StatefulSet scale-out — titan-12 and + titan-15 were already at/above CPU allocatable during this assessment. +5. **If and only if** the baseline is healthy and headroom allows, propose + scaling `hermes-chat-tenant` toward the already-configured 8 slots one + or two ordinals at a time (StatefulSet `podManagementPolicy: Parallel` + supports this), re-measuring node headroom and Switchyard/broker latency + after each step. +6. **Address the shared-broker ceiling separately from tenant-pod count**: + raising `HERMES_CLAUDE_BROKER_CONCURRENCY` above 2 is a distinct decision + gated by the underlying Claude Code provider's real rate limits, not by + Kubernetes resources — do not conflate it with StatefulSet scaling. +7. **Only after tenant/broker capacity is validated**, consider whether the + single-replica shared components (`hermes-switchyard`, `hermes-model-gate`, + `oauth2-proxy`) need a second replica or a PDB — note oauth2-proxy is the + only one of these that is currently stateless-safe to scale (Redis-backed + sessions); the others would need code changes first (in-process locks/ + state noted in §2). + +## 6. Rollback and observability gates + +- Every step above is a Git-reviewed manifest change applied via Flux, so + rollback is `git revert` + Flux reconcile, consistent with this repo's + existing GitOps model — no manual `kubectl edit` in any step. +- Gate each scaling step on: node CPU/mem headroom (§2.4 thresholds), the + new `/metrics` signal from step 2 above staying within the SLOs in §4.1, + and zero increase in `switchyard_errors_total` / `switchyard_classifier_fail_open_total` + rate versus the pre-change baseline. +- Because there is no staging environment (§2.5) and no PDB, treat every + step as a production change: schedule off-hours, announce before/after, + and keep the prior manifest revision ready to revert immediately if error + rate or node pressure crosses the gate. +- Because RBAC here is observer-only (§0), whoever executes a future load + test needs either elevated read access to confirm live pod/replica status + during the test, or must rely entirely on the node-metrics + Switchyard + signals already proven readable in this assessment. + +## 7. Recommendation + +**A load test against the current 4-user ceiling is reasonably safe to run +off-hours with dedicated test accounts, once §5 step 1 (the `TENANT_SLOTS` +mismatch) and step 2 (baseline metrics) are addressed** — it does not +require any replica/cap increase, and the blast radius is bounded to +resources already provisioned. + +**A test at 8 concurrent users, or any StatefulSet/broker scale-up, is not +yet safe to schedule.** Two of the nodes in the current chat-tenant affinity +pool are already at or above their allocatable CPU/memory (§2.4), there is +no PDB/HPA anywhere in this service, there is no staging environment to +absorb the risk, and the real throughput ceiling for Codex/local-model +routes and the actual behavior of the Claude-broker queue under load are +both unmeasured. Recommend running the 4-user baseline test first and using +its results (plus a fresh node-headroom check) to decide whether 8-user +scaling is warranted — this assessment does not make that call. + +## 8. Requested input / blockers to proceed further + +This assessment can be delivered without further input. To actually execute +the load test proposed in §4, a human owner needs to provide or approve: + +1. Sign-off to fix the `TENANT_SLOTS`/replica mismatch (§2.1) as a + correctness PR, separate from any capacity increase. +2. Dedicated test Keycloak identities (slot assignment is permanent — + production user accounts must not be used for load generation). +3. An approved off-hours maintenance window, since there is no staging + environment and every step in §5 is a production change. +4. A decision on whether to add `/metrics` instrumentation (§5 step 2) + before or in parallel with the first baseline test, since today's + Switchyard-only signal cannot attribute latency/errors to a specific + tenant pod. +5. If deeper live verification is wanted beyond node-level metrics + (e.g., live pod status/restart counts during a test), temporary + elevated read RBAC for the executing identity, since the current + `atlas-observer` grant cannot list pods/deployments in `hermes` (§0).