docs(hermes): add multi-user chat capacity assessment
Read-only investigation of the chat request path, replica/resource manifests, and live node metrics. Documents the confirmed 4-user tenant-slot ceiling, a TENANT_SLOTS=8 vs replicas=4 configuration drift, the shared Claude-broker concurrency=2 bottleneck, missing HPA/PDB/staging environment, and a proposed SLO/load-test and staged scaling plan pending human approval. No production manifests changed. Co-Authored-By: Hermes Agent <hermes-automation@bstein.dev>
This commit is contained in:
Hermes Agent
parent
2862594c62
commit
d8ccf4a6fa
349
docs/hermes_multiuser_capacity_assessment.md
Normal file
349
docs/hermes_multiuser_capacity_assessment.md
Normal file
@ -0,0 +1,349 @@
|
|||||||
|
# Hermes multi-user capacity assessment
|
||||||
|
|
||||||
|
Status: read-only investigation, no production changes made. This document
|
||||||
|
answers "can Hermes safely handle more simultaneous users" with evidence from
|
||||||
|
the current `titan-iac` manifests/code, plus non-invasive live cluster reads.
|
||||||
|
It does not increase any replica count or concurrency cap.
|
||||||
|
|
||||||
|
## 0. Method and access boundary
|
||||||
|
|
||||||
|
- Static evidence: `services/hermes/**` manifests and `services/hermes/router/*.go`,
|
||||||
|
`services/hermes/scripts/*.py` on branch `wt/t_65356568`, `git log`/`git blame`
|
||||||
|
for recent drift.
|
||||||
|
- Live evidence: read via the in-cluster `atlas-observer` kubeconfig context.
|
||||||
|
This identity (`system:serviceaccount:hermes:hermes-agent`) is **observer-only**:
|
||||||
|
`kubectl -n hermes get pods/deploy/sts/hpa/pdb` all returned `Forbidden`.
|
||||||
|
`kubectl -n hermes auth can-i --list` confirms the only granted verbs are
|
||||||
|
cluster-scoped `get/list/watch` on `namespaces`, `nodes`, `persistentvolumes`,
|
||||||
|
storage APIs, and `nodes.metrics.k8s.io`, plus API discovery/health endpoints.
|
||||||
|
There is no namespace-scoped read access to pods, deployments, replica counts,
|
||||||
|
or pod-level resource usage in `hermes` (or any namespace) from this session.
|
||||||
|
This matches the standing finding that hermes-agent RBAC is observer-only.
|
||||||
|
- What this means for this report: **all replica counts, resource
|
||||||
|
requests/limits, probes, and affinity rules below are read from the Git
|
||||||
|
manifests** (which is what Flux actually reconciles, so it is a trustworthy
|
||||||
|
source for desired state) rather than confirmed against live pod status.
|
||||||
|
**Node-level capacity and current node CPU/memory utilization are live,
|
||||||
|
real-time cluster evidence** (`nodes.metrics.k8s.io` + `nodes` are both
|
||||||
|
readable). One non-invasive HTTP probe against `https://chat.bstein.dev/`
|
||||||
|
confirmed the public endpoint is live and redirecting to auth (`302`) at
|
||||||
|
the time of writing.
|
||||||
|
- No pods, deployments, config, or replica counts were changed. No load test
|
||||||
|
was executed against any environment.
|
||||||
|
|
||||||
|
## 1. Architecture and request path
|
||||||
|
|
||||||
|
```
|
||||||
|
browser
|
||||||
|
-> Traefik Ingress (chat.hermes.bstein.dev / chat.bstein.dev)
|
||||||
|
-> oauth2-proxy-hermes-chat (Deployment, replicas: 1, oauth2-proxy.yaml:153)
|
||||||
|
session state: Redis, externalized (--session-store-type=redis,
|
||||||
|
redis://hermes-oauth-sessions...:6379/1, oauth2-proxy.yaml:203-204)
|
||||||
|
-> hermes-chat-router (Deployment, replicas: 1, chat-router.yaml:10)
|
||||||
|
Go binary, services/hermes/router/main.go
|
||||||
|
in-process, PVC-persisted identity -> slot map (tenants.json)
|
||||||
|
-> hermes-chat-tenant-<slot>.hermes-chat-tenant (StatefulSet, replicas: 4,
|
||||||
|
chat-statefulset.yaml:13) — one pod per Keycloak subject, pinned by ordinal
|
||||||
|
webui (8787) -> gateway (8642) -> per-tenant sandbox (chat-sandbox-<ordinal>:9080)
|
||||||
|
-> hermes-switchyard (Deployment, replicas: 1, switchyard-deployment.yaml:10)
|
||||||
|
shared model router for ALL tenants + triage + Kanban
|
||||||
|
-> hermes-claude-broker / hermes-codex-broker (containers in hermes-agent pod,
|
||||||
|
replicas: 1, agent-deployment.yaml:10) / hermes-model-gate (Deployment,
|
||||||
|
replicas: 1, model-gate-deployment.yaml:10, proxies to local Ollama)
|
||||||
|
-> hermes-stt / hermes-tts (Deployments, replicas: 1 each, voice-deployment.yaml,
|
||||||
|
pinned to node titan-21, shared GPU)
|
||||||
|
```
|
||||||
|
|
||||||
|
Every component on this path is `replicas: 1` **except** `hermes-chat-tenant`
|
||||||
|
(4). The StatefulSet is not a load-balanced pool: each pod is a fully isolated,
|
||||||
|
permanently-assigned per-user backend, not interchangeable capacity.
|
||||||
|
|
||||||
|
## 2. Known limits (confirmed from manifests/code)
|
||||||
|
|
||||||
|
### 2.1 Hard ceiling: 4 usable chat users today, and a live configuration bug
|
||||||
|
|
||||||
|
`services/hermes/router/main.go` (`tenantRouter.slotFor`, lines 153-176)
|
||||||
|
permanently assigns each Keycloak subject a slot `0..TENANT_SLOTS-1` on first
|
||||||
|
login, persists it to a PVC-backed JSON file, and never releases it. There is
|
||||||
|
no unassign/deprovision path in the router. Unit tests
|
||||||
|
(`router/main_test.go`, "expected the fixed private tenant pool to report
|
||||||
|
capacity") assert that once all slots are taken, `slotFor` returns an error.
|
||||||
|
|
||||||
|
**Confirmed drift**: `chat-router.yaml:70` sets `TENANT_SLOTS=8`, but
|
||||||
|
`chat-statefulset.yaml:13` sets `replicas: 4`. `git log -p` on both files
|
||||||
|
shows they were changed together at `18965a2f` (4->8), then
|
||||||
|
`chat-statefulset.yaml` was scaled down twice — `79baa7ec` ("return two chat
|
||||||
|
slots to fit the agent runtime", 8->6) and `e28b32bd` (6->4) — **without a
|
||||||
|
matching change to `TENANT_SLOTS`**, and `chat-sandbox.yaml` still defines 8
|
||||||
|
per-ordinal sandbox Deployments (`hermes-chat-sandbox-0` through `-7`).
|
||||||
|
Net effect: the 5th–8th distinct Keycloak login is durably assigned a slot
|
||||||
|
that has no backing StatefulSet pod. `main.go`'s proxy error path (around
|
||||||
|
line 382) returns a "starting" error to that user **permanently**, not just
|
||||||
|
during a cold start, because the pod will never exist at the current replica
|
||||||
|
count. This is a pre-existing configuration bug, independent of any future
|
||||||
|
scale-up decision, and it silently masks the real behavior of the capacity
|
||||||
|
ceiling (a 5th user does not get a clean "at capacity" rejection matching the
|
||||||
|
tested behavior — they get a StatefulSet-pod-not-found style failure).
|
||||||
|
**This assessment does not fix it** (per scope: no production changes), but
|
||||||
|
it should be the first fix considered before any scale-out, since it affects
|
||||||
|
correctness at the current replica count too.
|
||||||
|
|
||||||
|
### 2.2 The real shared bottleneck is not the 4 tenant pods, it's the brokers behind them
|
||||||
|
|
||||||
|
All 4 tenant pods, the triage/operator instance, and Kanban/CLI-lane traffic
|
||||||
|
funnel through the same single-replica `hermes-switchyard`, which routes to
|
||||||
|
the same single-replica `hermes-claude-broker` / `hermes-codex-broker`
|
||||||
|
(containers inside the one `hermes-agent` pod) or `hermes-model-gate`.
|
||||||
|
|
||||||
|
- `hermes-claude-broker` enforces a hard, code-level concurrency cap:
|
||||||
|
`HERMES_CLAUDE_BROKER_CONCURRENCY=2` (`agent-deployment.yaml:1160`) sizes a
|
||||||
|
`threading.BoundedSemaphore(2)` (`scripts/claude_oauth_broker.py:63`)
|
||||||
|
wrapped around a blocking `subprocess.run` call to the native Claude Code
|
||||||
|
CLI. Requests over the cap **block silently** (no 429/503 to the caller)
|
||||||
|
until a slot frees. This cap is shared cluster-wide across every chat
|
||||||
|
tenant, the triage instance, and indirectly the Kanban/CLI-lane workers
|
||||||
|
(same provider credentials, so heavy CLI-lane usage can throttle chat
|
||||||
|
responses via the provider's own rate limits even without touching this
|
||||||
|
semaphore directly).
|
||||||
|
- `hermes-codex-broker` and `hermes-model-gate` have **no equivalent
|
||||||
|
concurrency cap in code** — they are bounded only by container CPU/memory
|
||||||
|
`limits` (1 CPU / 1Gi for the codex-broker container, 250m/128Mi for
|
||||||
|
model-gate) and by the single upstream Ollama/Jetson instance's real
|
||||||
|
throughput, which is unknown (no load test exists).
|
||||||
|
|
||||||
|
**Implication**: scaling `hermes-chat-tenant` replicas alone (e.g. 4 -> 8)
|
||||||
|
does not proportionally increase real AI-response throughput. The
|
||||||
|
Claude-routed share of traffic is capped at 2 concurrent completions
|
||||||
|
regardless of how many tenant pods exist, and Codex/local-model throughput
|
||||||
|
under concurrent load is untested.
|
||||||
|
|
||||||
|
### 2.3 Single points of failure and deploy-time outage windows
|
||||||
|
|
||||||
|
- No `HorizontalPodAutoscaler` exists anywhere in this repository (confirmed
|
||||||
|
by a repo-wide grep, not just `services/hermes/`).
|
||||||
|
- No `PodDisruptionBudget` exists for any hermes component (the one PDB in
|
||||||
|
the repo is `infrastructure/traefik/pdb.yaml`, unrelated).
|
||||||
|
- `oauth2-proxy`, `hermes-chat-router`, `hermes-switchyard`,
|
||||||
|
`hermes-model-gate`, `hermes-agent` (brokers), and voice STT/TTS are all
|
||||||
|
`replicas: 1`. A pod restart/crash/node drain on any of them takes down
|
||||||
|
chat, voice, or model access for **all** users simultaneously, even though
|
||||||
|
the tenant StatefulSet gives each user an isolated conversation backend.
|
||||||
|
- `hermes` (main/triage), `hermes-chat-router`, `hermes-local-image`, and
|
||||||
|
`hermes-agent` all use `strategy: Recreate` — every rollout is a full
|
||||||
|
teardown-then-recreate, not a rolling update, adding a real availability
|
||||||
|
gap bounded by readiness/liveness `initialDelaySeconds` (up to 90s+ for the
|
||||||
|
main `hermes` deployment). The only genuinely zero-downtime rollout path
|
||||||
|
in this whole service is `hermes-switchyard`
|
||||||
|
(`RollingUpdate`, `maxSurge:1`, `maxUnavailable:0`) and the tenant
|
||||||
|
StatefulSet itself (`RollingUpdate`, `podManagementPolicy: Parallel`).
|
||||||
|
- OAuth session storage (`hermes-oauth-sessions`, a single Redis
|
||||||
|
`Deployment`, `replicas: 1`, `oauth-session-store.yaml`) has no
|
||||||
|
replication; its loss logs out every user cluster-wide (sessions are
|
||||||
|
externalized from oauth2-proxy but not made highly available).
|
||||||
|
|
||||||
|
### 2.4 The cluster is a heterogeneous home-lab/edge fleet with real, uneven headroom
|
||||||
|
|
||||||
|
Live `nodes.metrics.k8s.io` reads (non-invasive) show 21 nodes: mostly
|
||||||
|
Raspberry Pi 4/5 workers (~3.6-4 vCPU / ~6.5-8GB allocatable each), plus a
|
||||||
|
few larger amd64/Jetson-class nodes with shared GPUs (titan-20/21, titan-22,
|
||||||
|
titan-24) and one large amd64 node (titan-23, 48 vCPU/256GB, no GPU, not
|
||||||
|
labeled `worker`). This is **not** elastic cloud capacity — headroom is
|
||||||
|
finite and already uneven:
|
||||||
|
|
||||||
|
| Node | CPU used/alloc | Mem used/alloc | Note |
|
||||||
|
|---|---|---|---|
|
||||||
|
| titan-12 | 3.95 / 3.60 (**110%**) | 5.65 / 6.50 (87%) | rpi4, already over its CPU allocatable at snapshot time |
|
||||||
|
| titan-13 | 3.45 / 3.60 (96%) | 5.85 / 6.50 (90%) | rpi4, hot |
|
||||||
|
| titan-17 | 2.71 / 3.60 (75%) | 5.54 / 6.50 (85%) | rpi4, hot on memory |
|
||||||
|
| titan-07 | 1.23 / 3.60 (34%) | 5.24 / 6.63 (79%) | rpi5, memory-constrained |
|
||||||
|
| titan-04 | 0.35 / 3.60 (10%) | 1.69 / 6.63 (26%) | rpi5, most headroom in the pool |
|
||||||
|
| titan-21 (voice, GPU) | 0.91 / 6.00 (15%) | 8.02 / 14.56 (55%) | shared GPU node, moderate headroom |
|
||||||
|
| titan-23 (48 vCPU/256GB, no GPU label) | 0.74 / 48 (1.5%) | 22.9 / 251.6 (9%) | large headroom, not currently used by hermes affinity rules (arm64-only selectors exclude it) |
|
||||||
|
| titan-24 (GPU, Wolf/Hermes shared) | 1.16 / 24 (4.8%) | 22.6 / 62.7 (36%) | model-gate/Ollama contention with Wolf, per NOTES.md |
|
||||||
|
|
||||||
|
`chat-statefulset.yaml`'s node affinity (lines 54-74) requires `arm64` +
|
||||||
|
`node-role.kubernetes.io/worker=true` and already excludes 7 specific
|
||||||
|
hostnames (`titan-05,08,13,14,17,18,19`) — but two of the nodes it currently
|
||||||
|
*can* still land on, **titan-12 and titan-15, are already running at 75-110%
|
||||||
|
of allocatable CPU**. Any additional tenant/broker/router replica scheduled
|
||||||
|
into this same arm64 worker pool inherits that contention; it is not safe to
|
||||||
|
assume "add a replica" has free capacity behind it without checking node
|
||||||
|
headroom at scale-out time. The large amd64 node (titan-23) has abundant
|
||||||
|
headroom but is excluded by the current `arm64`-only affinity rules on every
|
||||||
|
hermes component — worth a design question, not something this assessment
|
||||||
|
changes.
|
||||||
|
|
||||||
|
### 2.5 No staging/canary environment exists
|
||||||
|
|
||||||
|
`services/hermes/kustomization.yaml` is a single flat kustomization with no
|
||||||
|
overlay split. `services/hermes-chat/` contains only a namespace and one PVC
|
||||||
|
(vestigial, no workload). `services/hermes-triage-demo/` is unrelated
|
||||||
|
(Ariadne's automated-repair demo loop, no chat workload). **Any load test
|
||||||
|
that exercises the real chat path runs against the production namespace and
|
||||||
|
the production `hermes-chat-tenant` StatefulSet** — there is no isolated
|
||||||
|
copy to test against.
|
||||||
|
|
||||||
|
### 2.6 No load-test tooling and thin production metrics exist today
|
||||||
|
|
||||||
|
- No k6/locust/vegeta/wrk/artillery/ab scripts exist anywhere in the repo
|
||||||
|
for hermes. A load test has to be built from scratch.
|
||||||
|
- No `ServiceMonitor`/`PodMonitor`/`PrometheusRule` objects exist for hermes
|
||||||
|
(annotation-based scraping only). `/metrics` exists for the `hermes-agent`
|
||||||
|
pod (CLI/provider quota usage, port 9010), `hermes-switchyard` (port 9005,
|
||||||
|
the richest signal — `switchyard_requests_total`,
|
||||||
|
`switchyard_model_call_latency_ms_bucket`,
|
||||||
|
`switchyard_routing_overhead_ms_bucket`, `switchyard_errors_total`,
|
||||||
|
`switchyard_client_responses_total`), and the CLI-lane metrics service
|
||||||
|
(port 9011, agent/Kanban-lane specific). **No `/metrics` endpoint exists
|
||||||
|
on `hermes-chat-tenant`, `hermes-model-gate`, `oauth2-proxy`,
|
||||||
|
`hermes-chat-router`, or voice** — there is no per-request latency/error
|
||||||
|
signal from the actual multi-user chat pods themselves.
|
||||||
|
- `services/monitoring/grafana-dashboard-ai.yaml` already has a P95
|
||||||
|
model-call-latency panel, a success/error-rate panel, and a per-pod CPU
|
||||||
|
panel filtered to `hermes-(agent|chat-tenant|switchyard|model-gate).*`,
|
||||||
|
all sourced from Switchyard metrics + cAdvisor — a usable baseline to
|
||||||
|
reuse, but nothing is wired to an alert threshold, and there is no
|
||||||
|
queue-depth or replica-health panel.
|
||||||
|
|
||||||
|
## 3. Unknowns (require an actual test to answer)
|
||||||
|
|
||||||
|
- Real max throughput of `hermes-switchyard`, `hermes-model-gate` -> Ollama,
|
||||||
|
and the STT/TTS GPU path under concurrent load — no data exists today.
|
||||||
|
- Real behavior/latency of the Claude broker's semaphore-2 queue under
|
||||||
|
contention (does it degrade gracefully to slow responses, or eventually
|
||||||
|
time out and error?) — no data exists today.
|
||||||
|
- Whether oauth2-proxy + Redis session store can handle concurrent
|
||||||
|
logins/refreshes at a higher user count (currently untested; the
|
||||||
|
Deployment is stateless-safe to scale, but has never been scaled).
|
||||||
|
Also whether Redis itself (`hermes-oauth-sessions`, 1Gi PVC, no HA)
|
||||||
|
becomes a bottleneck or SPOF under more simultaneous sessions.
|
||||||
|
- Actual current pod-level CPU/memory usage and restart history for any
|
||||||
|
hermes component — not observable with the current RBAC grant. Only
|
||||||
|
node-level aggregate usage is visible, not which pods are consuming it.
|
||||||
|
- Whether scaling `hermes-chat-tenant` toward its already-configured
|
||||||
|
`TENANT_SLOTS=8` (after fixing the mismatch) is actually safe given
|
||||||
|
current node headroom (titan-12/titan-15 pressure, see 2.4) — needs a
|
||||||
|
scheduling dry-run / headroom check at decision time, not assumed from
|
||||||
|
this snapshot.
|
||||||
|
|
||||||
|
## 4. SLO and load-test proposal (not yet executed — needs approval, see §6)
|
||||||
|
|
||||||
|
### 4.1 Target SLOs to validate (proposed, for review)
|
||||||
|
|
||||||
|
| Metric | Proposed target | Source |
|
||||||
|
|---|---|---|
|
||||||
|
| P95 end-to-end chat response latency | < 8s for local/Codex-routed, < 20s for Claude-routed (queued behind semaphore-2) | new synthetic client timer + existing `switchyard_model_call_latency_ms_bucket` |
|
||||||
|
| Error rate (5xx / proxy failures) | < 1% sustained | new synthetic client + `switchyard_errors_total` / `switchyard_client_responses_total` |
|
||||||
|
| Login/session success rate | 100% (oauth2-proxy + Redis) | synthetic client HTTP status |
|
||||||
|
| Node CPU/mem headroom during test | stay below 90% allocatable on any node hosting hermes pods | `nodes.metrics.k8s.io` (already provably readable) |
|
||||||
|
| Claude-broker queue depth / wait time | document actual behavior, no target yet (unknown baseline) | needs new instrumentation — none exists today |
|
||||||
|
|
||||||
|
### 4.2 Concurrency / ramp / mix proposal
|
||||||
|
|
||||||
|
- **Concurrency levels**: 1 (baseline), 2 (at the Claude-broker cap), 4
|
||||||
|
(at today's real tenant-pool ceiling), and — only if §2.1's mismatch is
|
||||||
|
resolved and node headroom is re-checked — 8 (at the configured
|
||||||
|
`TENANT_SLOTS`). Do not exceed the number of actually-backed tenant slots.
|
||||||
|
- **Ramp**: hold each level for at least 10 minutes after reaching steady
|
||||||
|
state before increasing, to separate cold-start effects (webui/gateway
|
||||||
|
startup, model warm-up) from steady-state capacity.
|
||||||
|
- **Traffic mix**: proportion requests across the three model routes
|
||||||
|
(Claude / Codex / local Ollama) matching real usage if known, otherwise
|
||||||
|
test each route in isolation first (to characterize each bottleneck
|
||||||
|
independently — semaphore-2 vs. no-cap-but-untested), then a blended mix.
|
||||||
|
- **Session behavior**: each synthetic user must be a distinct, dedicated
|
||||||
|
test Keycloak identity (slot assignment is permanent — do not burn real
|
||||||
|
user slots or reuse production accounts for load generation).
|
||||||
|
- **Failure injection (optional, staged)**: kill the single
|
||||||
|
`hermes-switchyard` or `hermes-model-gate` pod mid-test to confirm the
|
||||||
|
`Recreate`/no-PDB SPOF behavior matches expectations and measure recovery
|
||||||
|
time, only after the throughput characterization above is complete.
|
||||||
|
|
||||||
|
## 5. Safe staged scaling plan (for future execution, after human approval)
|
||||||
|
|
||||||
|
This is a plan to evaluate, not a plan already executed.
|
||||||
|
|
||||||
|
1. **Fix the `TENANT_SLOTS`/replica-count mismatch first** (§2.1). This is a
|
||||||
|
correctness fix at the *current* capacity, not a capacity increase, and
|
||||||
|
should land before any load test so results aren't confounded by the
|
||||||
|
existing 502-on-phantom-slot bug.
|
||||||
|
2. **Instrument before scaling**: add `/metrics` (or at minimum structured
|
||||||
|
access logs) to `hermes-chat-tenant`, `hermes-chat-router`, and
|
||||||
|
`hermes-model-gate` so a load test has a real per-request signal instead
|
||||||
|
of relying solely on Switchyard's aggregate view.
|
||||||
|
3. **Baseline test at today's real ceiling (4 users)**, off-hours, using
|
||||||
|
dedicated test Keycloak identities, per §4.
|
||||||
|
4. **Re-check node headroom** (§2.4) at decision time, not from this
|
||||||
|
snapshot, before considering any StatefulSet scale-out — titan-12 and
|
||||||
|
titan-15 were already at/above CPU allocatable during this assessment.
|
||||||
|
5. **If and only if** the baseline is healthy and headroom allows, propose
|
||||||
|
scaling `hermes-chat-tenant` toward the already-configured 8 slots one
|
||||||
|
or two ordinals at a time (StatefulSet `podManagementPolicy: Parallel`
|
||||||
|
supports this), re-measuring node headroom and Switchyard/broker latency
|
||||||
|
after each step.
|
||||||
|
6. **Address the shared-broker ceiling separately from tenant-pod count**:
|
||||||
|
raising `HERMES_CLAUDE_BROKER_CONCURRENCY` above 2 is a distinct decision
|
||||||
|
gated by the underlying Claude Code provider's real rate limits, not by
|
||||||
|
Kubernetes resources — do not conflate it with StatefulSet scaling.
|
||||||
|
7. **Only after tenant/broker capacity is validated**, consider whether the
|
||||||
|
single-replica shared components (`hermes-switchyard`, `hermes-model-gate`,
|
||||||
|
`oauth2-proxy`) need a second replica or a PDB — note oauth2-proxy is the
|
||||||
|
only one of these that is currently stateless-safe to scale (Redis-backed
|
||||||
|
sessions); the others would need code changes first (in-process locks/
|
||||||
|
state noted in §2).
|
||||||
|
|
||||||
|
## 6. Rollback and observability gates
|
||||||
|
|
||||||
|
- Every step above is a Git-reviewed manifest change applied via Flux, so
|
||||||
|
rollback is `git revert` + Flux reconcile, consistent with this repo's
|
||||||
|
existing GitOps model — no manual `kubectl edit` in any step.
|
||||||
|
- Gate each scaling step on: node CPU/mem headroom (§2.4 thresholds), the
|
||||||
|
new `/metrics` signal from step 2 above staying within the SLOs in §4.1,
|
||||||
|
and zero increase in `switchyard_errors_total` / `switchyard_classifier_fail_open_total`
|
||||||
|
rate versus the pre-change baseline.
|
||||||
|
- Because there is no staging environment (§2.5) and no PDB, treat every
|
||||||
|
step as a production change: schedule off-hours, announce before/after,
|
||||||
|
and keep the prior manifest revision ready to revert immediately if error
|
||||||
|
rate or node pressure crosses the gate.
|
||||||
|
- Because RBAC here is observer-only (§0), whoever executes a future load
|
||||||
|
test needs either elevated read access to confirm live pod/replica status
|
||||||
|
during the test, or must rely entirely on the node-metrics + Switchyard
|
||||||
|
signals already proven readable in this assessment.
|
||||||
|
|
||||||
|
## 7. Recommendation
|
||||||
|
|
||||||
|
**A load test against the current 4-user ceiling is reasonably safe to run
|
||||||
|
off-hours with dedicated test accounts, once §5 step 1 (the `TENANT_SLOTS`
|
||||||
|
mismatch) and step 2 (baseline metrics) are addressed** — it does not
|
||||||
|
require any replica/cap increase, and the blast radius is bounded to
|
||||||
|
resources already provisioned.
|
||||||
|
|
||||||
|
**A test at 8 concurrent users, or any StatefulSet/broker scale-up, is not
|
||||||
|
yet safe to schedule.** Two of the nodes in the current chat-tenant affinity
|
||||||
|
pool are already at or above their allocatable CPU/memory (§2.4), there is
|
||||||
|
no PDB/HPA anywhere in this service, there is no staging environment to
|
||||||
|
absorb the risk, and the real throughput ceiling for Codex/local-model
|
||||||
|
routes and the actual behavior of the Claude-broker queue under load are
|
||||||
|
both unmeasured. Recommend running the 4-user baseline test first and using
|
||||||
|
its results (plus a fresh node-headroom check) to decide whether 8-user
|
||||||
|
scaling is warranted — this assessment does not make that call.
|
||||||
|
|
||||||
|
## 8. Requested input / blockers to proceed further
|
||||||
|
|
||||||
|
This assessment can be delivered without further input. To actually execute
|
||||||
|
the load test proposed in §4, a human owner needs to provide or approve:
|
||||||
|
|
||||||
|
1. Sign-off to fix the `TENANT_SLOTS`/replica mismatch (§2.1) as a
|
||||||
|
correctness PR, separate from any capacity increase.
|
||||||
|
2. Dedicated test Keycloak identities (slot assignment is permanent —
|
||||||
|
production user accounts must not be used for load generation).
|
||||||
|
3. An approved off-hours maintenance window, since there is no staging
|
||||||
|
environment and every step in §5 is a production change.
|
||||||
|
4. A decision on whether to add `/metrics` instrumentation (§5 step 2)
|
||||||
|
before or in parallel with the first baseline test, since today's
|
||||||
|
Switchyard-only signal cannot attribute latency/errors to a specific
|
||||||
|
tenant pod.
|
||||||
|
5. If deeper live verification is wanted beyond node-level metrics
|
||||||
|
(e.g., live pod status/restart counts during a test), temporary
|
||||||
|
elevated read RBAC for the executing identity, since the current
|
||||||
|
`atlas-observer` grant cannot list pods/deployments in `hermes` (§0).
|
||||||
Loading…
x
Reference in New Issue
Block a user