docs(hermes): add multi-user chat capacity assessment
Read-only investigation of the chat request path, replica/resource manifests, and live node metrics. Documents the confirmed 4-user tenant-slot ceiling, a TENANT_SLOTS=8 vs replicas=4 configuration drift, the shared Claude-broker concurrency=2 bottleneck, missing HPA/PDB/staging environment, and a proposed SLO/load-test and staged scaling plan pending human approval. No production manifests changed. Co-Authored-By: Hermes Agent <hermes-automation@bstein.dev>
This commit is contained in:
Hermes Agent
parent
2862594c62
commit
d8ccf4a6fa
349
docs/hermes_multiuser_capacity_assessment.md
Normal file
349
docs/hermes_multiuser_capacity_assessment.md
Normal file
@ -0,0 +1,349 @@
|
||||
# Hermes multi-user capacity assessment
|
||||
|
||||
Status: read-only investigation, no production changes made. This document
|
||||
answers "can Hermes safely handle more simultaneous users" with evidence from
|
||||
the current `titan-iac` manifests/code, plus non-invasive live cluster reads.
|
||||
It does not increase any replica count or concurrency cap.
|
||||
|
||||
## 0. Method and access boundary
|
||||
|
||||
- Static evidence: `services/hermes/**` manifests and `services/hermes/router/*.go`,
|
||||
`services/hermes/scripts/*.py` on branch `wt/t_65356568`, `git log`/`git blame`
|
||||
for recent drift.
|
||||
- Live evidence: read via the in-cluster `atlas-observer` kubeconfig context.
|
||||
This identity (`system:serviceaccount:hermes:hermes-agent`) is **observer-only**:
|
||||
`kubectl -n hermes get pods/deploy/sts/hpa/pdb` all returned `Forbidden`.
|
||||
`kubectl -n hermes auth can-i --list` confirms the only granted verbs are
|
||||
cluster-scoped `get/list/watch` on `namespaces`, `nodes`, `persistentvolumes`,
|
||||
storage APIs, and `nodes.metrics.k8s.io`, plus API discovery/health endpoints.
|
||||
There is no namespace-scoped read access to pods, deployments, replica counts,
|
||||
or pod-level resource usage in `hermes` (or any namespace) from this session.
|
||||
This matches the standing finding that hermes-agent RBAC is observer-only.
|
||||
- What this means for this report: **all replica counts, resource
|
||||
requests/limits, probes, and affinity rules below are read from the Git
|
||||
manifests** (which is what Flux actually reconciles, so it is a trustworthy
|
||||
source for desired state) rather than confirmed against live pod status.
|
||||
**Node-level capacity and current node CPU/memory utilization are live,
|
||||
real-time cluster evidence** (`nodes.metrics.k8s.io` + `nodes` are both
|
||||
readable). One non-invasive HTTP probe against `https://chat.bstein.dev/`
|
||||
confirmed the public endpoint is live and redirecting to auth (`302`) at
|
||||
the time of writing.
|
||||
- No pods, deployments, config, or replica counts were changed. No load test
|
||||
was executed against any environment.
|
||||
|
||||
## 1. Architecture and request path
|
||||
|
||||
```
|
||||
browser
|
||||
-> Traefik Ingress (chat.hermes.bstein.dev / chat.bstein.dev)
|
||||
-> oauth2-proxy-hermes-chat (Deployment, replicas: 1, oauth2-proxy.yaml:153)
|
||||
session state: Redis, externalized (--session-store-type=redis,
|
||||
redis://hermes-oauth-sessions...:6379/1, oauth2-proxy.yaml:203-204)
|
||||
-> hermes-chat-router (Deployment, replicas: 1, chat-router.yaml:10)
|
||||
Go binary, services/hermes/router/main.go
|
||||
in-process, PVC-persisted identity -> slot map (tenants.json)
|
||||
-> hermes-chat-tenant-<slot>.hermes-chat-tenant (StatefulSet, replicas: 4,
|
||||
chat-statefulset.yaml:13) — one pod per Keycloak subject, pinned by ordinal
|
||||
webui (8787) -> gateway (8642) -> per-tenant sandbox (chat-sandbox-<ordinal>:9080)
|
||||
-> hermes-switchyard (Deployment, replicas: 1, switchyard-deployment.yaml:10)
|
||||
shared model router for ALL tenants + triage + Kanban
|
||||
-> hermes-claude-broker / hermes-codex-broker (containers in hermes-agent pod,
|
||||
replicas: 1, agent-deployment.yaml:10) / hermes-model-gate (Deployment,
|
||||
replicas: 1, model-gate-deployment.yaml:10, proxies to local Ollama)
|
||||
-> hermes-stt / hermes-tts (Deployments, replicas: 1 each, voice-deployment.yaml,
|
||||
pinned to node titan-21, shared GPU)
|
||||
```
|
||||
|
||||
Every component on this path is `replicas: 1` **except** `hermes-chat-tenant`
|
||||
(4). The StatefulSet is not a load-balanced pool: each pod is a fully isolated,
|
||||
permanently-assigned per-user backend, not interchangeable capacity.
|
||||
|
||||
## 2. Known limits (confirmed from manifests/code)
|
||||
|
||||
### 2.1 Hard ceiling: 4 usable chat users today, and a live configuration bug
|
||||
|
||||
`services/hermes/router/main.go` (`tenantRouter.slotFor`, lines 153-176)
|
||||
permanently assigns each Keycloak subject a slot `0..TENANT_SLOTS-1` on first
|
||||
login, persists it to a PVC-backed JSON file, and never releases it. There is
|
||||
no unassign/deprovision path in the router. Unit tests
|
||||
(`router/main_test.go`, "expected the fixed private tenant pool to report
|
||||
capacity") assert that once all slots are taken, `slotFor` returns an error.
|
||||
|
||||
**Confirmed drift**: `chat-router.yaml:70` sets `TENANT_SLOTS=8`, but
|
||||
`chat-statefulset.yaml:13` sets `replicas: 4`. `git log -p` on both files
|
||||
shows they were changed together at `18965a2f` (4->8), then
|
||||
`chat-statefulset.yaml` was scaled down twice — `79baa7ec` ("return two chat
|
||||
slots to fit the agent runtime", 8->6) and `e28b32bd` (6->4) — **without a
|
||||
matching change to `TENANT_SLOTS`**, and `chat-sandbox.yaml` still defines 8
|
||||
per-ordinal sandbox Deployments (`hermes-chat-sandbox-0` through `-7`).
|
||||
Net effect: the 5th–8th distinct Keycloak login is durably assigned a slot
|
||||
that has no backing StatefulSet pod. `main.go`'s proxy error path (around
|
||||
line 382) returns a "starting" error to that user **permanently**, not just
|
||||
during a cold start, because the pod will never exist at the current replica
|
||||
count. This is a pre-existing configuration bug, independent of any future
|
||||
scale-up decision, and it silently masks the real behavior of the capacity
|
||||
ceiling (a 5th user does not get a clean "at capacity" rejection matching the
|
||||
tested behavior — they get a StatefulSet-pod-not-found style failure).
|
||||
**This assessment does not fix it** (per scope: no production changes), but
|
||||
it should be the first fix considered before any scale-out, since it affects
|
||||
correctness at the current replica count too.
|
||||
|
||||
### 2.2 The real shared bottleneck is not the 4 tenant pods, it's the brokers behind them
|
||||
|
||||
All 4 tenant pods, the triage/operator instance, and Kanban/CLI-lane traffic
|
||||
funnel through the same single-replica `hermes-switchyard`, which routes to
|
||||
the same single-replica `hermes-claude-broker` / `hermes-codex-broker`
|
||||
(containers inside the one `hermes-agent` pod) or `hermes-model-gate`.
|
||||
|
||||
- `hermes-claude-broker` enforces a hard, code-level concurrency cap:
|
||||
`HERMES_CLAUDE_BROKER_CONCURRENCY=2` (`agent-deployment.yaml:1160`) sizes a
|
||||
`threading.BoundedSemaphore(2)` (`scripts/claude_oauth_broker.py:63`)
|
||||
wrapped around a blocking `subprocess.run` call to the native Claude Code
|
||||
CLI. Requests over the cap **block silently** (no 429/503 to the caller)
|
||||
until a slot frees. This cap is shared cluster-wide across every chat
|
||||
tenant, the triage instance, and indirectly the Kanban/CLI-lane workers
|
||||
(same provider credentials, so heavy CLI-lane usage can throttle chat
|
||||
responses via the provider's own rate limits even without touching this
|
||||
semaphore directly).
|
||||
- `hermes-codex-broker` and `hermes-model-gate` have **no equivalent
|
||||
concurrency cap in code** — they are bounded only by container CPU/memory
|
||||
`limits` (1 CPU / 1Gi for the codex-broker container, 250m/128Mi for
|
||||
model-gate) and by the single upstream Ollama/Jetson instance's real
|
||||
throughput, which is unknown (no load test exists).
|
||||
|
||||
**Implication**: scaling `hermes-chat-tenant` replicas alone (e.g. 4 -> 8)
|
||||
does not proportionally increase real AI-response throughput. The
|
||||
Claude-routed share of traffic is capped at 2 concurrent completions
|
||||
regardless of how many tenant pods exist, and Codex/local-model throughput
|
||||
under concurrent load is untested.
|
||||
|
||||
### 2.3 Single points of failure and deploy-time outage windows
|
||||
|
||||
- No `HorizontalPodAutoscaler` exists anywhere in this repository (confirmed
|
||||
by a repo-wide grep, not just `services/hermes/`).
|
||||
- No `PodDisruptionBudget` exists for any hermes component (the one PDB in
|
||||
the repo is `infrastructure/traefik/pdb.yaml`, unrelated).
|
||||
- `oauth2-proxy`, `hermes-chat-router`, `hermes-switchyard`,
|
||||
`hermes-model-gate`, `hermes-agent` (brokers), and voice STT/TTS are all
|
||||
`replicas: 1`. A pod restart/crash/node drain on any of them takes down
|
||||
chat, voice, or model access for **all** users simultaneously, even though
|
||||
the tenant StatefulSet gives each user an isolated conversation backend.
|
||||
- `hermes` (main/triage), `hermes-chat-router`, `hermes-local-image`, and
|
||||
`hermes-agent` all use `strategy: Recreate` — every rollout is a full
|
||||
teardown-then-recreate, not a rolling update, adding a real availability
|
||||
gap bounded by readiness/liveness `initialDelaySeconds` (up to 90s+ for the
|
||||
main `hermes` deployment). The only genuinely zero-downtime rollout path
|
||||
in this whole service is `hermes-switchyard`
|
||||
(`RollingUpdate`, `maxSurge:1`, `maxUnavailable:0`) and the tenant
|
||||
StatefulSet itself (`RollingUpdate`, `podManagementPolicy: Parallel`).
|
||||
- OAuth session storage (`hermes-oauth-sessions`, a single Redis
|
||||
`Deployment`, `replicas: 1`, `oauth-session-store.yaml`) has no
|
||||
replication; its loss logs out every user cluster-wide (sessions are
|
||||
externalized from oauth2-proxy but not made highly available).
|
||||
|
||||
### 2.4 The cluster is a heterogeneous home-lab/edge fleet with real, uneven headroom
|
||||
|
||||
Live `nodes.metrics.k8s.io` reads (non-invasive) show 21 nodes: mostly
|
||||
Raspberry Pi 4/5 workers (~3.6-4 vCPU / ~6.5-8GB allocatable each), plus a
|
||||
few larger amd64/Jetson-class nodes with shared GPUs (titan-20/21, titan-22,
|
||||
titan-24) and one large amd64 node (titan-23, 48 vCPU/256GB, no GPU, not
|
||||
labeled `worker`). This is **not** elastic cloud capacity — headroom is
|
||||
finite and already uneven:
|
||||
|
||||
| Node | CPU used/alloc | Mem used/alloc | Note |
|
||||
|---|---|---|---|
|
||||
| titan-12 | 3.95 / 3.60 (**110%**) | 5.65 / 6.50 (87%) | rpi4, already over its CPU allocatable at snapshot time |
|
||||
| titan-13 | 3.45 / 3.60 (96%) | 5.85 / 6.50 (90%) | rpi4, hot |
|
||||
| titan-17 | 2.71 / 3.60 (75%) | 5.54 / 6.50 (85%) | rpi4, hot on memory |
|
||||
| titan-07 | 1.23 / 3.60 (34%) | 5.24 / 6.63 (79%) | rpi5, memory-constrained |
|
||||
| titan-04 | 0.35 / 3.60 (10%) | 1.69 / 6.63 (26%) | rpi5, most headroom in the pool |
|
||||
| titan-21 (voice, GPU) | 0.91 / 6.00 (15%) | 8.02 / 14.56 (55%) | shared GPU node, moderate headroom |
|
||||
| titan-23 (48 vCPU/256GB, no GPU label) | 0.74 / 48 (1.5%) | 22.9 / 251.6 (9%) | large headroom, not currently used by hermes affinity rules (arm64-only selectors exclude it) |
|
||||
| titan-24 (GPU, Wolf/Hermes shared) | 1.16 / 24 (4.8%) | 22.6 / 62.7 (36%) | model-gate/Ollama contention with Wolf, per NOTES.md |
|
||||
|
||||
`chat-statefulset.yaml`'s node affinity (lines 54-74) requires `arm64` +
|
||||
`node-role.kubernetes.io/worker=true` and already excludes 7 specific
|
||||
hostnames (`titan-05,08,13,14,17,18,19`) — but two of the nodes it currently
|
||||
*can* still land on, **titan-12 and titan-15, are already running at 75-110%
|
||||
of allocatable CPU**. Any additional tenant/broker/router replica scheduled
|
||||
into this same arm64 worker pool inherits that contention; it is not safe to
|
||||
assume "add a replica" has free capacity behind it without checking node
|
||||
headroom at scale-out time. The large amd64 node (titan-23) has abundant
|
||||
headroom but is excluded by the current `arm64`-only affinity rules on every
|
||||
hermes component — worth a design question, not something this assessment
|
||||
changes.
|
||||
|
||||
### 2.5 No staging/canary environment exists
|
||||
|
||||
`services/hermes/kustomization.yaml` is a single flat kustomization with no
|
||||
overlay split. `services/hermes-chat/` contains only a namespace and one PVC
|
||||
(vestigial, no workload). `services/hermes-triage-demo/` is unrelated
|
||||
(Ariadne's automated-repair demo loop, no chat workload). **Any load test
|
||||
that exercises the real chat path runs against the production namespace and
|
||||
the production `hermes-chat-tenant` StatefulSet** — there is no isolated
|
||||
copy to test against.
|
||||
|
||||
### 2.6 No load-test tooling and thin production metrics exist today
|
||||
|
||||
- No k6/locust/vegeta/wrk/artillery/ab scripts exist anywhere in the repo
|
||||
for hermes. A load test has to be built from scratch.
|
||||
- No `ServiceMonitor`/`PodMonitor`/`PrometheusRule` objects exist for hermes
|
||||
(annotation-based scraping only). `/metrics` exists for the `hermes-agent`
|
||||
pod (CLI/provider quota usage, port 9010), `hermes-switchyard` (port 9005,
|
||||
the richest signal — `switchyard_requests_total`,
|
||||
`switchyard_model_call_latency_ms_bucket`,
|
||||
`switchyard_routing_overhead_ms_bucket`, `switchyard_errors_total`,
|
||||
`switchyard_client_responses_total`), and the CLI-lane metrics service
|
||||
(port 9011, agent/Kanban-lane specific). **No `/metrics` endpoint exists
|
||||
on `hermes-chat-tenant`, `hermes-model-gate`, `oauth2-proxy`,
|
||||
`hermes-chat-router`, or voice** — there is no per-request latency/error
|
||||
signal from the actual multi-user chat pods themselves.
|
||||
- `services/monitoring/grafana-dashboard-ai.yaml` already has a P95
|
||||
model-call-latency panel, a success/error-rate panel, and a per-pod CPU
|
||||
panel filtered to `hermes-(agent|chat-tenant|switchyard|model-gate).*`,
|
||||
all sourced from Switchyard metrics + cAdvisor — a usable baseline to
|
||||
reuse, but nothing is wired to an alert threshold, and there is no
|
||||
queue-depth or replica-health panel.
|
||||
|
||||
## 3. Unknowns (require an actual test to answer)
|
||||
|
||||
- Real max throughput of `hermes-switchyard`, `hermes-model-gate` -> Ollama,
|
||||
and the STT/TTS GPU path under concurrent load — no data exists today.
|
||||
- Real behavior/latency of the Claude broker's semaphore-2 queue under
|
||||
contention (does it degrade gracefully to slow responses, or eventually
|
||||
time out and error?) — no data exists today.
|
||||
- Whether oauth2-proxy + Redis session store can handle concurrent
|
||||
logins/refreshes at a higher user count (currently untested; the
|
||||
Deployment is stateless-safe to scale, but has never been scaled).
|
||||
Also whether Redis itself (`hermes-oauth-sessions`, 1Gi PVC, no HA)
|
||||
becomes a bottleneck or SPOF under more simultaneous sessions.
|
||||
- Actual current pod-level CPU/memory usage and restart history for any
|
||||
hermes component — not observable with the current RBAC grant. Only
|
||||
node-level aggregate usage is visible, not which pods are consuming it.
|
||||
- Whether scaling `hermes-chat-tenant` toward its already-configured
|
||||
`TENANT_SLOTS=8` (after fixing the mismatch) is actually safe given
|
||||
current node headroom (titan-12/titan-15 pressure, see 2.4) — needs a
|
||||
scheduling dry-run / headroom check at decision time, not assumed from
|
||||
this snapshot.
|
||||
|
||||
## 4. SLO and load-test proposal (not yet executed — needs approval, see §6)
|
||||
|
||||
### 4.1 Target SLOs to validate (proposed, for review)
|
||||
|
||||
| Metric | Proposed target | Source |
|
||||
|---|---|---|
|
||||
| P95 end-to-end chat response latency | < 8s for local/Codex-routed, < 20s for Claude-routed (queued behind semaphore-2) | new synthetic client timer + existing `switchyard_model_call_latency_ms_bucket` |
|
||||
| Error rate (5xx / proxy failures) | < 1% sustained | new synthetic client + `switchyard_errors_total` / `switchyard_client_responses_total` |
|
||||
| Login/session success rate | 100% (oauth2-proxy + Redis) | synthetic client HTTP status |
|
||||
| Node CPU/mem headroom during test | stay below 90% allocatable on any node hosting hermes pods | `nodes.metrics.k8s.io` (already provably readable) |
|
||||
| Claude-broker queue depth / wait time | document actual behavior, no target yet (unknown baseline) | needs new instrumentation — none exists today |
|
||||
|
||||
### 4.2 Concurrency / ramp / mix proposal
|
||||
|
||||
- **Concurrency levels**: 1 (baseline), 2 (at the Claude-broker cap), 4
|
||||
(at today's real tenant-pool ceiling), and — only if §2.1's mismatch is
|
||||
resolved and node headroom is re-checked — 8 (at the configured
|
||||
`TENANT_SLOTS`). Do not exceed the number of actually-backed tenant slots.
|
||||
- **Ramp**: hold each level for at least 10 minutes after reaching steady
|
||||
state before increasing, to separate cold-start effects (webui/gateway
|
||||
startup, model warm-up) from steady-state capacity.
|
||||
- **Traffic mix**: proportion requests across the three model routes
|
||||
(Claude / Codex / local Ollama) matching real usage if known, otherwise
|
||||
test each route in isolation first (to characterize each bottleneck
|
||||
independently — semaphore-2 vs. no-cap-but-untested), then a blended mix.
|
||||
- **Session behavior**: each synthetic user must be a distinct, dedicated
|
||||
test Keycloak identity (slot assignment is permanent — do not burn real
|
||||
user slots or reuse production accounts for load generation).
|
||||
- **Failure injection (optional, staged)**: kill the single
|
||||
`hermes-switchyard` or `hermes-model-gate` pod mid-test to confirm the
|
||||
`Recreate`/no-PDB SPOF behavior matches expectations and measure recovery
|
||||
time, only after the throughput characterization above is complete.
|
||||
|
||||
## 5. Safe staged scaling plan (for future execution, after human approval)
|
||||
|
||||
This is a plan to evaluate, not a plan already executed.
|
||||
|
||||
1. **Fix the `TENANT_SLOTS`/replica-count mismatch first** (§2.1). This is a
|
||||
correctness fix at the *current* capacity, not a capacity increase, and
|
||||
should land before any load test so results aren't confounded by the
|
||||
existing 502-on-phantom-slot bug.
|
||||
2. **Instrument before scaling**: add `/metrics` (or at minimum structured
|
||||
access logs) to `hermes-chat-tenant`, `hermes-chat-router`, and
|
||||
`hermes-model-gate` so a load test has a real per-request signal instead
|
||||
of relying solely on Switchyard's aggregate view.
|
||||
3. **Baseline test at today's real ceiling (4 users)**, off-hours, using
|
||||
dedicated test Keycloak identities, per §4.
|
||||
4. **Re-check node headroom** (§2.4) at decision time, not from this
|
||||
snapshot, before considering any StatefulSet scale-out — titan-12 and
|
||||
titan-15 were already at/above CPU allocatable during this assessment.
|
||||
5. **If and only if** the baseline is healthy and headroom allows, propose
|
||||
scaling `hermes-chat-tenant` toward the already-configured 8 slots one
|
||||
or two ordinals at a time (StatefulSet `podManagementPolicy: Parallel`
|
||||
supports this), re-measuring node headroom and Switchyard/broker latency
|
||||
after each step.
|
||||
6. **Address the shared-broker ceiling separately from tenant-pod count**:
|
||||
raising `HERMES_CLAUDE_BROKER_CONCURRENCY` above 2 is a distinct decision
|
||||
gated by the underlying Claude Code provider's real rate limits, not by
|
||||
Kubernetes resources — do not conflate it with StatefulSet scaling.
|
||||
7. **Only after tenant/broker capacity is validated**, consider whether the
|
||||
single-replica shared components (`hermes-switchyard`, `hermes-model-gate`,
|
||||
`oauth2-proxy`) need a second replica or a PDB — note oauth2-proxy is the
|
||||
only one of these that is currently stateless-safe to scale (Redis-backed
|
||||
sessions); the others would need code changes first (in-process locks/
|
||||
state noted in §2).
|
||||
|
||||
## 6. Rollback and observability gates
|
||||
|
||||
- Every step above is a Git-reviewed manifest change applied via Flux, so
|
||||
rollback is `git revert` + Flux reconcile, consistent with this repo's
|
||||
existing GitOps model — no manual `kubectl edit` in any step.
|
||||
- Gate each scaling step on: node CPU/mem headroom (§2.4 thresholds), the
|
||||
new `/metrics` signal from step 2 above staying within the SLOs in §4.1,
|
||||
and zero increase in `switchyard_errors_total` / `switchyard_classifier_fail_open_total`
|
||||
rate versus the pre-change baseline.
|
||||
- Because there is no staging environment (§2.5) and no PDB, treat every
|
||||
step as a production change: schedule off-hours, announce before/after,
|
||||
and keep the prior manifest revision ready to revert immediately if error
|
||||
rate or node pressure crosses the gate.
|
||||
- Because RBAC here is observer-only (§0), whoever executes a future load
|
||||
test needs either elevated read access to confirm live pod/replica status
|
||||
during the test, or must rely entirely on the node-metrics + Switchyard
|
||||
signals already proven readable in this assessment.
|
||||
|
||||
## 7. Recommendation
|
||||
|
||||
**A load test against the current 4-user ceiling is reasonably safe to run
|
||||
off-hours with dedicated test accounts, once §5 step 1 (the `TENANT_SLOTS`
|
||||
mismatch) and step 2 (baseline metrics) are addressed** — it does not
|
||||
require any replica/cap increase, and the blast radius is bounded to
|
||||
resources already provisioned.
|
||||
|
||||
**A test at 8 concurrent users, or any StatefulSet/broker scale-up, is not
|
||||
yet safe to schedule.** Two of the nodes in the current chat-tenant affinity
|
||||
pool are already at or above their allocatable CPU/memory (§2.4), there is
|
||||
no PDB/HPA anywhere in this service, there is no staging environment to
|
||||
absorb the risk, and the real throughput ceiling for Codex/local-model
|
||||
routes and the actual behavior of the Claude-broker queue under load are
|
||||
both unmeasured. Recommend running the 4-user baseline test first and using
|
||||
its results (plus a fresh node-headroom check) to decide whether 8-user
|
||||
scaling is warranted — this assessment does not make that call.
|
||||
|
||||
## 8. Requested input / blockers to proceed further
|
||||
|
||||
This assessment can be delivered without further input. To actually execute
|
||||
the load test proposed in §4, a human owner needs to provide or approve:
|
||||
|
||||
1. Sign-off to fix the `TENANT_SLOTS`/replica mismatch (§2.1) as a
|
||||
correctness PR, separate from any capacity increase.
|
||||
2. Dedicated test Keycloak identities (slot assignment is permanent —
|
||||
production user accounts must not be used for load generation).
|
||||
3. An approved off-hours maintenance window, since there is no staging
|
||||
environment and every step in §5 is a production change.
|
||||
4. A decision on whether to add `/metrics` instrumentation (§5 step 2)
|
||||
before or in parallel with the first baseline test, since today's
|
||||
Switchyard-only signal cannot attribute latency/errors to a specific
|
||||
tenant pod.
|
||||
5. If deeper live verification is wanted beyond node-level metrics
|
||||
(e.g., live pod status/restart counts during a test), temporary
|
||||
elevated read RBAC for the executing identity, since the current
|
||||
`atlas-observer` grant cannot list pods/deployments in `hermes` (§0).
|
||||
Loading…
x
Reference in New Issue
Block a user