docs(hermes): add multi-user chat capacity assessment #45

hermes-automation · 2026-08-21T12:40:27Z

hermes-automation commented

2026-08-21 12:40:27 +00:00

Summary

Read-only investigation of the Hermes chat request path (ingress -> oauth2-proxy -> chat router -> per-user tenant StatefulSet -> Switchyard -> model brokers/voice), replica and resource manifests, and non-invasive live cluster reads (node-level metrics only; this identity has no namespace-scoped pod/deployment read access).
Confirms the real capacity ceiling is 4 usable chat users today (permanent per-Keycloak-subject slot assignment in the tenant router), and documents a live configuration drift: chat-router.yaml sets TENANT_SLOTS to 8 while chat-statefulset.yaml has replicas 4, so a 5th-8th distinct login is durably assigned a slot with no backing pod.
Documents the shared Claude broker concurrency-2 semaphore as the real cross-tenant bottleneck, absence of any HPA/PDB/staging environment anywhere in the service, and current node CPU/memory headroom (two nodes in the chat-tenant scheduling pool are already at/above allocatable CPU).
Proposes an SLO/load-test plan, a staged scaling plan, and rollback/observability gates, and recommends a 4-user baseline load test only after the TENANT_SLOTS mismatch is fixed; recommends against an 8-user or replica-increase test until node headroom and shared-broker throughput are actually measured.
No production manifests changed, nothing merged or deployed. Opened as a draft per task instructions, since this is a durable, evidence-backed reference document, not ephemeral task output.

Test plan

Human review of the findings and the proposed SLO/load-test/staged-scaling plan
Decide whether to fix the TENANT_SLOTS/replicas mismatch as a separate correctness PR
Approve or decline a bounded off-hours 4-user load test using dedicated test accounts before any capacity-increasing change

Co-Authored-By: Hermes Agent hermes-automation@bstein.dev

## Summary - Read-only investigation of the Hermes chat request path (ingress -> oauth2-proxy -> chat router -> per-user tenant StatefulSet -> Switchyard -> model brokers/voice), replica and resource manifests, and non-invasive live cluster reads (node-level metrics only; this identity has no namespace-scoped pod/deployment read access). - Confirms the real capacity ceiling is 4 usable chat users today (permanent per-Keycloak-subject slot assignment in the tenant router), and documents a live configuration drift: chat-router.yaml sets TENANT_SLOTS to 8 while chat-statefulset.yaml has replicas 4, so a 5th-8th distinct login is durably assigned a slot with no backing pod. - Documents the shared Claude broker concurrency-2 semaphore as the real cross-tenant bottleneck, absence of any HPA/PDB/staging environment anywhere in the service, and current node CPU/memory headroom (two nodes in the chat-tenant scheduling pool are already at/above allocatable CPU). - Proposes an SLO/load-test plan, a staged scaling plan, and rollback/observability gates, and recommends a 4-user baseline load test only after the TENANT_SLOTS mismatch is fixed; recommends against an 8-user or replica-increase test until node headroom and shared-broker throughput are actually measured. - No production manifests changed, nothing merged or deployed. Opened as a draft per task instructions, since this is a durable, evidence-backed reference document, not ephemeral task output. ## Test plan - [ ] Human review of the findings and the proposed SLO/load-test/staged-scaling plan - [ ] Decide whether to fix the TENANT_SLOTS/replicas mismatch as a separate correctness PR - [ ] Approve or decline a bounded off-hours 4-user load test using dedicated test accounts before any capacity-increasing change Co-Authored-By: Hermes Agent <hermes-automation@bstein.dev>

hermes-automation added 2 commits 2026-08-21 12:40:28 +00:00

fix(monitoring): measure Atlas availability honestly across telemetry gaps 2862594c62

The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.

The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.

The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.

The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.

Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.

docs(hermes): add multi-user chat capacity assessment d8ccf4a6fa

Read-only investigation of the chat request path, replica/resource
manifests, and live node metrics. Documents the confirmed 4-user
tenant-slot ceiling, a TENANT_SLOTS=8 vs replicas=4 configuration
drift, the shared Claude-broker concurrency=2 bottleneck, missing
HPA/PDB/staging environment, and a proposed SLO/load-test and staged
scaling plan pending human approval. No production manifests changed.

Co-Authored-By: Hermes Agent <hermes-automation@bstein.dev>

bstein approved these changes 2026-08-21 23:14:28 +00:00

bstein added 1 commit 2026-08-21 23:14:37 +00:00

Merge branch 'main' into hermes/t_65356568-multiuser-capacity-assessment

Tests / Declarative: Post Actions failed: 41, skipped: 19, passed: 2740

Details

8ce45159de

bstein changed title from ~~WIP: docs(hermes): add multi-user chat capacity assessment~~ to docs(hermes): add multi-user chat capacity assessment

2026-08-21 23:14:42 +00:00

bstein merged commit 4c9f48cbbc into main

2026-08-21 23:14:48 +00:00

bstein referenced this issue from a commit

2026-08-21 23:14:50 +00:00

Merge pull request 'docs(hermes): add multi-user chat capacity assessment' (#45) from hermes/t_65356568-multiuser-capacity-assessment into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: titan/atlas-iac#45