Supersede draft PR #26 with a merge-safe prerequisite: bake and preload the amy, irina, and claude Piper models, route only validated server-side language to fixed voices, and leave the live voice deployment manifest unchanged.
Remove the pinned WebUI speaker selector and its persisted preference, omit client voice fields from every outbound TTS path, and keep hands-free Voice Mode and the conversation instrument intact. Hostile or legacy voice fields remain ignored by the Piper server.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Read-only investigation of the chat request path, replica/resource
manifests, and live node metrics. Documents the confirmed 4-user
tenant-slot ceiling, a TENANT_SLOTS=8 vs replicas=4 configuration
drift, the shared Claude-broker concurrency=2 bottleneck, missing
HPA/PDB/staging environment, and a proposed SLO/load-test and staged
scaling plan pending human approval. No production manifests changed.
Co-Authored-By: Hermes Agent <hermes-automation@bstein.dev>
Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.
The rename is additive, matching the shape #38 restored for chat and triage.
CoreDNS, both agent Ingresses and the hermes-sites certificate now serve BOTH
names, so merging this cannot take away the endpoint the operator uses to
reach the coordinator. Retiring agent.hermes.bstein.dev is a separate,
separately scheduled change. No redirect middleware is added.
What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak agent proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default
What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
rollback only needs the oauth2-proxy --redirect-url reverted and does not
require re-running the ensure job
The agent client passes its legacy origin through the optional fourth argument
#38 added to ensure_proxy_client, so no second mechanism is introduced. The
immutable ensure Job goes -11 -> -12 because #38 already consumed -11 and that
run has completed; without a further bump this change would never be applied.
Login on the new host fails until the -12 Job completes.
Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.
#38's public-host continuity test now covers the agent proxy's dual origins
rather than asserting the agent surface was untouched by the rename.
Knowledge catalogs and diagrams regenerated with `make knowledge`.
`make knowledge` output on main no longer matched the manifests it renders
from. The legacy chat/triage hosts restored by #38 were missing from the
committed HTTP catalogs and diagrams, along with Flux kustomizations and
Grafana panels added since the last regeneration.
Pure `make knowledge` run against unmodified main, separated into its own
commit so the hostname rename that follows reviews as a hostname rename and
nothing else. No hand edits.
The #34 rename dropped the legacy chat/triage names from the certificate
SANs, the hermes-sites Ingress, the CoreDNS overrides and the Keycloak
ensure script at the same time, so nothing failed loudly: DNS and TLS
still looked healthy while the legacy hosts served 404 and the renamed
hosts could not finish a login.
Pin the invariant that makes that silent: a public host is either served
by all four layers or by none. The table of hosts is the contract, so
retiring a name stays a deliberate edit rather than a side effect.
Verified to catch the regression: against the pre-fix tree these fail for
both legacy hosts on all four layers (9 failures); against this branch
the suite is green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR #34 renamed the public chat/triage hosts in place rather than adding
the new names, so chat.hermes.bstein.dev and triage.hermes.bstein.dev
were dropped from the certificate SANs, the hermes-sites Ingress rules
and the CoreDNS overrides at once. Both legacy hosts now answer 404 with
Traefik's default self-signed certificate, and the renamed hosts cannot
complete a login because the Keycloak clients still carry the old
redirect URIs, so chat and triage are unreachable on every hostname.
Make the rename additive, which is the rollback path the post-merge
runbook asks for when the OIDC step fails:
- put the legacy names back on hermes-sites-tls and on the Ingress,
pointing at the same oauth2-proxy backends
- restore both CoreDNS host overrides for in-cluster resolution
- teach ensure_proxy_client to register an optional legacy origin, so
hermes-chat-proxy and hermes-triage-proxy accept the old and new
redirect URIs, web origins and post-logout origins at the same time
while rootUrl stays on the canonical new host
- bump the immutable ensure Job so Flux reruns the script
Serving both names is deliberate: oauth2-proxy cookies are host-bound,
so redirecting the legacy hosts would silently drop live sessions.
Retiring them stays a separate, explicit change.
Supersedes #36, which only bumped the Job and would have left the
legacy hosts dark.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Capacity/auth/quota failover for cli-auto previously called select_route
with a hardcoded manual lane (cli-{alternate}-{effort}), so the retry
boundary was classified as switchyard-manual instead of going through
Jetson automatic classification, making the routing evidence misleading.
Now the retry calls select_route(context, "cli-auto", exclude_provider=...)
so the boundary stays automatically classified with an explicit
failed-provider exclusion. If the classifier reclassifies to a lower
effort than the original route, the lane re-pins the chosen provider at
the original effort floor so a capacity failure never silently downgrades
a high/xhigh task. Manual lanes (assignee != cli-auto) remain unchanged
and still fail closed without switching providers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Returning to chat.hermes.bstein.dev after a Keycloak logout/login showed
"This session is unavailable to this account. Start a new chat." even
though the session was intact and owned by the same subject.
The banner comes from the continuity fallback the router injects into
every chat page. It polled `/api/sessions/<id>` and
`/api/sessions/<id>/messages` — routes that belong to the Hermes agent
dashboard (added by scripts/patch_web_session_activity.py, applied only
in agent-deployment.yaml). The router proxies browser traffic to the
tenant Hermes WebUI instead, whose only session read is
`GET /api/session?session_id=<id>`; the dashboard paths are unrouted
there, so server.py answered its generic 404 for every poll and the
fallback reported a false ownership failure.
The script runs only on a full document load of `/session/<id>`, which is
exactly what the OIDC round-trip produces when oauth2-proxy returns the
browser to `rd=/session/<id>` — hence the "only after relogin" symptom.
Poll the WebUI contract instead, and let its own answers decide what the
banner claims: 409 `session_profile_mismatch` is the single response that
means the session is outside this account's active scope, 404 now means
the conversation is no longer stored, and 401/403 still re-enter OIDC.
The steady-state poll drops to one request and backs off to 3s/15s now
that it reaches a real endpoint on the tenant Raspberry Pi.
`boundSessionSnapshot` follows the same move: it caps the WebUI envelope
`{"session": {..., "messages": [...]}}`, relaying every other session key
verbatim rather than re-serializing a fixed struct that would silently
drop metadata the banner depends on.
Isolation is unchanged and now covered: the router still resolves the
slot from the salted Keycloak subject, overwrites any client-supplied
X-Hermes-Tenant-Identity, and forwards only the two tenant cookies.
Tests: relogin keeps a stable slot and resolves the durable session; a
second subject replaying the owner's session id, WebUI cookie and a
forged tenant header gets 404 from its own backend and never reaches the
owner's; the legacy dashboard paths are pinned as permanent 404s against
a stub of the deployed WebUI dispatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.
The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.
The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.
The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.
Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.
