Merge pull request 'docs(hermes): add multi-user chat capacity assessment' (#45) from hermes/t_65356568-multiuser-capacity-assessment into main
Reviewed-on: atlas/titan-iac#45 Reviewed-by: bstein <bstein@noreply.scm.bstein.dev>
This commit is contained in:
commit
4c9f48cbbc
349
docs/hermes_multiuser_capacity_assessment.md
Normal file
349
docs/hermes_multiuser_capacity_assessment.md
Normal file
@ -0,0 +1,349 @@
|
||||
# Hermes multi-user capacity assessment
|
||||
|
||||
Status: read-only investigation, no production changes made. This document
|
||||
answers "can Hermes safely handle more simultaneous users" with evidence from
|
||||
the current `titan-iac` manifests/code, plus non-invasive live cluster reads.
|
||||
It does not increase any replica count or concurrency cap.
|
||||
|
||||
## 0. Method and access boundary
|
||||
|
||||
- Static evidence: `services/hermes/**` manifests and `services/hermes/router/*.go`,
|
||||
`services/hermes/scripts/*.py` on branch `wt/t_65356568`, `git log`/`git blame`
|
||||
for recent drift.
|
||||
- Live evidence: read via the in-cluster `atlas-observer` kubeconfig context.
|
||||
This identity (`system:serviceaccount:hermes:hermes-agent`) is **observer-only**:
|
||||
`kubectl -n hermes get pods/deploy/sts/hpa/pdb` all returned `Forbidden`.
|
||||
`kubectl -n hermes auth can-i --list` confirms the only granted verbs are
|
||||
cluster-scoped `get/list/watch` on `namespaces`, `nodes`, `persistentvolumes`,
|
||||
storage APIs, and `nodes.metrics.k8s.io`, plus API discovery/health endpoints.
|
||||
There is no namespace-scoped read access to pods, deployments, replica counts,
|
||||
or pod-level resource usage in `hermes` (or any namespace) from this session.
|
||||
This matches the standing finding that hermes-agent RBAC is observer-only.
|
||||
- What this means for this report: **all replica counts, resource
|
||||
requests/limits, probes, and affinity rules below are read from the Git
|
||||
manifests** (which is what Flux actually reconciles, so it is a trustworthy
|
||||
source for desired state) rather than confirmed against live pod status.
|
||||
**Node-level capacity and current node CPU/memory utilization are live,
|
||||
real-time cluster evidence** (`nodes.metrics.k8s.io` + `nodes` are both
|
||||
readable). One non-invasive HTTP probe against `https://chat.bstein.dev/`
|
||||
confirmed the public endpoint is live and redirecting to auth (`302`) at
|
||||
the time of writing.
|
||||
- No pods, deployments, config, or replica counts were changed. No load test
|
||||
was executed against any environment.
|
||||
|
||||
## 1. Architecture and request path
|
||||
|
||||
```
|
||||
browser
|
||||
-> Traefik Ingress (chat.hermes.bstein.dev / chat.bstein.dev)
|
||||
-> oauth2-proxy-hermes-chat (Deployment, replicas: 1, oauth2-proxy.yaml:153)
|
||||
session state: Redis, externalized (--session-store-type=redis,
|
||||
redis://hermes-oauth-sessions...:6379/1, oauth2-proxy.yaml:203-204)
|
||||
-> hermes-chat-router (Deployment, replicas: 1, chat-router.yaml:10)
|
||||
Go binary, services/hermes/router/main.go
|
||||
in-process, PVC-persisted identity -> slot map (tenants.json)
|
||||
-> hermes-chat-tenant-<slot>.hermes-chat-tenant (StatefulSet, replicas: 4,
|
||||
chat-statefulset.yaml:13) — one pod per Keycloak subject, pinned by ordinal
|
||||
webui (8787) -> gateway (8642) -> per-tenant sandbox (chat-sandbox-<ordinal>:9080)
|
||||
-> hermes-switchyard (Deployment, replicas: 1, switchyard-deployment.yaml:10)
|
||||
shared model router for ALL tenants + triage + Kanban
|
||||
-> hermes-claude-broker / hermes-codex-broker (containers in hermes-agent pod,
|
||||
replicas: 1, agent-deployment.yaml:10) / hermes-model-gate (Deployment,
|
||||
replicas: 1, model-gate-deployment.yaml:10, proxies to local Ollama)
|
||||
-> hermes-stt / hermes-tts (Deployments, replicas: 1 each, voice-deployment.yaml,
|
||||
pinned to node titan-21, shared GPU)
|
||||
```
|
||||
|
||||
Every component on this path is `replicas: 1` **except** `hermes-chat-tenant`
|
||||
(4). The StatefulSet is not a load-balanced pool: each pod is a fully isolated,
|
||||
permanently-assigned per-user backend, not interchangeable capacity.
|
||||
|
||||
## 2. Known limits (confirmed from manifests/code)
|
||||
|
||||
### 2.1 Hard ceiling: 4 usable chat users today, and a live configuration bug
|
||||
|
||||
`services/hermes/router/main.go` (`tenantRouter.slotFor`, lines 153-176)
|
||||
permanently assigns each Keycloak subject a slot `0..TENANT_SLOTS-1` on first
|
||||
login, persists it to a PVC-backed JSON file, and never releases it. There is
|
||||
no unassign/deprovision path in the router. Unit tests
|
||||
(`router/main_test.go`, "expected the fixed private tenant pool to report
|
||||
capacity") assert that once all slots are taken, `slotFor` returns an error.
|
||||
|
||||
**Confirmed drift**: `chat-router.yaml:70` sets `TENANT_SLOTS=8`, but
|
||||
`chat-statefulset.yaml:13` sets `replicas: 4`. `git log -p` on both files
|
||||
shows they were changed together at `18965a2f` (4->8), then
|
||||
`chat-statefulset.yaml` was scaled down twice — `79baa7ec` ("return two chat
|
||||
slots to fit the agent runtime", 8->6) and `e28b32bd` (6->4) — **without a
|
||||
matching change to `TENANT_SLOTS`**, and `chat-sandbox.yaml` still defines 8
|
||||
per-ordinal sandbox Deployments (`hermes-chat-sandbox-0` through `-7`).
|
||||
Net effect: the 5th–8th distinct Keycloak login is durably assigned a slot
|
||||
that has no backing StatefulSet pod. `main.go`'s proxy error path (around
|
||||
line 382) returns a "starting" error to that user **permanently**, not just
|
||||
during a cold start, because the pod will never exist at the current replica
|
||||
count. This is a pre-existing configuration bug, independent of any future
|
||||
scale-up decision, and it silently masks the real behavior of the capacity
|
||||
ceiling (a 5th user does not get a clean "at capacity" rejection matching the
|
||||
tested behavior — they get a StatefulSet-pod-not-found style failure).
|
||||
**This assessment does not fix it** (per scope: no production changes), but
|
||||
it should be the first fix considered before any scale-out, since it affects
|
||||
correctness at the current replica count too.
|
||||
|
||||
### 2.2 The real shared bottleneck is not the 4 tenant pods, it's the brokers behind them
|
||||
|
||||
All 4 tenant pods, the triage/operator instance, and Kanban/CLI-lane traffic
|
||||
funnel through the same single-replica `hermes-switchyard`, which routes to
|
||||
the same single-replica `hermes-claude-broker` / `hermes-codex-broker`
|
||||
(containers inside the one `hermes-agent` pod) or `hermes-model-gate`.
|
||||
|
||||
- `hermes-claude-broker` enforces a hard, code-level concurrency cap:
|
||||
`HERMES_CLAUDE_BROKER_CONCURRENCY=2` (`agent-deployment.yaml:1160`) sizes a
|
||||
`threading.BoundedSemaphore(2)` (`scripts/claude_oauth_broker.py:63`)
|
||||
wrapped around a blocking `subprocess.run` call to the native Claude Code
|
||||
CLI. Requests over the cap **block silently** (no 429/503 to the caller)
|
||||
until a slot frees. This cap is shared cluster-wide across every chat
|
||||
tenant, the triage instance, and indirectly the Kanban/CLI-lane workers
|
||||
(same provider credentials, so heavy CLI-lane usage can throttle chat
|
||||
responses via the provider's own rate limits even without touching this
|
||||
semaphore directly).
|
||||
- `hermes-codex-broker` and `hermes-model-gate` have **no equivalent
|
||||
concurrency cap in code** — they are bounded only by container CPU/memory
|
||||
`limits` (1 CPU / 1Gi for the codex-broker container, 250m/128Mi for
|
||||
model-gate) and by the single upstream Ollama/Jetson instance's real
|
||||
throughput, which is unknown (no load test exists).
|
||||
|
||||
**Implication**: scaling `hermes-chat-tenant` replicas alone (e.g. 4 -> 8)
|
||||
does not proportionally increase real AI-response throughput. The
|
||||
Claude-routed share of traffic is capped at 2 concurrent completions
|
||||
regardless of how many tenant pods exist, and Codex/local-model throughput
|
||||
under concurrent load is untested.
|
||||
|
||||
### 2.3 Single points of failure and deploy-time outage windows
|
||||
|
||||
- No `HorizontalPodAutoscaler` exists anywhere in this repository (confirmed
|
||||
by a repo-wide grep, not just `services/hermes/`).
|
||||
- No `PodDisruptionBudget` exists for any hermes component (the one PDB in
|
||||
the repo is `infrastructure/traefik/pdb.yaml`, unrelated).
|
||||
- `oauth2-proxy`, `hermes-chat-router`, `hermes-switchyard`,
|
||||
`hermes-model-gate`, `hermes-agent` (brokers), and voice STT/TTS are all
|
||||
`replicas: 1`. A pod restart/crash/node drain on any of them takes down
|
||||
chat, voice, or model access for **all** users simultaneously, even though
|
||||
the tenant StatefulSet gives each user an isolated conversation backend.
|
||||
- `hermes` (main/triage), `hermes-chat-router`, `hermes-local-image`, and
|
||||
`hermes-agent` all use `strategy: Recreate` — every rollout is a full
|
||||
teardown-then-recreate, not a rolling update, adding a real availability
|
||||
gap bounded by readiness/liveness `initialDelaySeconds` (up to 90s+ for the
|
||||
main `hermes` deployment). The only genuinely zero-downtime rollout path
|
||||
in this whole service is `hermes-switchyard`
|
||||
(`RollingUpdate`, `maxSurge:1`, `maxUnavailable:0`) and the tenant
|
||||
StatefulSet itself (`RollingUpdate`, `podManagementPolicy: Parallel`).
|
||||
- OAuth session storage (`hermes-oauth-sessions`, a single Redis
|
||||
`Deployment`, `replicas: 1`, `oauth-session-store.yaml`) has no
|
||||
replication; its loss logs out every user cluster-wide (sessions are
|
||||
externalized from oauth2-proxy but not made highly available).
|
||||
|
||||
### 2.4 The cluster is a heterogeneous home-lab/edge fleet with real, uneven headroom
|
||||
|
||||
Live `nodes.metrics.k8s.io` reads (non-invasive) show 21 nodes: mostly
|
||||
Raspberry Pi 4/5 workers (~3.6-4 vCPU / ~6.5-8GB allocatable each), plus a
|
||||
few larger amd64/Jetson-class nodes with shared GPUs (titan-20/21, titan-22,
|
||||
titan-24) and one large amd64 node (titan-23, 48 vCPU/256GB, no GPU, not
|
||||
labeled `worker`). This is **not** elastic cloud capacity — headroom is
|
||||
finite and already uneven:
|
||||
|
||||
| Node | CPU used/alloc | Mem used/alloc | Note |
|
||||
|---|---|---|---|
|
||||
| titan-12 | 3.95 / 3.60 (**110%**) | 5.65 / 6.50 (87%) | rpi4, already over its CPU allocatable at snapshot time |
|
||||
| titan-13 | 3.45 / 3.60 (96%) | 5.85 / 6.50 (90%) | rpi4, hot |
|
||||
| titan-17 | 2.71 / 3.60 (75%) | 5.54 / 6.50 (85%) | rpi4, hot on memory |
|
||||
| titan-07 | 1.23 / 3.60 (34%) | 5.24 / 6.63 (79%) | rpi5, memory-constrained |
|
||||
| titan-04 | 0.35 / 3.60 (10%) | 1.69 / 6.63 (26%) | rpi5, most headroom in the pool |
|
||||
| titan-21 (voice, GPU) | 0.91 / 6.00 (15%) | 8.02 / 14.56 (55%) | shared GPU node, moderate headroom |
|
||||
| titan-23 (48 vCPU/256GB, no GPU label) | 0.74 / 48 (1.5%) | 22.9 / 251.6 (9%) | large headroom, not currently used by hermes affinity rules (arm64-only selectors exclude it) |
|
||||
| titan-24 (GPU, Wolf/Hermes shared) | 1.16 / 24 (4.8%) | 22.6 / 62.7 (36%) | model-gate/Ollama contention with Wolf, per NOTES.md |
|
||||
|
||||
`chat-statefulset.yaml`'s node affinity (lines 54-74) requires `arm64` +
|
||||
`node-role.kubernetes.io/worker=true` and already excludes 7 specific
|
||||
hostnames (`titan-05,08,13,14,17,18,19`) — but two of the nodes it currently
|
||||
*can* still land on, **titan-12 and titan-15, are already running at 75-110%
|
||||
of allocatable CPU**. Any additional tenant/broker/router replica scheduled
|
||||
into this same arm64 worker pool inherits that contention; it is not safe to
|
||||
assume "add a replica" has free capacity behind it without checking node
|
||||
headroom at scale-out time. The large amd64 node (titan-23) has abundant
|
||||
headroom but is excluded by the current `arm64`-only affinity rules on every
|
||||
hermes component — worth a design question, not something this assessment
|
||||
changes.
|
||||
|
||||
### 2.5 No staging/canary environment exists
|
||||
|
||||
`services/hermes/kustomization.yaml` is a single flat kustomization with no
|
||||
overlay split. `services/hermes-chat/` contains only a namespace and one PVC
|
||||
(vestigial, no workload). `services/hermes-triage-demo/` is unrelated
|
||||
(Ariadne's automated-repair demo loop, no chat workload). **Any load test
|
||||
that exercises the real chat path runs against the production namespace and
|
||||
the production `hermes-chat-tenant` StatefulSet** — there is no isolated
|
||||
copy to test against.
|
||||
|
||||
### 2.6 No load-test tooling and thin production metrics exist today
|
||||
|
||||
- No k6/locust/vegeta/wrk/artillery/ab scripts exist anywhere in the repo
|
||||
for hermes. A load test has to be built from scratch.
|
||||
- No `ServiceMonitor`/`PodMonitor`/`PrometheusRule` objects exist for hermes
|
||||
(annotation-based scraping only). `/metrics` exists for the `hermes-agent`
|
||||
pod (CLI/provider quota usage, port 9010), `hermes-switchyard` (port 9005,
|
||||
the richest signal — `switchyard_requests_total`,
|
||||
`switchyard_model_call_latency_ms_bucket`,
|
||||
`switchyard_routing_overhead_ms_bucket`, `switchyard_errors_total`,
|
||||
`switchyard_client_responses_total`), and the CLI-lane metrics service
|
||||
(port 9011, agent/Kanban-lane specific). **No `/metrics` endpoint exists
|
||||
on `hermes-chat-tenant`, `hermes-model-gate`, `oauth2-proxy`,
|
||||
`hermes-chat-router`, or voice** — there is no per-request latency/error
|
||||
signal from the actual multi-user chat pods themselves.
|
||||
- `services/monitoring/grafana-dashboard-ai.yaml` already has a P95
|
||||
model-call-latency panel, a success/error-rate panel, and a per-pod CPU
|
||||
panel filtered to `hermes-(agent|chat-tenant|switchyard|model-gate).*`,
|
||||
all sourced from Switchyard metrics + cAdvisor — a usable baseline to
|
||||
reuse, but nothing is wired to an alert threshold, and there is no
|
||||
queue-depth or replica-health panel.
|
||||
|
||||
## 3. Unknowns (require an actual test to answer)
|
||||
|
||||
- Real max throughput of `hermes-switchyard`, `hermes-model-gate` -> Ollama,
|
||||
and the STT/TTS GPU path under concurrent load — no data exists today.
|
||||
- Real behavior/latency of the Claude broker's semaphore-2 queue under
|
||||
contention (does it degrade gracefully to slow responses, or eventually
|
||||
time out and error?) — no data exists today.
|
||||
- Whether oauth2-proxy + Redis session store can handle concurrent
|
||||
logins/refreshes at a higher user count (currently untested; the
|
||||
Deployment is stateless-safe to scale, but has never been scaled).
|
||||
Also whether Redis itself (`hermes-oauth-sessions`, 1Gi PVC, no HA)
|
||||
becomes a bottleneck or SPOF under more simultaneous sessions.
|
||||
- Actual current pod-level CPU/memory usage and restart history for any
|
||||
hermes component — not observable with the current RBAC grant. Only
|
||||
node-level aggregate usage is visible, not which pods are consuming it.
|
||||
- Whether scaling `hermes-chat-tenant` toward its already-configured
|
||||
`TENANT_SLOTS=8` (after fixing the mismatch) is actually safe given
|
||||
current node headroom (titan-12/titan-15 pressure, see 2.4) — needs a
|
||||
scheduling dry-run / headroom check at decision time, not assumed from
|
||||
this snapshot.
|
||||
|
||||
## 4. SLO and load-test proposal (not yet executed — needs approval, see §6)
|
||||
|
||||
### 4.1 Target SLOs to validate (proposed, for review)
|
||||
|
||||
| Metric | Proposed target | Source |
|
||||
|---|---|---|
|
||||
| P95 end-to-end chat response latency | < 8s for local/Codex-routed, < 20s for Claude-routed (queued behind semaphore-2) | new synthetic client timer + existing `switchyard_model_call_latency_ms_bucket` |
|
||||
| Error rate (5xx / proxy failures) | < 1% sustained | new synthetic client + `switchyard_errors_total` / `switchyard_client_responses_total` |
|
||||
| Login/session success rate | 100% (oauth2-proxy + Redis) | synthetic client HTTP status |
|
||||
| Node CPU/mem headroom during test | stay below 90% allocatable on any node hosting hermes pods | `nodes.metrics.k8s.io` (already provably readable) |
|
||||
| Claude-broker queue depth / wait time | document actual behavior, no target yet (unknown baseline) | needs new instrumentation — none exists today |
|
||||
|
||||
### 4.2 Concurrency / ramp / mix proposal
|
||||
|
||||
- **Concurrency levels**: 1 (baseline), 2 (at the Claude-broker cap), 4
|
||||
(at today's real tenant-pool ceiling), and — only if §2.1's mismatch is
|
||||
resolved and node headroom is re-checked — 8 (at the configured
|
||||
`TENANT_SLOTS`). Do not exceed the number of actually-backed tenant slots.
|
||||
- **Ramp**: hold each level for at least 10 minutes after reaching steady
|
||||
state before increasing, to separate cold-start effects (webui/gateway
|
||||
startup, model warm-up) from steady-state capacity.
|
||||
- **Traffic mix**: proportion requests across the three model routes
|
||||
(Claude / Codex / local Ollama) matching real usage if known, otherwise
|
||||
test each route in isolation first (to characterize each bottleneck
|
||||
independently — semaphore-2 vs. no-cap-but-untested), then a blended mix.
|
||||
- **Session behavior**: each synthetic user must be a distinct, dedicated
|
||||
test Keycloak identity (slot assignment is permanent — do not burn real
|
||||
user slots or reuse production accounts for load generation).
|
||||
- **Failure injection (optional, staged)**: kill the single
|
||||
`hermes-switchyard` or `hermes-model-gate` pod mid-test to confirm the
|
||||
`Recreate`/no-PDB SPOF behavior matches expectations and measure recovery
|
||||
time, only after the throughput characterization above is complete.
|
||||
|
||||
## 5. Safe staged scaling plan (for future execution, after human approval)
|
||||
|
||||
This is a plan to evaluate, not a plan already executed.
|
||||
|
||||
1. **Fix the `TENANT_SLOTS`/replica-count mismatch first** (§2.1). This is a
|
||||
correctness fix at the *current* capacity, not a capacity increase, and
|
||||
should land before any load test so results aren't confounded by the
|
||||
existing 502-on-phantom-slot bug.
|
||||
2. **Instrument before scaling**: add `/metrics` (or at minimum structured
|
||||
access logs) to `hermes-chat-tenant`, `hermes-chat-router`, and
|
||||
`hermes-model-gate` so a load test has a real per-request signal instead
|
||||
of relying solely on Switchyard's aggregate view.
|
||||
3. **Baseline test at today's real ceiling (4 users)**, off-hours, using
|
||||
dedicated test Keycloak identities, per §4.
|
||||
4. **Re-check node headroom** (§2.4) at decision time, not from this
|
||||
snapshot, before considering any StatefulSet scale-out — titan-12 and
|
||||
titan-15 were already at/above CPU allocatable during this assessment.
|
||||
5. **If and only if** the baseline is healthy and headroom allows, propose
|
||||
scaling `hermes-chat-tenant` toward the already-configured 8 slots one
|
||||
or two ordinals at a time (StatefulSet `podManagementPolicy: Parallel`
|
||||
supports this), re-measuring node headroom and Switchyard/broker latency
|
||||
after each step.
|
||||
6. **Address the shared-broker ceiling separately from tenant-pod count**:
|
||||
raising `HERMES_CLAUDE_BROKER_CONCURRENCY` above 2 is a distinct decision
|
||||
gated by the underlying Claude Code provider's real rate limits, not by
|
||||
Kubernetes resources — do not conflate it with StatefulSet scaling.
|
||||
7. **Only after tenant/broker capacity is validated**, consider whether the
|
||||
single-replica shared components (`hermes-switchyard`, `hermes-model-gate`,
|
||||
`oauth2-proxy`) need a second replica or a PDB — note oauth2-proxy is the
|
||||
only one of these that is currently stateless-safe to scale (Redis-backed
|
||||
sessions); the others would need code changes first (in-process locks/
|
||||
state noted in §2).
|
||||
|
||||
## 6. Rollback and observability gates
|
||||
|
||||
- Every step above is a Git-reviewed manifest change applied via Flux, so
|
||||
rollback is `git revert` + Flux reconcile, consistent with this repo's
|
||||
existing GitOps model — no manual `kubectl edit` in any step.
|
||||
- Gate each scaling step on: node CPU/mem headroom (§2.4 thresholds), the
|
||||
new `/metrics` signal from step 2 above staying within the SLOs in §4.1,
|
||||
and zero increase in `switchyard_errors_total` / `switchyard_classifier_fail_open_total`
|
||||
rate versus the pre-change baseline.
|
||||
- Because there is no staging environment (§2.5) and no PDB, treat every
|
||||
step as a production change: schedule off-hours, announce before/after,
|
||||
and keep the prior manifest revision ready to revert immediately if error
|
||||
rate or node pressure crosses the gate.
|
||||
- Because RBAC here is observer-only (§0), whoever executes a future load
|
||||
test needs either elevated read access to confirm live pod/replica status
|
||||
during the test, or must rely entirely on the node-metrics + Switchyard
|
||||
signals already proven readable in this assessment.
|
||||
|
||||
## 7. Recommendation
|
||||
|
||||
**A load test against the current 4-user ceiling is reasonably safe to run
|
||||
off-hours with dedicated test accounts, once §5 step 1 (the `TENANT_SLOTS`
|
||||
mismatch) and step 2 (baseline metrics) are addressed** — it does not
|
||||
require any replica/cap increase, and the blast radius is bounded to
|
||||
resources already provisioned.
|
||||
|
||||
**A test at 8 concurrent users, or any StatefulSet/broker scale-up, is not
|
||||
yet safe to schedule.** Two of the nodes in the current chat-tenant affinity
|
||||
pool are already at or above their allocatable CPU/memory (§2.4), there is
|
||||
no PDB/HPA anywhere in this service, there is no staging environment to
|
||||
absorb the risk, and the real throughput ceiling for Codex/local-model
|
||||
routes and the actual behavior of the Claude-broker queue under load are
|
||||
both unmeasured. Recommend running the 4-user baseline test first and using
|
||||
its results (plus a fresh node-headroom check) to decide whether 8-user
|
||||
scaling is warranted — this assessment does not make that call.
|
||||
|
||||
## 8. Requested input / blockers to proceed further
|
||||
|
||||
This assessment can be delivered without further input. To actually execute
|
||||
the load test proposed in §4, a human owner needs to provide or approve:
|
||||
|
||||
1. Sign-off to fix the `TENANT_SLOTS`/replica mismatch (§2.1) as a
|
||||
correctness PR, separate from any capacity increase.
|
||||
2. Dedicated test Keycloak identities (slot assignment is permanent —
|
||||
production user accounts must not be used for load generation).
|
||||
3. An approved off-hours maintenance window, since there is no staging
|
||||
environment and every step in §5 is a production change.
|
||||
4. A decision on whether to add `/metrics` instrumentation (§5 step 2)
|
||||
before or in parallel with the first baseline test, since today's
|
||||
Switchyard-only signal cannot attribute latency/errors to a specific
|
||||
tenant pod.
|
||||
5. If deeper live verification is wanted beyond node-level metrics
|
||||
(e.g., live pod status/restart counts during a test), temporary
|
||||
elevated read RBAC for the executing identity, since the current
|
||||
`atlas-observer` grant cannot list pods/deployments in `hermes` (§0).
|
||||
@ -148,7 +148,7 @@ INFRA_REGEX = f"^({'|'.join(INFRA_PATTERNS)})$"
|
||||
CP_ALLOWED_NS = INFRA_REGEX
|
||||
LONGHORN_NODE_REGEX = "titan-1[2-9]|titan-2[2-4]"
|
||||
ALL_NODE_REGEX = "|".join(CONTROL_ALL + WORKER_NODES)
|
||||
GAUGE_WIDTHS = [4, 3, 3, 4, 3, 3, 4]
|
||||
GAUGE_WIDTHS = [3, 3, 3, 3, 3, 3, 3, 3]
|
||||
CONTROL_WORKLOADS_EXPR = (
|
||||
f'sum(kube_pod_info{{node=~"{CONTROL_REGEX}",namespace!~"{CP_ALLOWED_NS}"}}) or on() vector(0)'
|
||||
)
|
||||
@ -467,21 +467,18 @@ UPTIME_WINDOW = "365d"
|
||||
UPTIME_RECORDING_METRIC = (
|
||||
f'atlas:availability:ratio_{UPTIME_WINDOW}{{scope="atlas",definition="request-v4"}}'
|
||||
)
|
||||
AVAILABILITY_REQUESTS_1H_EXPR = (
|
||||
'sum(increase(traefik_entrypoint_requests_total{'
|
||||
'entrypoint="websecure",protocol="http",code=~"[1-5].."}[1h]))'
|
||||
)
|
||||
AVAILABILITY_FAILURES_1H_EXPR = (
|
||||
'sum(increase(traefik_entrypoint_requests_total{'
|
||||
'entrypoint="websecure",protocol="http",code=~"5.."}[1h]))'
|
||||
)
|
||||
UPTIME_LIVE_FALLBACK_EXPR = (
|
||||
f"(1 - (({AVAILABILITY_FAILURES_1H_EXPR} or on() vector(0)) / "
|
||||
f"clamp_min({AVAILABILITY_REQUESTS_1H_EXPR}, 1)))"
|
||||
)
|
||||
UPTIME_RECORDING_EXPR = (
|
||||
f"(last_over_time({UPTIME_RECORDING_METRIC}[48h]) "
|
||||
f"or on() {UPTIME_LIVE_FALLBACK_EXPR})"
|
||||
# The 2026-08-18 storage outage proved a fallback here lies twice at once:
|
||||
# when the yearly sample went stale the panel silently rendered the last hour
|
||||
# of Traefik traffic under a 365d title, while the unmeasured gap vanished
|
||||
# from the yearly ratio as if it had been observed. A stale rollup must render
|
||||
# no value and page (atlas-availability-rollup-stale), never substitute a
|
||||
# different measurement under the same label.
|
||||
UPTIME_RECORDING_EXPR = f"last_over_time({UPTIME_RECORDING_METRIC}[48h])"
|
||||
# Days of daily availability rollups the yearly figure actually rests on,
|
||||
# published by the same rollup job so freshness matches the ratio sample.
|
||||
UPTIME_COVERAGE_EXPR = (
|
||||
"last_over_time(atlas:availability:coverage_days_365d"
|
||||
'{scope="atlas",definition="request-v4"}[48h])'
|
||||
)
|
||||
|
||||
# Tie-breaker to deterministically pick one node per namespace when shares tie.
|
||||
@ -511,6 +508,18 @@ UPTIME_PERCENT_THRESHOLDS = {
|
||||
{"color": "blue", "value": 0.99999},
|
||||
],
|
||||
}
|
||||
# Coverage is a disclosure, not an SLO: a short history is honest, a shrinking
|
||||
# one means telemetry is being lost. The request-v4 definition was backfilled
|
||||
# to 2026-05-01, so the figure legitimately sits well below a full year.
|
||||
UPTIME_COVERAGE_THRESHOLDS = {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{"color": "red", "value": None},
|
||||
{"color": "orange", "value": 7},
|
||||
{"color": "yellow", "value": 30},
|
||||
{"color": "green", "value": 90},
|
||||
],
|
||||
}
|
||||
PROBLEM_TABLE_EXPR = (
|
||||
"(time() - kube_pod_created{pod!=\"\"}) "
|
||||
"* on(namespace,pod) group_left(node) kube_pod_info "
|
||||
@ -1910,7 +1919,8 @@ OVERVIEW_PANEL_DESCRIPTIONS = {
|
||||
"Control Plane Ready": "Control-plane nodes currently Ready; full count is good, lower means Kubernetes core capacity is missing.",
|
||||
"Control Plane Workloads": "Non-core pods running on control-plane nodes; zero is good because control nodes should stay focused.",
|
||||
"Stuck Terminating": "Pods that Kubernetes cannot finish deleting; zero is good, growth means cleanup or storage may be stuck.",
|
||||
"Atlas Availability (365d)": "Request-weighted Atlas ingress availability; every server-side 5xx response counts as a failed request.",
|
||||
"Atlas Availability (365d window)": "Request-weighted Atlas ingress availability over measured days only; telemetry gaps are excluded from the ratio, never counted as downtime.",
|
||||
"Availability Coverage (days)": "Days of rollup history behind the availability figure; a telemetry gap lowers this instead of lowering availability.",
|
||||
"Problem Pods": "Current-service pods Pending for more than 15 minutes or in an actionable failed phase. Completed Jobs and retained Veles migration workloads are kept on drill-down dashboards but excluded here.",
|
||||
"CrashLoop / ImagePull": "Current-service pods stuck in CrashLoopBackOff or ImagePullBackOff for more than 15 minutes. Retained Veles migration workloads remain visible on the Pods dashboard.",
|
||||
"Workers Ready": "Worker nodes currently Ready; full count is good, lower means less place to run services.",
|
||||
@ -2143,7 +2153,7 @@ def build_overview():
|
||||
},
|
||||
{
|
||||
"id": 27,
|
||||
"title": "Atlas Availability (365d)",
|
||||
"title": "Atlas Availability (365d window)",
|
||||
"expr": UPTIME_PERCENT_EXPR,
|
||||
"kind": "stat",
|
||||
"thresholds": UPTIME_PERCENT_THRESHOLDS,
|
||||
@ -2151,7 +2161,19 @@ def build_overview():
|
||||
"decimals": 4,
|
||||
"text_mode": "value",
|
||||
"instant": True,
|
||||
"description": "Rolling request-weighted availability at the Atlas HTTPS ingress: responses below 500 divided by all HTTP responses. Every server-side 5xx is a real failed request; client 4xx responses count as served. Replica counts, Grafana health, and monitoring gaps are not inputs, so they cannot lower availability unless user traffic actually receives a 5xx. A daily rollup job publishes one annual sample from deduplicated daily totals; Grafana keeps it for up to 48 hours so one delayed retry cannot cause a fallback, and only uses the same one-hour request SLI before history exists.",
|
||||
"description": "Request-weighted availability at the Atlas HTTPS ingress over the trailing year: responses below 500 divided by all HTTP responses, read from the daily rollup sample. Every server-side 5xx is a real failed request; client 4xx responses count as served. Only measured days enter the ratio: a telemetry gap such as the 2026-08-18 metrics-storage outage is excluded from both sides, never converted into downtime or uptime, and the Availability Coverage panel states how many days the figure actually rests on. If the rollup stops publishing, this panel goes stale and renders no value instead of quietly substituting a one-hour live ratio under a yearly title; the atlas-availability-rollup-stale alert pages first.",
|
||||
},
|
||||
{
|
||||
"id": 36,
|
||||
"title": "Availability Coverage (days)",
|
||||
"expr": UPTIME_COVERAGE_EXPR,
|
||||
"kind": "stat",
|
||||
"thresholds": UPTIME_COVERAGE_THRESHOLDS,
|
||||
"unit": "none",
|
||||
"decimals": 0,
|
||||
"text_mode": "value",
|
||||
"instant": True,
|
||||
"description": "Days of daily rollups the availability figure to the left actually rests on, out of the 365-day window. A telemetry gap such as the 2026-08-18 metrics-storage outage lowers this number instead of moving availability, which is what keeps an observability failure from being reported as Atlas downtime. The request-v4 definition begins on 2026-05-01, so coverage stays below a full year until that history accumulates.",
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
|
||||
82
scripts/tests/test_availability_measurement.py
Normal file
82
scripts/tests/test_availability_measurement.py
Normal file
@ -0,0 +1,82 @@
|
||||
"""Keep the public availability figure honest about what it measured."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
# On 2026-08-18 the metrics volume filled and the availability pipeline failed
|
||||
# in both directions at once: the stale yearly sample silently fell back to a
|
||||
# one-hour Traefik ratio wearing the 365d title, and the 34-hour telemetry gap
|
||||
# vanished from the yearly ratio as if it had been measured. These guardrails
|
||||
# pin the corrections.
|
||||
|
||||
|
||||
def _overview_panels() -> list[dict]:
|
||||
"""Return the generated Overview dashboard's top-level panels."""
|
||||
dashboard = json.loads(
|
||||
(REPO_ROOT / "services/monitoring/dashboards/atlas-overview.json").read_text()
|
||||
)
|
||||
return dashboard["panels"]
|
||||
|
||||
|
||||
def _panel(title_prefix: str) -> dict:
|
||||
return next(
|
||||
panel
|
||||
for panel in _overview_panels()
|
||||
if str(panel.get("title", "")).startswith(title_prefix)
|
||||
)
|
||||
|
||||
|
||||
def _panel_exprs(panel: dict) -> str:
|
||||
return " ".join(target.get("expr", "") for target in panel.get("targets", []))
|
||||
|
||||
|
||||
def _alert_rules() -> dict[str, dict]:
|
||||
"""Return every provisioned Grafana alert rule keyed by uid."""
|
||||
manifest = next(
|
||||
document
|
||||
for document in yaml.safe_load_all(
|
||||
(REPO_ROOT / "services/monitoring/grafana-alerting-config.yaml").read_text()
|
||||
)
|
||||
if document
|
||||
)
|
||||
groups = yaml.safe_load(manifest["data"]["rules.yaml"])["groups"]
|
||||
return {rule["uid"]: rule for group in groups for rule in group["rules"]}
|
||||
|
||||
|
||||
def test_availability_panel_never_swaps_measurements() -> None:
|
||||
"""A stale yearly rollup must render empty, not a 1h ratio in disguise."""
|
||||
panel = _panel("Atlas Availability")
|
||||
exprs = _panel_exprs(panel)
|
||||
assert "atlas:availability:ratio_365d" in exprs
|
||||
assert "traefik_entrypoint_requests_total" not in exprs
|
||||
|
||||
|
||||
def test_availability_coverage_is_disclosed() -> None:
|
||||
"""The overview must state how many days the yearly figure rests on."""
|
||||
panel = _panel("Availability Coverage")
|
||||
assert "atlas:availability:coverage_days_365d" in _panel_exprs(panel)
|
||||
|
||||
|
||||
def test_stale_availability_rollup_pages() -> None:
|
||||
"""A silent rollup once hid 34 hours of lost publishes; staleness must page."""
|
||||
rules = _alert_rules()
|
||||
stale = rules["atlas-availability-rollup-stale"]
|
||||
assert "atlas:availability:ratio_365d" in stale["data"][0]["model"]["expr"]
|
||||
assert stale["noDataState"] == "Alerting"
|
||||
assert stale["execErrState"] == "Alerting"
|
||||
assert stale["labels"]["severity"] == "warning"
|
||||
|
||||
|
||||
def test_rollup_proves_its_publish_survived() -> None:
|
||||
"""A read-only store accepts imports and drops them; the job must read back."""
|
||||
source = (
|
||||
REPO_ROOT / "services/monitoring/scripts/availability_rollup.py"
|
||||
).read_text()
|
||||
assert "def verify_stored" in source
|
||||
assert "verify_stored(OUTPUT_METRIC" in source
|
||||
assert "atlas:availability:coverage_days_365d" in source
|
||||
@ -43,10 +43,14 @@ def test_calculate_availability_uses_all_server_failures() -> None:
|
||||
|
||||
|
||||
def test_render_metric_publishes_only_the_request_v4_series() -> None:
|
||||
"""Render the single series selected by the Grafana panel."""
|
||||
"""Render the two series selected by the Grafana overview panels."""
|
||||
mod = load_module()
|
||||
|
||||
assert mod.render_metric(0.9995, 1234) == (
|
||||
assert mod.render_metric(mod.OUTPUT_METRIC, 0.9995, 1234) == (
|
||||
'atlas:availability:ratio_365d{definition="request-v4",scope="atlas",'
|
||||
'rollup="yearly"} 0.999500000000 1234\n'
|
||||
)
|
||||
assert mod.render_metric(mod.COVERAGE_METRIC, 109, 1234) == (
|
||||
'atlas:availability:coverage_days_365d{definition="request-v4",scope="atlas",'
|
||||
'rollup="yearly"} 109.000000000000 1234\n'
|
||||
)
|
||||
|
||||
@ -42,16 +42,18 @@ def test_node_filter_and_expr_helpers():
|
||||
def test_overview_availability_panel_uses_recorded_365d_rollup():
|
||||
mod = load_module()
|
||||
dashboard = mod.build_overview()
|
||||
panel = next(panel for panel in flatten_panels(dashboard["panels"]) if panel["id"] == 27)
|
||||
panels_by_id = {panel["id"]: panel for panel in flatten_panels(dashboard["panels"])}
|
||||
panel = panels_by_id[27]
|
||||
|
||||
assert panel["title"] == "Atlas Availability (365d)"
|
||||
assert panel["title"] == "Atlas Availability (365d window)"
|
||||
availability_expr = panel["targets"][0]["expr"]
|
||||
assert (
|
||||
'last_over_time(atlas:availability:ratio_365d{scope="atlas",definition="request-v4"}[48h])'
|
||||
in availability_expr
|
||||
availability_expr
|
||||
== 'last_over_time(atlas:availability:ratio_365d{scope="atlas",definition="request-v4"}[48h])'
|
||||
)
|
||||
assert 'code=~"5.."' in availability_expr
|
||||
assert 'code=~"[1-5].."' in availability_expr
|
||||
# A stale rollup must render nothing rather than silently substituting the
|
||||
# last hour of Traefik traffic under a yearly title, as it did on 2026-08-18.
|
||||
assert "traefik_entrypoint_requests_total" not in availability_expr
|
||||
assert "atlas:availability:failures_1d" not in availability_expr
|
||||
assert "atlas:availability:requests_1d" not in availability_expr
|
||||
assert "sum_over_time" not in availability_expr
|
||||
@ -59,9 +61,15 @@ def test_overview_availability_panel_uses_recorded_365d_rollup():
|
||||
assert "kube_deployment_status_replicas_available" not in availability_expr
|
||||
assert panel["targets"][0]["instant"] is True
|
||||
assert "Every server-side 5xx" in panel["description"]
|
||||
assert "Replica counts, Grafana health" in panel["description"]
|
||||
assert "daily rollup job publishes one annual sample" in panel["description"]
|
||||
assert "keeps it for up to 48 hours" in panel["description"]
|
||||
assert "never converted into downtime or uptime" in panel["description"]
|
||||
|
||||
coverage = panels_by_id[36]
|
||||
assert coverage["title"] == "Availability Coverage (days)"
|
||||
assert (
|
||||
coverage["targets"][0]["expr"]
|
||||
== 'last_over_time(atlas:availability:coverage_days_365d{scope="atlas",definition="request-v4"}[48h])'
|
||||
)
|
||||
assert "lowers this number instead of moving availability" in coverage["description"]
|
||||
|
||||
def test_overview_uses_readable_quality_power_and_gitops_panels():
|
||||
mod = load_module()
|
||||
|
||||
@ -17,7 +17,7 @@
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
@ -72,7 +72,7 @@
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 4,
|
||||
"x": 3,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -148,7 +148,7 @@
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 7,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -216,20 +216,20 @@
|
||||
{
|
||||
"id": 27,
|
||||
"type": "stat",
|
||||
"title": "Atlas Availability (365d)",
|
||||
"title": "Atlas Availability (365d window)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "atlas-vm"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 4,
|
||||
"x": 10,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(last_over_time(atlas:availability:ratio_365d{scope=\"atlas\",definition=\"request-v4\"}[48h]) or on() (1 - ((sum(increase(traefik_entrypoint_requests_total{entrypoint=\"websecure\",protocol=\"http\",code=~\"5..\"}[1h])) or on() vector(0)) / clamp_min(sum(increase(traefik_entrypoint_requests_total{entrypoint=\"websecure\",protocol=\"http\",code=~\"[1-5]..\"}[1h])), 1))))",
|
||||
"expr": "last_over_time(atlas:availability:ratio_365d{scope=\"atlas\",definition=\"request-v4\"}[48h])",
|
||||
"refId": "A",
|
||||
"instant": true
|
||||
}
|
||||
@ -286,7 +286,78 @@
|
||||
},
|
||||
"textMode": "value"
|
||||
},
|
||||
"description": "Rolling request-weighted availability at the Atlas HTTPS ingress: responses below 500 divided by all HTTP responses. Every server-side 5xx is a real failed request; client 4xx responses count as served. Replica counts, Grafana health, and monitoring gaps are not inputs, so they cannot lower availability unless user traffic actually receives a 5xx. A daily rollup job publishes one annual sample from deduplicated daily totals; Grafana keeps it for up to 48 hours so one delayed retry cannot cause a fallback, and only uses the same one-hour request SLI before history exists."
|
||||
"description": "Request-weighted availability at the Atlas HTTPS ingress over the trailing year: responses below 500 divided by all HTTP responses, read from the daily rollup sample. Every server-side 5xx is a real failed request; client 4xx responses count as served. Only measured days enter the ratio: a telemetry gap such as the 2026-08-18 metrics-storage outage is excluded from both sides, never converted into downtime or uptime, and the Availability Coverage panel states how many days the figure actually rests on. If the rollup stops publishing, this panel goes stale and renders no value instead of quietly substituting a one-hour live ratio under a yearly title; the atlas-availability-rollup-stale alert pages first."
|
||||
},
|
||||
{
|
||||
"id": 36,
|
||||
"type": "stat",
|
||||
"title": "Availability Coverage (days)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "atlas-vm"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "last_over_time(atlas:availability:coverage_days_365d{scope=\"atlas\",definition=\"request-v4\"}[48h])",
|
||||
"refId": "A",
|
||||
"instant": true
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "dark-red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "dark-orange",
|
||||
"value": 7
|
||||
},
|
||||
{
|
||||
"color": "dark-yellow",
|
||||
"value": 30
|
||||
},
|
||||
{
|
||||
"color": "dark-green",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none",
|
||||
"custom": {
|
||||
"displayMode": "auto"
|
||||
},
|
||||
"decimals": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value"
|
||||
},
|
||||
"description": "Days of daily rollups the availability figure to the left actually rests on, out of the 365-day window. A telemetry gap such as the 2026-08-18 metrics-storage outage lowers this number instead of moving availability, which is what keeps an observability failure from being reported as Atlas downtime. The request-v4 definition begins on 2026-05-01, so coverage stays below a full year until that history accumulates."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
@ -299,7 +370,7 @@
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 14,
|
||||
"x": 15,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -375,7 +446,7 @@
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 17,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -450,8 +521,8 @@
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
|
||||
@ -398,6 +398,58 @@ data:
|
||||
summary: "VictoriaMetrics stored no samples for 15m; every dashboard is about to read empty"
|
||||
labels:
|
||||
severity: critical
|
||||
# The rollup CronJob publishes one yearly availability sample per
|
||||
# day, and the Overview panel refuses to substitute another
|
||||
# measurement when that sample goes stale. Staleness therefore has
|
||||
# to page before the panel goes blank at the 48h lookback.
|
||||
- uid: atlas-availability-rollup-stale
|
||||
title: "Atlas availability rollup is stale (>26h)"
|
||||
condition: C
|
||||
for: "30m"
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: atlas-vm
|
||||
model:
|
||||
intervalMs: 60000
|
||||
maxDataPoints: 43200
|
||||
expr: (time() - tlast_over_time(atlas:availability:ratio_365d{scope="atlas",definition="request-v4"}[3d])) or on() vector(999999)
|
||||
legendFormat: seconds since publish
|
||||
datasource:
|
||||
type: prometheus
|
||||
uid: atlas-vm
|
||||
- refId: B
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
expression: A
|
||||
intervalMs: 60000
|
||||
maxDataPoints: 43200
|
||||
reducer: last
|
||||
type: reduce
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
expression: B
|
||||
intervalMs: 60000
|
||||
maxDataPoints: 43200
|
||||
type: threshold
|
||||
conditions:
|
||||
- evaluator:
|
||||
params: [93600]
|
||||
type: gt
|
||||
operator:
|
||||
type: and
|
||||
reducer:
|
||||
type: last
|
||||
type: query
|
||||
noDataState: Alerting
|
||||
execErrState: Alerting
|
||||
annotations:
|
||||
summary: "Atlas availability rollup has not published in >26h; the Overview availability panel goes blank at 48h"
|
||||
labels:
|
||||
severity: warning
|
||||
- orgId: 1
|
||||
name: maintenance
|
||||
folder: Alerts
|
||||
|
||||
@ -26,7 +26,7 @@ data:
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
@ -81,7 +81,7 @@ data:
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 4,
|
||||
"x": 3,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -157,7 +157,7 @@ data:
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 7,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -225,20 +225,20 @@ data:
|
||||
{
|
||||
"id": 27,
|
||||
"type": "stat",
|
||||
"title": "Atlas Availability (365d)",
|
||||
"title": "Atlas Availability (365d window)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "atlas-vm"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 4,
|
||||
"x": 10,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(last_over_time(atlas:availability:ratio_365d{scope=\"atlas\",definition=\"request-v4\"}[48h]) or on() (1 - ((sum(increase(traefik_entrypoint_requests_total{entrypoint=\"websecure\",protocol=\"http\",code=~\"5..\"}[1h])) or on() vector(0)) / clamp_min(sum(increase(traefik_entrypoint_requests_total{entrypoint=\"websecure\",protocol=\"http\",code=~\"[1-5]..\"}[1h])), 1))))",
|
||||
"expr": "last_over_time(atlas:availability:ratio_365d{scope=\"atlas\",definition=\"request-v4\"}[48h])",
|
||||
"refId": "A",
|
||||
"instant": true
|
||||
}
|
||||
@ -295,7 +295,78 @@ data:
|
||||
},
|
||||
"textMode": "value"
|
||||
},
|
||||
"description": "Rolling request-weighted availability at the Atlas HTTPS ingress: responses below 500 divided by all HTTP responses. Every server-side 5xx is a real failed request; client 4xx responses count as served. Replica counts, Grafana health, and monitoring gaps are not inputs, so they cannot lower availability unless user traffic actually receives a 5xx. A daily rollup job publishes one annual sample from deduplicated daily totals; Grafana keeps it for up to 48 hours so one delayed retry cannot cause a fallback, and only uses the same one-hour request SLI before history exists."
|
||||
"description": "Request-weighted availability at the Atlas HTTPS ingress over the trailing year: responses below 500 divided by all HTTP responses, read from the daily rollup sample. Every server-side 5xx is a real failed request; client 4xx responses count as served. Only measured days enter the ratio: a telemetry gap such as the 2026-08-18 metrics-storage outage is excluded from both sides, never converted into downtime or uptime, and the Availability Coverage panel states how many days the figure actually rests on. If the rollup stops publishing, this panel goes stale and renders no value instead of quietly substituting a one-hour live ratio under a yearly title; the atlas-availability-rollup-stale alert pages first."
|
||||
},
|
||||
{
|
||||
"id": 36,
|
||||
"type": "stat",
|
||||
"title": "Availability Coverage (days)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "atlas-vm"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "last_over_time(atlas:availability:coverage_days_365d{scope=\"atlas\",definition=\"request-v4\"}[48h])",
|
||||
"refId": "A",
|
||||
"instant": true
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "dark-red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "dark-orange",
|
||||
"value": 7
|
||||
},
|
||||
{
|
||||
"color": "dark-yellow",
|
||||
"value": 30
|
||||
},
|
||||
{
|
||||
"color": "dark-green",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none",
|
||||
"custom": {
|
||||
"displayMode": "auto"
|
||||
},
|
||||
"decimals": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value"
|
||||
},
|
||||
"description": "Days of daily rollups the availability figure to the left actually rests on, out of the 365-day window. A telemetry gap such as the 2026-08-18 metrics-storage outage lowers this number instead of moving availability, which is what keeps an observability failure from being reported as Atlas downtime. The request-v4 definition begins on 2026-05-01, so coverage stays below a full year until that history accumulates."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
@ -308,7 +379,7 @@ data:
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 14,
|
||||
"x": 15,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -384,7 +455,7 @@ data:
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 17,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
@ -459,8 +530,8 @@ data:
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
|
||||
@ -20,7 +20,12 @@ DEFINITION = "request-v4"
|
||||
REQUESTS_METRIC = "atlas:availability:requests_1d"
|
||||
FAILURES_METRIC = "atlas:availability:failures_1d"
|
||||
OUTPUT_METRIC = "atlas:availability:ratio_365d"
|
||||
COVERAGE_METRIC = "atlas:availability:coverage_days_365d"
|
||||
WINDOW_DAYS = 365
|
||||
# Freshly imported samples sit in the in-memory buffer briefly before they
|
||||
# become searchable, so the read-back check retries instead of failing fast.
|
||||
VERIFY_ATTEMPTS = 6
|
||||
VERIFY_DELAY_SECONDS = 10
|
||||
|
||||
|
||||
def parse_export(lines: Iterable[bytes]) -> dict[int, float]:
|
||||
@ -35,8 +40,8 @@ def parse_export(lines: Iterable[bytes]) -> dict[int, float]:
|
||||
return points
|
||||
|
||||
|
||||
def fetch_rollup(metric: str, start: datetime, end: datetime) -> dict[int, float]:
|
||||
"""Stream one compact rollup series from VictoriaMetrics."""
|
||||
def fetch_series(metric: str, start: datetime, end: datetime) -> dict[int, float]:
|
||||
"""Stream one compact series from VictoriaMetrics."""
|
||||
matcher = (
|
||||
f'{{__name__="{metric}",scope="{SCOPE}",definition="{DEFINITION}"}}'
|
||||
)
|
||||
@ -60,19 +65,19 @@ def calculate_availability(requests: float, failures: float) -> float:
|
||||
return max(0.0, min(1.0, 1.0 - (failures / requests)))
|
||||
|
||||
|
||||
def render_metric(value: float, timestamp_ms: int) -> str:
|
||||
def render_metric(metric: str, value: float, timestamp_ms: int) -> str:
|
||||
"""Render one VictoriaMetrics Prometheus-import sample."""
|
||||
return (
|
||||
f'{OUTPUT_METRIC}{{definition="{DEFINITION}",scope="{SCOPE}",'
|
||||
f'{metric}{{definition="{DEFINITION}",scope="{SCOPE}",'
|
||||
f'rollup="yearly"}} {value:.12f} {timestamp_ms}\n'
|
||||
)
|
||||
|
||||
|
||||
def publish(value: float, timestamp_ms: int) -> None:
|
||||
"""Write the calculated annual ratio to VictoriaMetrics."""
|
||||
def publish(metric: str, value: float, timestamp_ms: int) -> None:
|
||||
"""Write one calculated yearly sample to VictoriaMetrics."""
|
||||
request = Request(
|
||||
f"{VM_URL}/api/v1/import/prometheus",
|
||||
data=render_metric(value, timestamp_ms).encode(),
|
||||
data=render_metric(metric, value, timestamp_ms).encode(),
|
||||
headers={"Content-Type": "text/plain"},
|
||||
method="POST",
|
||||
)
|
||||
@ -81,21 +86,53 @@ def publish(value: float, timestamp_ms: int) -> None:
|
||||
raise RuntimeError(f"VictoriaMetrics import returned HTTP {response.status}")
|
||||
|
||||
|
||||
def verify_stored(metric: str, value: float, timestamp_ms: int) -> None:
|
||||
"""Prove the sample landed; a read-only store accepts writes and drops them.
|
||||
|
||||
During the 2026-08-18 storage outage every import returned success while
|
||||
VictoriaMetrics silently discarded the samples, so this job reported
|
||||
healthy runs across a day and a half of lost publishes. Reading the sample
|
||||
back is the only evidence the write survived.
|
||||
"""
|
||||
when = datetime.fromtimestamp(timestamp_ms / 1000, tz=timezone.utc)
|
||||
for attempt in range(VERIFY_ATTEMPTS):
|
||||
if attempt:
|
||||
time.sleep(VERIFY_DELAY_SECONDS)
|
||||
points = fetch_series(
|
||||
metric, when - timedelta(minutes=5), when + timedelta(minutes=5)
|
||||
)
|
||||
stored = points.get(timestamp_ms)
|
||||
if stored is not None and abs(stored - value) < 1e-9:
|
||||
return
|
||||
raise RuntimeError(
|
||||
f"{metric} sample at {timestamp_ms} is not readable after publish; "
|
||||
"VictoriaMetrics accepted and then discarded the write "
|
||||
"(is /storage read-only?)"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Rebuild and publish the rolling request-availability sample."""
|
||||
"""Rebuild and publish the rolling request-availability samples."""
|
||||
end = datetime.now(timezone.utc)
|
||||
start = end - timedelta(days=WINDOW_DAYS)
|
||||
requests = sum(fetch_rollup(REQUESTS_METRIC, start, end).values())
|
||||
failures = sum(fetch_rollup(FAILURES_METRIC, start, end).values())
|
||||
availability = calculate_availability(requests, failures)
|
||||
requests = fetch_series(REQUESTS_METRIC, start, end)
|
||||
failures = fetch_series(FAILURES_METRIC, start, end)
|
||||
availability = calculate_availability(
|
||||
sum(requests.values()), sum(failures.values())
|
||||
)
|
||||
coverage_days = float(len(requests))
|
||||
timestamp_ms = time.time_ns() // 1_000_000
|
||||
publish(availability, timestamp_ms)
|
||||
publish(OUTPUT_METRIC, availability, timestamp_ms)
|
||||
publish(COVERAGE_METRIC, coverage_days, timestamp_ms)
|
||||
verify_stored(OUTPUT_METRIC, availability, timestamp_ms)
|
||||
verify_stored(COVERAGE_METRIC, coverage_days, timestamp_ms)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"requests": requests,
|
||||
"failures": failures,
|
||||
"availability_percent": availability * 100,
|
||||
"coverage_days": coverage_days,
|
||||
"failures": sum(failures.values()),
|
||||
"requests": sum(requests.values()),
|
||||
"timestamp_ms": timestamp_ms,
|
||||
},
|
||||
sort_keys=True,
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user