22 Commits

Author SHA1 Message Date
Hermes Agent
2862594c62 fix(monitoring): measure Atlas availability honestly across telemetry gaps
The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.

The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.

The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.

The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.

Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.
2026-08-20 02:03:43 +00:00
jenkins
46c50814ad fix(monitoring): retain short Jetson GPU activity 2026-08-09 14:38:51 -03:00
jenkins
24308f4870 monitoring: remove legacy availability series 2026-08-04 22:49:48 -03:00
jenkins
ba4b871a62 monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
b1ecfe96e4 ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
95dc2c5007 monitoring(gpu): add process-level utilization attribution 2026-05-22 02:28:08 -03:00
fe5307ff2c ci(metrics): use Pushgateway PUT for suite payload replacement 2026-04-19 16:10:20 -03:00
45fde5e2bc quality: standardize suite checks and add SonarQube stack 2026-04-19 14:18:58 -03:00
eadab2c2e2 monitoring(metrics): normalize platform gate contract and pegasus suite name 2026-04-18 16:34:20 -03:00
99eda351df monitoring/jenkins: add pegasus CI job and separate health probe suite 2026-04-10 03:26:51 -03:00
530f440679 monitoring: add suite probe metrics and align fan labels 2026-04-09 20:10:52 -03:00
c0073b08cc monitoring: fix tegrastats regexes 2026-01-27 16:44:00 -03:00
0a64708b3d monitoring: expose jetson scrape line length 2026-01-27 16:38:09 -03:00
aacfc8f28c monitoring: read tegrastats per scrape 2026-01-27 16:34:31 -03:00
3b20290561 monitoring: read jetson stats on demand 2026-01-27 16:27:45 -03:00
eb809524b5 monitoring: refresh jetson stats on scrape 2026-01-27 16:23:23 -03:00
0fbbbf39e9 monitoring: fix jetson gpu metrics 2026-01-27 16:19:54 -03:00
5393585f3e monitoring: fix jetson metrics newlines 2026-01-26 22:50:33 -03:00
995050f544 monitoring: unify jetson gpu metrics 2026-01-26 22:26:24 -03:00
dd0b4e28e7 vault: inject comms and grafana secrets 2026-01-14 22:29:27 -03:00
e897858d97 monitoring: move grafana smtp to vault 2026-01-14 06:41:34 -03:00
6fa2203561 iac: externalize ConfigMap scripts 2026-01-13 10:00:19 -03:00