270 Commits

Author SHA1 Message Date
Hermes Agent
2862594c62 fix(monitoring): measure Atlas availability honestly across telemetry gaps
The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.

The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.

The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.

The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.

Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.
2026-08-20 02:03:43 +00:00
jenkins
46c50814ad fix(monitoring): retain short Jetson GPU activity 2026-08-09 14:38:51 -03:00
jenkins
a2564f8611 monitoring: retain daily availability through retries 2026-08-04 21:47:30 -03:00
jenkins
ba4b871a62 monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
af10f7aadc monitoring: add daily availability rollups 2026-08-04 21:24:47 -03:00
jenkins
5f692ecd3d monitoring: add compact availability fallback 2026-08-04 21:14:06 -03:00
jenkins
83696d46d2 monitoring: use request success for availability 2026-08-04 21:12:31 -03:00
jenkins
6b1d6388db monitoring: measure gateway availability 2026-08-04 21:02:19 -03:00
jenkins
026db898fc monitoring: preserve serving availability history 2026-08-04 18:39:36 -03:00
jenkins
8d16b790af monitoring: fall back to live serving state 2026-08-04 18:16:13 -03:00
jenkins
5fabd1a84b monitoring: avoid unstable volume hosts 2026-08-04 18:06:43 -03:00
jenkins
e9b4404e02 monitoring: render test category zero state 2026-08-04 12:34:08 -03:00
jenkins
a00175b376 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
jenkins
4dab530baf monitoring(nodes): clamp CPU charts to physical range 2026-08-02 19:36:44 -03:00
jenkins
f2328f12e4 monitoring(gpu): attribute Jetson activity by allocation 2026-08-02 04:26:33 -03:00
jenkins
b1ecfe96e4 ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
3255936aeb monitoring(gpu): report time-weighted namespace usage 2026-08-02 03:08:20 -03:00
jenkins
48773efbb1 monitoring: filter node dashboards to real nodes 2026-07-14 20:24:45 -03:00
jenkins
c8707c002d monitoring: collapse duplicate Typhon climate series 2026-07-14 19:01:53 -03:00
jenkins
bea20b9c97 monitoring: track titan-23 and pin tiny apps to workers 2026-07-14 18:14:38 -03:00
jenkins
255b05bf44 monitoring(testing): restore lesavka suite visibility 2026-06-05 01:04:56 -03:00
jenkins
56a29fce03 monitoring(testing): clean up dashboard health signals 2026-06-04 16:09:08 -03:00
jenkins
04175d33ab monitoring(gpu): show activity share by namespace 2026-05-22 04:22:51 -03:00
jenkins
250a850871 monitoring(gpu): count monitored GPU pool devices 2026-05-22 03:23:36 -03:00
jenkins
491da8a7f5 monitoring(gpu): add pool utilization counters 2026-05-22 03:09:10 -03:00
jenkins
8002181f36 monitoring(gpu): normalize utilization pie to pool capacity 2026-05-22 02:55:24 -03:00
jenkins
ddb49dc62d monitoring(gpu): hide zero-utilization namespaces 2026-05-22 02:35:51 -03:00
jenkins
95dc2c5007 monitoring(gpu): add process-level utilization attribution 2026-05-22 02:28:08 -03:00
jenkins
2a1e4b3c49 monitoring(gpu): attribute utilization to namespaces 2026-05-22 01:46:32 -03:00
jenkins
5ddb428a96 monitoring(gpu): show utilization with idle fallback 2026-05-21 15:26:02 -03:00
jenkins
1cc2372791 monitoring(gpu): clarify reservation accounting 2026-05-21 13:04:58 -03:00
jenkins
6d5da627c5 monitoring(testing): backfill category health rollups 2026-05-20 14:39:07 -03:00
jenkins
eb1838e29f monitoring(testing): memoize dashboard freshness panels 2026-05-20 12:45:50 -03:00
jenkins
1372ea7581 monitoring(testing): memoize slow dashboard panels 2026-05-20 11:52:25 -03:00
jenkins
2d6a1b5eaa monitoring(testing): surface current gate health 2026-05-20 11:01:28 -03:00
jenkins
8da1bb6d53 ops: harden ci placement and gpu idle reporting 2026-05-20 04:27:26 -03:00
jenkins
e5d00c4eab monitoring: hide idle gpu share during activity 2026-05-19 23:30:22 -03:00
jenkins
095b93ca8c monitoring(testing): dedupe run counters by scrape target 2026-05-19 07:46:06 -03:00
jenkins
a85b508ba8 monitoring(testing): prefer fresh coverage metrics 2026-05-19 06:31:04 -03:00
jenkins
35ac110ed8 monitoring(testing): clarify category panel window 2026-05-19 02:29:16 -03:00
jenkins
1ee847b6d3 monitoring(testing): clarify CI run and test history labels 2026-05-18 21:04:14 -03:00
jenkins
2e5f3e61de monitoring(testing): clarify CI run health labels 2026-05-18 14:18:56 -03:00
jenkins
4a1ebb1a84 monitoring(overview): simplify test category lanes 2026-05-18 11:42:42 -03:00
jenkins
eb16e6f02b monitoring(testing): restrict category panels to taxonomy 2026-05-17 18:26:57 -03:00
jenkins
1515274f1e monitoring(overview): restore category timeline 2026-05-17 18:01:37 -03:00
jenkins
c0520d6fa9 monitoring(overview): clean up category pass-rate panel 2026-05-17 16:47:17 -03:00
jenkins
23ec64092f monitoring(testing): prefer fresh suite quality samples 2026-05-17 05:54:28 -03:00
jenkins
c6c8b3feef monitoring: clarify fresh suite signal 2026-05-17 00:28:07 -03:00
jenkins
c117648a04 monitoring: use fresh quality snapshots 2026-05-16 20:57:56 -03:00
jenkins
01e70afbde monitoring: count only canonical reporting suites 2026-05-16 17:26:20 -03:00