22 Commits

Author SHA1 Message Date
Hermes Agent
2862594c62 fix(monitoring): measure Atlas availability honestly across telemetry gaps
The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.

The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.

The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.

The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.

Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.
2026-08-20 02:03:43 +00:00
Hermes Agent
c4eac8ceee fix(monitoring): keep VictoriaMetrics writable 2026-08-19 10:51:44 +00:00
jenkins
a00175b376 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
2391477619 monitoring: alert on soteria backup job creation spikes 2026-04-17 01:09:25 -03:00
9448093c4e monitoring(alerts): make soteria backup health rule driver-agnostic 2026-04-13 02:36:39 -03:00
5d402458b0 maintenance(soteria): roll react ui image and wire b2 monitoring 2026-04-12 20:04:35 -03:00
4566e601b3 monitoring(alerts): watch soteria authz denial spikes 2026-04-12 15:07:54 -03:00
2e97ba062c maintenance(soteria): harden ingress path and add backup alerts 2026-04-12 15:07:54 -03:00
3ce7b2eeb7 maintenance/monitoring: wire reciprocal metis hecate key + dampen alert flapping 2026-04-05 13:51:57 -03:00
03ae79df3e maintenance: harden sd-write controls and recovery workflow 2026-03-31 00:06:44 -03:00
8006540645 monitoring: raise rootfs warning threshold to 85 percent 2026-03-30 18:41:05 -03:00
0aeb08d375 monitoring: fix noisy grafana email alerts and reload rules 2026-03-30 18:33:02 -03:00
35d5d5a1a3 monitoring: fix grafana alert exec state 2026-01-27 23:34:11 -03:00
9a978c5e72 monitoring: tune cpu and maintenance alerts 2026-01-27 23:23:42 -03:00
993702afee monitoring: alert on VM outage 2026-01-23 11:51:28 -03:00
e897858d97 monitoring: move grafana smtp to vault 2026-01-14 06:41:34 -03:00
879ff7c16b monitoring: fix infra scopes and add jetson metrics 2026-01-11 23:46:24 -03:00
0e36e8ce12 knowledge: relocate metis doc; monitoring: add cpu high alert 2026-01-11 08:59:51 -03:00
f500e81606 monitoring: maintenance panels, extra alerts, update overview 2026-01-11 02:28:39 -03:00
4a01632f6b monitoring: add alert rules and include titan-20/21 in dashboards 2026-01-11 02:02:47 -03:00
ea7f1bfb5a monitoring: fix grafana alerting root policy 2026-01-11 01:40:07 -03:00
6ac61e7b44 monitoring: wire grafana smtp sync and alerting provisioning 2026-01-11 00:29:20 -03:00