2862594c62
The 2026-08-18 metrics-storage outage exposed two defects in the availability pipeline that distorted the figure in opposite directions at once. The Overview panel fell back to a live one-hour Traefik ratio whenever the yearly rollup sample went stale for 48h, and rendered it under the same "365d" title. When the rollup stopped publishing on 2026-08-18 the panel quietly swapped a 365-day measurement for a 60-minute one and read 99.74% instead of the recorded 99.95%. The fallback is removed: a stale rollup now renders no value, and a new atlas-availability-rollup-stale alert pages at 26h, well before the panel goes blank at 48h. The yearly ratio also silently excluded the 34-hour telemetry gap, because missing days contribute zero requests and zero failures. Absent data was read as "nothing happened" — had Atlas genuinely been down in that window, the figure would still have said 99.95%. Availability keeps its measured-days-only definition, which is correct, but coverage is now published alongside it and shown in a new panel, so a telemetry gap lowers disclosed coverage instead of vanishing. The title reads "365d window" to stop implying 365 days of data exist; request-v4 begins 2026-05-01. The rollup job reported healthy runs across a day and a half of lost publishes: a read-only VictoriaMetrics accepts an import and discards it. It now reads each sample back and fails loudly when the write did not survive. Not addressed here: availability is still measured from inside the platform via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry down", and misses failures that never reach Traefik (DNS, TLS, node dead). An external synthetic prober is the real fix and needs a hosting decision.
57 lines
1.9 KiB
Python
57 lines
1.9 KiB
Python
"""Unit tests for the Atlas availability publisher."""
|
|
|
|
import importlib.util
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
|
|
def load_module():
|
|
"""Load the service-owned rollup module without packaging it."""
|
|
path = (
|
|
Path(__file__).resolve().parents[2]
|
|
/ "services/monitoring/scripts/availability_rollup.py"
|
|
)
|
|
spec = importlib.util.spec_from_file_location("availability_rollup", path)
|
|
module = importlib.util.module_from_spec(spec)
|
|
assert spec.loader is not None
|
|
spec.loader.exec_module(module)
|
|
return module
|
|
|
|
|
|
def test_parse_export_deduplicates_replay_boundaries() -> None:
|
|
"""Keep only the final value when replay chunks share a timestamp."""
|
|
mod = load_module()
|
|
lines = [
|
|
(json.dumps({"timestamps": [1000, 2000], "values": [2, 3]}) + "\n").encode(),
|
|
(json.dumps({"timestamps": [2000, 3000], "values": [3, 5]}) + "\n").encode(),
|
|
]
|
|
|
|
assert mod.parse_export(lines) == {1000: 2.0, 2000: 3.0, 3000: 5.0}
|
|
|
|
|
|
def test_calculate_availability_uses_all_server_failures() -> None:
|
|
"""Calculate one bounded successful-request ratio."""
|
|
mod = load_module()
|
|
|
|
assert mod.calculate_availability(1000, 2) == pytest.approx(0.998)
|
|
with pytest.raises(ValueError):
|
|
mod.calculate_availability(0, 0)
|
|
with pytest.raises(ValueError):
|
|
mod.calculate_availability(100, -1)
|
|
|
|
|
|
def test_render_metric_publishes_only_the_request_v4_series() -> None:
|
|
"""Render the two series selected by the Grafana overview panels."""
|
|
mod = load_module()
|
|
|
|
assert mod.render_metric(mod.OUTPUT_METRIC, 0.9995, 1234) == (
|
|
'atlas:availability:ratio_365d{definition="request-v4",scope="atlas",'
|
|
'rollup="yearly"} 0.999500000000 1234\n'
|
|
)
|
|
assert mod.render_metric(mod.COVERAGE_METRIC, 109, 1234) == (
|
|
'atlas:availability:coverage_days_365d{definition="request-v4",scope="atlas",'
|
|
'rollup="yearly"} 109.000000000000 1234\n'
|
|
)
|