fix(monitoring): keep VictoriaMetrics writable so public dashboards render #23

hermes-automation · 2026-08-19T10:58:36Z

hermes-automation commented

2026-08-19 10:58:36 +00:00

The public dashboards rendered "No data" on every panel. Root cause was not the
dashboards: the VictoriaMetrics volume filled, the database flipped to read-only,
and it stopped storing samples at 2026-08-18T15:25Z. Grafana, the anonymous org,
the atlas-vm datasource, both public dashboards and all 119 scrape targets were
healthy throughout.

Evidence at the time of diagnosis:
vm_free_disk_space_bytes{path="/storage"} 5214208 (VictoriaMetrics stops at a 10MB floor)
vm_storage_is_read_only{path="/storage"} 1
vm_rows_added_to_storage_total 0
count(up) last non-empty sample 2026-08-18T15:25Z

Two repository defects made this silent and inevitable:

The declared volume read 100Gi while the bound volume in the cluster was
200Gi, and a full year of retention needs roughly 210Gi at current
cardinality. Retention never pruned anything. Raised to 400Gi; astreae has
several TB free per storage node.
Nothing alerted on the database refusing writes. Added three rules that watch
the write path rather than liveness: free headroom, read-only state, and
stored rows. Each alerts on absent data, because a stalled database stops
producing the very series the rules read, and would otherwise resolve into
silence.

Verification:
scripts/tests: 70 passed
new guardrails failed first for the right reasons (100 >= 400, missing rule uids)
kustomize build services/monitoring renders 61 documents and the 400Gi value

This change alone does not clear the outage. Expanding the bound claim in the
monitoring namespace is a cluster mutation this account has no authority for and
is left for human action.

The public dashboards rendered "No data" on every panel. Root cause was not the dashboards: the VictoriaMetrics volume filled, the database flipped to read-only, and it stopped storing samples at 2026-08-18T15:25Z. Grafana, the anonymous org, the atlas-vm datasource, both public dashboards and all 119 scrape targets were healthy throughout. Evidence at the time of diagnosis: vm_free_disk_space_bytes{path="/storage"} 5214208 (VictoriaMetrics stops at a 10MB floor) vm_storage_is_read_only{path="/storage"} 1 vm_rows_added_to_storage_total 0 count(up) last non-empty sample 2026-08-18T15:25Z Two repository defects made this silent and inevitable: 1. The declared volume read 100Gi while the bound volume in the cluster was 200Gi, and a full year of retention needs roughly 210Gi at current cardinality. Retention never pruned anything. Raised to 400Gi; astreae has several TB free per storage node. 2. Nothing alerted on the database refusing writes. Added three rules that watch the write path rather than liveness: free headroom, read-only state, and stored rows. Each alerts on absent data, because a stalled database stops producing the very series the rules read, and would otherwise resolve into silence. Verification: scripts/tests: 70 passed new guardrails failed first for the right reasons (100 >= 400, missing rule uids) kustomize build services/monitoring renders 61 documents and the 400Gi value This change alone does not clear the outage. Expanding the bound claim in the monitoring namespace is a cluster mutation this account has no authority for and is left for human action.

hermes-automation added 1 commit 2026-08-19 10:58:37 +00:00

fix(monitoring): keep VictoriaMetrics writable c4eac8ceee

bstein changed title from ~~WIP: fix(monitoring): keep VictoriaMetrics writable so public dashboards render~~ to fix(monitoring): keep VictoriaMetrics writable so public dashboards render

2026-08-19 23:26:22 +00:00

bstein scheduled this pull request to auto merge when all checks succeed 2026-08-19 23:26:33 +00:00

bstein approved these changes 2026-08-20 00:44:17 +00:00

bstein scheduled this pull request to auto merge when all checks succeed 2026-08-20 00:44:52 +00:00

bstein added 1 commit 2026-08-20 00:49:44 +00:00

Merge branch 'main' into hermes/fix-grafana-no-data-vm-storage

Tests / Declarative: Post Actions failed: 35, skipped: 7, passed: 2586

Details

2b683d9775

bstein merged commit 9872ee0995 into main

2026-08-20 00:50:00 +00:00

bstein referenced this issue from a commit