AGENTS.md

@@ -1,68 +0,0 @@
-
-
-Repository Guidelines
-
-## Project Structure & Module Organization
-- `infrastructure/`: cluster-scoped building blocks (core, flux-system, traefik, longhorn). Add new platform features by mirroring this layout.
-- `services/`: workload manifests per app (`services/gitea/`, etc.) with `kustomization.yaml` plus one file per kind; keep diffs small and focused.
-- `dockerfiles/` hosts bespoke images, while `scripts/` stores operational Fish/Bash helpers—extend these directories instead of relying on ad-hoc commands.
-
-## Build, Test, and Development Commands
-- `kustomize build services/<app>` (or `kubectl kustomize ...`) renders manifests exactly as Flux will.
-- `kubectl apply --server-side --dry-run=client -k services/<app>` checks schema compatibility without touching the cluster.
-- `flux reconcile kustomization <name> --namespace flux-system --with-source` pulls the latest Git state after merges or hotfixes.
-- `fish scripts/flux_hammer.fish --help` explains the recovery tool; read it before running against production workloads.
-
-## Coding Style & Naming Conventions
-- YAML uses two-space indents; retain the leading path comment (e.g. `# services/gitea/deployment.yaml`) to speed code review.
-- Keep resource names lowercase kebab-case, align labels/selectors, and mirror namespaces with directory names.
-- List resources in `kustomization.yaml` from namespace/config, through storage, then workloads and networking for predictable diffs.
-- Scripts start with `#!/usr/bin/env fish` or bash, stay executable, and follow snake_case names such as `flux_hammer.fish`.
-
-## Testing Guidelines
-- Run `kustomize build` and the dry-run apply for every service you touch; capture failures before opening a PR.
-- `flux diff kustomization <name> --path services/<app>` previews reconciliations—link notable output when behavior shifts.
-- Docker edits: `docker build -f dockerfiles/Dockerfile.monerod .` (swap the file you changed) to verify image builds.
-
-## Commit & Pull Request Guidelines
-- Keep commit subjects short, present-tense, and optionally scoped (`gpu(titan-24): add RuntimeClass`); squash fixups before review.
-- Describe linked issues, affected services, and required operator steps (e.g. `flux reconcile kustomization services-gitea`) in the PR body.
-- Focus each PR on one kustomization or service and update `infrastructure/flux-system` when Flux must track new folders.
-- Record the validation you ran (dry-runs, diffs, builds) and add screenshots only when ingress or UI behavior changes.
-
-## Security & Configuration Tips
-- Never commit credentials; use Vault workflows (`services/vault/`) or SOPS-encrypted manifests wired through `infrastructure/flux-system`.
-- Node selectors and tolerations gate workloads to hardware like `hardware: rpi4`; confirm labels before scaling or renaming nodes.
-- Pin external images by digest or rely on Flux image automation to follow approved tags and avoid drift.
-
-## Dashboard roadmap / context (2025-12-02)
-- Atlas dashboards are generated via `scripts/dashboards_render_atlas.py --build`, which writes JSON under `services/monitoring/dashboards/` and ConfigMaps under `services/monitoring/`. Keep the Grafana manifests in sync by regenerating after edits.
-- Atlas Overview panels are paired with internal dashboards (pods, nodes, storage, network, GPU). A new `atlas-gpu` internal dashboard holds the detailed GPU metrics that feed the overview share pie.
-- Old Grafana folders (`Atlas Storage`, `Atlas SRE`, `Atlas Public`, `Atlas Nodes`) should be removed in Grafana UI when convenient; only `Atlas Overview` and `Atlas Internal` should remain provisioned.
-- Future work: add a separate generator (e.g., `dashboards_render_oceanus.py`) for SUI/oceanus validation dashboards, mirroring the atlas pattern of internal dashboards feeding a public overview.
-
-## Monitoring state (2025-12-03)
-- dcgm-exporter DaemonSet pulls `registry.bstein.dev/monitoring/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04` with nvidia runtime/imagePullSecret; titan-24 exports metrics, titan-22 remains NotReady.
-- Atlas Overview is the Grafana home (1h range, 1m refresh), Overview folder UID `overview`, internal folder `atlas-internal` (oceanus-internal stub).
-- Panels standardized via generator; hottest row compressed, worker/control rows taller, root disk row taller and top12 bar gauge with labels. GPU share pie uses 1h avg_over_time to persist idle activity.
-- Internal dashboards are provisioned without Viewer role; if anonymous still sees them, restart Grafana and tighten auth if needed.
-
-## Upcoming priorities (SSO/storage/mail)
-- Establish SSO (Keycloak or similar) and federate Grafana, Gitea, Zot, Nextcloud, Pegasus/Jellyfin; keep Vaultwarden separate until safe.
-- Add Nextcloud (limit to rpi5 workers) with office suite; integrate with SSO; plan storage class and ingress.
-- Plan mail: mostly self-hosted, relay through trusted provider for outbound; integrate with services (Nextcloud, Vaultwarden, etc.) for notifications and account flows.
-
-## SSO plan sketch (2025-12-03)
-- IdP: use Keycloak (preferred) in a new `sso` namespace, Bitnami or codecentric chart with Postgres backing store (single PVC), ingress `sso.bstein.dev`, admin user bound to brad@bstein.dev; stick with local DB initially (no external IdP).
-- Auth flow goals: Grafana (OIDC), Gitea (OAuth2/Keycloak), Zot (via Traefik forward-auth/oauth2-proxy), Jellyfin/Pegasus via Jellyfin OAuth/OpenID plugin (map existing usernames; run migration to pre-create users in Keycloak with same usernames/emails and temporary passwords), Pegasus keeps using Jellyfin tokens.
-- Steps to implement:
-  1) Add service folder `services/keycloak/` (namespace, PVC, HelmRelease, ingress, secret for admin creds). Verify with kustomize + Flux reconcile.
-  2) Seed realm `atlas` with users (import CSV/realm). Create client for Grafana (public/implicit), Gitea (confidential), and a “jellyfin” client for the OAuth plugin; set email for brad@bstein.dev as admin.
-  3) Reconfigure Grafana to OIDC (disable anonymous to internal folders, leave Overview public via folder permissions). Reconfigure Gitea to OIDC (app.ini).
-  4) Add Traefik forward-auth (oauth2-proxy) in front of Zot and any other services needing headers-based auth.
-  5) Deploy Jellyfin OpenID plugin; map Keycloak users to existing Jellyfin usernames; communicate password reset path.
-- Migration caution: do not delete existing local creds until SSO validated; keep Pegasus working via Jellyfin tokens during transition.
-
-## Postgres centralization (2025-12-03)
-- Prefer a shared in-cluster Postgres deployment with per-service databases to reduce resource sprawl on Pi nodes. Use it for services that can easily point at an external DB.
-- Candidates to migrate to shared Postgres: Keycloak (realm DB), Gitea (git DB), Nextcloud (app DB), possibly Grafana (if persistence needed beyond current provisioner), Jitsi prosody/JVB state (if external DB supported). Keep tightly-coupled or lightweight embedded DBs as-is when migration is painful or not supported.

clusters/atlas/flux-system/gotk-sync.yaml

@@ -8,7 +8,7 @@ metadata:
 spec:
   interval: 1m0s
   ref:
-    branch: feature/atlas-monitoring
+    branch: main
   secretRef:
     name: flux-system-gitea
   url: ssh://git@scm.bstein.dev:2242/bstein/titan-iac.git

clusters/atlas/flux-system/platform/monitoring/kustomization.yaml

@@ -11,4 +11,4 @@ spec:
   sourceRef:
     kind: GitRepository
     name: flux-system
-  wait: false
+  wait: true

infrastructure/traefik/deployment.yaml

@@ -39,12 +39,6 @@ items:
           - --metrics.prometheus.addEntryPointsLabels=true
           - --metrics.prometheus.addRoutersLabels=true
           - --metrics.prometheus.addServicesLabels=true
-          - --entrypoints.web.transport.respondingTimeouts.readTimeout=0s
-          - --entrypoints.web.transport.respondingTimeouts.writeTimeout=0s
-          - --entrypoints.web.transport.respondingTimeouts.idleTimeout=0s
-          - --entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s
-          - --entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0s
-          - --entrypoints.websecure.transport.respondingTimeouts.idleTimeout=0s
           - --entrypoints.metrics.address=:9100
           - --metrics.prometheus.entryPoint=metrics
           image: traefik:v3.3.3

scripts/styx_prep_nvme_luks.sh

@@ -1,575 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# --- CONFIG (edit if needed) ---
-# Leave NVME empty → script will auto-detect the SSK dock.
-NVME="${NVME:-}"
-FLAVOR="${FLAVOR:-desktop}"
-# Persistent cache so the image survives reboots.
-IMG_DIR="${IMG_DIR:-/var/cache/styx-rpi}"
-IMG_FILE="${IMG_FILE:-ubuntu-24.04.3-preinstalled-${FLAVOR}-arm64+raspi.img}"
-IMG_BOOT_MNT="${IMG_BOOT_MNT:-/mnt/img-boot}"
-IMG_ROOT_MNT="${IMG_ROOT_MNT:-/mnt/img-root}"
-TGT_ROOT="/mnt/target-root"
-TGT_BOOT="/mnt/target-boot"
-
-STYX_USER="styx"
-STYX_HOSTNAME="titan-ag"
-STYX_PASS="TempPass#123"                # will be forced to change on first login via cloud-init
-SSH_PUBKEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOb8oMX6u0z3sH/p/WBGlvPXXdbGETCKzWYwR/dd6fZb titan-bastion"
-
-# Video / input prefs
-DSI_FLAGS="video=DSI-1:800x480@60D video=HDMI-A-1:off video=HDMI-A-2:off"
-
-# --- Helpers ---
-fatal(){ echo "ERROR: $*" >&2; exit 1; }
-need(){ command -v "$1" >/dev/null || fatal "Missing tool: $1"; }
-
-require_root(){ [[ $EUID -eq 0 ]] || exec sudo -E "$0" "$@"; }
-
-part() {
-  local n="$1"
-  if [[ "$NVME" =~ [0-9]$ ]]; then
-    echo "${NVME}p${n}"
-  else
-    echo "${NVME}${n}"
-  fi
-}
-
-auto_detect_target_disk() {
-  # If user already set NVME, validate and return
-  if [[ -n "${NVME:-}" ]]; then
-    [[ -b "$NVME" ]] || fatal "NVME='$NVME' is not a block device"
-    return
-  fi
-
-  # Prefer stable by-id symlinks
-  local byid
-  byid=$(ls -1 /dev/disk/by-id/usb-SSK* 2>/dev/null | head -n1 || true)
-  if [[ -n "$byid" ]]; then
-    NVME=$(readlink -f "$byid")
-  else
-    # Heuristic via lsblk -S: look for USB with SSK/Ingram/Storage in vendor/model
-    NVME=$(lsblk -S -p -o NAME,TRAN,VENDOR,MODEL | \
-             awk '/ usb / && ($3 ~ /SSK|Ingram/i || $4 ~ /SSK|Storage/i){print $1; exit}')
-  fi
-
-  [[ -n "${NVME:-}" && -b "$NVME" ]] || fatal "Could not auto-detect SSK USB NVMe dock. Export NVME=/dev/sdX and re-run."
-  echo "Auto-detected target disk: $NVME"
-}
-
-preflight_cleanup() {
-  local img="$IMG_DIR/$IMG_FILE"
-
-  # 1) Unmount image mountpoints and detach only loops for this IMG
-  umount -lf "$IMG_BOOT_MNT" "$IMG_ROOT_MNT" 2>/dev/null || true
-  # losetup -j exits non-zero if no association → tolerate it
-  { losetup -j "$img" | cut -d: -f1 | xargs -r losetup -d; } 2>/dev/null || true
-
-  # 2) Unmount our target mounts
-  umount -lf "$TGT_ROOT/boot/firmware" "$TGT_BOOT" "$TGT_ROOT" 2>/dev/null || true
-
-  # 3) Unmount the actual target partitions if mounted anywhere (tolerate 'not found')
-  for p in "$(part 1)" "$(part 2)"; do
-    # findmnt returns 1 when no match → capture and iterate if any
-    while read -r mnt; do
-      [ -n "$mnt" ] && umount -lf "$mnt" 2>/dev/null || true
-    done < <(findmnt -rno TARGET -S "$p" 2>/dev/null || true)
-  done
-
-  # 4) Close dm-crypt mapping (if it exists)
-  cryptsetup luksClose cryptroot 2>/dev/null || true
-  dmsetup remove -f cryptroot 2>/dev/null || true
-
-  # 5) Let udev settle
-  command -v udevadm >/dev/null && udevadm settle || true
-}
-
-guard_target_device() {
-  # Refuse to operate if NVME appears to be the current system disk
-  local root_src root_disk
-  root_src=$(findmnt -no SOURCE /)
-  root_disk=$(lsblk -no pkname "$root_src" 2>/dev/null || true)
-  if [[ -n "$root_disk" && "/dev/$root_disk" == "$NVME" ]]; then
-    fatal "Refusing to operate on system disk ($NVME). Pick the external NVMe."
-  fi
-}
-
-need_host_fido2() {
-  if ! command -v fido2-token >/dev/null 2>&1; then
-    echo "Host is missing fido2-token. On Arch: sudo pacman -S libfido2"
-    echo "On Debian/Ubuntu host: sudo apt-get install fido2-tools"
-    exit 1
-  fi
-}
-
-ensure_image() {
-  mkdir -p "$IMG_DIR"
-  chmod 755 "$IMG_DIR"
-
-  local BASE="https://cdimage.ubuntu.com/releases/noble/release"
-  local XZ="ubuntu-24.04.3-preinstalled-${FLAVOR}-arm64+raspi.img.xz"
-
-  # If the decompressed .img is missing, fetch/decompress into the cache.
-  if [[ ! -f "$IMG_DIR/$IMG_FILE" ]]; then
-    need curl; need unxz  # Arch: pacman -S curl xz   |  Ubuntu: apt-get install curl xz-utils
-    if [[ ! -f "$IMG_DIR/$XZ" ]]; then
-      echo "Fetching image…"
-      curl -fL -o "$IMG_DIR/$XZ" "$BASE/$XZ"
-    fi
-    echo "Decompressing to $IMG_DIR/$IMG_FILE …"
-    # Keep the .xz for future runs; stream-decompress to the .img
-    if command -v unxz >/dev/null 2>&1; then
-      unxz -c "$IMG_DIR/$XZ" > "$IMG_DIR/$IMG_FILE"
-    else
-      need xz
-      xz -dc "$IMG_DIR/$XZ" > "$IMG_DIR/$IMG_FILE"
-    fi
-    sync
-  else
-    echo "Using cached image: $IMG_DIR/$IMG_FILE"
-  fi
-}
-
-ensure_binfmt_aarch64(){
-  # Register qemu-aarch64 for chrooted ARM64 apt runs
-  if [[ ! -e /proc/sys/fs/binfmt_misc/qemu-aarch64 ]]; then
-    need docker
-    systemctl enable --now docker >/dev/null 2>&1 || true
-    docker run --rm --privileged tonistiigi/binfmt --install arm64 >/dev/null
-  fi
-  if [[ ! -x /usr/local/bin/qemu-aarch64-static ]]; then
-    docker rm -f qemu-static >/dev/null 2>&1 || true
-    docker create --name qemu-static docker.io/multiarch/qemu-user-static:latest >/dev/null
-    docker cp qemu-static:/usr/bin/qemu-aarch64-static /usr/local/bin/
-    install -D -m755 /usr/local/bin/qemu-aarch64-static /usr/local/bin/qemu-aarch64-static
-    docker rm qemu-static >/dev/null
-  fi
-}
-
-open_image() {
-  [[ -r "$IMG_DIR/$IMG_FILE" ]] || fatal "Image not found: $IMG_DIR/$IMG_FILE"
-  mkdir -p "$IMG_BOOT_MNT" "$IMG_ROOT_MNT"
-
-  # Pre-clean: detach any previous loop(s) for this image (tolerate absence)
-  umount -lf "$IMG_BOOT_MNT" 2>/dev/null || true
-  umount -lf "$IMG_ROOT_MNT" 2>/dev/null || true
-  # If no loop is attached, losetup -j returns non-zero → swallow it
-  mapfile -t OLD < <({ losetup -j "$IMG_DIR/$IMG_FILE" | cut -d: -f1; } 2>/dev/null || true)
-  for L in "${OLD[@]:-}"; do losetup -d "$L" 2>/dev/null || true; done
-  command -v udevadm >/dev/null && udevadm settle || true
-
-  # Attach with partition scan; wait for partition nodes to exist
-  LOOP=$(losetup --find --show --partscan "$IMG_DIR/$IMG_FILE") || fatal "losetup failed"
-  command -v udevadm >/dev/null && udevadm settle || true
-  for _ in {1..25}; do
-    [[ -b "${LOOP}p1" && -b "${LOOP}p2" ]] && break
-    sleep 0.1
-    command -v udevadm >/dev/null && udevadm settle || true
-  done
-  [[ -b "${LOOP}p1" ]] || fatal "loop partitions not present for $LOOP"
-
-  # Cleanup on exit: unmount first, then detach loop (tolerate absence)
-  trap 'umount -lf "'"$IMG_BOOT_MNT"'" "'"$IMG_ROOT_MNT"'" 2>/dev/null; losetup -d "'"$LOOP"'" 2>/dev/null' EXIT
-
-  # Mount image partitions read-only
-  mount -o ro "${LOOP}p1" "$IMG_BOOT_MNT"
-  mount -o ro "${LOOP}p2" "$IMG_ROOT_MNT"
-
-  # Sanity checks without using failing pipelines
-  # start*.elf must exist
-  if ! compgen -G "$IMG_BOOT_MNT/start*.elf" > /dev/null; then
-    fatal "start*.elf not found in image"
-  fi
-  # vmlinuz-* must exist
-  if ! compgen -G "$IMG_ROOT_MNT/boot/vmlinuz-*" > /dev/null; then
-    fatal "vmlinuz-* not found in image root"
-  fi
-}
-
-confirm_and_wipe(){
-  lsblk -o NAME,SIZE,MODEL,TRAN,LABEL "$NVME"
-  read -rp "Type EXACTLY 'WIPE' to destroy ALL DATA on $NVME: " ACK
-  [[ "$ACK" == "WIPE" ]] || fatal "Aborted"
-  wipefs -a "$NVME"
-  sgdisk -Zo "$NVME"
-  # GPT: 1: 1MiB..513MiB vfat ESP; 2: rest LUKS
-  parted -s "$NVME" mklabel gpt \
-    mkpart system-boot fat32 1MiB 513MiB set 1 esp on \
-    mkpart cryptroot 513MiB 100%
-  partprobe "$NVME"; sleep 1
-  mkfs.vfat -F32 -n system-boot "$(part 1)"
-}
-
-setup_luks(){
-  echo "Create LUKS2 on $(part 2) (you will be prompted for a passphrase; keep it as fallback)"
-  need cryptsetup
-  cryptsetup luksFormat --type luks2 "$(part 2)"
-  cryptsetup open "$(part 2)" cryptroot
-  mkfs.ext4 -L rootfs /dev/mapper/cryptroot
-}
-
-mount_targets(){
-  mkdir -p "$TGT_ROOT" "$TGT_BOOT"
-  mount /dev/mapper/cryptroot "$TGT_ROOT"
-  mkdir -p "$TGT_ROOT/boot/firmware"
-  mount "$(part 1)" "$TGT_BOOT"
-  mount --bind "$TGT_BOOT" "$TGT_ROOT/boot/firmware"
-}
-
-rsync_root_and_boot(){
-  need rsync
-  rsync -aAXH --numeric-ids --delete \
-  --exclude='/boot/firmware' --exclude='/boot/firmware/**' \
-  --exclude='/dev/*' --exclude='/proc/*' --exclude='/sys/*' \
-  --exclude='/run/*' --exclude='/tmp/*' --exclude='/mnt/*' \
-  --exclude='/media/*' --exclude='/lost+found' \
-  "$IMG_ROOT_MNT"/ "$TGT_ROOT"/
-  rsync -aH --delete "$IMG_BOOT_MNT"/ "$TGT_ROOT/boot/firmware"/
-}
-
-write_crypttab_fstab(){
-  LUUID=$(blkid -s UUID -o value "$(part 2)")
-  printf 'cryptroot UUID=%s none luks,discard,fido2-device=auto\n' "$LUUID" > "$TGT_ROOT/etc/crypttab"
-  cat > "$TGT_ROOT/etc/fstab" <<EOF
-/dev/mapper/cryptroot  /               ext4  defaults,discard,errors=remount-ro  0 1
-LABEL=system-boot      /boot/firmware  vfat  defaults,umask=0077                 0 1
-EOF
-}
-
-fix_firmware_files(){
-  local C="$TGT_ROOT/boot/firmware/config.txt"
-  local CL="$TGT_ROOT/boot/firmware/cmdline.txt"
-  [[ -f "$C" ]] || fatal "missing $C"
-
-  # Always boot the uncompressed Pi 5 kernel
-  if grep -q '^kernel=' "$C"; then
-    sed -i 's#^kernel=.*#kernel=kernel_2712.img#' "$C"
-  else
-    sed -i '1i kernel=kernel_2712.img' "$C"
-  fi
-
-  # Ensure initramfs and cmdline indirection are set
-  grep -q '^initramfs ' "$C" || echo 'initramfs initrd.img followkernel' >> "$C"
-  grep -q '^cmdline=cmdline.txt' "$C" || sed -i '1i cmdline=cmdline.txt' "$C"
-
-  # Display & buses (Pi 5)
-  grep -q '^dtoverlay=vc4-kms-v3d-pi5' "$C" || echo 'dtoverlay=vc4-kms-v3d-pi5' >> "$C"
-  grep -q '^dtparam=i2c_arm=on'        "$C" || echo 'dtparam=i2c_arm=on' >> "$C"
-  grep -q '^dtparam=pciex1=on'         "$C" || echo 'dtparam=pciex1=on' >> "$C"
-  grep -q '^dtparam=pciex1_gen=2'      "$C" || echo 'dtparam=pciex1_gen=2' >> "$C"
-  grep -q '^enable_uart=1'             "$C" || echo 'enable_uart=1' >> "$C"
-
-  # Minimal, correct dracut hints using the bare UUID
-  local LUUID; LUUID=$(blkid -s UUID -o value "$(part 2)")
-  : > "$CL"
-  {
-    echo -n "rd.luks.uuid=$LUUID rd.luks.name=$LUUID=cryptroot "
-    echo -n "root=/dev/mapper/cryptroot rootfstype=ext4 rootwait fixrtc "
-    echo    "console=serial0,115200 console=tty1 ds=nocloud;s=file:///boot/firmware/ ${DSI_FLAGS} rd.debug"
-  } >> "$CL"
-}
-
-seed_cloud_init(){
-  # NoCloud seed to create user, lock down SSH, set hostname, and enable avahi.
-  cat > "$TGT_ROOT/boot/firmware/user-data" <<EOF
-#cloud-config
-hostname: $STYX_HOSTNAME
-manage_etc_hosts: true
-users:
-  - name: $STYX_USER
-    gecos: "$STYX_USER"
-    shell: /bin/bash
-    groups: [sudo,video,i2c]
-    sudo: ALL=(ALL) NOPASSWD:ALL
-    lock_passwd: false
-    ssh_authorized_keys:
-      - $SSH_PUBKEY
-chpasswd:
-  list: |
-    $STYX_USER:$STYX_PASS
-  expire: true
-ssh_pwauth: false
-package_update: true
-packages: [openssh-server, avahi-daemon]
-runcmd:
-  - systemctl enable --now ssh
-  - systemctl enable --now avahi-daemon || true
-EOF
-
-  # Minimal meta-data for NoCloud
-  date +%s | awk '{print "instance-id: iid-titan-ag-"$1"\nlocal-hostname: '"$STYX_HOSTNAME"'"}' \
-    > "$TGT_ROOT/boot/firmware/meta-data"
-}
-
-prep_chroot_mounts(){
-  for d in dev proc sys; do mount --bind "/$d" "$TGT_ROOT/$d"; done
-  mount -t devpts devpts "$TGT_ROOT/dev/pts"
-  # Replace the usual resolv.conf symlink with a real file for apt to work
-  rm -f "$TGT_ROOT/etc/resolv.conf"
-  cp /etc/resolv.conf "$TGT_ROOT/etc/resolv.conf"
-
-  # Block service starts (no systemd in chroot)
-  cat > "$TGT_ROOT/usr/sbin/policy-rc.d" <<'EOP'
-#!/bin/sh
-exit 101
-EOP
-  chmod +x "$TGT_ROOT/usr/sbin/policy-rc.d"
-
-  # Ensure qemu static is present inside chroot
-  install -D -m755 /usr/local/bin/qemu-aarch64-static "$TGT_ROOT/usr/bin/qemu-aarch64-static"
-}
-
-in_chroot(){
-  chroot "$TGT_ROOT" /usr/bin/qemu-aarch64-static /bin/bash -lc '
-set -euo pipefail
-export DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC
-
-# --- APT sources (ports) ---
-cat > /etc/apt/sources.list <<'"'"'EOS'"'"'
-deb http://ports.ubuntu.com/ubuntu-ports noble main restricted universe multiverse
-deb http://ports.ubuntu.com/ubuntu-ports noble-updates main restricted universe multiverse
-deb http://ports.ubuntu.com/ubuntu-ports noble-security main restricted universe multiverse
-EOS
-
-apt-get update
-
-# --- Remove snaps and pin them off ---
-apt-get -y purge snapd || true
-rm -rf /snap /var/snap /var/lib/snapd /home/*/snap || true
-mkdir -p /etc/apt/preferences.d
-cat > /etc/apt/preferences.d/nosnap.pref <<'"'"'EOS'"'"'
-Package: snapd
-Pin: release *
-Pin-Priority: -10
-EOS
-
-# --- Base tools (no flash-kernel; we use dracut) ---
-apt-get install -y --no-install-recommends \
-  openssh-client openssh-server openssh-sftp-server avahi-daemon \
-  cryptsetup dracut fido2-tools libfido2-1 i2c-tools \
-  python3-smbus python3-pil zbar-tools qrencode lm-sensors \
-  file zstd lz4 || true
-
-# Camera apps: try rpicam-apps; otherwise basic libcamera tools
-apt-get install -y rpicam-apps || apt-get install -y libcamera-tools || true
-
-# --- Persistent journal so we can read logs after failed boot ---
-mkdir -p /etc/systemd/journald.conf.d
-cat > /etc/systemd/journald.conf.d/99-persistent.conf <<'"'"'EOS'"'"'
-[Journal]
-Storage=persistent
-EOS
-
-# --- SSH hardening (ensure file exists even if package was half-installed) ---
-if [ ! -f /etc/ssh/sshd_config ]; then
-  mkdir -p /etc/ssh
-  cat > /etc/ssh/sshd_config <<'"'"'EOS'"'"'
-PermitRootLogin no
-PasswordAuthentication no
-KbdInteractiveAuthentication no
-PubkeyAuthentication yes
-# Accept defaults for the rest
-EOS
-fi
-sed -i -e "s/^#\?PasswordAuthentication .*/PasswordAuthentication no/" \
-       -e "s/^#\?KbdInteractiveAuthentication .*/KbdInteractiveAuthentication no/" \
-       -e "s/^#\?PermitRootLogin .*/PermitRootLogin no/" \
-       -e "s/^#\?PubkeyAuthentication .*/PubkeyAuthentication yes/" /etc/ssh/sshd_config || true
-
-# --- Hostname & hosts ---
-echo "'"$STYX_HOSTNAME"'" > /etc/hostname
-if grep -q "^127\\.0\\.1\\.1" /etc/hosts; then
-  sed -i "s/^127\\.0\\.1\\.1.*/127.0.1.1\t'"$STYX_HOSTNAME"'/" /etc/hosts
-else
-  echo -e "127.0.1.1\t'"$STYX_HOSTNAME"'" >> /etc/hosts
-fi
-
-# --- Enable services on first boot ---
-mkdir -p /etc/systemd/system/multi-user.target.wants
-ln -sf /lib/systemd/system/ssh.service /etc/systemd/system/multi-user.target.wants/ssh.service
-ln -sf /lib/systemd/system/avahi-daemon.service /etc/systemd/system/multi-user.target.wants/avahi-daemon.service || true
-
-# --- Ensure i2c group ---
-getent group i2c >/dev/null || groupadd i2c
-
-# --- Dracut configuration (generic, not host-only) ---
-mkdir -p /etc/dracut.conf.d
-cat > /etc/dracut.conf.d/00-hostonly.conf <<'"'"'EOS'"'"'
-hostonly=no
-EOS
-cat > /etc/dracut.conf.d/10-systemd-crypt.conf <<'"'"'EOS'"'"'
-add_dracutmodules+=" systemd crypt "
-EOS
-cat > /etc/dracut.conf.d/20-drivers.conf <<'"'"'EOS'"'"'
-add_drivers+=" nvme xhci_pci xhci_hcd usbhid hid_generic hid "
-EOS
-cat > /etc/dracut.conf.d/30-fido2.conf <<'"'"'EOS'"'"'
-install_items+="/usr/bin/systemd-cryptsetup /usr/bin/fido2-token /usr/lib/*/libfido2.so* /usr/lib/*/libcbor.so*"
-EOS
-
-# --- Build initramfs and place it where firmware expects it ---
-KVER=$(ls -1 /lib/modules | sort -V | tail -n1)
-dracut --force /boot/initramfs-$KVER.img $KVER
-ln -sf initramfs-$KVER.img /boot/initrd.img
-ln -sf initramfs-$KVER.img /boot/initrd.img-$KVER
-cp -a /boot/initramfs-$KVER.img /boot/firmware/initrd.img
-
-# --- Create uncompressed kernel for Pi 5 firmware ---
-if [ -f "/usr/lib/linux-image-$KVER/Image" ]; then
-  cp -a "/usr/lib/linux-image-$KVER/Image" /boot/firmware/kernel_2712.img
-else
-  FMT=$(file -b "/boot/vmlinuz-$KVER" || true)
-  case "$FMT" in
-    *Zstandard*|*zstd*) zstd -dc "/boot/vmlinuz-$KVER" > /boot/firmware/kernel_2712.img ;;
-    *LZ4*)              lz4  -dc "/boot/vmlinuz-$KVER" > /boot/firmware/kernel_2712.img ;;
-    *gzip*)             zcat     "/boot/vmlinuz-$KVER" > /boot/firmware/kernel_2712.img ;;
-    *)                  cp -a "/boot/vmlinuz-$KVER" /boot/firmware/kernel_2712.img ;;
-  esac
-fi
-
-# --- Ensure Pi 5 DTB is present on the boot partition ---
-DTB=$(find /lib/firmware -type f -name "bcm2712-rpi-5-b.dtb" | sort | tail -n1 || true)
-[ -n "$DTB" ] && cp -a "$DTB" /boot/firmware/
-
-# --- Dracut hook to copy rdsosreport.txt to the FAT partition on failure ---
-mkdir -p /usr/lib/dracut/modules.d/99copylog
-cat > /usr/lib/dracut/modules.d/99copylog/module-setup.sh <<'"'"'EOS'"'"'
-#!/bin/bash
-check() { return 0; }
-depends() { echo base; return 0; }
-install() {
-  # Guard $moddir for nounset; derive if absent
-  local mdir="${moddir:-$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)}"
-  inst_hook emergency 99 "$mdir/copylog.sh"
-}
-EOS
-chmod +x /usr/lib/dracut/modules.d/99copylog/module-setup.sh
-
-cat > /usr/lib/dracut/modules.d/99copylog/copylog.sh <<'"'"'EOS'"'"'
-#!/bin/sh
-set -e
-for dev in /dev/nvme0n1p1 /dev/sda1 /dev/sdb1 /dev/mmcblk0p1; do
-  [ -b "$dev" ] || continue
-  mkdir -p /mnt/bootfat
-  if mount -t vfat "$dev" /mnt/bootfat 2>/dev/null; then
-    if [ -s /run/initramfs/rdsosreport.txt ]; then
-      cp -f /run/initramfs/rdsosreport.txt /mnt/bootfat/rdsosreport.txt 2>/dev/null || true
-      sync || true
-    fi
-    umount /mnt/bootfat || true
-    break
-  fi
-done
-EOS
-chmod +x /usr/lib/dracut/modules.d/99copylog/copylog.sh
-
-# Rebuild to ensure the copylog module is included
-dracut --force /boot/initramfs-$KVER.img $KVER
-ln -sf initramfs-$KVER.img /boot/initrd.img
-cp -a /boot/initramfs-$KVER.img /boot/firmware/initrd.img
-
-true
-'
-}
-
-verify_boot_assets(){
-  echo "---- verify boot assets on FAT ----"
-  file "$TGT_ROOT/boot/firmware/kernel_2712.img" || true
-  ls -lh "$TGT_ROOT/boot/firmware/initrd.img" || true
-  echo "-- config.txt (key lines) --"
-  grep -E '^(kernel|initramfs|cmdline)=|^dtoverlay=|^dtparam=' "$TGT_ROOT/boot/firmware/config.txt" || true
-  echo "-- cmdline.txt --"
-  cat "$TGT_ROOT/boot/firmware/cmdline.txt" || true
-  echo "-- firmware blobs (sample) --"
-  ls -1 "$TGT_ROOT/boot/firmware"/start*.elf "$TGT_ROOT/boot/firmware"/fixup*.dat | head -n 8 || true
-  echo "-- Pi5 DTB --"
-  ls -l "$TGT_ROOT/boot/firmware/"*rpi-5-b.dtb || true
-}
-
-enroll_fido_tokens(){
-  echo "Enrolling FIDO2 Solo keys into $(part 2) ..."
-  need systemd-cryptenroll
-  need fido2-token
-
-  # Collect all hidraw paths from both output styles (some distros print 'Device: /dev/hidrawX')
-  mapfile -t DEVS < <(
-    fido2-token -L \
-      | sed -n 's,^\(/dev/hidraw[0-9]\+\):.*,\1,p; s,^Device:[[:space:]]\+/dev/hidraw\([0-9]\+\).*,/dev/hidraw\1,p' \
-      | sort -u
-  )
-
-  if (( ${#DEVS[@]} == 0 )); then
-    echo "No FIDO2 tokens detected; skipping enrollment (you can enroll later)."
-    echo "Example later: systemd-cryptenroll $(part 2) --fido2-device=/dev/hidrawX --fido2-with-client-pin=no"
-    return 0
-  fi
-
-  # Recommend keeping exactly ONE key plugged during first enrollment to avoid ambiguity.
-  if (( ${#DEVS[@]} > 1 )); then
-    echo "Note: multiple FIDO2 tokens present: ${DEVS[*]}"
-    echo "If enrollment fails, try with only one key inserted."
-  fi
-
-  local rc=0
-  for D in "${DEVS[@]}"; do
-    echo "-> Enrolling $D (you should be asked to touch the key)"
-    if ! SYSTEMD_LOG_LEVEL=debug systemd-cryptenroll "$(part 2)" \
-          --fido2-device="$D" \
-          --fido2-with-client-pin=no \
-          --fido2-with-user-presence=yes \
-          --fido2-with-user-verification=no \
-          --label="solo-$(basename "$D")"; then
-      echo "WARN: enrollment failed for $D"
-      rc=1
-    fi
-  done
-
-  echo "Tokens enrolled (if any):"
-  systemd-cryptenroll "$(part 2)" --list || true
-  return $rc
-}
-
-cleanup(){
-  rm -f "$TGT_ROOT/usr/sbin/policy-rc.d" || true
-  umount -lf "$TGT_ROOT/dev/pts" 2>/dev/null || true
-  for d in dev proc sys; do umount -lf "$TGT_ROOT/$d" 2>/dev/null || true; done
-  umount -lf "$TGT_ROOT/boot/firmware" 2>/dev/null || true
-  umount -lf "$TGT_BOOT" 2>/dev/null || true
-  umount -lf "$TGT_ROOT" 2>/dev/null || true
-  cryptsetup close cryptroot 2>/dev/null || true
-  umount -lf "$IMG_BOOT_MNT" 2>/dev/null || true
-  umount -lf "$IMG_ROOT_MNT" 2>/dev/null || true
-}
-
-main(){
-  require_root
-  need losetup; need parted; need rsync
-  auto_detect_target_disk
-  echo "Target disk: $NVME"
-  ensure_binfmt_aarch64
-  ensure_image
-  preflight_cleanup
-  guard_target_device
-  open_image
-  confirm_and_wipe
-  setup_luks
-  mount_targets
-  rsync_root_and_boot
-  write_crypttab_fstab
-  fix_firmware_files
-  seed_cloud_init
-  prep_chroot_mounts
-  in_chroot
-  verify_boot_assets
-  need_host_fido2
-  enroll_fido_tokens
-  cleanup
-  echo "✅ NVMe prepared."
-  echo "   Install in the Pi 5 and boot with no SD."
-  echo "   Expect LUKS to unlock automatically with a Solo key inserted;"
-  echo "   passphrase fallback remains. Hostname: ${STYX_HOSTNAME}  User: ${STYX_USER}"
-  echo "   On first boot, reach it via: ssh -i ~/.ssh/id_ed25519_titan styx@titan-ag.local"
-}
-
-main "$@"

services/monitoring/README.md

@@ -1,28 +0,0 @@
-# services/monitoring
-
-## Grafana admin secret
-
-The Grafana Helm release expects a pre-existing secret named `grafana-admin`
-in the `monitoring` namespace. Create or rotate it with:
-
-```bash
-kubectl create secret generic grafana-admin \
-  --namespace monitoring \
-  --from-literal=admin-user=admin \
-  --from-literal=admin-password='REPLACE_ME'
-```
-
-Update the password whenever you rotate credentials.
-
-## DCGM exporter image
-
-The NVIDIA GPU metrics DaemonSet expects `registry.bstein.dev/monitoring/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04`, mirrored from `docker.io/nvidia/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04`. Refresh it in Zot when bumping versions:
-
-```bash
-skopeo copy \
-  --all \
-  docker://docker.io/nvidia/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04 \
-  docker://registry.bstein.dev/monitoring/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04
-```
-
-When finished mirroring from the control-plane, you can remove temporary tooling with `sudo apt-get purge -y skopeo && sudo apt-get autoremove -y` and clear `~/.config/containers/auth.json`.

services/monitoring/dashboards/atlas-gpu.json

@@ -1,184 +0,0 @@
-{
-  "uid": "atlas-gpu",
-  "title": "Atlas GPU",
-  "folderUid": "atlas-internal",
-  "editable": true,
-  "panels": [
-    {
-      "id": 1,
-      "type": "piechart",
-      "title": "Namespace GPU Share",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "100 * ( ( (sum by (namespace) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{namespace!=\"\",pod!=\"\"}[1h]))) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) ) and on(namespace) ( (topk(10, ( sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) ) + (sum(container_memory_working_set_bytes{namespace!=\"\",pod!=\"\",container!=\"\"}) by (namespace) / 1e9) + ((sum((kube_pod_container_resource_requests{namespace!=\"\",resource=\"nvidia.com/gpu\"} or kube_pod_container_resource_limits{namespace!=\"\",resource=\"nvidia.com/gpu\"})) by (namespace)) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) * 100)) >= bool 0) ) ) / clamp_min(sum( ( (sum by (namespace) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{namespace!=\"\",pod!=\"\"}[1h]))) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) ) and on(namespace) ( (topk(10, ( sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) ) + (sum(container_memory_working_set_bytes{namespace!=\"\",pod!=\"\",container!=\"\"}) by (namespace) / 1e9) + ((sum((kube_pod_container_resource_requests{namespace!=\"\",resource=\"nvidia.com/gpu\"} or kube_pod_container_resource_limits{namespace!=\"\",resource=\"nvidia.com/gpu\"})) by (namespace)) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) * 100)) >= bool 0) ) ), 1)",
-          "refId": "A",
-          "legendFormat": "{{namespace}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent",
-          "color": {
-            "mode": "palette-classic"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "right"
-        },
-        "pieType": "pie",
-        "displayLabels": [
-          "percent"
-        ],
-        "tooltip": {
-          "mode": "single"
-        },
-        "colorScheme": "interpolateSpectral",
-        "colorBy": "value",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        }
-      }
-    },
-    {
-      "id": 2,
-      "type": "timeseries",
-      "title": "GPU Util by Namespace",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(DCGM_FI_DEV_GPU_UTIL{namespace!=\"\",pod!=\"\"}) by (namespace)",
-          "refId": "A",
-          "legendFormat": "{{namespace}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 3,
-      "type": "timeseries",
-      "title": "GPU Util by Node",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "targets": [
-        {
-          "expr": "sum by (Hostname) (DCGM_FI_DEV_GPU_UTIL{pod!=\"\"})",
-          "refId": "A",
-          "legendFormat": "{{Hostname}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 4,
-      "type": "table",
-      "title": "Top Pods by GPU Util",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "targets": [
-        {
-          "expr": "topk(10, sum(DCGM_FI_DEV_GPU_UTIL{pod!=\"\"}) by (namespace,pod,Hostname))",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      },
-      "transformations": [
-        {
-          "id": "labelsToFields",
-          "options": {}
-        }
-      ]
-    }
-  ],
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "annotations": {
-    "list": []
-  },
-  "schemaVersion": 39,
-  "style": "dark",
-  "tags": [
-    "atlas",
-    "gpu"
-  ]
-}

services/monitoring/dashboards/atlas-network.json

@@ -1,445 +0,0 @@
-{
-  "uid": "atlas-network",
-  "title": "Atlas Network",
-  "folderUid": "atlas-internal",
-  "editable": true,
-  "panels": [
-    {
-      "id": 1,
-      "type": "stat",
-      "title": "Ingress Traffic",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 0,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(node_network_receive_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "Bps",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 2,
-      "type": "stat",
-      "title": "Egress Traffic",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 8,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(node_network_transmit_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "Bps",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 3,
-      "type": "stat",
-      "title": "Intra-Cluster Traffic",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 16,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace!=\"traefik\",pod!=\"\"}[5m]) + rate(container_network_transmit_bytes_total{namespace!=\"traefik\",pod!=\"\"}[5m])) or on() vector(0)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "Bps",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 4,
-      "type": "stat",
-      "title": "Top Router req/s",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 0,
-        "y": 4
-      },
-      "targets": [
-        {
-          "expr": "topk(1, sum by (router) (rate(traefik_router_requests_total[5m])))",
-          "refId": "A",
-          "legendFormat": "{{router}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "req/s",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 5,
-      "type": "timeseries",
-      "title": "Per-Node Throughput",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 8
-      },
-      "targets": [
-        {
-          "expr": "avg by (node) ((sum(rate(node_network_transmit_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0) + sum(rate(node_network_receive_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 6,
-      "type": "table",
-      "title": "Top Namespaces",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "targets": [
-        {
-          "expr": "topk(10, sum(rate(container_network_transmit_bytes_total{namespace!=\"\"}[5m]) + rate(container_network_receive_bytes_total{namespace!=\"\"}[5m])) by (namespace))",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      },
-      "transformations": [
-        {
-          "id": "labelsToFields",
-          "options": {}
-        }
-      ]
-    },
-    {
-      "id": 7,
-      "type": "table",
-      "title": "Top Pods",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "targets": [
-        {
-          "expr": "topk(10, sum(rate(container_network_transmit_bytes_total{pod!=\"\"}[5m]) + rate(container_network_receive_bytes_total{pod!=\"\"}[5m])) by (namespace,pod))",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      },
-      "transformations": [
-        {
-          "id": "labelsToFields",
-          "options": {}
-        }
-      ]
-    },
-    {
-      "id": 8,
-      "type": "timeseries",
-      "title": "Traefik Routers (req/s)",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 0,
-        "y": 25
-      },
-      "targets": [
-        {
-          "expr": "topk(10, sum by (router) (rate(traefik_router_requests_total[5m])))",
-          "refId": "A",
-          "legendFormat": "{{router}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "req/s"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 9,
-      "type": "timeseries",
-      "title": "Traefik Entrypoints (req/s)",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 12,
-        "y": 25
-      },
-      "targets": [
-        {
-          "expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total[5m]))",
-          "refId": "A",
-          "legendFormat": "{{entrypoint}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "req/s"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    }
-  ],
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "annotations": {
-    "list": []
-  },
-  "schemaVersion": 39,
-  "style": "dark",
-  "tags": [
-    "atlas",
-    "network"
-  ]
-}

services/monitoring/dashboards/atlas-nodes.json

@@ -1,395 +0,0 @@
-{
-  "uid": "atlas-nodes",
-  "title": "Atlas Nodes",
-  "folderUid": "atlas-internal",
-  "editable": true,
-  "panels": [
-    {
-      "id": 1,
-      "type": "stat",
-      "title": "Worker Nodes Ready",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 0,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(kube_node_status_condition{condition=\"Ready\",status=\"true\",node=~\"titan-04|titan-05|titan-06|titan-07|titan-08|titan-09|titan-10|titan-11|titan-12|titan-13|titan-14|titan-15|titan-16|titan-17|titan-18|titan-19|titan-22|titan-24\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto",
-            "valueSuffix": "/18"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 2,
-      "type": "stat",
-      "title": "Control Plane Ready",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 8,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(kube_node_status_condition{condition=\"Ready\",status=\"true\",node=~\"titan-0a|titan-0b|titan-0c\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto",
-            "valueSuffix": "/3"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 3,
-      "type": "stat",
-      "title": "Control Plane Workloads",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 8,
-        "x": 16,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(kube_pod_info{node=~\"titan-0a|titan-0b|titan-0c\",namespace!~\"kube-system|kube-public|kube-node-lease|longhorn-system|monitoring|flux-system\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 4,
-      "type": "timeseries",
-      "title": "Node CPU",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 24,
-        "x": 0,
-        "y": 4
-      },
-      "targets": [
-        {
-          "expr": "avg by (node) (((1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right",
-          "calcs": [
-            "last"
-          ]
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 5,
-      "type": "timeseries",
-      "title": "Node RAM",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 24,
-        "x": 0,
-        "y": 13
-      },
-      "targets": [
-        {
-          "expr": "avg by (node) ((avg by (instance) ((node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right",
-          "calcs": [
-            "last"
-          ]
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 6,
-      "type": "timeseries",
-      "title": "Control Plane (incl. titan-db) CPU",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 0,
-        "y": 22
-      },
-      "targets": [
-        {
-          "expr": "(avg by (node) (((1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-0a|titan-0b|titan-0c|titan-db\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 7,
-      "type": "timeseries",
-      "title": "Control Plane (incl. titan-db) RAM",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 12,
-        "y": 22
-      },
-      "targets": [
-        {
-          "expr": "(avg by (node) ((avg by (instance) ((node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-0a|titan-0b|titan-0c|titan-db\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      }
-    },
-    {
-      "id": 8,
-      "type": "timeseries",
-      "title": "Root Filesystem Usage",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 24,
-        "x": 0,
-        "y": 31
-      },
-      "targets": [
-        {
-          "expr": "avg by (node) ((avg by (instance) ((1 - (node_filesystem_avail_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay\"} / node_filesystem_size_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay\"})) * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "timeFrom": "30d"
-    }
-  ],
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "annotations": {
-    "list": []
-  },
-  "schemaVersion": 39,
-  "style": "dark",
-  "tags": [
-    "atlas",
-    "nodes"
-  ]
-}

services/monitoring/dashboards/atlas-pods.json

@@ -1,377 +0,0 @@
-{
-  "uid": "atlas-pods",
-  "title": "Atlas Pods",
-  "folderUid": "atlas-internal",
-  "editable": true,
-  "panels": [
-    {
-      "id": 1,
-      "type": "stat",
-      "title": "Problem Pods",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 6,
-        "x": 0,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(max by (namespace,pod) (kube_pod_status_phase{phase!~\"Running|Succeeded\"}))",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 2,
-      "type": "stat",
-      "title": "CrashLoop / ImagePull",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 6,
-        "x": 6,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(max by (namespace,pod) (kube_pod_container_status_waiting_reason{reason=~\"CrashLoopBackOff|ImagePullBackOff\"}))",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 3,
-      "type": "stat",
-      "title": "Stuck Terminating (>10m)",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 6,
-        "x": 12,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(max by (namespace,pod) (((time() - kube_pod_deletion_timestamp{pod!=\"\"}) > bool 600) and on(namespace,pod) (kube_pod_deletion_timestamp{pod!=\"\"} > bool 0)))",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 4,
-      "type": "stat",
-      "title": "Control Plane Workloads",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 6,
-        "x": 18,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(kube_pod_info{node=~\"titan-0a|titan-0b|titan-0c\",namespace!~\"kube-system|kube-public|kube-node-lease|longhorn-system|monitoring|flux-system\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "none",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 5,
-      "type": "table",
-      "title": "Pods Not Running",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 4
-      },
-      "targets": [
-        {
-          "expr": "(time() - kube_pod_created{pod!=\"\"}) * on(namespace,pod) group_left(node) kube_pod_info * on(namespace,pod) group_left(phase) max by (namespace,pod,phase) (kube_pod_status_phase{phase!~\"Running|Succeeded\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      },
-      "transformations": [
-        {
-          "id": "labelsToFields",
-          "options": {}
-        }
-      ]
-    },
-    {
-      "id": 6,
-      "type": "table",
-      "title": "CrashLoop / ImagePull",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "targets": [
-        {
-          "expr": "(time() - kube_pod_created{pod!=\"\"}) * on(namespace,pod) group_left(node) kube_pod_info * on(namespace,pod,container) group_left(reason) max by (namespace,pod,container,reason) (kube_pod_container_status_waiting_reason{reason=~\"CrashLoopBackOff|ImagePullBackOff\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      },
-      "transformations": [
-        {
-          "id": "labelsToFields",
-          "options": {}
-        }
-      ]
-    },
-    {
-      "id": 7,
-      "type": "table",
-      "title": "Terminating >10m",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "targets": [
-        {
-          "expr": "(((time() - kube_pod_deletion_timestamp{pod!=\"\"}) and on(namespace,pod) (kube_pod_deletion_timestamp{pod!=\"\"} > bool 0)) * on(namespace,pod) group_left(node) kube_pod_info)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      },
-      "transformations": [
-        {
-          "id": "labelsToFields",
-          "options": {}
-        },
-        {
-          "id": "filterByValue",
-          "options": {
-            "match": "Value",
-            "operator": "gt",
-            "value": 600
-          }
-        }
-      ]
-    }
-  ],
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "annotations": {
-    "list": []
-  },
-  "schemaVersion": 39,
-  "style": "dark",
-  "tags": [
-    "atlas",
-    "pods"
-  ]
-}

services/monitoring/dashboards/atlas-storage.json

@@ -1,419 +0,0 @@
-{
-  "uid": "atlas-storage",
-  "title": "Atlas Storage",
-  "folderUid": "atlas-internal",
-  "editable": true,
-  "panels": [
-    {
-      "id": 1,
-      "type": "stat",
-      "title": "Astreae Usage",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 0,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) * 100)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "percentage",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 70
-              },
-              {
-                "color": "red",
-                "value": 85
-              }
-            ]
-          },
-          "unit": "percent",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 2,
-      "type": "stat",
-      "title": "Asteria Usage",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 6,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) * 100)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "percentage",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 70
-              },
-              {
-                "color": "red",
-                "value": 85
-              }
-            ]
-          },
-          "unit": "percent",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 3,
-      "type": "stat",
-      "title": "Astreae Free",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 12,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "decbytes",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 4,
-      "type": "stat",
-      "title": "Asteria Free",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 18,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"})",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "rgba(115, 115, 115, 1)",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          },
-          "unit": "decbytes",
-          "custom": {
-            "displayMode": "auto"
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "value"
-      }
-    },
-    {
-      "id": 5,
-      "type": "timeseries",
-      "title": "Astreae Per-Node Usage",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 0,
-        "y": 5
-      },
-      "targets": [
-        {
-          "expr": "(avg by (node) ((avg by (instance) ((1 - (node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"} / node_filesystem_size_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"})) * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-1[2-9]|titan-2[24]\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "timeFrom": "30d"
-    },
-    {
-      "id": 6,
-      "type": "timeseries",
-      "title": "Asteria Per-Node Usage",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 12,
-        "y": 5
-      },
-      "targets": [
-        {
-          "expr": "(avg by (node) ((avg by (instance) ((1 - (node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"} / node_filesystem_size_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"})) * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-1[2-9]|titan-2[24]\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-          "refId": "A",
-          "legendFormat": "{{node}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "timeFrom": "30d"
-    },
-    {
-      "id": 7,
-      "type": "timeseries",
-      "title": "Astreae Usage History",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 0,
-        "y": 14
-      },
-      "targets": [
-        {
-          "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) * 100)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "timeFrom": "90d"
-    },
-    {
-      "id": 8,
-      "type": "timeseries",
-      "title": "Asteria Usage History",
-      "datasource": {
-        "type": "prometheus",
-        "uid": "atlas-vm"
-      },
-      "gridPos": {
-        "h": 9,
-        "w": 12,
-        "x": 12,
-        "y": 14
-      },
-      "targets": [
-        {
-          "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) * 100)",
-          "refId": "A"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        },
-        "overrides": []
-      },
-      "options": {
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "timeFrom": "90d"
-    }
-  ],
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "annotations": {
-    "list": []
-  },
-  "schemaVersion": 39,
-  "style": "dark",
-  "tags": [
-    "atlas",
-    "storage"
-  ]
-}

services/monitoring/dcgm-exporter.yaml

@@ -1,80 +0,0 @@
-# services/monitoring/dcgm-exporter.yaml
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: dcgm-exporter
-  namespace: monitoring
-  labels:
-    app: dcgm-exporter
-spec:
-  selector:
-    matchLabels:
-      app: dcgm-exporter
-  updateStrategy:
-    rollingUpdate:
-      maxUnavailable: 2
-  template:
-    metadata:
-      labels:
-        app: dcgm-exporter
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "9400"
-    spec:
-      serviceAccountName: default
-      runtimeClassName: nvidia
-      affinity:
-        nodeAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-            nodeSelectorTerms:
-              - matchExpressions:
-                  - key: kubernetes.io/hostname
-                    operator: In
-                    values:
-                      - titan-20
-                      - titan-21
-                      - titan-22
-                      - titan-24
-      tolerations:
-        - operator: Exists
-      containers:
-        - name: dcgm-exporter
-          image: registry.bstein.dev/monitoring/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04
-          imagePullPolicy: Always
-          ports:
-            - name: metrics
-              containerPort: 9400
-          env:
-            - name: DCGM_EXPORTER_KUBERNETES
-              value: "true"
-          securityContext:
-            privileged: true
-          resources:
-            requests:
-              cpu: 50m
-              memory: 64Mi
-          volumeMounts:
-            - name: pod-resources
-              mountPath: /var/lib/kubelet/pod-resources
-      imagePullSecrets:
-        - name: zot-regcred
-      volumes:
-        - name: pod-resources
-          hostPath:
-            path: /var/lib/kubelet/pod-resources
-            type: Directory
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: dcgm-exporter
-  namespace: monitoring
-  labels:
-    app: dcgm-exporter
-spec:
-  selector:
-    app: dcgm-exporter
-  ports:
-    - name: metrics
-      port: 9400
-      targetPort: metrics

services/monitoring/grafana-dashboard-gpu.yaml

@@ -1,193 +0,0 @@
-# services/monitoring/grafana-dashboard-gpu.yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-dashboard-gpu
-  labels:
-    grafana_dashboard: "1"
-data:
-  atlas-gpu.json: |
-    {
-      "uid": "atlas-gpu",
-      "title": "Atlas GPU",
-      "folderUid": "atlas-internal",
-      "editable": true,
-      "panels": [
-        {
-          "id": 1,
-          "type": "piechart",
-          "title": "Namespace GPU Share",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 12,
-            "x": 0,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "100 * ( ( (sum by (namespace) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{namespace!=\"\",pod!=\"\"}[1h]))) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) ) and on(namespace) ( (topk(10, ( sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) ) + (sum(container_memory_working_set_bytes{namespace!=\"\",pod!=\"\",container!=\"\"}) by (namespace) / 1e9) + ((sum((kube_pod_container_resource_requests{namespace!=\"\",resource=\"nvidia.com/gpu\"} or kube_pod_container_resource_limits{namespace!=\"\",resource=\"nvidia.com/gpu\"})) by (namespace)) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) * 100)) >= bool 0) ) ) / clamp_min(sum( ( (sum by (namespace) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{namespace!=\"\",pod!=\"\"}[1h]))) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) ) and on(namespace) ( (topk(10, ( sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) ) + (sum(container_memory_working_set_bytes{namespace!=\"\",pod!=\"\",container!=\"\"}) by (namespace) / 1e9) + ((sum((kube_pod_container_resource_requests{namespace!=\"\",resource=\"nvidia.com/gpu\"} or kube_pod_container_resource_limits{namespace!=\"\",resource=\"nvidia.com/gpu\"})) by (namespace)) or on(namespace) (sum(rate(container_cpu_usage_seconds_total{namespace!=\"\",pod!=\"\",container!=\"\"}[5m])) by (namespace) * 0) * 100)) >= bool 0) ) ), 1)",
-              "refId": "A",
-              "legendFormat": "{{namespace}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent",
-              "color": {
-                "mode": "palette-classic"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "list",
-              "placement": "right"
-            },
-            "pieType": "pie",
-            "displayLabels": [
-              "percent"
-            ],
-            "tooltip": {
-              "mode": "single"
-            },
-            "colorScheme": "interpolateSpectral",
-            "colorBy": "value",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            }
-          }
-        },
-        {
-          "id": 2,
-          "type": "timeseries",
-          "title": "GPU Util by Namespace",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 12,
-            "x": 12,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(DCGM_FI_DEV_GPU_UTIL{namespace!=\"\",pod!=\"\"}) by (namespace)",
-              "refId": "A",
-              "legendFormat": "{{namespace}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 3,
-          "type": "timeseries",
-          "title": "GPU Util by Node",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 12,
-            "x": 0,
-            "y": 8
-          },
-          "targets": [
-            {
-              "expr": "sum by (Hostname) (DCGM_FI_DEV_GPU_UTIL{pod!=\"\"})",
-              "refId": "A",
-              "legendFormat": "{{Hostname}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 4,
-          "type": "table",
-          "title": "Top Pods by GPU Util",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 12,
-            "x": 12,
-            "y": 8
-          },
-          "targets": [
-            {
-              "expr": "topk(10, sum(DCGM_FI_DEV_GPU_UTIL{pod!=\"\"}) by (namespace,pod,Hostname))",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "showHeader": true
-          },
-          "transformations": [
-            {
-              "id": "labelsToFields",
-              "options": {}
-            }
-          ]
-        }
-      ],
-      "time": {
-        "from": "now-12h",
-        "to": "now"
-      },
-      "annotations": {
-        "list": []
-      },
-      "schemaVersion": 39,
-      "style": "dark",
-      "tags": [
-        "atlas",
-        "gpu"
-      ]
-    }

services/monitoring/grafana-dashboard-network.yaml

@@ -1,454 +0,0 @@
-# services/monitoring/grafana-dashboard-network.yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-dashboard-network
-  labels:
-    grafana_dashboard: "1"
-data:
-  atlas-network.json: |
-    {
-      "uid": "atlas-network",
-      "title": "Atlas Network",
-      "folderUid": "atlas-internal",
-      "editable": true,
-      "panels": [
-        {
-          "id": 1,
-          "type": "stat",
-          "title": "Ingress Traffic",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 0,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(rate(node_network_receive_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "Bps",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 2,
-          "type": "stat",
-          "title": "Egress Traffic",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 8,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(rate(node_network_transmit_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "Bps",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 3,
-          "type": "stat",
-          "title": "Intra-Cluster Traffic",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 16,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(rate(container_network_receive_bytes_total{namespace!=\"traefik\",pod!=\"\"}[5m]) + rate(container_network_transmit_bytes_total{namespace!=\"traefik\",pod!=\"\"}[5m])) or on() vector(0)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "Bps",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 4,
-          "type": "stat",
-          "title": "Top Router req/s",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 0,
-            "y": 4
-          },
-          "targets": [
-            {
-              "expr": "topk(1, sum by (router) (rate(traefik_router_requests_total[5m])))",
-              "refId": "A",
-              "legendFormat": "{{router}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "req/s",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 5,
-          "type": "timeseries",
-          "title": "Per-Node Throughput",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 24,
-            "x": 0,
-            "y": 8
-          },
-          "targets": [
-            {
-              "expr": "avg by (node) ((sum(rate(node_network_transmit_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0) + sum(rate(node_network_receive_bytes_total{device!~\"lo|cni.*|veth.*|flannel.*|docker.*|virbr.*|vxlan.*|wg.*\"}[5m])) or on() vector(0)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "Bps"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 6,
-          "type": "table",
-          "title": "Top Namespaces",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 0,
-            "y": 16
-          },
-          "targets": [
-            {
-              "expr": "topk(10, sum(rate(container_network_transmit_bytes_total{namespace!=\"\"}[5m]) + rate(container_network_receive_bytes_total{namespace!=\"\"}[5m])) by (namespace))",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "Bps"
-            },
-            "overrides": []
-          },
-          "options": {
-            "showHeader": true
-          },
-          "transformations": [
-            {
-              "id": "labelsToFields",
-              "options": {}
-            }
-          ]
-        },
-        {
-          "id": 7,
-          "type": "table",
-          "title": "Top Pods",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 12,
-            "y": 16
-          },
-          "targets": [
-            {
-              "expr": "topk(10, sum(rate(container_network_transmit_bytes_total{pod!=\"\"}[5m]) + rate(container_network_receive_bytes_total{pod!=\"\"}[5m])) by (namespace,pod))",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "Bps"
-            },
-            "overrides": []
-          },
-          "options": {
-            "showHeader": true
-          },
-          "transformations": [
-            {
-              "id": "labelsToFields",
-              "options": {}
-            }
-          ]
-        },
-        {
-          "id": 8,
-          "type": "timeseries",
-          "title": "Traefik Routers (req/s)",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 0,
-            "y": 25
-          },
-          "targets": [
-            {
-              "expr": "topk(10, sum by (router) (rate(traefik_router_requests_total[5m])))",
-              "refId": "A",
-              "legendFormat": "{{router}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "req/s"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 9,
-          "type": "timeseries",
-          "title": "Traefik Entrypoints (req/s)",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 12,
-            "y": 25
-          },
-          "targets": [
-            {
-              "expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total[5m]))",
-              "refId": "A",
-              "legendFormat": "{{entrypoint}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "req/s"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        }
-      ],
-      "time": {
-        "from": "now-12h",
-        "to": "now"
-      },
-      "annotations": {
-        "list": []
-      },
-      "schemaVersion": 39,
-      "style": "dark",
-      "tags": [
-        "atlas",
-        "network"
-      ]
-    }

services/monitoring/grafana-dashboard-nodes.yaml

@@ -1,404 +0,0 @@
-# services/monitoring/grafana-dashboard-nodes.yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-dashboard-nodes
-  labels:
-    grafana_dashboard: "1"
-data:
-  atlas-nodes.json: |
-    {
-      "uid": "atlas-nodes",
-      "title": "Atlas Nodes",
-      "folderUid": "atlas-internal",
-      "editable": true,
-      "panels": [
-        {
-          "id": 1,
-          "type": "stat",
-          "title": "Worker Nodes Ready",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 0,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(kube_node_status_condition{condition=\"Ready\",status=\"true\",node=~\"titan-04|titan-05|titan-06|titan-07|titan-08|titan-09|titan-10|titan-11|titan-12|titan-13|titan-14|titan-15|titan-16|titan-17|titan-18|titan-19|titan-22|titan-24\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto",
-                "valueSuffix": "/18"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 2,
-          "type": "stat",
-          "title": "Control Plane Ready",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 8,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(kube_node_status_condition{condition=\"Ready\",status=\"true\",node=~\"titan-0a|titan-0b|titan-0c\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto",
-                "valueSuffix": "/3"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 3,
-          "type": "stat",
-          "title": "Control Plane Workloads",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 8,
-            "x": 16,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(kube_pod_info{node=~\"titan-0a|titan-0b|titan-0c\",namespace!~\"kube-system|kube-public|kube-node-lease|longhorn-system|monitoring|flux-system\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 4,
-          "type": "timeseries",
-          "title": "Node CPU",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 24,
-            "x": 0,
-            "y": 4
-          },
-          "targets": [
-            {
-              "expr": "avg by (node) (((1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right",
-              "calcs": [
-                "last"
-              ]
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 5,
-          "type": "timeseries",
-          "title": "Node RAM",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 24,
-            "x": 0,
-            "y": 13
-          },
-          "targets": [
-            {
-              "expr": "avg by (node) ((avg by (instance) ((node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right",
-              "calcs": [
-                "last"
-              ]
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 6,
-          "type": "timeseries",
-          "title": "Control Plane (incl. titan-db) CPU",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 0,
-            "y": 22
-          },
-          "targets": [
-            {
-              "expr": "(avg by (node) (((1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-0a|titan-0b|titan-0c|titan-db\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 7,
-          "type": "timeseries",
-          "title": "Control Plane (incl. titan-db) RAM",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 12,
-            "y": 22
-          },
-          "targets": [
-            {
-              "expr": "(avg by (node) ((avg by (instance) ((node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-0a|titan-0b|titan-0c|titan-db\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          }
-        },
-        {
-          "id": 8,
-          "type": "timeseries",
-          "title": "Root Filesystem Usage",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 24,
-            "x": 0,
-            "y": 31
-          },
-          "targets": [
-            {
-              "expr": "avg by (node) ((avg by (instance) ((1 - (node_filesystem_avail_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay\"} / node_filesystem_size_bytes{mountpoint=\"/\",fstype!~\"tmpfs|overlay\"})) * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          },
-          "timeFrom": "30d"
-        }
-      ],
-      "time": {
-        "from": "now-12h",
-        "to": "now"
-      },
-      "annotations": {
-        "list": []
-      },
-      "schemaVersion": 39,
-      "style": "dark",
-      "tags": [
-        "atlas",
-        "nodes"
-      ]
-    }

services/monitoring/grafana-dashboard-pods.yaml

@@ -1,386 +0,0 @@
-# services/monitoring/grafana-dashboard-pods.yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-dashboard-pods
-  labels:
-    grafana_dashboard: "1"
-data:
-  atlas-pods.json: |
-    {
-      "uid": "atlas-pods",
-      "title": "Atlas Pods",
-      "folderUid": "atlas-internal",
-      "editable": true,
-      "panels": [
-        {
-          "id": 1,
-          "type": "stat",
-          "title": "Problem Pods",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 6,
-            "x": 0,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(max by (namespace,pod) (kube_pod_status_phase{phase!~\"Running|Succeeded\"}))",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "red",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 2,
-          "type": "stat",
-          "title": "CrashLoop / ImagePull",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 6,
-            "x": 6,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(max by (namespace,pod) (kube_pod_container_status_waiting_reason{reason=~\"CrashLoopBackOff|ImagePullBackOff\"}))",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "red",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 3,
-          "type": "stat",
-          "title": "Stuck Terminating (>10m)",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 6,
-            "x": 12,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(max by (namespace,pod) (((time() - kube_pod_deletion_timestamp{pod!=\"\"}) > bool 600) and on(namespace,pod) (kube_pod_deletion_timestamp{pod!=\"\"} > bool 0)))",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "red",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 4,
-          "type": "stat",
-          "title": "Control Plane Workloads",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 4,
-            "w": 6,
-            "x": 18,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(kube_pod_info{node=~\"titan-0a|titan-0b|titan-0c\",namespace!~\"kube-system|kube-public|kube-node-lease|longhorn-system|monitoring|flux-system\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "red",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "none",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 5,
-          "type": "table",
-          "title": "Pods Not Running",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 4
-          },
-          "targets": [
-            {
-              "expr": "(time() - kube_pod_created{pod!=\"\"}) * on(namespace,pod) group_left(node) kube_pod_info * on(namespace,pod) group_left(phase) max by (namespace,pod,phase) (kube_pod_status_phase{phase!~\"Running|Succeeded\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "s"
-            },
-            "overrides": []
-          },
-          "options": {
-            "showHeader": true
-          },
-          "transformations": [
-            {
-              "id": "labelsToFields",
-              "options": {}
-            }
-          ]
-        },
-        {
-          "id": 6,
-          "type": "table",
-          "title": "CrashLoop / ImagePull",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 14
-          },
-          "targets": [
-            {
-              "expr": "(time() - kube_pod_created{pod!=\"\"}) * on(namespace,pod) group_left(node) kube_pod_info * on(namespace,pod,container) group_left(reason) max by (namespace,pod,container,reason) (kube_pod_container_status_waiting_reason{reason=~\"CrashLoopBackOff|ImagePullBackOff\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "s"
-            },
-            "overrides": []
-          },
-          "options": {
-            "showHeader": true
-          },
-          "transformations": [
-            {
-              "id": "labelsToFields",
-              "options": {}
-            }
-          ]
-        },
-        {
-          "id": 7,
-          "type": "table",
-          "title": "Terminating >10m",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 24
-          },
-          "targets": [
-            {
-              "expr": "(((time() - kube_pod_deletion_timestamp{pod!=\"\"}) and on(namespace,pod) (kube_pod_deletion_timestamp{pod!=\"\"} > bool 0)) * on(namespace,pod) group_left(node) kube_pod_info)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "s"
-            },
-            "overrides": []
-          },
-          "options": {
-            "showHeader": true
-          },
-          "transformations": [
-            {
-              "id": "labelsToFields",
-              "options": {}
-            },
-            {
-              "id": "filterByValue",
-              "options": {
-                "match": "Value",
-                "operator": "gt",
-                "value": 600
-              }
-            }
-          ]
-        }
-      ],
-      "time": {
-        "from": "now-12h",
-        "to": "now"
-      },
-      "annotations": {
-        "list": []
-      },
-      "schemaVersion": 39,
-      "style": "dark",
-      "tags": [
-        "atlas",
-        "pods"
-      ]
-    }

services/monitoring/grafana-dashboard-storage.yaml

@@ -1,428 +0,0 @@
-# services/monitoring/grafana-dashboard-storage.yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-dashboard-storage
-  labels:
-    grafana_dashboard: "1"
-data:
-  atlas-storage.json: |
-    {
-      "uid": "atlas-storage",
-      "title": "Atlas Storage",
-      "folderUid": "atlas-internal",
-      "editable": true,
-      "panels": [
-        {
-          "id": 1,
-          "type": "stat",
-          "title": "Astreae Usage",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 5,
-            "w": 6,
-            "x": 0,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) * 100)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "percentage",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "yellow",
-                    "value": 70
-                  },
-                  {
-                    "color": "red",
-                    "value": 85
-                  }
-                ]
-              },
-              "unit": "percent",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 2,
-          "type": "stat",
-          "title": "Asteria Usage",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 5,
-            "w": 6,
-            "x": 6,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) * 100)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "percentage",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "yellow",
-                    "value": 70
-                  },
-                  {
-                    "color": "red",
-                    "value": 85
-                  }
-                ]
-              },
-              "unit": "percent",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 3,
-          "type": "stat",
-          "title": "Astreae Free",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 5,
-            "w": 6,
-            "x": 12,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "decbytes",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 4,
-          "type": "stat",
-          "title": "Asteria Free",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 5,
-            "w": 6,
-            "x": 18,
-            "y": 0
-          },
-          "targets": [
-            {
-              "expr": "sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"})",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "mappings": [],
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "rgba(115, 115, 115, 1)",
-                    "value": null
-                  },
-                  {
-                    "color": "green",
-                    "value": 1
-                  }
-                ]
-              },
-              "unit": "decbytes",
-              "custom": {
-                "displayMode": "auto"
-              }
-            },
-            "overrides": []
-          },
-          "options": {
-            "colorMode": "value",
-            "graphMode": "area",
-            "justifyMode": "center",
-            "reduceOptions": {
-              "calcs": [
-                "lastNotNull"
-              ],
-              "fields": "",
-              "values": false
-            },
-            "textMode": "value"
-          }
-        },
-        {
-          "id": 5,
-          "type": "timeseries",
-          "title": "Astreae Per-Node Usage",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 0,
-            "y": 5
-          },
-          "targets": [
-            {
-              "expr": "(avg by (node) ((avg by (instance) ((1 - (node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"} / node_filesystem_size_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"})) * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-1[2-9]|titan-2[24]\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          },
-          "timeFrom": "30d"
-        },
-        {
-          "id": 6,
-          "type": "timeseries",
-          "title": "Asteria Per-Node Usage",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 12,
-            "y": 5
-          },
-          "targets": [
-            {
-              "expr": "(avg by (node) ((avg by (instance) ((1 - (node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"} / node_filesystem_size_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"})) * 100)) * on(instance) group_left(node) label_replace(node_uname_info{nodename!=\"\"}, \"node\", \"$1\", \"nodename\", \"(.*)\"))) * on(node) group_left() label_replace(node_uname_info{nodename=~\"titan-1[2-9]|titan-2[24]\"}, \"node\", \"$1\", \"nodename\", \"(.*)\")",
-              "refId": "A",
-              "legendFormat": "{{node}}"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "right"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          },
-          "timeFrom": "30d"
-        },
-        {
-          "id": 7,
-          "type": "timeseries",
-          "title": "Astreae Usage History",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 0,
-            "y": 14
-          },
-          "targets": [
-            {
-              "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/astreae\",fstype!~\"tmpfs|overlay\"}) * 100)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "bottom"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          },
-          "timeFrom": "90d"
-        },
-        {
-          "id": 8,
-          "type": "timeseries",
-          "title": "Asteria Usage History",
-          "datasource": {
-            "type": "prometheus",
-            "uid": "atlas-vm"
-          },
-          "gridPos": {
-            "h": 9,
-            "w": 12,
-            "x": 12,
-            "y": 14
-          },
-          "targets": [
-            {
-              "expr": "100 - (sum(node_filesystem_avail_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) / sum(node_filesystem_size_bytes{mountpoint=\"/mnt/asteria\",fstype!~\"tmpfs|overlay\"}) * 100)",
-              "refId": "A"
-            }
-          ],
-          "fieldConfig": {
-            "defaults": {
-              "unit": "percent"
-            },
-            "overrides": []
-          },
-          "options": {
-            "legend": {
-              "displayMode": "table",
-              "placement": "bottom"
-            },
-            "tooltip": {
-              "mode": "multi"
-            }
-          },
-          "timeFrom": "90d"
-        }
-      ],
-      "time": {
-        "from": "now-12h",
-        "to": "now"
-      },
-      "annotations": {
-        "list": []
-      },
-      "schemaVersion": 39,
-      "style": "dark",
-      "tags": [
-        "atlas",
-        "storage"
-      ]
-    }

services/monitoring/grafana-folders.yaml

@@ -1,35 +0,0 @@
-# services/monitoring/grafana-folders.yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-folders
-  labels:
-    app.kubernetes.io/name: grafana
-    app.kubernetes.io/component: folders
-data:
-  folders.yaml: |
-    apiVersion: 1
-    folders:
-      - uid: overview
-        title: Overview
-        permissions:
-          - role: Viewer
-            permission: View
-          - role: Editor
-            permission: Edit
-          - role: Admin
-            permission: Admin
-      - uid: atlas-internal
-        title: Atlas Internal
-        permissions:
-          - role: Editor
-            permission: View
-          - role: Admin
-            permission: Admin
-      - uid: oceanus-internal
-        title: Oceanus Internal
-        permissions:
-          - role: Editor
-            permission: View
-          - role: Admin
-            permission: Admin

services/monitoring/helmrelease.yaml

@@ -71,7 +71,8 @@ spec:
 
       persistentVolume:
         enabled: true
-        size: 100Gi
+        size: 100Gi              # adjust; uses default StorageClass (Longhorn)
+        # storageClassName: ""   # set if you want a specific class
 
       # Enable built-in Kubernetes scraping
       scrape:
@@ -209,187 +210,3 @@ spec:
                 - action: keep
                   source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app_kubernetes_io_part_of]
                   regex: flux-system;flux
-            - job_name: "titan-db"
-              static_configs:
-                - targets: ["titan-db:9100"]
-              relabel_configs:
-                - source_labels: [__address__]
-                  target_label: instance
-              metric_relabel_configs:
-                - source_labels: [instance]
-                  target_label: node
-                  replacement: titan-db
-
----
-
-apiVersion: helm.toolkit.fluxcd.io/v2
-kind: HelmRelease
-metadata:
-  name: grafana
-  namespace: monitoring
-spec:
-  interval: 15m
-  chart:
-    spec:
-      chart: grafana
-      version: "~8.5.0"
-      sourceRef:
-        kind: HelmRepository
-        name: grafana
-        namespace: flux-system
-  values:
-    admin:
-      existingSecret: grafana-admin
-      userKey: admin-user
-      passwordKey: admin-password
-    persistence:
-      enabled: true
-      size: 20Gi
-      storageClassName: astreae
-    service:
-      type: ClusterIP
-    env:
-      GF_AUTH_ANONYMOUS_ENABLED: "true"
-      GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
-      GF_SECURITY_ALLOW_EMBEDDING: "true"
-    grafana.ini:
-      server:
-        domain: metrics.bstein.dev
-        root_url: https://metrics.bstein.dev/
-      dashboards:
-        default_home_dashboard_path: /var/lib/grafana/dashboards/overview/atlas-overview.json
-      auth.anonymous:
-        hide_version: true
-      users:
-        default_theme: dark
-    ingress:
-      enabled: true
-      ingressClassName: traefik
-      annotations:
-        cert-manager.io/cluster-issuer: letsencrypt
-      hosts:
-        - metrics.bstein.dev
-      path: /
-      tls:
-        - secretName: grafana-metrics-tls
-          hosts:
-            - metrics.bstein.dev
-    datasources:
-      datasources.yaml:
-        apiVersion: 1
-        datasources:
-          - name: VictoriaMetrics
-            type: prometheus
-            access: proxy
-            url: http://victoria-metrics-single-server:8428
-            isDefault: true
-            jsonData:
-              timeInterval: "15s"
-            uid: atlas-vm
-    dashboardProviders:
-      dashboardproviders.yaml:
-        apiVersion: 1
-        providers:
-          - name: overview
-            orgId: 1
-            folder: Overview
-            type: file
-            disableDeletion: false
-            editable: false
-            options:
-              path: /var/lib/grafana/dashboards/overview
-          - name: pods
-            orgId: 1
-            folder: Atlas Internal
-            type: file
-            disableDeletion: false
-            editable: true
-            options:
-              path: /var/lib/grafana/dashboards/pods
-          - name: nodes
-            orgId: 1
-            folder: Atlas Internal
-            type: file
-            disableDeletion: false
-            editable: true
-            options:
-              path: /var/lib/grafana/dashboards/nodes
-          - name: storage
-            orgId: 1
-            folder: Atlas Internal
-            type: file
-            disableDeletion: false
-            editable: true
-            options:
-              path: /var/lib/grafana/dashboards/storage
-          - name: gpu
-            orgId: 1
-            folder: Atlas Internal
-            type: file
-            disableDeletion: false
-            editable: true
-            options:
-              path: /var/lib/grafana/dashboards/gpu
-          - name: network
-            orgId: 1
-            folder: Atlas Internal
-            type: file
-            disableDeletion: false
-            editable: true
-            options:
-              path: /var/lib/grafana/dashboards/network
-    dashboardsConfigMaps:
-      overview: grafana-dashboard-overview
-      pods: grafana-dashboard-pods
-      nodes: grafana-dashboard-nodes
-      storage: grafana-dashboard-storage
-      gpu: grafana-dashboard-gpu
-      network: grafana-dashboard-network
-    extraConfigmapMounts:
-      - name: grafana-folders
-        mountPath: /etc/grafana/provisioning/folders
-        configMap: grafana-folders
-        readOnly: true
-
----
-
-apiVersion: helm.toolkit.fluxcd.io/v2
-kind: HelmRelease
-metadata:
-  name: alertmanager
-  namespace: monitoring
-spec:
-  interval: 15m
-  chart:
-    spec:
-      chart: alertmanager
-      version: "~1.9.0"
-      sourceRef:
-        kind: HelmRepository
-        name: prometheus
-        namespace: flux-system
-  values:
-    ingress:
-      enabled: true
-      ingressClassName: traefik
-      annotations:
-        cert-manager.io/cluster-issuer: letsencrypt
-      hosts:
-        - host: alerts.bstein.dev
-          paths:
-            - path: /
-              pathType: Prefix
-      tls:
-        - secretName: alerts-bstein-dev-tls
-          hosts:
-            - alerts.bstein.dev
-    config:
-      global:
-        resolve_timeout: 5m
-      route:
-        receiver: default
-        group_wait: 30s
-        group_interval: 5m
-        repeat_interval: 2h
-      receivers:
-        - name: default

services/monitoring/kustomization.yaml

@@ -5,12 +5,4 @@ namespace: monitoring
 resources:
   - namespace.yaml
   - rbac.yaml
-  - grafana-dashboard-overview.yaml
-  - grafana-dashboard-pods.yaml
-  - grafana-dashboard-nodes.yaml
-  - grafana-dashboard-storage.yaml
-  - grafana-dashboard-network.yaml
-  - grafana-dashboard-gpu.yaml
-  - dcgm-exporter.yaml
-  - grafana-folders.yaml
   - helmrelease.yaml