bstein/titan-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,825 Commits 27 Branches 0 Tags
Commit Graph

532 Commits

Author SHA1 Message Date
jenkins
e3e40f4618 feat(monitoring): make Hermes triage email mean something 
Every human_required escalation already files an issue in the failing
service's own repository, and mailing on each one made the inbox the loudest
and least informative output of the system. Replace the blanket alert with
two narrow ones: a repair that ran and failed, which is the only case where
the automation acted and left things no better, and an escalation still
untouched after six hours.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-06 00:26:34 -03:00
jenkins
fc0d41056d fix(scripts): dashboard renderer wrote outside the repo after the layout move 
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.

Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 21:41:02 -03:00
jenkins
2c11b87fdb fix(monitoring): Alertmanager must HELO with a FQDN for Mailu 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 21:33:11 -03:00
jenkins
84e9849e46 feat(hermes-triage): real-repo patch proposals + Alertmanager email escalation 
- Ariadne: per-repo code config for metis, lesavka, soteria,
  bstein-dev-home and ariadne, each with its own base branch, source path
  prefixes and file suffixes so a proposal can only touch that repo's
  source tree
- Alertmanager: the only receiver was an empty "default", so every alert
  fired into a void. HermesTriageHumanRequired now routes to an email
  receiver via Mailu's in-cluster local-domain relay, with resolved
  notices; scoped to service=hermes-triage so nothing else mails yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 21:30:34 -03:00
jenkins
363adedd75 feat(hermes-triage): human-required alert rule + Ariadne autotriage config 
- vmalert: hermes-triage.yaml alert group (HermesTriageHumanRequired on
  ariadne_hermes_triage_incident{status="human_required"}, 1m interval),
  wire Alertmanager notifier into vmalert-atlas-availability
- Ariadne deployment: hermes autotriage env (observe mode: autotriage on,
  autoremediation OFF), 1m schedule, allowlist hermes-triage-demo only,
  HERMES_API_KEY from optional hermes-api-server-key Secret
 2026-08-05 16:51:48 -03:00
jenkins
f4097c6e59 Merge origin/main into layout migration 
Reconciles the services layout migration with ~991 upstream commits:
- Remote content wins for cassandra/cassandra-auth, monitoring dashboards,
  vmalert availability rules, veles, vault auth script, dashboard render
  script and tests (request-v4 availability definition)
- Layout paths win for structure: keycloak/bstein-dev-home job dirs use
  bootstrap-jobs/validation-jobs; cassandra realm jobs live in
  cassandra-auth (removed keycloak duplicates)
- Union: applications CR list gains hermes-chat and cassandra
  image-automation
- Fixed post-migration paths in dashboard test module loader and
  hermes-access job header

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 16:26:36 -03:00
jenkins
0da9e4c82d refactor: restructure services layout, retire oceanus, add aether scaffolding 
- Move flat service manifests into structured subdirs (apps/, bootstrap-jobs/,
  repair-jobs/, migration-jobs/, validation-jobs/, node-ops/, networking/)
- Retire oneoffs/ directories across services
- Remove oceanus cluster and its host roles; add aether cluster + terraform scaffolding
- Reorganize scripts/ into ops/, render/, sync/, manual-tests/
- Add Makefile with render/validate/test/flux targets and repo-structure tests
- Update flux-system application CRs to the new paths
- Add hermes-automated-triage-24h-plan knowledge doc (+ comms mirror)
- Refresh knowledge catalogs, dashboards, vmalert rules, quality contract

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 16:21:36 -03:00
jenkins
9b8a29022d Revert "monitoring(network): organize Traefik traffic lanes" 
This reverts commit eeec8b72dc885950dd116d6dc2ad6574c1cbe6b1.
 2026-08-05 12:06:23 -03:00
jenkins
eeec8b72dc monitoring(network): organize Traefik traffic lanes 2026-08-05 12:02:51 -03:00
jenkins
ea53bec75d monitoring: remove legacy availability series 2026-08-04 22:49:48 -03:00
jenkins
663aa3e4f1 monitoring: retain daily availability through retries 2026-08-04 21:47:30 -03:00
jenkins
2691581cbc monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
a8dda9815d monitoring: add daily availability rollups 2026-08-04 21:24:47 -03:00
jenkins
7c47a92610 monitoring: add compact availability fallback 2026-08-04 21:14:06 -03:00
jenkins
f724e72a96 monitoring: use request success for availability 2026-08-04 21:12:31 -03:00
jenkins
4906fb73d8 monitoring: measure gateway availability 2026-08-04 21:02:19 -03:00
jenkins
302fe78681 monitoring: preserve serving availability history 2026-08-04 18:39:36 -03:00
jenkins
368de58769 monitoring: fall back to live serving state 2026-08-04 18:16:13 -03:00
jenkins
bbab7330ce monitoring: avoid unstable volume hosts 2026-08-04 18:06:43 -03:00
jenkins
5f9c313f0a monitoring: render test category zero state 2026-08-04 12:34:08 -03:00
jenkins
17b25c4429 monitoring: reload vmalert rules automatically 2026-08-04 12:24:38 -03:00
jenkins
b0599f4318 monitoring: prevent query pool starvation 2026-08-04 12:22:23 -03:00
jenkins
95bc7b6f17 monitoring(grafana): use Texas timezone 2026-08-03 15:58:16 -03:00
jenkins
8db88d79f4 fix(monitoring): reload provisioned Grafana rules 2026-08-03 04:05:33 -03:00
jenkins
e8f75f5cc9 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
jenkins
604f91cff3 monitoring(nodes): clamp CPU charts to physical range 2026-08-02 19:36:44 -03:00
jenkins
82b8a1c899 monitoring(gpu): attribute Jetson activity by allocation 2026-08-02 04:26:33 -03:00
jenkins
4991493d3b ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
2fe327c7eb monitoring(gpu): use process-level pod attribution 2026-08-02 03:31:02 -03:00
jenkins
e794ccf254 monitoring(gpu): report time-weighted namespace usage 2026-08-02 03:08:20 -03:00
jenkins
b26a4f8c66 feat(cassandra): add migration Grafana rollup 2026-07-25 00:29:03 -03:00
jenkins
766c96f54d monitoring: filter node dashboards to real nodes 2026-07-14 20:24:45 -03:00
jenkins
dad4448ebb monitoring: collapse duplicate Typhon climate series 2026-07-14 19:01:53 -03:00
jenkins
8c0e848e30 monitoring: scope endpoint node relabel to node-exporter 2026-07-14 18:57:51 -03:00
jenkins
575cae700d monitoring: relabel pod-network node-exporter names 2026-07-14 18:30:43 -03:00
jenkins
c02f36e24c monitoring: move node-exporter host port 2026-07-14 18:23:52 -03:00
jenkins
d1c1172c1d monitoring: avoid node-exporter host port collisions 2026-07-14 18:18:03 -03:00
jenkins
c2a4b42b3d monitoring: track titan-23 and pin tiny apps to workers 2026-07-14 18:14:38 -03:00
jenkins
15eb7cdba8 maintenance: expire completed one-off jobs 2026-07-14 17:10:18 -03:00
jenkins
52920d0f8b monitoring: give victoria metrics recovery headroom 2026-06-18 22:23:48 -03:00
jenkins
a9ddc80e36 recovery(ananke): keep flux holds and place metrics on longhorn nodes 2026-06-18 22:08:14 -03:00
jenkins
28356e89fc monitoring: keep nvidia exporter off oceanus 2026-06-09 00:55:11 -03:00
jenkins
654900b8a2 veles: stage atlas infrastructure 2026-06-09 00:46:46 -03:00
jenkins
4fd8a00d4a monitoring(testing): cap history panel ranges 2026-06-05 13:22:29 -03:00
jenkins
75d002dc88 monitoring(testing): cap expensive dashboard queries 2026-06-05 13:15:12 -03:00
jenkins
a2ecdef536 monitoring(testing): restore lesavka suite visibility 2026-06-05 01:04:56 -03:00
jenkins
f2ad8cca4c monitoring(testing): clean up dashboard health signals 2026-06-04 16:09:08 -03:00
jenkins
5e27384ea2 monitoring(gpu): show activity share by namespace 2026-05-22 04:22:51 -03:00
jenkins
d21b61f6d9 monitoring(gpu): count monitored GPU pool devices 2026-05-22 03:23:36 -03:00
jenkins
b367c6dea3 monitoring: keep quality probe on worker nodes 2026-05-22 03:16:01 -03:00