titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity
5,549 Commits 81 Branches 0 Tags
Commit Graph

540 Commits

Author SHA1 Message Date
Hermes Agent
c4eac8ceee fix(monitoring): keep VictoriaMetrics writable 2026-08-19 10:51:44 +00:00
jenkins
612cefad23 monitoring(ai): add provider quota operations dashboard 2026-08-16 05:13:20 -03:00
jenkins
963c14940c gpu(titan-24): pin image guard to local exporter 2026-08-11 06:10:39 -03:00
jenkins
da128b0487 monitoring: retain Pi fallback for VictoriaMetrics 2026-08-11 01:40:28 -03:00
jenkins
e9a3b1fd0e monitoring: keep VictoriaMetrics on Pi 5 workers 2026-08-11 01:36:50 -03:00
jenkins
46c50814ad fix(monitoring): retain short Jetson GPU activity 2026-08-09 14:38:51 -03:00
jenkins
1f893fba27 monitoring(testing): retain scalar branch labels 2026-08-07 20:07:51 -03:00
jenkins
55d8831f8c monitoring(testing): keep quality rollups current 2026-08-07 20:02:13 -03:00
jenkins
2b0fdd1a03 feat(monitoring): make Hermes triage email mean something 
Every human_required escalation already files an issue in the failing
service's own repository, and mailing on each one made the inbox the loudest
and least informative output of the system. Replace the blanket alert with
two narrow ones: a repair that ran and failed, which is the only case where
the automation acted and left things no better, and an escalation still
untouched after six hours.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-06 00:26:34 -03:00
jenkins
1894078b62 fix(scripts): dashboard renderer wrote outside the repo after the layout move 
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.

Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 21:41:02 -03:00
jenkins
84ff647898 fix(monitoring): Alertmanager must HELO with a FQDN for Mailu 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 21:33:11 -03:00
jenkins
664e436575 feat(hermes-triage): real-repo patch proposals + Alertmanager email escalation 
- Ariadne: per-repo code config for metis, lesavka, soteria,
  bstein-dev-home and ariadne, each with its own base branch, source path
  prefixes and file suffixes so a proposal can only touch that repo's
  source tree
- Alertmanager: the only receiver was an empty "default", so every alert
  fired into a void. HermesTriageHumanRequired now routes to an email
  receiver via Mailu's in-cluster local-domain relay, with resolved
  notices; scoped to service=hermes-triage so nothing else mails yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 21:30:34 -03:00
jenkins
f4e2da317c feat(hermes-triage): human-required alert rule + Ariadne autotriage config 
- vmalert: hermes-triage.yaml alert group (HermesTriageHumanRequired on
  ariadne_hermes_triage_incident{status="human_required"}, 1m interval),
  wire Alertmanager notifier into vmalert-atlas-availability
- Ariadne deployment: hermes autotriage env (observe mode: autotriage on,
  autoremediation OFF), 1m schedule, allowlist hermes-triage-demo only,
  HERMES_API_KEY from optional hermes-api-server-key Secret
 2026-08-05 16:51:48 -03:00
jenkins
c4b4e8a404 Merge origin/main into layout migration 
Reconciles the services layout migration with ~991 upstream commits:
- Remote content wins for cassandra/cassandra-auth, monitoring dashboards,
  vmalert availability rules, veles, vault auth script, dashboard render
  script and tests (request-v4 availability definition)
- Layout paths win for structure: keycloak/bstein-dev-home job dirs use
  bootstrap-jobs/validation-jobs; cassandra realm jobs live in
  cassandra-auth (removed keycloak duplicates)
- Union: applications CR list gains hermes-chat and cassandra
  image-automation
- Fixed post-migration paths in dashboard test module loader and
  hermes-access job header

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 16:26:36 -03:00
jenkins
8f45f47e60 refactor: restructure services layout, retire oceanus, add aether scaffolding 
- Move flat service manifests into structured subdirs (apps/, bootstrap-jobs/,
  repair-jobs/, migration-jobs/, validation-jobs/, node-ops/, networking/)
- Retire oneoffs/ directories across services
- Remove oceanus cluster and its host roles; add aether cluster + terraform scaffolding
- Reorganize scripts/ into ops/, render/, sync/, manual-tests/
- Add Makefile with render/validate/test/flux targets and repo-structure tests
- Update flux-system application CRs to the new paths
- Add hermes-automated-triage-24h-plan knowledge doc (+ comms mirror)
- Refresh knowledge catalogs, dashboards, vmalert rules, quality contract

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 16:21:36 -03:00
jenkins
798e3fee84 Revert "monitoring(network): organize Traefik traffic lanes" 
This reverts commit eeec8b72dc885950dd116d6dc2ad6574c1cbe6b1.
 2026-08-05 12:06:23 -03:00
jenkins
48253cdaf0 monitoring(network): organize Traefik traffic lanes 2026-08-05 12:02:51 -03:00
jenkins
24308f4870 monitoring: remove legacy availability series 2026-08-04 22:49:48 -03:00
jenkins
a2564f8611 monitoring: retain daily availability through retries 2026-08-04 21:47:30 -03:00
jenkins
ba4b871a62 monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
af10f7aadc monitoring: add daily availability rollups 2026-08-04 21:24:47 -03:00
jenkins
5f692ecd3d monitoring: add compact availability fallback 2026-08-04 21:14:06 -03:00
jenkins
83696d46d2 monitoring: use request success for availability 2026-08-04 21:12:31 -03:00
jenkins
6b1d6388db monitoring: measure gateway availability 2026-08-04 21:02:19 -03:00
jenkins
026db898fc monitoring: preserve serving availability history 2026-08-04 18:39:36 -03:00
jenkins
8d16b790af monitoring: fall back to live serving state 2026-08-04 18:16:13 -03:00
jenkins
5fabd1a84b monitoring: avoid unstable volume hosts 2026-08-04 18:06:43 -03:00
jenkins
e9b4404e02 monitoring: render test category zero state 2026-08-04 12:34:08 -03:00
jenkins
7c6e7f9736 monitoring: reload vmalert rules automatically 2026-08-04 12:24:38 -03:00
jenkins
e354f3f83a monitoring: prevent query pool starvation 2026-08-04 12:22:23 -03:00
jenkins
1c070d96d4 monitoring(grafana): use Texas timezone 2026-08-03 15:58:16 -03:00
jenkins
8e7327a7ef fix(monitoring): reload provisioned Grafana rules 2026-08-03 04:05:33 -03:00
jenkins
a00175b376 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
jenkins
4dab530baf monitoring(nodes): clamp CPU charts to physical range 2026-08-02 19:36:44 -03:00
jenkins
f2328f12e4 monitoring(gpu): attribute Jetson activity by allocation 2026-08-02 04:26:33 -03:00
jenkins
b1ecfe96e4 ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
df953359b9 monitoring(gpu): use process-level pod attribution 2026-08-02 03:31:02 -03:00
jenkins
3255936aeb monitoring(gpu): report time-weighted namespace usage 2026-08-02 03:08:20 -03:00
jenkins
a155847915 feat(cassandra): add migration Grafana rollup 2026-07-25 00:29:03 -03:00
jenkins
48773efbb1 monitoring: filter node dashboards to real nodes 2026-07-14 20:24:45 -03:00
jenkins
c8707c002d monitoring: collapse duplicate Typhon climate series 2026-07-14 19:01:53 -03:00
jenkins
618fd01cc5 monitoring: scope endpoint node relabel to node-exporter 2026-07-14 18:57:51 -03:00
jenkins
921edc257c monitoring: relabel pod-network node-exporter names 2026-07-14 18:30:43 -03:00
jenkins
e03598caa2 monitoring: move node-exporter host port 2026-07-14 18:23:52 -03:00
jenkins
4f4a5f99b7 monitoring: avoid node-exporter host port collisions 2026-07-14 18:18:03 -03:00
jenkins
bea20b9c97 monitoring: track titan-23 and pin tiny apps to workers 2026-07-14 18:14:38 -03:00
jenkins
6d200d79ae maintenance: expire completed one-off jobs 2026-07-14 17:10:18 -03:00
jenkins
8d3402beb7 monitoring: give victoria metrics recovery headroom 2026-06-18 22:23:48 -03:00
jenkins
e7ec333fb0 recovery(ananke): keep flux holds and place metrics on longhorn nodes 2026-06-18 22:08:14 -03:00
jenkins
b395fb6abb monitoring: keep nvidia exporter off oceanus 2026-06-09 00:55:11 -03:00