493 Commits

Author SHA1 Message Date
jenkins
b6c4b2cd48 fix(demo): show durable evidence at detection, not live cluster state
The detection stage read the fixture ConfigMap, which is a live value. Every
stage describes a moment that has passed, and by the time detection is
narrated the repair has already run - so it printed 'healthy' and implied the
fixture had never failed, contradicting the premise of the whole run. Replay
made it certain rather than occasional.

It now prints the incident's recorded state changes, which stay true
afterwards:

    21:48:03 detected         {result: FAILURE, ...}
    21:48:18 diagnosed        {classification: known_demo_fixture_failure, ...}
    21:48:18 repairing        {action: repair_demo_fixture}
    21:48:18 awaiting_rebuild {repair: configmap_patch, ...}
    21:49:03 resolved         {resolved_by_build: 35}

The fixture read stays on the response stage, where healthy is the point.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:52:08 -03:00
jenkins
664d009546 feat(demo): show the diagnosis and the policy result, in the chart's own words
The Hermes stage proved only that a pod was running, when the substantive
evidence is the JSON Hermes returned; it now reads that from the audit trail,
so classification, confidence and requested action are on screen. The policy
gates stage never showed the verdict it was describing; authorized and
authorize_reason now appear with it.

The chart paths were paraphrases. They are now the node labels from
mermaid/TestAutomation.mmd verbatim - Response check, Scoped repair guard,
Action authorizer, Action registry, Scoped ConfigMap repair, Action result -
so a line on screen can be found on the diagram. The checklist carries them too.

The Optional source proposal branch was silently absent rather than explained.
The route stage now states it was not taken and why: a fixture repair is an
operational action, and source fixes are not actions at all.

Also widened the log window and its timeout; the fetch occasionally exceeded
25s and dropped a poll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:14:13 -03:00
jenkins
2c5d163233 fix(demo): stop the monitor losing steps to poll timing
It read only the newest tick from a --tail=400 window. Ariadne now emits
roughly two hundred lines per tick, so that window held about two ticks, and a
step that lasted a single tick - the repair, most importantly - vanished if a
poll landed after it. Build 30 was triaged, repaired and resolved correctly
while the monitor printed nothing at all.

The window is now time-based, and every tick inside it is replayed rather than
just the last. mark() was already idempotent, so replaying is safe and the
monitor recovers stages it was not running for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 17:32:51 -03:00
jenkins
515b4eaff7 feat(demo): narrate the monitor by echoing the commands it runs
An audience watching a dashboard has to take the result on trust. Watching
kubectl run against the cluster and reading the raw answer is the difference
between a demonstration and an assertion, so every read is now printed before
it runs and its output shown beneath.

Each stage also names the service UI worth looking at when something changes
there: the Jenkins job as the build goes red and again when the rebuild starts
on its own, the Hermes dashboard as the agent run appears, the Grafana triage
panels and the filed issues at the end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:41:35 -03:00
jenkins
46c8a0675f feat(demo): name the chart branch the monitor is taking
An audience following mermaid/TestAutomation.mmd needs to know which edge off
the route diamond was taken, which was the one thing the output left them to
infer. It now quotes the chart's own labels - authorized action, human
required, and the optional code path - and the docstring states where the
reporting is coarser than the diagram.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:25:39 -03:00
jenkins
e6e0d84272 feat(demo): reset deletes the demo repositories' issues, PRs and branches
A rerun should start from nothing. DEMO_REPOS is the entire blast radius and
is deliberately explicit: a real service's issues are genuine triage records,
and clearing them to tidy a demo would destroy the evidence the system exists
to produce.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:24:53 -03:00
jenkins
aa93b0fe04 fix(demo): print the monitor summary once per incident
The resolving tick remains the newest one until another incident opens, so the
completed checklist reprinted on every poll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:07:03 -03:00
jenkins
ab16b5ca6b feat(demo): add reset and monitor commands
reset restores only what the demo itself creates: the fixture ConfigMap, the
demo repository's open repair pull requests and their branches. Real service
repositories are deliberately untouched - those issues are genuine triage
records, and deleting them to tidy a demo would destroy the evidence the
system exists to produce.

monitor streams the flow in the stages of mermaid/TestAutomation.mmd, so a
second terminal can be followed against the chart: Detect and gather, Hermes
analysis, Ariadne policy gates, Ariadne response, Inspectable outputs. Each
stage prints the evidence that moved it, since the claim the chart makes is
that every step is answerable from data rather than asserted. Read-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:57:02 -03:00
jenkins
954be9d34a fix(demo): print the triage allowlist as a list, not raw config
The allowlist is the outermost safety boundary - a job absent from it is never
touched, whatever fails - so it is worth reading at a glance rather than as a
comma-separated setting value echoed verbatim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:48:57 -03:00
jenkins
92be86211d fix(demo): source local credentials and stop waiting on a job that no longer exists
The script demanded JENKINS_USER and JENKINS_TOKEN in the environment and said
only 'set JENKINS_USER' when they were missing, which is not enough to act on.
It now sources scripts/ops/hermes_triage_demo.env, git-ignored so it can hold
real tokens, and names that file when credentials are absent. An example file
records what belongs in it.

The fixture command also still polled for a hermes-demo-repair-<build> Job.
That Job stopped existing when the repair became an in-process ConfigMap
patch, so the command would have waited its full 400 seconds and then reported
nothing. It now watches the fixture returning to healthy, which is what
actually happens.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:45:15 -03:00
jenkins
ea206e98fc feat(demo): preflight the agent-pool cap and open code-demo PRs
Two conditions silently break a rehearsal. A saturated Kubernetes agent pool
leaves the demo build queued reporting that all nodes are offline, and an open
hermes-repair PR makes the duplicate guard refuse a new proposal. Report both
so the operator sees them before starting rather than mid-demo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:19:37 -03:00
jenkins
1894078b62 fix(scripts): dashboard renderer wrote outside the repo after the layout move
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.

Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 21:41:02 -03:00
jenkins
0731ab5d9d feat(hermes-code): multibranch validation for hermes-repair/* proposal branches
Adds demo driver script and branch-level test gate so a Hermes-proposed
pull request carries a green build before human merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 19:46:30 -03:00
jenkins
c4b4e8a404 Merge origin/main into layout migration
Reconciles the services layout migration with ~991 upstream commits:
- Remote content wins for cassandra/cassandra-auth, monitoring dashboards,
  vmalert availability rules, veles, vault auth script, dashboard render
  script and tests (request-v4 availability definition)
- Layout paths win for structure: keycloak/bstein-dev-home job dirs use
  bootstrap-jobs/validation-jobs; cassandra realm jobs live in
  cassandra-auth (removed keycloak duplicates)
- Union: applications CR list gains hermes-chat and cassandra
  image-automation
- Fixed post-migration paths in dashboard test module loader and
  hermes-access job header

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 16:26:36 -03:00
jenkins
8f45f47e60 refactor: restructure services layout, retire oceanus, add aether scaffolding
- Move flat service manifests into structured subdirs (apps/, bootstrap-jobs/,
  repair-jobs/, migration-jobs/, validation-jobs/, node-ops/, networking/)
- Retire oneoffs/ directories across services
- Remove oceanus cluster and its host roles; add aether cluster + terraform scaffolding
- Reorganize scripts/ into ops/, render/, sync/, manual-tests/
- Add Makefile with render/validate/test/flux targets and repo-structure tests
- Update flux-system application CRs to the new paths
- Add hermes-automated-triage-24h-plan knowledge doc (+ comms mirror)
- Refresh knowledge catalogs, dashboards, vmalert rules, quality contract

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 16:21:36 -03:00
jenkins
798e3fee84 Revert "monitoring(network): organize Traefik traffic lanes"
This reverts commit eeec8b72dc885950dd116d6dc2ad6574c1cbe6b1.
2026-08-05 12:06:23 -03:00
jenkins
48253cdaf0 monitoring(network): organize Traefik traffic lanes 2026-08-05 12:02:51 -03:00
jenkins
24308f4870 monitoring: remove legacy availability series 2026-08-04 22:49:48 -03:00
jenkins
a2564f8611 monitoring: retain daily availability through retries 2026-08-04 21:47:30 -03:00
jenkins
ba4b871a62 monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
af10f7aadc monitoring: add daily availability rollups 2026-08-04 21:24:47 -03:00
jenkins
5f692ecd3d monitoring: add compact availability fallback 2026-08-04 21:14:06 -03:00
jenkins
83696d46d2 monitoring: use request success for availability 2026-08-04 21:12:31 -03:00
jenkins
6b1d6388db monitoring: measure gateway availability 2026-08-04 21:02:19 -03:00
jenkins
026db898fc monitoring: preserve serving availability history 2026-08-04 18:39:36 -03:00
jenkins
8d16b790af monitoring: fall back to live serving state 2026-08-04 18:16:13 -03:00
jenkins
5fabd1a84b monitoring: avoid unstable volume hosts 2026-08-04 18:06:43 -03:00
jenkins
e9b4404e02 monitoring: render test category zero state 2026-08-04 12:34:08 -03:00
jenkins
7c6e7f9736 monitoring: reload vmalert rules automatically 2026-08-04 12:24:38 -03:00
jenkins
e354f3f83a monitoring: prevent query pool starvation 2026-08-04 12:22:23 -03:00
jenkins
a00175b376 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
jenkins
4dab530baf monitoring(nodes): clamp CPU charts to physical range 2026-08-02 19:36:44 -03:00
jenkins
f2328f12e4 monitoring(gpu): attribute Jetson activity by allocation 2026-08-02 04:26:33 -03:00
jenkins
b1ecfe96e4 ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
df953359b9 monitoring(gpu): use process-level pod attribution 2026-08-02 03:31:02 -03:00
jenkins
3255936aeb monitoring(gpu): report time-weighted namespace usage 2026-08-02 03:08:20 -03:00
jenkins
880736b4ff agent: replace OpenClaw with Hermes 2026-07-21 21:02:44 -03:00
jenkins
48773efbb1 monitoring: filter node dashboards to real nodes 2026-07-14 20:24:45 -03:00
jenkins
c8707c002d monitoring: collapse duplicate Typhon climate series 2026-07-14 19:01:53 -03:00
jenkins
bea20b9c97 monitoring: track titan-23 and pin tiny apps to workers 2026-07-14 18:14:38 -03:00
jenkins
e7ec333fb0 recovery(ananke): keep flux holds and place metrics on longhorn nodes 2026-06-18 22:08:14 -03:00
jenkins
a6aa513818 recovery(ananke): reassert final flux hold after drain 2026-06-18 21:09:40 -03:00
jenkins
1eeaae8bd8 recovery(ananke): leave flux root stopped on final hold 2026-06-18 20:54:16 -03:00
jenkins
646e5643e4 recovery(ananke): thaw critical flux without root apply 2026-06-18 20:37:00 -03:00
jenkins
b3051cc24f recovery(ananke): fetch flux source before root thaw 2026-06-18 20:27:03 -03:00
jenkins
076339f880 recovery(ananke): quiet flux root apply window 2026-06-18 20:19:03 -03:00
jenkins
3916d5bf69 recovery(ananke): apply flux hold before suspension 2026-06-18 19:58:11 -03:00
jenkins
a8eb9ecca9 recovery(ananke): apply root flux recovery hold 2026-06-18 19:40:19 -03:00
jenkins
5ac3f6813b recovery(ananke): finalize flux holds without races 2026-06-18 19:11:22 -03:00
jenkins
d3a100bb3a recovery(ananke): verify flux suspension during thaw 2026-06-18 18:53:44 -03:00