548 Commits

Author SHA1 Message Date
jenkins
0af41aa995 feat(ariadne): declare the fix categories, and show them beside the actions
The three categories now appear in the deployment next to the action
allowlist, and the monitor prints both at the policy gate so the difference is
visible during a demo rather than asserted: two ids Ariadne may execute on its
own authority, three categories it may only ask Hermes to propose a patch for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 20:46:55 -03:00
jenkins
8b42e80839 fix(demo): make the monitor's evidence correct for the code demo too
Several stages assumed the fixture job. On a code-demo run they would have
read the fixture ConfigMap, which is unrelated to that job, and asserted that
a field changing from unhealthy to healthy was the repair - flatly false when
the outcome is a pull request. The route stage would also have claimed the
source-proposal branch was not taken during the run that takes it.

The stages now branch on the job: the route explains that a source fix is not
a registered action and so takes the proposal branch; the response points at
the pull request recorded on the incident; and the verify stage says the
branch build validates the proposal and that nothing merges without a human
whatever it reports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 20:07:10 -03:00
jenkins
ba53dbab80 fix(gitea): raise the memory ceiling; time-bound the demo cleanup calls
Gitea wedged at 2073Mi against a 2Gi limit: its API stopped answering even on
its own loopback, and the repeated SSH LoginGraceTime drops in its log were
starvation symptoms rather than a separate fault. Raised to 3Gi.

The reset command's Gitea calls had no --max-time, so a slow service became an
indefinite hang with no output - the script appeared frozen after 'clearing
bstein/hermes-code-demo'. They now fail after 25 seconds and say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 20:04:42 -03:00
jenkins
c251791d47 fix(demo): frame the control decision and the branch endings clearly
Five corrections to how the monitor explains itself, plus code-demo support.

The authorization is now described as what it is: a match between an action
Ariadne already has code to perform, an action id Hermes is permitted to
request, and the action Hermes actually recommended. Hermes cannot add to that
registry, and the recommendation proceeds only because it names something
already in it. The editorial line about pull requests is gone.

The repair now reads as a change rather than a value: the field said unhealthy
when the build failed and says healthy because Ariadne just patched it.

The rebuild stage says plainly that the operational branch is complete and
that neither the human-required response nor the optional source proposal is
entered, rather than leaving the reader to infer it from silence.

Inspectable outputs states that on the chart it is the 'records and artifacts'
edge out of the whole response box, not out of one branch - every path ends
there. The layout makes it look like it belongs to the proposal path alone.

 follows hermes-code-demo, which takes the proposal branch: the
stages now read as a patch proposal and a pull request instead of being
mislabelled an escalation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 19:18:31 -03:00
jenkins
4bde434beb feat(demo): show what each source contributed to the bundle
The bundle stage printed the job allowlist, which says nothing about what was
actually sent. It now samples the bundle itself, rebuilt with the same
collector Ariadne used - durable, because a finished build's console does not
change - and trimmed hard, since the point is to show what each source
contributes rather than to reprint it:

    jenkins.console_failures : 1 region(s), truncated=False
       | ERROR: Demo fixture check failed for incident hermes-triage-demo/34
    jenkins.failed_tests     : 0
    log_evidence.records     : 35 from OpenSearch kube-*
       | [jenkins] Using /home/jenkins/agent/remoting as a remoting work directory

The diagnosis now also carries run_id and run_seconds, so the run can be
opened in the Hermes dashboard to see the prompt it was given and every tool
call it made - the request, not just the answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:57:44 -03:00
jenkins
b6c4b2cd48 fix(demo): show durable evidence at detection, not live cluster state
The detection stage read the fixture ConfigMap, which is a live value. Every
stage describes a moment that has passed, and by the time detection is
narrated the repair has already run - so it printed 'healthy' and implied the
fixture had never failed, contradicting the premise of the whole run. Replay
made it certain rather than occasional.

It now prints the incident's recorded state changes, which stay true
afterwards:

    21:48:03 detected         {result: FAILURE, ...}
    21:48:18 diagnosed        {classification: known_demo_fixture_failure, ...}
    21:48:18 repairing        {action: repair_demo_fixture}
    21:48:18 awaiting_rebuild {repair: configmap_patch, ...}
    21:49:03 resolved         {resolved_by_build: 35}

The fixture read stays on the response stage, where healthy is the point.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:52:08 -03:00
jenkins
664d009546 feat(demo): show the diagnosis and the policy result, in the chart's own words
The Hermes stage proved only that a pod was running, when the substantive
evidence is the JSON Hermes returned; it now reads that from the audit trail,
so classification, confidence and requested action are on screen. The policy
gates stage never showed the verdict it was describing; authorized and
authorize_reason now appear with it.

The chart paths were paraphrases. They are now the node labels from
mermaid/TestAutomation.mmd verbatim - Response check, Scoped repair guard,
Action authorizer, Action registry, Scoped ConfigMap repair, Action result -
so a line on screen can be found on the diagram. The checklist carries them too.

The Optional source proposal branch was silently absent rather than explained.
The route stage now states it was not taken and why: a fixture repair is an
operational action, and source fixes are not actions at all.

Also widened the log window and its timeout; the fetch occasionally exceeded
25s and dropped a poll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:14:13 -03:00
jenkins
2c5d163233 fix(demo): stop the monitor losing steps to poll timing
It read only the newest tick from a --tail=400 window. Ariadne now emits
roughly two hundred lines per tick, so that window held about two ticks, and a
step that lasted a single tick - the repair, most importantly - vanished if a
poll landed after it. Build 30 was triaged, repaired and resolved correctly
while the monitor printed nothing at all.

The window is now time-based, and every tick inside it is replayed rather than
just the last. mark() was already idempotent, so replaying is safe and the
monitor recovers stages it was not running for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 17:32:51 -03:00
jenkins
515b4eaff7 feat(demo): narrate the monitor by echoing the commands it runs
An audience watching a dashboard has to take the result on trust. Watching
kubectl run against the cluster and reading the raw answer is the difference
between a demonstration and an assertion, so every read is now printed before
it runs and its output shown beneath.

Each stage also names the service UI worth looking at when something changes
there: the Jenkins job as the build goes red and again when the rebuild starts
on its own, the Hermes dashboard as the agent run appears, the Grafana triage
panels and the filed issues at the end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:41:35 -03:00
jenkins
46c8a0675f feat(demo): name the chart branch the monitor is taking
An audience following mermaid/TestAutomation.mmd needs to know which edge off
the route diamond was taken, which was the one thing the output left them to
infer. It now quotes the chart's own labels - authorized action, human
required, and the optional code path - and the docstring states where the
reporting is coarser than the diagram.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:25:39 -03:00
jenkins
e6e0d84272 feat(demo): reset deletes the demo repositories' issues, PRs and branches
A rerun should start from nothing. DEMO_REPOS is the entire blast radius and
is deliberately explicit: a real service's issues are genuine triage records,
and clearing them to tidy a demo would destroy the evidence the system exists
to produce.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:24:53 -03:00
jenkins
aa93b0fe04 fix(demo): print the monitor summary once per incident
The resolving tick remains the newest one until another incident opens, so the
completed checklist reprinted on every poll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:07:03 -03:00
jenkins
ab16b5ca6b feat(demo): add reset and monitor commands
reset restores only what the demo itself creates: the fixture ConfigMap, the
demo repository's open repair pull requests and their branches. Real service
repositories are deliberately untouched - those issues are genuine triage
records, and deleting them to tidy a demo would destroy the evidence the
system exists to produce.

monitor streams the flow in the stages of mermaid/TestAutomation.mmd, so a
second terminal can be followed against the chart: Detect and gather, Hermes
analysis, Ariadne policy gates, Ariadne response, Inspectable outputs. Each
stage prints the evidence that moved it, since the claim the chart makes is
that every step is answerable from data rather than asserted. Read-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:57:02 -03:00
jenkins
954be9d34a fix(demo): print the triage allowlist as a list, not raw config
The allowlist is the outermost safety boundary - a job absent from it is never
touched, whatever fails - so it is worth reading at a glance rather than as a
comma-separated setting value echoed verbatim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:48:57 -03:00
jenkins
92be86211d fix(demo): source local credentials and stop waiting on a job that no longer exists
The script demanded JENKINS_USER and JENKINS_TOKEN in the environment and said
only 'set JENKINS_USER' when they were missing, which is not enough to act on.
It now sources scripts/ops/hermes_triage_demo.env, git-ignored so it can hold
real tokens, and names that file when credentials are absent. An example file
records what belongs in it.

The fixture command also still polled for a hermes-demo-repair-<build> Job.
That Job stopped existing when the repair became an in-process ConfigMap
patch, so the command would have waited its full 400 seconds and then reported
nothing. It now watches the fixture returning to healthy, which is what
actually happens.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:45:15 -03:00
jenkins
ea206e98fc feat(demo): preflight the agent-pool cap and open code-demo PRs
Two conditions silently break a rehearsal. A saturated Kubernetes agent pool
leaves the demo build queued reporting that all nodes are offline, and an open
hermes-repair PR makes the duplicate guard refuse a new proposal. Report both
so the operator sees them before starting rather than mid-demo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:19:37 -03:00
jenkins
1894078b62 fix(scripts): dashboard renderer wrote outside the repo after the layout move
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.

Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 21:41:02 -03:00
jenkins
0731ab5d9d feat(hermes-code): multibranch validation for hermes-repair/* proposal branches
Adds demo driver script and branch-level test gate so a Hermes-proposed
pull request carries a green build before human merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 19:46:30 -03:00
jenkins
c4b4e8a404 Merge origin/main into layout migration
Reconciles the services layout migration with ~991 upstream commits:
- Remote content wins for cassandra/cassandra-auth, monitoring dashboards,
  vmalert availability rules, veles, vault auth script, dashboard render
  script and tests (request-v4 availability definition)
- Layout paths win for structure: keycloak/bstein-dev-home job dirs use
  bootstrap-jobs/validation-jobs; cassandra realm jobs live in
  cassandra-auth (removed keycloak duplicates)
- Union: applications CR list gains hermes-chat and cassandra
  image-automation
- Fixed post-migration paths in dashboard test module loader and
  hermes-access job header

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 16:26:36 -03:00
jenkins
8f45f47e60 refactor: restructure services layout, retire oceanus, add aether scaffolding
- Move flat service manifests into structured subdirs (apps/, bootstrap-jobs/,
  repair-jobs/, migration-jobs/, validation-jobs/, node-ops/, networking/)
- Retire oneoffs/ directories across services
- Remove oceanus cluster and its host roles; add aether cluster + terraform scaffolding
- Reorganize scripts/ into ops/, render/, sync/, manual-tests/
- Add Makefile with render/validate/test/flux targets and repo-structure tests
- Update flux-system application CRs to the new paths
- Add hermes-automated-triage-24h-plan knowledge doc (+ comms mirror)
- Refresh knowledge catalogs, dashboards, vmalert rules, quality contract

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 16:21:36 -03:00
jenkins
798e3fee84 Revert "monitoring(network): organize Traefik traffic lanes"
This reverts commit eeec8b72dc885950dd116d6dc2ad6574c1cbe6b1.
2026-08-05 12:06:23 -03:00
jenkins
48253cdaf0 monitoring(network): organize Traefik traffic lanes 2026-08-05 12:02:51 -03:00
jenkins
24308f4870 monitoring: remove legacy availability series 2026-08-04 22:49:48 -03:00
jenkins
a2564f8611 monitoring: retain daily availability through retries 2026-08-04 21:47:30 -03:00
jenkins
ba4b871a62 monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
af10f7aadc monitoring: add daily availability rollups 2026-08-04 21:24:47 -03:00
jenkins
5f692ecd3d monitoring: add compact availability fallback 2026-08-04 21:14:06 -03:00
jenkins
83696d46d2 monitoring: use request success for availability 2026-08-04 21:12:31 -03:00
jenkins
6b1d6388db monitoring: measure gateway availability 2026-08-04 21:02:19 -03:00
jenkins
026db898fc monitoring: preserve serving availability history 2026-08-04 18:39:36 -03:00
jenkins
8d16b790af monitoring: fall back to live serving state 2026-08-04 18:16:13 -03:00
jenkins
5fabd1a84b monitoring: avoid unstable volume hosts 2026-08-04 18:06:43 -03:00
jenkins
e9b4404e02 monitoring: render test category zero state 2026-08-04 12:34:08 -03:00
jenkins
7c6e7f9736 monitoring: reload vmalert rules automatically 2026-08-04 12:24:38 -03:00
jenkins
e354f3f83a monitoring: prevent query pool starvation 2026-08-04 12:22:23 -03:00
jenkins
a00175b376 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
jenkins
4dab530baf monitoring(nodes): clamp CPU charts to physical range 2026-08-02 19:36:44 -03:00
jenkins
f2328f12e4 monitoring(gpu): attribute Jetson activity by allocation 2026-08-02 04:26:33 -03:00
jenkins
b1ecfe96e4 ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
df953359b9 monitoring(gpu): use process-level pod attribution 2026-08-02 03:31:02 -03:00
jenkins
3255936aeb monitoring(gpu): report time-weighted namespace usage 2026-08-02 03:08:20 -03:00
jenkins
880736b4ff agent: replace OpenClaw with Hermes 2026-07-21 21:02:44 -03:00
jenkins
48773efbb1 monitoring: filter node dashboards to real nodes 2026-07-14 20:24:45 -03:00
jenkins
c8707c002d monitoring: collapse duplicate Typhon climate series 2026-07-14 19:01:53 -03:00
jenkins
bea20b9c97 monitoring: track titan-23 and pin tiny apps to workers 2026-07-14 18:14:38 -03:00
jenkins
e7ec333fb0 recovery(ananke): keep flux holds and place metrics on longhorn nodes 2026-06-18 22:08:14 -03:00
jenkins
a6aa513818 recovery(ananke): reassert final flux hold after drain 2026-06-18 21:09:40 -03:00
jenkins
1eeaae8bd8 recovery(ananke): leave flux root stopped on final hold 2026-06-18 20:54:16 -03:00
jenkins
646e5643e4 recovery(ananke): thaw critical flux without root apply 2026-06-18 20:37:00 -03:00
jenkins
b3051cc24f recovery(ananke): fetch flux source before root thaw 2026-06-18 20:27:03 -03:00