498 Commits

Author SHA1 Message Date
jenkins
7f5508c930 feat(ariadne): declare the fix categories, and show them beside the actions
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
The three categories now appear in the deployment next to the action
allowlist, and the monitor prints both at the policy gate so the difference is
visible during a demo rather than asserted: two ids Ariadne may execute on its
own authority, three categories it may only ask Hermes to propose a patch for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 20:46:55 -03:00
jenkins
4fe4162dad fix(demo): make the monitor's evidence correct for the code demo too
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
Several stages assumed the fixture job. On a code-demo run they would have
read the fixture ConfigMap, which is unrelated to that job, and asserted that
a field changing from unhealthy to healthy was the repair - flatly false when
the outcome is a pull request. The route stage would also have claimed the
source-proposal branch was not taken during the run that takes it.

The stages now branch on the job: the route explains that a source fix is not
a registered action and so takes the proposal branch; the response points at
the pull request recorded on the incident; and the verify stage says the
branch build validates the proposal and that nothing merges without a human
whatever it reports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 20:07:10 -03:00
jenkins
4ffcae8b3c fix(gitea): raise the memory ceiling; time-bound the demo cleanup calls
Gitea wedged at 2073Mi against a 2Gi limit: its API stopped answering even on
its own loopback, and the repeated SSH LoginGraceTime drops in its log were
starvation symptoms rather than a separate fault. Raised to 3Gi.

The reset command's Gitea calls had no --max-time, so a slow service became an
indefinite hang with no output - the script appeared frozen after 'clearing
bstein/hermes-code-demo'. They now fail after 25 seconds and say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 20:04:42 -03:00
jenkins
4e3be5f7e4 fix(demo): frame the control decision and the branch endings clearly
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
Five corrections to how the monitor explains itself, plus code-demo support.

The authorization is now described as what it is: a match between an action
Ariadne already has code to perform, an action id Hermes is permitted to
request, and the action Hermes actually recommended. Hermes cannot add to that
registry, and the recommendation proceeds only because it names something
already in it. The editorial line about pull requests is gone.

The repair now reads as a change rather than a value: the field said unhealthy
when the build failed and says healthy because Ariadne just patched it.

The rebuild stage says plainly that the operational branch is complete and
that neither the human-required response nor the optional source proposal is
entered, rather than leaving the reader to infer it from silence.

Inspectable outputs states that on the chart it is the 'records and artifacts'
edge out of the whole response box, not out of one branch - every path ends
there. The layout makes it look like it belongs to the proposal path alone.

 follows hermes-code-demo, which takes the proposal branch: the
stages now read as a patch proposal and a pull request instead of being
mislabelled an escalation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 19:18:31 -03:00
jenkins
7d40355ff6 feat(demo): show what each source contributed to the bundle
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
The bundle stage printed the job allowlist, which says nothing about what was
actually sent. It now samples the bundle itself, rebuilt with the same
collector Ariadne used - durable, because a finished build's console does not
change - and trimmed hard, since the point is to show what each source
contributes rather than to reprint it:

    jenkins.console_failures : 1 region(s), truncated=False
       | ERROR: Demo fixture check failed for incident hermes-triage-demo/34
    jenkins.failed_tests     : 0
    log_evidence.records     : 35 from OpenSearch kube-*
       | [jenkins] Using /home/jenkins/agent/remoting as a remoting work directory

The diagnosis now also carries run_id and run_seconds, so the run can be
opened in the Hermes dashboard to see the prompt it was given and every tool
call it made - the request, not just the answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:57:44 -03:00
jenkins
bbb4f6b185 fix(demo): show durable evidence at detection, not live cluster state
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
The detection stage read the fixture ConfigMap, which is a live value. Every
stage describes a moment that has passed, and by the time detection is
narrated the repair has already run - so it printed 'healthy' and implied the
fixture had never failed, contradicting the premise of the whole run. Replay
made it certain rather than occasional.

It now prints the incident's recorded state changes, which stay true
afterwards:

    21:48:03 detected         {result: FAILURE, ...}
    21:48:18 diagnosed        {classification: known_demo_fixture_failure, ...}
    21:48:18 repairing        {action: repair_demo_fixture}
    21:48:18 awaiting_rebuild {repair: configmap_patch, ...}
    21:49:03 resolved         {resolved_by_build: 35}

The fixture read stays on the response stage, where healthy is the point.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:52:08 -03:00
jenkins
6da0cba96d feat(demo): show the diagnosis and the policy result, in the chart's own words
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
The Hermes stage proved only that a pod was running, when the substantive
evidence is the JSON Hermes returned; it now reads that from the audit trail,
so classification, confidence and requested action are on screen. The policy
gates stage never showed the verdict it was describing; authorized and
authorize_reason now appear with it.

The chart paths were paraphrases. They are now the node labels from
mermaid/TestAutomation.mmd verbatim - Response check, Scoped repair guard,
Action authorizer, Action registry, Scoped ConfigMap repair, Action result -
so a line on screen can be found on the diagram. The checklist carries them too.

The Optional source proposal branch was silently absent rather than explained.
The route stage now states it was not taken and why: a fixture repair is an
operational action, and source fixes are not actions at all.

Also widened the log window and its timeout; the fetch occasionally exceeded
25s and dropped a poll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 18:14:13 -03:00
jenkins
39cc744d19 fix(demo): stop the monitor losing steps to poll timing
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
It read only the newest tick from a --tail=400 window. Ariadne now emits
roughly two hundred lines per tick, so that window held about two ticks, and a
step that lasted a single tick - the repair, most importantly - vanished if a
poll landed after it. Build 30 was triaged, repaired and resolved correctly
while the monitor printed nothing at all.

The window is now time-based, and every tick inside it is replayed rather than
just the last. mark() was already idempotent, so replaying is safe and the
monitor recovers stages it was not running for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 17:32:51 -03:00
jenkins
16d771aa69 feat(demo): narrate the monitor by echoing the commands it runs
An audience watching a dashboard has to take the result on trust. Watching
kubectl run against the cluster and reading the raw answer is the difference
between a demonstration and an assertion, so every read is now printed before
it runs and its output shown beneath.

Each stage also names the service UI worth looking at when something changes
there: the Jenkins job as the build goes red and again when the rebuild starts
on its own, the Hermes dashboard as the agent run appears, the Grafana triage
panels and the filed issues at the end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:41:35 -03:00
jenkins
db14a9d6a8 feat(demo): name the chart branch the monitor is taking
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
An audience following mermaid/TestAutomation.mmd needs to know which edge off
the route diamond was taken, which was the one thing the output left them to
infer. It now quotes the chart's own labels - authorized action, human
required, and the optional code path - and the docstring states where the
reporting is coarser than the diagram.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:25:39 -03:00
jenkins
e93dc7b8a8 feat(demo): reset deletes the demo repositories' issues, PRs and branches
A rerun should start from nothing. DEMO_REPOS is the entire blast radius and
is deliberately explicit: a real service's issues are genuine triage records,
and clearing them to tidy a demo would destroy the evidence the system exists
to produce.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:24:53 -03:00
jenkins
e9c2d84dbc fix(demo): print the monitor summary once per incident
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
The resolving tick remains the newest one until another incident opens, so the
completed checklist reprinted on every poll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:07:03 -03:00
jenkins
93ff096fa1 feat(demo): add reset and monitor commands
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
reset restores only what the demo itself creates: the fixture ConfigMap, the
demo repository's open repair pull requests and their branches. Real service
repositories are deliberately untouched - those issues are genuine triage
records, and deleting them to tidy a demo would destroy the evidence the
system exists to produce.

monitor streams the flow in the stages of mermaid/TestAutomation.mmd, so a
second terminal can be followed against the chart: Detect and gather, Hermes
analysis, Ariadne policy gates, Ariadne response, Inspectable outputs. Each
stage prints the evidence that moved it, since the claim the chart makes is
that every step is answerable from data rather than asserted. Read-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:57:02 -03:00
jenkins
cd974baeeb fix(demo): print the triage allowlist as a list, not raw config
Some checks failed
Tests / Declarative: Post Actions testing.tests.test_repo_structure.test_knowledge_service_mirror_matches_source failed
The allowlist is the outermost safety boundary - a job absent from it is never
touched, whatever fails - so it is worth reading at a glance rather than as a
comma-separated setting value echoed verbatim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:48:57 -03:00
jenkins
5364cc9666 fix(demo): source local credentials and stop waiting on a job that no longer exists
The script demanded JENKINS_USER and JENKINS_TOKEN in the environment and said
only 'set JENKINS_USER' when they were missing, which is not enough to act on.
It now sources scripts/ops/hermes_triage_demo.env, git-ignored so it can hold
real tokens, and names that file when credentials are absent. An example file
records what belongs in it.

The fixture command also still polled for a hermes-demo-repair-<build> Job.
That Job stopped existing when the repair became an in-process ConfigMap
patch, so the command would have waited its full 400 seconds and then reported
nothing. It now watches the fixture returning to healthy, which is what
actually happens.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 15:45:15 -03:00
jenkins
b233630007 feat(demo): preflight the agent-pool cap and open code-demo PRs
Two conditions silently break a rehearsal. A saturated Kubernetes agent pool
leaves the demo build queued reporting that all nodes are offline, and an open
hermes-repair PR makes the duplicate guard refuse a new proposal. Report both
so the operator sees them before starting rather than mid-demo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:19:37 -03:00
jenkins
fc0d41056d fix(scripts): dashboard renderer wrote outside the repo after the layout move
ROOT still used parents[1], which resolved to scripts/ once the renderer
moved into scripts/render/. Every --build run wrote a phantom
scripts/services/monitoring tree and silently left the real dashboards
untouched. Points at the repo root again and removes the stray tree.

Also adds Hermes triage panels to the Atlas Testing dashboard: open
escalations awaiting a human, automated actions succeeded, Hermes
diagnosis latency, actions by result, and incident state by job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 21:41:02 -03:00
jenkins
a555fc0c96 feat(hermes-code): multibranch validation for hermes-repair/* proposal branches
Adds demo driver script and branch-level test gate so a Hermes-proposed
pull request carries a green build before human merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 19:46:30 -03:00
jenkins
f4097c6e59 Merge origin/main into layout migration
Reconciles the services layout migration with ~991 upstream commits:
- Remote content wins for cassandra/cassandra-auth, monitoring dashboards,
  vmalert availability rules, veles, vault auth script, dashboard render
  script and tests (request-v4 availability definition)
- Layout paths win for structure: keycloak/bstein-dev-home job dirs use
  bootstrap-jobs/validation-jobs; cassandra realm jobs live in
  cassandra-auth (removed keycloak duplicates)
- Union: applications CR list gains hermes-chat and cassandra
  image-automation
- Fixed post-migration paths in dashboard test module loader and
  hermes-access job header

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 16:26:36 -03:00
jenkins
0da9e4c82d refactor: restructure services layout, retire oceanus, add aether scaffolding
- Move flat service manifests into structured subdirs (apps/, bootstrap-jobs/,
  repair-jobs/, migration-jobs/, validation-jobs/, node-ops/, networking/)
- Retire oneoffs/ directories across services
- Remove oceanus cluster and its host roles; add aether cluster + terraform scaffolding
- Reorganize scripts/ into ops/, render/, sync/, manual-tests/
- Add Makefile with render/validate/test/flux targets and repo-structure tests
- Update flux-system application CRs to the new paths
- Add hermes-automated-triage-24h-plan knowledge doc (+ comms mirror)
- Refresh knowledge catalogs, dashboards, vmalert rules, quality contract

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 16:21:36 -03:00
jenkins
9b8a29022d Revert "monitoring(network): organize Traefik traffic lanes"
This reverts commit eeec8b72dc885950dd116d6dc2ad6574c1cbe6b1.
2026-08-05 12:06:23 -03:00
jenkins
eeec8b72dc monitoring(network): organize Traefik traffic lanes 2026-08-05 12:02:51 -03:00
jenkins
ea53bec75d monitoring: remove legacy availability series 2026-08-04 22:49:48 -03:00
jenkins
663aa3e4f1 monitoring: retain daily availability through retries 2026-08-04 21:47:30 -03:00
jenkins
2691581cbc monitoring: publish availability outside query pool 2026-08-04 21:40:24 -03:00
jenkins
a8dda9815d monitoring: add daily availability rollups 2026-08-04 21:24:47 -03:00
jenkins
7c47a92610 monitoring: add compact availability fallback 2026-08-04 21:14:06 -03:00
jenkins
f724e72a96 monitoring: use request success for availability 2026-08-04 21:12:31 -03:00
jenkins
4906fb73d8 monitoring: measure gateway availability 2026-08-04 21:02:19 -03:00
jenkins
302fe78681 monitoring: preserve serving availability history 2026-08-04 18:39:36 -03:00
jenkins
368de58769 monitoring: fall back to live serving state 2026-08-04 18:16:13 -03:00
jenkins
bbab7330ce monitoring: avoid unstable volume hosts 2026-08-04 18:06:43 -03:00
jenkins
5f9c313f0a monitoring: render test category zero state 2026-08-04 12:34:08 -03:00
jenkins
17b25c4429 monitoring: reload vmalert rules automatically 2026-08-04 12:24:38 -03:00
jenkins
b0599f4318 monitoring: prevent query pool starvation 2026-08-04 12:22:23 -03:00
jenkins
e8f75f5cc9 feat(hermes): add actionable Atlas triage skills 2026-08-03 03:58:37 -03:00
jenkins
604f91cff3 monitoring(nodes): clamp CPU charts to physical range 2026-08-02 19:36:44 -03:00
jenkins
82b8a1c899 monitoring(gpu): attribute Jetson activity by allocation 2026-08-02 04:26:33 -03:00
jenkins
4991493d3b ai(hermes): add operator guide and current GPU shares 2026-08-02 03:59:32 -03:00
jenkins
2fe327c7eb monitoring(gpu): use process-level pod attribution 2026-08-02 03:31:02 -03:00
jenkins
e794ccf254 monitoring(gpu): report time-weighted namespace usage 2026-08-02 03:08:20 -03:00
jenkins
97ecb36b56 agent: replace OpenClaw with Hermes 2026-07-21 21:02:44 -03:00
jenkins
766c96f54d monitoring: filter node dashboards to real nodes 2026-07-14 20:24:45 -03:00
jenkins
dad4448ebb monitoring: collapse duplicate Typhon climate series 2026-07-14 19:01:53 -03:00
jenkins
c2a4b42b3d monitoring: track titan-23 and pin tiny apps to workers 2026-07-14 18:14:38 -03:00
jenkins
a9ddc80e36 recovery(ananke): keep flux holds and place metrics on longhorn nodes 2026-06-18 22:08:14 -03:00
jenkins
6b777b8c74 recovery(ananke): reassert final flux hold after drain 2026-06-18 21:09:40 -03:00
jenkins
62d9791b76 recovery(ananke): leave flux root stopped on final hold 2026-06-18 20:54:16 -03:00
jenkins
ea333f6648 recovery(ananke): thaw critical flux without root apply 2026-06-18 20:37:00 -03:00
jenkins
6d3c59ad8d recovery(ananke): fetch flux source before root thaw 2026-06-18 20:27:03 -03:00