352 Commits

Author SHA1 Message Date
jenkins
c0b806e5d2 release(hermes): automate validated image promotion 2026-08-23 13:41:58 -03:00
jenkins
435e258e45 rollout(hermes): activate Claude health hysteresis 2026-08-23 11:19:48 -03:00
jenkins
6b378fd410 fix(hermes): preserve proven Claude authentication 2026-08-23 11:18:18 -03:00
jenkins
1e66a0bedf fix(hermes): tolerate worker lease handoff 2026-08-23 11:01:20 -03:00
jenkins
e9e355f25a resources: release Pi scheduler headroom 2026-08-23 10:53:57 -03:00
jenkins
b7be05427e fix(hermes): place arm64 agent on titan-08 2026-08-23 10:45:51 -03:00
jenkins
f1739e7108 fix(hermes): keep sessions off storage nodes 2026-08-23 10:43:42 -03:00
jenkins
271f3e8c32 ops: spread saturated node workloads 2026-08-23 10:37:01 -03:00
jenkins
fe7832efaf hermes: retain dual-provider quota health
Some checks failed
Tests / Declarative: Post Actions failed: 39, skipped: 19, passed: 2757
2026-08-23 02:52:42 -03:00
jenkins
81b3e6b992 hermes: expose Claude remaining quota 2026-08-23 00:42:39 -03:00
jenkins
d036062519 hermes: stabilize AI quota collection 2026-08-23 00:14:58 -03:00
jenkins
3da6a10174 hermes: remove provider-biased auto fallbacks 2026-08-22 23:54:40 -03:00
jenkins
69ad76bebc hermes: keep distributed workers on Claude 2026-08-22 23:02:15 -03:00
jenkins
8acb99d100 hermes: make Claude setup-token access durable 2026-08-22 22:52:28 -03:00
jenkins
db0cb86ca6 hermes: stabilize Claude access health probes 2026-08-22 18:07:58 -03:00
jenkins
9bffc077e8 hermes: avoid recursive home ownership rollout
Some checks failed
Tests / Declarative: Post Actions failed: 40, skipped: 19, passed: 2746
2026-08-22 17:38:42 -03:00
jenkins
bd50a7e4ad hermes: persist Claude subscription access 2026-08-22 17:32:30 -03:00
jenkins
fd42e892f7 hermes: avoid flapping worker node 2026-08-22 16:17:33 -03:00
jenkins
fdf53b264b hermes: fit owner pod on healthy workers 2026-08-22 15:55:30 -03:00
jenkins
46a44241c1 hermes: reclaim healthy worker capacity 2026-08-22 15:42:56 -03:00
jenkins
8a8df5ee4f hermes: add stateful accelerator fallback 2026-08-22 15:32:21 -03:00
ed98278981 Merge branch 'main' into feature/hermes-domain-rename-agent-worker-v2
Some checks failed
Tests / Declarative: Post Actions failed: 40, skipped: 19, passed: 2743
2026-08-21 23:15:16 +00:00
77905ff1db Merge branch 'main' into fix/cli-auto-failover-effort
Some checks failed
Tests / Declarative: Post Actions failed: 40, skipped: 19, passed: 2737
2026-08-21 23:12:12 +00:00
Hermes Agent
820872e117 feat(hermes-voice): route Whisper language to multilingual Piper
Port the original #27 detected-language pipeline onto the verified PR #39 prerequisite while preserving the current-main conversation instrument and host continuity changes.

Keep voice selection server-side with no user selector or client voice field. Reuse 207c16ab only for its stricter exact-code trust boundary, omitting malformed or absent language so Piper defaults to Amy.
2026-08-21 13:58:50 +00:00
Hermes Agent
724656d841 feat(hermes-tts): prepare fixed multilingual voice policy
Supersede draft PR #26 with a merge-safe prerequisite: bake and preload the amy, irina, and claude Piper models, route only validated server-side language to fixed voices, and leave the live voice deployment manifest unchanged.

Remove the pinned WebUI speaker selector and its persisted preference, omit client voice fields from every outbound TTS path, and keep hands-free Voice Mode and the conversation instrument intact. Hostile or legacy voice fields remain ignored by the Piper server.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-21 13:45:31 +00:00
Hermes Agent
94106bf252 refactor(hermes): rename the owner agent host to worker.bstein.dev
Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.

The rename is additive, matching the shape #38 restored for chat and triage.
CoreDNS, both agent Ingresses and the hermes-sites certificate now serve BOTH
names, so merging this cannot take away the endpoint the operator uses to
reach the coordinator. Retiring agent.hermes.bstein.dev is a separate,
separately scheduled change. No redirect middleware is added.

What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak agent proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default

What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
  rollback only needs the oauth2-proxy --redirect-url reverted and does not
  require re-running the ensure job

The agent client passes its legacy origin through the optional fourth argument
#38 added to ensure_proxy_client, so no second mechanism is introduced. The
immutable ensure Job goes -11 -> -12 because #38 already consumed -11 and that
run has completed; without a further bump this change would never be applied.
Login on the new host fails until the -12 Job completes.

Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.

#38's public-host continuity test now covers the agent proxy's dual origins
rather than asserting the agent surface was untouched by the rename.

Knowledge catalogs and diagrams regenerated with `make knowledge`.
2026-08-21 10:29:46 +00:00
Hermes Agent
f4f51323f6 hermes: restore legacy chat/triage hosts alongside the renamed ones
PR #34 renamed the public chat/triage hosts in place rather than adding
the new names, so chat.hermes.bstein.dev and triage.hermes.bstein.dev
were dropped from the certificate SANs, the hermes-sites Ingress rules
and the CoreDNS overrides at once. Both legacy hosts now answer 404 with
Traefik's default self-signed certificate, and the renamed hosts cannot
complete a login because the Keycloak clients still carry the old
redirect URIs, so chat and triage are unreachable on every hostname.

Make the rename additive, which is the rollback path the post-merge
runbook asks for when the OIDC step fails:

- put the legacy names back on hermes-sites-tls and on the Ingress,
  pointing at the same oauth2-proxy backends
- restore both CoreDNS host overrides for in-cluster resolution
- teach ensure_proxy_client to register an optional legacy origin, so
  hermes-chat-proxy and hermes-triage-proxy accept the old and new
  redirect URIs, web origins and post-logout origins at the same time
  while rootUrl stays on the canonical new host
- bump the immutable ensure Job so Flux reruns the script

Serving both names is deliberate: oauth2-proxy cookies are host-bound,
so redirecting the legacy hosts would silently drop live sessions.
Retiring them stays a separate, explicit change.

Supersedes #36, which only bumped the Job and would have left the
legacy hosts dark.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 08:49:09 +00:00
66a537ddbe Merge branch 'main' into feature/hermes-direct-cli-lane-concurrency-2-replacement 2026-08-21 07:57:35 +00:00
Hermes Agent
4277aa6a02 fix(hermes): allow two direct CLI lane workers 2026-08-21 07:11:55 +00:00
Hermes Agent
79be4d1c71 refactor(hermes): rename chat and triage public hosts 2026-08-21 06:43:19 +00:00
Hermes Agent
d579c08cbe fix(hermes): cli-auto capacity failover uses automatic Switchyard reclassification
Capacity/auth/quota failover for cli-auto previously called select_route
with a hardcoded manual lane (cli-{alternate}-{effort}), so the retry
boundary was classified as switchyard-manual instead of going through
Jetson automatic classification, making the routing evidence misleading.

Now the retry calls select_route(context, "cli-auto", exclude_provider=...)
so the boundary stays automatically classified with an explicit
failed-provider exclusion. If the classifier reclassifies to a lower
effort than the original route, the lane re-pins the chosen provider at
the original effort floor so a capacity failure never silently downgrades
a high/xhigh task. Manual lanes (assignee != cli-auto) remain unchanged
and still fail closed without switching providers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-21 05:04:55 +00:00
Hermes Agent
dda2d5fb60 fix(hermes): persist per-profile Kanban concurrency 2026-08-21 00:27:38 +00:00
Hermes Agent
d22588dddb fix(hermes): poll the session contract the chat tenants actually serve
Returning to chat.hermes.bstein.dev after a Keycloak logout/login showed
"This session is unavailable to this account. Start a new chat." even
though the session was intact and owned by the same subject.

The banner comes from the continuity fallback the router injects into
every chat page. It polled `/api/sessions/<id>` and
`/api/sessions/<id>/messages` — routes that belong to the Hermes agent
dashboard (added by scripts/patch_web_session_activity.py, applied only
in agent-deployment.yaml). The router proxies browser traffic to the
tenant Hermes WebUI instead, whose only session read is
`GET /api/session?session_id=<id>`; the dashboard paths are unrouted
there, so server.py answered its generic 404 for every poll and the
fallback reported a false ownership failure.

The script runs only on a full document load of `/session/<id>`, which is
exactly what the OIDC round-trip produces when oauth2-proxy returns the
browser to `rd=/session/<id>` — hence the "only after relogin" symptom.

Poll the WebUI contract instead, and let its own answers decide what the
banner claims: 409 `session_profile_mismatch` is the single response that
means the session is outside this account's active scope, 404 now means
the conversation is no longer stored, and 401/403 still re-enter OIDC.
The steady-state poll drops to one request and backs off to 3s/15s now
that it reaches a real endpoint on the tenant Raspberry Pi.

`boundSessionSnapshot` follows the same move: it caps the WebUI envelope
`{"session": {..., "messages": [...]}}`, relaying every other session key
verbatim rather than re-serializing a fixed struct that would silently
drop metadata the banner depends on.

Isolation is unchanged and now covered: the router still resolves the
slot from the salted Keycloak subject, overwrites any client-supplied
X-Hermes-Tenant-Identity, and forwards only the two tenant cookies.

Tests: relogin keeps a stable slot and resolves the durable session; a
second subject replaying the owner's session id, WebUI cookie and a
forged tenant header gets 404 from its own backend and never reaches the
owner's; the legacy dashboard paths are pinned as permanent 404s against
a stub of the deployed WebUI dispatch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 18:14:55 +00:00
jenkins
e28b32bd4e hermes: run the reviewed agent image and arm the takeover
Some checks failed
Tests / Declarative: Post Actions failed: 40, skipped: 7, passed: 2578
Pins hermes-agent to sha256:4a385fbd, built by hermes-agent-image #11 from
this exact main revision through the daemonless Kaniko lane. Enables
kanban.auto_supervise and returns chat tenancy to the documented four slots.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 23:55:14 -03:00
jenkins
b6ae6225f6 hermes: source handoff forge evidence through the scm broker
The acceptance harness pinned a forge client that has never existed in any
commit or pod (/opt/coordinator/gitea_api.py, digest f0943db4..., GIT/POST
grammar, an askpass helper). Every Gitea-backed check was therefore
unrunnable as merged. Point the harness at the credential-isolated SCM
broker client that actually ships in the agent pod.

- policy: GITEA_CLIENT=/opt/scm/gitea_api.py; trust /opt/scm/ instead of
  the phantom /opt/coordinator/; admit the client's real grammar
  (`read <api-path>`, exactly one path) with the same atlas/titan-iac pin
  and dot-segment rejection; bare HTTP methods are refused in every mode.
  The armed POST/PATCH/DELETE windows remain but are documented as
  deferred: the deployed client cannot execute them.
- exec: pin the client digest to the sha256 of
  services/hermes/scm-common/scripts/gitea_api.py — the exact file the
  hermes-scm-boundary-v2 ConfigMap mounts at /opt/scm/gitea_api.py — so
  the pin is derivable from merged source and equal to the deployed
  client. gitea_api.py gains a narrow /api/v1/user identity read in
  _authorize_read (see below), so the pin is the NEW source hash
  76efd16dedbeb74425b12fbbdbfaa391854771292077e0463bf22706855ae6dc.
  Drop the dangling GIT_ASKPASS (no helper exists; broker git needs
  none) and swap /opt/coordinator for /opt/scm in SAFE_PATH.
- checks: all forge/baseline/lineage probes use (client, "read", path).
  The SELF-vantage identity checks now truthfully assert the *broker's*
  forge identity (the only one the platform can exercise) is not an
  administrator and holds push-scoped, non-administrative repository
  authority; the administrative-route check asserts the broker read
  allowlist's live refusal of branch_protections. The remote-main step
  keeps `origin` (the broker remote exists only in pool workspaces and
  the broker origin is cluster-local); its rationale now tells the
  operator to ensure origin fetchability.
- gitea_api.py/_authorize_read: allow exactly `/api/v1/user` (no query,
  no sibling routes) as operation "identity" so the harness can prove
  the broker identity is not an administrator. The broker imports the
  same module, so one reviewed edit covers both sides of the boundary.
- rules: DENIAL_MARKERS now match the client's real refusal lines
  ("SCM broker request failed with HTTP 400/403" and the no-credential
  rejection) and drop "gitea api returned http 403", which the client
  never emits; a broker 404 is deliberately not denial evidence.
- ephemeral: index/verification reads use the real grammar; manual
  cleanup guidance now says close/delete require operator forge
  credentials (the client exposes no mutation besides create-draft);
  armed mode is documented as deferred until the probes are rebuilt on
  the broker's bounded mutation surface.
- docs: broker vantage/evidence section, operator prerequisites (broker
  healthy, no /vault/secrets/gitea-token anywhere on the harness path,
  current ConfigMap mount, operator-side client + origin fetchability),
  armed-mode deferral.
- tests: read-grammar accepted / GET refused in every mode, /opt/scm
  attestation pin proven equal to the merged source digest, real
  denial-marker matching, /api/v1/user identity route bounds; the
  repository-pin mutant probe speaks the new grammar. Full handoff +
  gitea + broker families pass (952 tests), mutation gate 13/13, per-file
  line+branch coverage >=95%, all touched sources within the 500-line cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 18:21:38 -03:00
jenkins
e85fc7e633 hermes: deploy friendly automatic model labeling to consumer chat
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 18:21:37 -03:00
jenkins
3ee7368003 hermes: deploy decluttered consumer chat router
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:59:44 -03:00
jenkins
12f4fcc35f hermes: ship the new lane modules to the coordinator mounts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:57:10 -03:00
jenkins
79baa7ecb9 hermes: return two chat slots to fit the agent runtime
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:33:24 -03:00
jenkins
0a4ff7f1f6 hermes: read node access keys through CSI symlinks
The Vault CSI secrets-store projects /vault/secrets/node-ssh-public-key
as a symlink chain (file -> ..data/file -> ..<timestamp>/file), which
the O_NOFOLLOW hardened reader refuses with ELOOP, crash-looping the
hermes-node-ssh-access key reconciler on most nodes.

Resolve the projection safely instead of relaxing the reader: realpath
the secret strictly (loops and dangling links become a clean refusal),
refuse any resolution that escapes the read-only mount, then apply the
unchanged O_NOFOLLOW regular-file read to the resolved target. Host
account databases and authorized_keys keep the strict no-symlink read.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:28:25 -03:00
jenkins
ea97d37524 hermes: derive execution pool key with a supported template function
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:19:28 -03:00
jenkins
de221a0ae2 hermes: fit agent pod on constrained workers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 16:05:50 -03:00
jenkins
9749e65042 Merge remote-tracking branch 'origin/feature/hermes-chat-smoothness' into feature/hermes-inpod-supervisor 2026-08-18 07:03:41 -03:00
jenkins
f83c59a1c9 hermes: declutter chat to a chat-only surface
Extend the chat router HTML-injection bridge so tenants see a clean
chat + conversation-history experience instead of the full agent
cockpit. hideChatAdministration now hides the rail buttons and panels
for logs, insights, memory, skills, workspaces, todos, tasks and
profiles (alongside the existing Kanban), and bridgeCSS hides the same
[data-panel] targets pre-JS to avoid a flash. Also hide the redundant
legacy settings model dropdown (#settingsModel) via bridgeCSS while
keeping the composer routing chip. Chat, history/sessions, new-chat,
composer and the model chip are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-18 06:55:43 -03:00
jenkins
d1e8cda5ed hermes: make supervisor terminal emissions idempotent across ticks
Fix reviewer BLOCK: the stateless poll loop re-planned every terminal card each
tick, so SHIP and the two cycle-limit escalations - whose source cards stay in
the done state - re-fired their comment/block on every pass (~2880/day/chain).

- Add a persistent emission Ledger (/opt/data/supervisor/emitted.json): SHIP and
  escalate perform their action only when the (action, target) key is absent,
  then record it, so each fires at most once and survives a pod restart. Spawns
  remain self-limiting via the existing dedup scan.
- Cycle-limit escalations now target the SOURCE review/repair card (not the impl
  root), so the card also leaves the done state and is skipped next tick even if
  block_task round-trips imperfectly.
- Harden the _spawn TypeError fallback: re-run the dedup scan before retrying so
  a post-insert TypeError can never double-insert.
- Add across-ticks idempotency tests (10x supervise_once -> exactly one
  comment/flag/block) plus ledger persistence/corruption and spawn-guard tests.

Both modules stay 100% line+branch, <500 LOC. Stacks on the merge train
(base 5f27e50c).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-18 06:51:36 -03:00
jenkins
2b2c8bda14 Merge remote-tracking branch 'origin/wt/t_1e95ea6d' into feature/hermes-chat-smoothness 2026-08-18 06:46:48 -03:00
jenkins
a9ee1070be Merge remote-tracking branch 'origin/feature/hermes-full-handoff-acceptance' into wt/t_1e95ea6d
# Conflicts:
#	services/hermes/agent-deployment.yaml
#	testing/tests/test_hermes_auto_router.py
2026-08-18 06:41:13 -03:00
jenkins
6a7009b31c hermes: add in-pod autonomous kanban supervisor
Drive the cross-card implement->review->repair->re-review chain from inside
the pod so unattended runs no longer stall once the first implementation card
completes. Today that chain exists only as an external codex-shepherd session;
this adds a bounded in-pod poll loop that reads board state via
hermes_cli.kanban_db and creates Kanban follow-up cards (subscription lanes
only) with no provider/metered path of its own.

- kanban_supervisor.py (I/O shell) + supervisor_policy.py (pure state machine):
  impl-done+PR -> review; review SHIP -> mark impl ready-for-human (never
  merges/approves/clears WIP); review BLOCK -> bounded repair; repair new
  commit -> re-review. Fail-closed on unparseable/ambiguous state; per-
  (parent, head_commit) dedup safe beside the external shepherd; bounded review
  <->repair cycle count and max concurrent chains.
- Deployed as a hardened non-root sidecar (drop ALL caps, read-only rootfs, no
  runtime-access/credential mount) alongside model-steward; scripts packaged in
  the coordinator configMapGenerator.
- Gated by new kanban.auto_supervise config key (default false, re-read each
  tick like auto_decompose) so it is inert until the external shepherd retires.
- Fix latent goal_max_turns NULL fallback in cli_lane_execution (1 -> documented
  default 20).
- 62 new behavioral tests at 100% line+branch on both modules.

Stacks on the merge train (base 5f27e50c).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-18 06:28:49 -03:00
jenkins
8036101f0b Merge remote-tracking branch 'origin/feature/hermes-review-goal-semantics' into feature/hermes-distributed-worker-pool
# Conflicts:
#	scripts/tests/test_dashboards_render_atlas_drilldowns.py
#	scripts/tests/test_dashboards_render_jobs.py
#	services/hermes/scm-common/scripts/scm_broker.py
#	services/hermes/scripts/cli_lane_dispatch.py
#	services/hermes/scripts/cli_lane_execution.py
#	testing/quality_contract.json
#	testing/tests/test_hermes_agent_access.py
#	testing/tests/test_hermes_agent_security.py
#	testing/tests/test_hermes_chat_config.py
#	testing/tests/test_hermes_chat_images.py
#	testing/tests/test_hermes_chat_provider_auth.py
#	testing/tests/test_hermes_chat_quality.py
#	testing/tests/test_hermes_chat_support.py
#	testing/tests/test_hermes_chat_voice.py
#	testing/tests/test_hermes_cli_finalization_edges.py
#	testing/tests/test_hermes_cli_foundation_coverage.py
#	testing/tests/test_hermes_cli_lanes_configuration.py
#	testing/tests/test_hermes_cli_recovery_edges.py
#	testing/tests/test_hermes_cli_retention_edges.py
#	testing/tests/test_hermes_coordinator.py
#	testing/tests/test_hermes_coordinator_boards.py
#	testing/tests/test_hermes_coordinator_support.py
2026-08-18 01:43:39 -03:00
jenkins
7db9f61659 Merge remote-tracking branch 'origin/feature/hermes-cli-auto-quota-routing' into feature/hermes-review-goal-semantics
# Conflicts:
#	services/hermes/scripts/cli_lane_runner.py
2026-08-18 01:18:06 -03:00