Returning to chat.hermes.bstein.dev after a Keycloak logout/login showed
"This session is unavailable to this account. Start a new chat." even
though the session was intact and owned by the same subject.
The banner comes from the continuity fallback the router injects into
every chat page. It polled `/api/sessions/<id>` and
`/api/sessions/<id>/messages` — routes that belong to the Hermes agent
dashboard (added by scripts/patch_web_session_activity.py, applied only
in agent-deployment.yaml). The router proxies browser traffic to the
tenant Hermes WebUI instead, whose only session read is
`GET /api/session?session_id=<id>`; the dashboard paths are unrouted
there, so server.py answered its generic 404 for every poll and the
fallback reported a false ownership failure.
The script runs only on a full document load of `/session/<id>`, which is
exactly what the OIDC round-trip produces when oauth2-proxy returns the
browser to `rd=/session/<id>` — hence the "only after relogin" symptom.
Poll the WebUI contract instead, and let its own answers decide what the
banner claims: 409 `session_profile_mismatch` is the single response that
means the session is outside this account's active scope, 404 now means
the conversation is no longer stored, and 401/403 still re-enter OIDC.
The steady-state poll drops to one request and backs off to 3s/15s now
that it reaches a real endpoint on the tenant Raspberry Pi.
`boundSessionSnapshot` follows the same move: it caps the WebUI envelope
`{"session": {..., "messages": [...]}}`, relaying every other session key
verbatim rather than re-serializing a fixed struct that would silently
drop metadata the banner depends on.
Isolation is unchanged and now covered: the router still resolves the
slot from the salted Keycloak subject, overwrites any client-supplied
X-Hermes-Tenant-Identity, and forwards only the two tenant cookies.
Tests: relogin keeps a stable slot and resolves the durable session; a
second subject replaying the owner's session id, WebUI cookie and a
forged tenant header gets 404 from its own backend and never reaches the
owner's; the legacy dashboard paths are pinned as permanent 404s against
a stub of the deployed WebUI dispatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pins hermes-agent to sha256:4a385fbd, built by hermes-agent-image #11 from
this exact main revision through the daemonless Kaniko lane. Enables
kanban.auto_supervise and returns chat tenancy to the documented four slots.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The acceptance harness pinned a forge client that has never existed in any
commit or pod (/opt/coordinator/gitea_api.py, digest f0943db4..., GIT/POST
grammar, an askpass helper). Every Gitea-backed check was therefore
unrunnable as merged. Point the harness at the credential-isolated SCM
broker client that actually ships in the agent pod.
- policy: GITEA_CLIENT=/opt/scm/gitea_api.py; trust /opt/scm/ instead of
the phantom /opt/coordinator/; admit the client's real grammar
(`read <api-path>`, exactly one path) with the same atlas/titan-iac pin
and dot-segment rejection; bare HTTP methods are refused in every mode.
The armed POST/PATCH/DELETE windows remain but are documented as
deferred: the deployed client cannot execute them.
- exec: pin the client digest to the sha256 of
services/hermes/scm-common/scripts/gitea_api.py — the exact file the
hermes-scm-boundary-v2 ConfigMap mounts at /opt/scm/gitea_api.py — so
the pin is derivable from merged source and equal to the deployed
client. gitea_api.py gains a narrow /api/v1/user identity read in
_authorize_read (see below), so the pin is the NEW source hash
76efd16dedbeb74425b12fbbdbfaa391854771292077e0463bf22706855ae6dc.
Drop the dangling GIT_ASKPASS (no helper exists; broker git needs
none) and swap /opt/coordinator for /opt/scm in SAFE_PATH.
- checks: all forge/baseline/lineage probes use (client, "read", path).
The SELF-vantage identity checks now truthfully assert the *broker's*
forge identity (the only one the platform can exercise) is not an
administrator and holds push-scoped, non-administrative repository
authority; the administrative-route check asserts the broker read
allowlist's live refusal of branch_protections. The remote-main step
keeps `origin` (the broker remote exists only in pool workspaces and
the broker origin is cluster-local); its rationale now tells the
operator to ensure origin fetchability.
- gitea_api.py/_authorize_read: allow exactly `/api/v1/user` (no query,
no sibling routes) as operation "identity" so the harness can prove
the broker identity is not an administrator. The broker imports the
same module, so one reviewed edit covers both sides of the boundary.
- rules: DENIAL_MARKERS now match the client's real refusal lines
("SCM broker request failed with HTTP 400/403" and the no-credential
rejection) and drop "gitea api returned http 403", which the client
never emits; a broker 404 is deliberately not denial evidence.
- ephemeral: index/verification reads use the real grammar; manual
cleanup guidance now says close/delete require operator forge
credentials (the client exposes no mutation besides create-draft);
armed mode is documented as deferred until the probes are rebuilt on
the broker's bounded mutation surface.
- docs: broker vantage/evidence section, operator prerequisites (broker
healthy, no /vault/secrets/gitea-token anywhere on the harness path,
current ConfigMap mount, operator-side client + origin fetchability),
armed-mode deferral.
- tests: read-grammar accepted / GET refused in every mode, /opt/scm
attestation pin proven equal to the merged source digest, real
denial-marker matching, /api/v1/user identity route bounds; the
repository-pin mutant probe speaks the new grammar. Full handoff +
gitea + broker families pass (952 tests), mutation gate 13/13, per-file
line+branch coverage >=95%, all touched sources within the 500-line cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Vault CSI secrets-store projects /vault/secrets/node-ssh-public-key
as a symlink chain (file -> ..data/file -> ..<timestamp>/file), which
the O_NOFOLLOW hardened reader refuses with ELOOP, crash-looping the
hermes-node-ssh-access key reconciler on most nodes.
Resolve the projection safely instead of relaxing the reader: realpath
the secret strictly (loops and dangling links become a clean refusal),
refuse any resolution that escapes the read-only mount, then apply the
unchanged O_NOFOLLOW regular-file read to the resolved target. Host
account databases and authorized_keys keep the strict no-symlink read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the chat router HTML-injection bridge so tenants see a clean
chat + conversation-history experience instead of the full agent
cockpit. hideChatAdministration now hides the rail buttons and panels
for logs, insights, memory, skills, workspaces, todos, tasks and
profiles (alongside the existing Kanban), and bridgeCSS hides the same
[data-panel] targets pre-JS to avoid a flash. Also hide the redundant
legacy settings model dropdown (#settingsModel) via bridgeCSS while
keeping the composer routing chip. Chat, history/sessions, new-chat,
composer and the model chip are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix reviewer BLOCK: the stateless poll loop re-planned every terminal card each
tick, so SHIP and the two cycle-limit escalations - whose source cards stay in
the done state - re-fired their comment/block on every pass (~2880/day/chain).
- Add a persistent emission Ledger (/opt/data/supervisor/emitted.json): SHIP and
escalate perform their action only when the (action, target) key is absent,
then record it, so each fires at most once and survives a pod restart. Spawns
remain self-limiting via the existing dedup scan.
- Cycle-limit escalations now target the SOURCE review/repair card (not the impl
root), so the card also leaves the done state and is skipped next tick even if
block_task round-trips imperfectly.
- Harden the _spawn TypeError fallback: re-run the dedup scan before retrying so
a post-insert TypeError can never double-insert.
- Add across-ticks idempotency tests (10x supervise_once -> exactly one
comment/flag/block) plus ledger persistence/corruption and spawn-guard tests.
Both modules stay 100% line+branch, <500 LOC. Stacks on the merge train
(base 5f27e50c).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drive the cross-card implement->review->repair->re-review chain from inside
the pod so unattended runs no longer stall once the first implementation card
completes. Today that chain exists only as an external codex-shepherd session;
this adds a bounded in-pod poll loop that reads board state via
hermes_cli.kanban_db and creates Kanban follow-up cards (subscription lanes
only) with no provider/metered path of its own.
- kanban_supervisor.py (I/O shell) + supervisor_policy.py (pure state machine):
impl-done+PR -> review; review SHIP -> mark impl ready-for-human (never
merges/approves/clears WIP); review BLOCK -> bounded repair; repair new
commit -> re-review. Fail-closed on unparseable/ambiguous state; per-
(parent, head_commit) dedup safe beside the external shepherd; bounded review
<->repair cycle count and max concurrent chains.
- Deployed as a hardened non-root sidecar (drop ALL caps, read-only rootfs, no
runtime-access/credential mount) alongside model-steward; scripts packaged in
the coordinator configMapGenerator.
- Gated by new kanban.auto_supervise config key (default false, re-read each
tick like auto_decompose) so it is inert until the external shepherd retires.
- Fix latent goal_max_turns NULL fallback in cli_lane_execution (1 -> documented
default 20).
- 62 new behavioral tests at 100% line+branch on both modules.
Stacks on the merge train (base 5f27e50c).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CAPACITY_PATTERN (the gate that sets result.capacity_failure) lacked the
bare unauthorized/forbidden/401/403 signals that classify_capacity_failure
already recognizes, so an auth blip surfacing only as "403 Forbidden"
was blocked as capability instead of transient: no auto failover, no
health cooldown. Add 401|403|unauthorized|forbidden to the gate so it
matches the classifier; reason classification still distinguishes auth
from quota/rate-limit/transport.
Tests: an auto card failing with only "403 Forbidden" now fails over to
the other provider, records an auth cooldown (authenticated:false), and
classifies the fallback reason as auth in metrics; a manual card with the
same failure still fails closed as transient. Router-outage tests moved to
test_hermes_cli_router_outage.py to keep both files <500 LOC.
Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The watchdog can only close a live socket, but getaddrinfo runs before
any socket exists and ignores socket timeouts, so a slow resolver
outlived the stream deadline. Resolve under the same absolute deadline
in a joinable worker before the real connect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A quoted, space-bearing #include path failed the include regex and fell
through to the comment branch, leaving a second authority file
unenumerated. Detect any include directive before the comment rule and
fail closed on every form except the exact bare includedir into the
audited /etc/sudoers.d.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A regex literal embedding a quote desynced the string tokenizer and
swallowed an unconditional Result.YES, so the grant read as unscoped and
slipped past the literal-identity check. Fail closed on any slash outside
a string or comment; neither regex nor division belongs in an
identity-scoped grant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Based on PR #15 (fix/hermes-result-decomposition-reliability); stacked
on the decomposed cli_lane modules.
- cli_lane_quota: soft-exclude a provider from NEW cli-auto work below
the remaining-quota threshold (both-below prefers more remaining;
fetch failure fails open with a metric).
- cli_lane_health: lane now writes provider health (G7) with classified
failure reasons splitting the capacity conflation (quota/auth/
rate-limit/transport) and cooldown hysteresis; re-admission only on
full cooldown expiry, passed quota reset, or fresh success (G4).
- cli_lane_routing: capacity-limited health now excludes a provider
(G3); cooldown/reset-aware re-admission.
- cli_lane_failover: explicit cli-codex-*/cli-claude-* assignees fail
closed as transient instead of switching providers (G5); fallback
depth stays bounded at two hosted providers (G1) with effort
preserved; Switchyard outages block transient, not capability (G9).
- cli_lane_metrics: route-decision/fallback counters, quota and
soft-exclusion gauges, pod-local scrape server (G6).
- cli_lane_provider: worker env drops ANTHROPIC_API_KEY, CLAUDE_API_KEY,
OPENAI_API_KEY, API_SERVER_KEY so no metered path exists (G10).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add kanban routing keys (provider_quota_min_remaining_percent: 15,
provider_capacity_cooldown_seconds: 300, provider_auth_cooldown_seconds:
3600), a lane-metrics port/Service on 9011 with service-annotation
scraping, monitoring ingress for the new port, and the lane's quota
metrics URL env. Based on PR #15 (fix/hermes-result-decomposition-
reliability); stacked because this work rides on the decomposed lane.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand sudoers User_Alias chains so aliases, wildcards, netgroups, and
undefined names cannot smuggle authority to the Hermes account, and
require polkit grants to scope through exact literal identity
comparisons: computed strings, bracket lookups, subject aliasing,
operator-built values, and unconditional or wildcard grants fail closed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inflate every receive-pack object under strict pack, size, and checksum
bounds, resolve deltas against in-pack bases only, and scan the real
decompressed payloads for runtime-token forms, private keys, SSH key
material, and known provider token formats. Thin packs are rejected so
no pushed content escapes the scan, and upstream Git exchanges now run
under one absolute stream deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run every Gitea API and broker control exchange inside a killable helper
process whose connect, send, and read share one absolute wall-clock
deadline, and add a watchdog that force-closes streaming connections at
expiry. Redirects are rejected before authentication headers can move.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repairs the blockers from the independent review of the previous head. The
role-aware verdict contract was correct but reachable only through one call
site and only on cards written with real newlines, so most real review cards
never used it.
* The role-blind call is no longer a weaker classifier that can reject before
the role-aware judge runs. Without a card there is no defensible
role-dependent judgement, so `unfinished_result_reason()` applies only the
card-independent checks. That closes the short-circuit at every call site,
including the one PR15 moves to `cli_lane_execution`, and it makes a
journalled terminal record accepted under one version of these semantics
re-validate under any other instead of being quarantined into a re-dispatch
of an already-accepted task.
* The lane resolves the role from the card and passes it, and records it in
the Kanban metadata. The verdict contract binds only where the lane can buy
another turn: in single-shot mode a rejection discards the worker's real
result, so review cards keep the relaxation without gaining any rejection
single-shot mode did not already have.
* Card scope expands the literal \n escapes the board stores in one-line
bodies, so explicit `Hermes-Task-Role` / `Hermes-Expected-Output` directives
are honoured on the 6 of 78 live cards that carry no real newline, and the
read-only, verdict and mutation heuristics stop being cut apart by them.
* Card scope now ends at the first non-card H2 and at the runner's controller
evidence, which is emitted under its own heading. Goal-controller rejection
history can no longer sit inside the card, and an upstream heading rename
fails closed instead of admitting history into role resolution.
* The inference recognises the SHIP/BLOCK-shaped deliverables real cards
actually use: 21 of 78 live cards resolve to review, up from 10, with no
implementation card misclassified. Cards asking for a findings list rather
than a verdict deliberately stay on the model judge, since the verdict is
the review contract's only gate.
* A declared review role no longer outranks mutation evidence: a report that
changed files falls back to the implementation regime.
* Judge reasons go through the agent runtime's canonical redactor, extended
for the two shapes it deliberately passes through and this lane handles -
`scheme://user:secret@host` and a credential named in prose - while a 40-hex
commit SHA survives as evidence.
Regressions cover the recovered t_dbdcd739 incident, a verbatim snapshot of
every live card on three boards with a hand-labelled expected role, the
upgrade and single-shot properties against the previous gate, the upstream
context-heading contract, and the end-to-end `execute_claim` shape that used
to burn every goal turn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document what an operator needs to know about the repairs: why the ordinal
workspace claim is ReadWriteMany and why colocation is only a preference, that
`lease_failed` is retryable rather than terminal and what a persistent deferred
park means, why a retry publishes an attempt-scoped ref and what to expect from
the extra drafts, and why the best-effort pool is deliberately absent from the
hermes Kustomization health checks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Independent review t_5975c06a blocked this branch on a P1: a Kanban write that
failed while a lease expired left a `lease_failed` row that was invisible to
every pass, immortal to garbage collection, and fatal to the coordinator. It
poisoned `reconcile()` forever with a conflicting-duplicate primary key,
produced a spurious capability `block_task` from `dispatch()`, and -- because
startup maintenance ran unguarded before the port bound, against a store on a
PVC -- crash-looped the coordinator with no automatic recovery.
`lease_failed` is now a retryable state that every maintenance pass drains, and
a row only reaches a terminal state on authoritative evidence about its exact
Kanban run, so nothing is collected before its outcome is known and nothing is
silently dropped. Each row, task, and board is processed in isolation, and a
coordinator-side fault is never converted into a Kanban mutation. Startup runs
through the same guarded cycle as the steady-state loop.
The wire protocol and the durable store are now separate modules, and the
maintenance passes moved out of the coordinator, so each file stays under the
managed line ceiling with room for the recovery logic.
Also closes three consequential handoff risks the same review raised:
* mediator-N pinned itself hard to worker-N while sharing a ReadWriteOnce
claim, so a drain or preemption that moved only the lower-priority worker
deadlocked the ordinal on Multi-Attach until an operator deleted a Pod. The
shared workspace is now ReadWriteMany (as the hermes-chat tenant workspaces
already are on the same class), colocation is a preference, and the mediator
shares the worker's preemption priority, so each Pod reschedules on its own.
* the broker permits only branch creation, so a retry that added commits could
never submit and the run's work was discarded with the failure. Submission
now targets a fresh attempt- or content-scoped ref in the same reviewed
namespace -- never an update -- and is idempotent under replay. A refused
submission downgrades the result and says why instead of unwinding the run.
* the provider CLIs were reinstalled into an emptyDir on every Pod start inside
the 10m Flux health window for the whole hermes app. They now install once
per pinned version onto a durable volume, re-verified against the real
binaries and time-bounded, and the best-effort pool no longer gates the
health of the app its dependents wait on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The distributed worker pool stacks on exactly two open pull requests, in this
order: PR 14 supplies the broker-only SCM boundary the mediators route through
and the hermes-scm-boundary-v2 ConfigMap they mount, and PR 15 supplies the
cli_lane_* decomposition -- including canonical_run_id and the eligibility
predicate on claim_ready -- that the coordinator depends on. Neither can be
dropped without breaking a fixed P0 boundary, so both are carried here as
prerequisites and this branch must not merge before them.
PR 16 (agent image release lane) and PR 19 (full-handoff acceptance harness)
are NOT prerequisites and are deliberately absent, so reviewing this branch no
longer means approving them.
PR 14 and PR 15 conflict with each other in nine paths. Each is resolved to the
resolution already reviewed on this branch at 4d4cf1bd.
The local goal judge scored every worker report against "did the reviewed
implementation reach a shippable state". A read-only reviewer that returned a
completed BLOCK verdict with findings was therefore resumed turn after turn with
an instruction to repair code it was forbidden to touch (observed live on
t_dbdcd739), burning subscription capacity and risking an unbounded loop.
Completion is now judged against the action the card assigned:
* Cards declare their role explicitly with Hermes-Task-Role / Hermes-Expected-
Output metadata. Pre-contract cards fall back to a narrow inference that needs
a read-only scope, a requested verdict, no requested mutation deliverable, and
a report that changed no files.
* Role resolution reads only the card itself. Prior attempts, parent results,
cross-task history and comments appended to the worker context can no longer
reassign the role.
* Review, audit and diagnostic cards finalize deterministically on a truthful
SHIP or BLOCK verdict with evidence, and fail closed on a missing,
unrecognized or self-contradictory verdict, on a BLOCK without findings, and
on a verdict without evidence. Every rejection reason carries the read-only
guard, so a resumed review is never told to edit the reviewed implementation.
* Implementation cards keep the fail-closed model judge unchanged, including the
unfinished-work heuristic and judge-unavailable rejection.
* All judge reasons are bounded, single-line and secret-redacted before they
reach Kanban metadata, comments and continuation prompts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
