Post-merge fixes after rebasing the distributed worker pool onto the
review-goal-semantics train tip:
- Pool SCM submission tests target the train's relocated receive-pack
scanner: FEATURE_REF_RE now lives in receive_pack_scan, bodies are
built via the shared _receive_command helper (valid pack), and the
update-rejection assertion matches the train's message.
- Take the train's canonical test_hermes_scm_broker,
test_hermes_cli_dispatch_runtime and test_hermes_cli_execution_edges,
which exercise the train's broker/dispatch/execution behavior.
- Runtime staging tests patch os.fchown alongside os.chown so the
UID-10000 _write_secret path passes under a non-root gate runner
(production ownership behavior unchanged).
- Split the jenkins build-evidence contracts out of
test_hermes_runtime_access into test_hermes_runtime_evidence to keep
both files under the 500-LOC hygiene ceiling after the merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CAPACITY_PATTERN (the gate that sets result.capacity_failure) lacked the
bare unauthorized/forbidden/401/403 signals that classify_capacity_failure
already recognizes, so an auth blip surfacing only as "403 Forbidden"
was blocked as capability instead of transient: no auto failover, no
health cooldown. Add 401|403|unauthorized|forbidden to the gate so it
matches the classifier; reason classification still distinguishes auth
from quota/rate-limit/transport.
Tests: an auto card failing with only "403 Forbidden" now fails over to
the other provider, records an auth cooldown (authenticated:false), and
classifies the fallback reason as auth in metrics; a manual card with the
same failure still fails closed as transient. Router-outage tests moved to
test_hermes_cli_router_outage.py to keep both files <500 LOC.
Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The watchdog can only close a live socket, but getaddrinfo runs before
any socket exists and ignores socket timeouts, so a slow resolver
outlived the stream deadline. Resolve under the same absolute deadline
in a joinable worker before the real connect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A quoted, space-bearing #include path failed the include regex and fell
through to the comment branch, leaving a second authority file
unenumerated. Detect any include directive before the comment rule and
fail closed on every form except the exact bare includedir into the
audited /etc/sudoers.d.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A regex literal embedding a quote desynced the string tokenizer and
swallowed an unconditional Result.YES, so the grant read as unscoped and
slipped past the literal-identity check. Fail closed on any slash outside
a string or comment; neither regex nor division belongs in an
identity-scoped grant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deterministic coverage for the quota-aware lane: threshold boundaries
(14.9/15/15.1), both-below preference, fetch-failure fail-open, cooldown
elapsed-vs-not hysteresis, quota-reset recovery (never for auth),
explicit fail-closed in both directions, bounded double-failure block,
failure-reason classification, metrics emission, and worker env key
stripping. Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on PR #15 (fix/hermes-result-decomposition-reliability); stacked
on the decomposed cli_lane modules.
- cli_lane_quota: soft-exclude a provider from NEW cli-auto work below
the remaining-quota threshold (both-below prefers more remaining;
fetch failure fails open with a metric).
- cli_lane_health: lane now writes provider health (G7) with classified
failure reasons splitting the capacity conflation (quota/auth/
rate-limit/transport) and cooldown hysteresis; re-admission only on
full cooldown expiry, passed quota reset, or fresh success (G4).
- cli_lane_routing: capacity-limited health now excludes a provider
(G3); cooldown/reset-aware re-admission.
- cli_lane_failover: explicit cli-codex-*/cli-claude-* assignees fail
closed as transient instead of switching providers (G5); fallback
depth stays bounded at two hosted providers (G1) with effort
preserved; Switchyard outages block transient, not capability (G9).
- cli_lane_metrics: route-decision/fallback counters, quota and
soft-exclusion gauges, pod-local scrape server (G6).
- cli_lane_provider: worker env drops ANTHROPIC_API_KEY, CLAUDE_API_KEY,
OPENAI_API_KEY, API_SERVER_KEY so no metered path exists (G10).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add kanban routing keys (provider_quota_min_remaining_percent: 15,
provider_capacity_cooldown_seconds: 300, provider_auth_cooldown_seconds:
3600), a lane-metrics port/Service on 9011 with service-annotation
scraping, monitoring ingress for the new port, and the lane's quota
metrics URL env. Based on PR #15 (fix/hermes-result-decomposition-
reliability); stacked because this work rides on the decomposed lane.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand sudoers User_Alias chains so aliases, wildcards, netgroups, and
undefined names cannot smuggle authority to the Hermes account, and
require polkit grants to scope through exact literal identity
comparisons: computed strings, bracket lookups, subject aliasing,
operator-built values, and unconditional or wildcard grants fail closed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inflate every receive-pack object under strict pack, size, and checksum
bounds, resolve deltas against in-pack bases only, and scan the real
decompressed payloads for runtime-token forms, private keys, SSH key
material, and known provider token formats. Thin packs are rejected so
no pushed content escapes the scan, and upstream Git exchanges now run
under one absolute stream deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run every Gitea API and broker control exchange inside a killable helper
process whose connect, send, and read share one absolute wall-clock
deadline, and add a watchdog that force-closes streaming connections at
expiry. Redirects are rejected before authentication headers can move.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the reimplemented gobwas/glob matcher with exact-literal rule
matching that fails closed on any special or malformed pattern, accepts
legacy zero priorities, and rejects duplicate primary-branch rules.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The coverage check read only line-rate, so a file could pass with weak
branch coverage. Load both Cobertura rates and fail any tracked file
below the 95% floor on either metric, failing closed when branch
evidence is absent. Prove enforcement end to end with a synthetic
fully-lined but branch-weak file failing run_profile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exercise the mailu sync retry, attribute, and skip branches, the
listener non-object JSON path, and the hygiene conftest skip; drop the
unreachable inverted-range clamp in the semgrep report (the line helper
already floors the end line) and pin that behavior with a test. Exclude
the mailu __main__ guards from measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cover the uncovered branches in the image release script (redirect
handler, policy-read credential and size bounds, count and shape
rejections, EOF manifest scan) and in the Harbor immutability helper
(scope-failure verification, absent robot duration, stale-read retries,
created-rule ID binding, bounded startup retries). Exclude the
__main__ guard like the sibling release script.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repairs the blockers from the independent review of the previous head. The
role-aware verdict contract was correct but reachable only through one call
site and only on cards written with real newlines, so most real review cards
never used it.
* The role-blind call is no longer a weaker classifier that can reject before
the role-aware judge runs. Without a card there is no defensible
role-dependent judgement, so `unfinished_result_reason()` applies only the
card-independent checks. That closes the short-circuit at every call site,
including the one PR15 moves to `cli_lane_execution`, and it makes a
journalled terminal record accepted under one version of these semantics
re-validate under any other instead of being quarantined into a re-dispatch
of an already-accepted task.
* The lane resolves the role from the card and passes it, and records it in
the Kanban metadata. The verdict contract binds only where the lane can buy
another turn: in single-shot mode a rejection discards the worker's real
result, so review cards keep the relaxation without gaining any rejection
single-shot mode did not already have.
* Card scope expands the literal \n escapes the board stores in one-line
bodies, so explicit `Hermes-Task-Role` / `Hermes-Expected-Output` directives
are honoured on the 6 of 78 live cards that carry no real newline, and the
read-only, verdict and mutation heuristics stop being cut apart by them.
* Card scope now ends at the first non-card H2 and at the runner's controller
evidence, which is emitted under its own heading. Goal-controller rejection
history can no longer sit inside the card, and an upstream heading rename
fails closed instead of admitting history into role resolution.
* The inference recognises the SHIP/BLOCK-shaped deliverables real cards
actually use: 21 of 78 live cards resolve to review, up from 10, with no
implementation card misclassified. Cards asking for a findings list rather
than a verdict deliberately stay on the model judge, since the verdict is
the review contract's only gate.
* A declared review role no longer outranks mutation evidence: a report that
changed files falls back to the implementation regime.
* Judge reasons go through the agent runtime's canonical redactor, extended
for the two shapes it deliberately passes through and this lane handles -
`scheme://user:secret@host` and a credential named in prose - while a 40-hex
commit SHA survives as evidence.
Regressions cover the recovered t_dbdcd739 incident, a verbatim snapshot of
every live card on three boards with a hand-labelled expected role, the
upgrade and single-shot properties against the previous gate, the upstream
context-heading contract, and the end-to-end `execute_claim` shape that used
to burn every goal turn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document what an operator needs to know about the repairs: why the ordinal
workspace claim is ReadWriteMany and why colocation is only a preference, that
`lease_failed` is retryable rather than terminal and what a persistent deferred
park means, why a retry publishes an attempt-scoped ref and what to expect from
the extra drafts, and why the best-effort pool is deliberately absent from the
hermes Kustomization health checks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Independent review t_5975c06a blocked this branch on a P1: a Kanban write that
failed while a lease expired left a `lease_failed` row that was invisible to
every pass, immortal to garbage collection, and fatal to the coordinator. It
poisoned `reconcile()` forever with a conflicting-duplicate primary key,
produced a spurious capability `block_task` from `dispatch()`, and -- because
startup maintenance ran unguarded before the port bound, against a store on a
PVC -- crash-looped the coordinator with no automatic recovery.
`lease_failed` is now a retryable state that every maintenance pass drains, and
a row only reaches a terminal state on authoritative evidence about its exact
Kanban run, so nothing is collected before its outcome is known and nothing is
silently dropped. Each row, task, and board is processed in isolation, and a
coordinator-side fault is never converted into a Kanban mutation. Startup runs
through the same guarded cycle as the steady-state loop.
The wire protocol and the durable store are now separate modules, and the
maintenance passes moved out of the coordinator, so each file stays under the
managed line ceiling with room for the recovery logic.
Also closes three consequential handoff risks the same review raised:
* mediator-N pinned itself hard to worker-N while sharing a ReadWriteOnce
claim, so a drain or preemption that moved only the lower-priority worker
deadlocked the ordinal on Multi-Attach until an operator deleted a Pod. The
shared workspace is now ReadWriteMany (as the hermes-chat tenant workspaces
already are on the same class), colocation is a preference, and the mediator
shares the worker's preemption priority, so each Pod reschedules on its own.
* the broker permits only branch creation, so a retry that added commits could
never submit and the run's work was discarded with the failure. Submission
now targets a fresh attempt- or content-scoped ref in the same reviewed
namespace -- never an update -- and is idempotent under replay. A refused
submission downgrades the result and says why instead of unwinding the run.
* the provider CLIs were reinstalled into an emptyDir on every Pod start inside
the 10m Flux health window for the whole hermes app. They now install once
per pinned version onto a durable volume, re-verified against the real
binaries and time-bounded, and the best-effort pool no longer gates the
health of the app its dependents wait on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The distributed worker pool stacks on exactly two open pull requests, in this
order: PR 14 supplies the broker-only SCM boundary the mediators route through
and the hermes-scm-boundary-v2 ConfigMap they mount, and PR 15 supplies the
cli_lane_* decomposition -- including canonical_run_id and the eligibility
predicate on claim_ready -- that the coordinator depends on. Neither can be
dropped without breaking a fixed P0 boundary, so both are carried here as
prerequisites and this branch must not merge before them.
PR 16 (agent image release lane) and PR 19 (full-handoff acceptance harness)
are NOT prerequisites and are deliberately absent, so reviewing this branch no
longer means approving them.
PR 14 and PR 15 conflict with each other in nine paths. Each is resolved to the
resolution already reviewed on this branch at 4d4cf1bd.
Refresh #12 and #18 to their live Gitea heads, record #19's replaced head
exactly, and replace the pairwise conflict set with a freshly computed one
over all 28 combinations. #18+#19 is now clean because #19 no longer
duplicates the repo-wide hygiene legacy-exception mechanism that #18 also
carries; #12+#19 is new because #12 advanced. Name the exact overlapping
files per conflicting pair and how to resolve each additively.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evaluate_names_absent returned PASS when its step exited 0 with no output,
so five mandatory checks - the ones asserting that provider API keys, forge
credentials, a cluster-admin binding, and shared coordinator state are
absent - could report a pass on no evidence and turn a NO_GO into a GO.
Both name rules now resolve their step through one guard in _line_step, so
zero observations are NOT_RUN. Regressions pin all five real catalog specs
plus both reachable silence paths: a POSIX pipeline whose status comes from
its last stage, and a drifted kubectl -o jsonpath. The pool claim projection
emits one <volume>=<claim> line per template volume so a volume without a
PVC still counts as an observation rather than reading as drift.
Also closes the review's reachable hardening and evidence defects:
- pin Gitea paths to atlas/titan-iac on an exact segment boundary and
reject relative segments, including percent-encoded ones
- forbid impersonation structurally in every mode and vantage; the inner
command of kubectl exec is re-checked rather than exempted, and
validate_catalog no longer guards only the operator vantage
- drop flux and helm from the binary allowlist; they had no pinned release
digest, so no allowlisted binary can now be admitted that the executor
would refuse to attest
- remove the inert --concurrency and --expect-telegram-sessions flags and
the dead concurrency bound; Telegram continuity stays mandatory
- read the ephemeral pull index page by page, treat the create response as
an authoritative source for the pull number, close every number either
source names, and surface residue_ref plus exact manual_cleanup commands
when creation is uncertain
- keep executable_path and executable_sha256 on unrecorded bulk-evidence
steps so withholding bytes never withholds binary attestation
- revert the repo-wide hygiene legacy-exception mechanism; the contract
change here is purely additive and the four pre-existing over-cap files
are left to the canonical contract change in PR #14/#15
- correct the runbook ruff format scope so the documented command passes
Split hermes_handoff_arming.py out of hermes_handoff_ephemeral.py to keep
both modules under the 500-line cap. All 16 handoff modules hold at least
95% line and branch coverage; the mutation gate is 13/13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The local goal judge scored every worker report against "did the reviewed
implementation reach a shippable state". A read-only reviewer that returned a
completed BLOCK verdict with findings was therefore resumed turn after turn with
an instruction to repair code it was forbidden to touch (observed live on
t_dbdcd739), burning subscription capacity and risking an unbounded loop.
Completion is now judged against the action the card assigned:
* Cards declare their role explicitly with Hermes-Task-Role / Hermes-Expected-
Output metadata. Pre-contract cards fall back to a narrow inference that needs
a read-only scope, a requested verdict, no requested mutation deliverable, and
a report that changed no files.
* Role resolution reads only the card itself. Prior attempts, parent results,
cross-task history and comments appended to the worker context can no longer
reassign the role.
* Review, audit and diagnostic cards finalize deterministically on a truthful
SHIP or BLOCK verdict with evidence, and fail closed on a missing,
unrecognized or self-contradictory verdict, on a BLOCK without findings, and
on a verdict without evidence. Every rejection reason carries the read-only
guard, so a resumed review is never told to edit the reviewed implementation.
* Implementation cards keep the fail-closed model judge unchanged, including the
unfinished-work heuristic and judge-unavailable rejection.
* All judge reasons are bounded, single-line and secret-redacted before they
reach Kanban metadata, comments and continuation prompts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
