CAPACITY_PATTERN (the gate that sets result.capacity_failure) lacked the
bare unauthorized/forbidden/401/403 signals that classify_capacity_failure
already recognizes, so an auth blip surfacing only as "403 Forbidden"
was blocked as capability instead of transient: no auto failover, no
health cooldown. Add 401|403|unauthorized|forbidden to the gate so it
matches the classifier; reason classification still distinguishes auth
from quota/rate-limit/transport.
Tests: an auto card failing with only "403 Forbidden" now fails over to
the other provider, records an auth cooldown (authenticated:false), and
classifies the fallback reason as auth in metrics; a manual card with the
same failure still fails closed as transient. Router-outage tests moved to
test_hermes_cli_router_outage.py to keep both files <500 LOC.
Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The watchdog can only close a live socket, but getaddrinfo runs before
any socket exists and ignores socket timeouts, so a slow resolver
outlived the stream deadline. Resolve under the same absolute deadline
in a joinable worker before the real connect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A quoted, space-bearing #include path failed the include regex and fell
through to the comment branch, leaving a second authority file
unenumerated. Detect any include directive before the comment rule and
fail closed on every form except the exact bare includedir into the
audited /etc/sudoers.d.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A regex literal embedding a quote desynced the string tokenizer and
swallowed an unconditional Result.YES, so the grant read as unscoped and
slipped past the literal-identity check. Fail closed on any slash outside
a string or comment; neither regex nor division belongs in an
identity-scoped grant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deterministic coverage for the quota-aware lane: threshold boundaries
(14.9/15/15.1), both-below preference, fetch-failure fail-open, cooldown
elapsed-vs-not hysteresis, quota-reset recovery (never for auth),
explicit fail-closed in both directions, bounded double-failure block,
failure-reason classification, metrics emission, and worker env key
stripping. Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on PR #15 (fix/hermes-result-decomposition-reliability); stacked
on the decomposed cli_lane modules.
- cli_lane_quota: soft-exclude a provider from NEW cli-auto work below
the remaining-quota threshold (both-below prefers more remaining;
fetch failure fails open with a metric).
- cli_lane_health: lane now writes provider health (G7) with classified
failure reasons splitting the capacity conflation (quota/auth/
rate-limit/transport) and cooldown hysteresis; re-admission only on
full cooldown expiry, passed quota reset, or fresh success (G4).
- cli_lane_routing: capacity-limited health now excludes a provider
(G3); cooldown/reset-aware re-admission.
- cli_lane_failover: explicit cli-codex-*/cli-claude-* assignees fail
closed as transient instead of switching providers (G5); fallback
depth stays bounded at two hosted providers (G1) with effort
preserved; Switchyard outages block transient, not capability (G9).
- cli_lane_metrics: route-decision/fallback counters, quota and
soft-exclusion gauges, pod-local scrape server (G6).
- cli_lane_provider: worker env drops ANTHROPIC_API_KEY, CLAUDE_API_KEY,
OPENAI_API_KEY, API_SERVER_KEY so no metered path exists (G10).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add kanban routing keys (provider_quota_min_remaining_percent: 15,
provider_capacity_cooldown_seconds: 300, provider_auth_cooldown_seconds:
3600), a lane-metrics port/Service on 9011 with service-annotation
scraping, monitoring ingress for the new port, and the lane's quota
metrics URL env. Based on PR #15 (fix/hermes-result-decomposition-
reliability); stacked because this work rides on the decomposed lane.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand sudoers User_Alias chains so aliases, wildcards, netgroups, and
undefined names cannot smuggle authority to the Hermes account, and
require polkit grants to scope through exact literal identity
comparisons: computed strings, bracket lookups, subject aliasing,
operator-built values, and unconditional or wildcard grants fail closed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inflate every receive-pack object under strict pack, size, and checksum
bounds, resolve deltas against in-pack bases only, and scan the real
decompressed payloads for runtime-token forms, private keys, SSH key
material, and known provider token formats. Thin packs are rejected so
no pushed content escapes the scan, and upstream Git exchanges now run
under one absolute stream deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run every Gitea API and broker control exchange inside a killable helper
process whose connect, send, and read share one absolute wall-clock
deadline, and add a watchdog that force-closes streaming connections at
expiry. Redirects are rejected before authentication headers can move.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the reimplemented gobwas/glob matcher with exact-literal rule
matching that fails closed on any special or malformed pattern, accepts
legacy zero priorities, and rejects duplicate primary-branch rules.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The coverage check read only line-rate, so a file could pass with weak
branch coverage. Load both Cobertura rates and fail any tracked file
below the 95% floor on either metric, failing closed when branch
evidence is absent. Prove enforcement end to end with a synthetic
fully-lined but branch-weak file failing run_profile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exercise the mailu sync retry, attribute, and skip branches, the
listener non-object JSON path, and the hygiene conftest skip; drop the
unreachable inverted-range clamp in the semgrep report (the line helper
already floors the end line) and pin that behavior with a test. Exclude
the mailu __main__ guards from measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cover the uncovered branches in the image release script (redirect
handler, policy-read credential and size bounds, count and shape
rejections, EOF manifest scan) and in the Harbor immutability helper
(scope-failure verification, absent robot duration, stale-read retries,
created-rule ID binding, bounded startup retries). Exclude the
__main__ guard like the sibling release script.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repairs the blockers from the independent review of the previous head. The
role-aware verdict contract was correct but reachable only through one call
site and only on cards written with real newlines, so most real review cards
never used it.
* The role-blind call is no longer a weaker classifier that can reject before
the role-aware judge runs. Without a card there is no defensible
role-dependent judgement, so `unfinished_result_reason()` applies only the
card-independent checks. That closes the short-circuit at every call site,
including the one PR15 moves to `cli_lane_execution`, and it makes a
journalled terminal record accepted under one version of these semantics
re-validate under any other instead of being quarantined into a re-dispatch
of an already-accepted task.
* The lane resolves the role from the card and passes it, and records it in
the Kanban metadata. The verdict contract binds only where the lane can buy
another turn: in single-shot mode a rejection discards the worker's real
result, so review cards keep the relaxation without gaining any rejection
single-shot mode did not already have.
* Card scope expands the literal \n escapes the board stores in one-line
bodies, so explicit `Hermes-Task-Role` / `Hermes-Expected-Output` directives
are honoured on the 6 of 78 live cards that carry no real newline, and the
read-only, verdict and mutation heuristics stop being cut apart by them.
* Card scope now ends at the first non-card H2 and at the runner's controller
evidence, which is emitted under its own heading. Goal-controller rejection
history can no longer sit inside the card, and an upstream heading rename
fails closed instead of admitting history into role resolution.
* The inference recognises the SHIP/BLOCK-shaped deliverables real cards
actually use: 21 of 78 live cards resolve to review, up from 10, with no
implementation card misclassified. Cards asking for a findings list rather
than a verdict deliberately stay on the model judge, since the verdict is
the review contract's only gate.
* A declared review role no longer outranks mutation evidence: a report that
changed files falls back to the implementation regime.
* Judge reasons go through the agent runtime's canonical redactor, extended
for the two shapes it deliberately passes through and this lane handles -
`scheme://user:secret@host` and a credential named in prose - while a 40-hex
commit SHA survives as evidence.
Regressions cover the recovered t_dbdcd739 incident, a verbatim snapshot of
every live card on three boards with a hand-labelled expected role, the
upgrade and single-shot properties against the previous gate, the upstream
context-heading contract, and the end-to-end `execute_claim` shape that used
to burn every goal turn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The local goal judge scored every worker report against "did the reviewed
implementation reach a shippable state". A read-only reviewer that returned a
completed BLOCK verdict with findings was therefore resumed turn after turn with
an instruction to repair code it was forbidden to touch (observed live on
t_dbdcd739), burning subscription capacity and risking an unbounded loop.
Completion is now judged against the action the card assigned:
* Cards declare their role explicitly with Hermes-Task-Role / Hermes-Expected-
Output metadata. Pre-contract cards fall back to a narrow inference that needs
a read-only scope, a requested verdict, no requested mutation deliverable, and
a report that changed no files.
* Role resolution reads only the card itself. Prior attempts, parent results,
cross-task history and comments appended to the worker context can no longer
reassign the role.
* Review, audit and diagnostic cards finalize deterministically on a truthful
SHIP or BLOCK verdict with evidence, and fail closed on a missing,
unrecognized or self-contradictory verdict, on a BLOCK without findings, and
on a verdict without evidence. Every rejection reason carries the read-only
guard, so a resumed review is never told to edit the reviewed implementation.
* Implementation cards keep the fail-closed model judge unchanged, including the
unfinished-work heuristic and judge-unavailable rejection.
* All judge reasons are bounded, single-line and secret-redacted before they
reach Kanban metadata, comments and continuation prompts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sticky-block gate added in the previous commit classifies a task from
the `created` event payload that upstream `create_task` writes. That
producer is code we do not own, so trusting it silently was the gap: if
upstream renamed the key, dropped it, or stopped deriving it from
`initial_status`, the image would still build and ship a consumer that
mis-classifies every task it reads.
Anchor the producer contract at build time, before the regression suite
runs, with three assert-only preconditions: the `initial_status="blocked"`
park resolves `task_status` to `"blocked"`, every non-park creation
resolves it to something else, and the `created` event carries that same
variable under `"status"`. None of them rewrite the producer.
Textual anchors cannot see dataflow, so add the runtime net the reviewer
asked for. The suite now drives the real API: create + claim an ordinary
task, trip the circuit breaker once at failure_limit=1 so it parks with a
`gave_up` event (leaving its own `created` event as the most recent
create/block/unblock row), then recompute at failure_limit=2 and require
promotion to ready. That case is red under an unconditional-true created
predicate and red under producer drift that labels every created event
blocked, while the explicit block/unblock, dependency-promotion and
circuit-breaker-at-current-limit cases stay green. Non-blocked and
malformed created payloads are pinned as controls, and the gate now
rejects non-dict payloads rather than trusting `.get`.
Also make the live placement correction durable: titan-04 is cordoned
after repeated kernel undervoltage and kubelet failure and titan-19 was
probe/Longhorn unstable under worker load, so both join the hard NotIn
list; titan-05 is healthy but sits at 3592m/3600m requested CPU, so the
main hermes container gives back 50m (350m -> 300m) to schedule there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
