CAPACITY_PATTERN (the gate that sets result.capacity_failure) lacked the
bare unauthorized/forbidden/401/403 signals that classify_capacity_failure
already recognizes, so an auth blip surfacing only as "403 Forbidden"
was blocked as capability instead of transient: no auto failover, no
health cooldown. Add 401|403|unauthorized|forbidden to the gate so it
matches the classifier; reason classification still distinguishes auth
from quota/rate-limit/transport.
Tests: an auto card failing with only "403 Forbidden" now fails over to
the other provider, records an auth cooldown (authenticated:false), and
classifies the fallback reason as auth in metrics; a manual card with the
same failure still fails closed as transient. Router-outage tests moved to
test_hermes_cli_router_outage.py to keep both files <500 LOC.
Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The watchdog can only close a live socket, but getaddrinfo runs before
any socket exists and ignores socket timeouts, so a slow resolver
outlived the stream deadline. Resolve under the same absolute deadline
in a joinable worker before the real connect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A quoted, space-bearing #include path failed the include regex and fell
through to the comment branch, leaving a second authority file
unenumerated. Detect any include directive before the comment rule and
fail closed on every form except the exact bare includedir into the
audited /etc/sudoers.d.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A regex literal embedding a quote desynced the string tokenizer and
swallowed an unconditional Result.YES, so the grant read as unscoped and
slipped past the literal-identity check. Fail closed on any slash outside
a string or comment; neither regex nor division belongs in an
identity-scoped grant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deterministic coverage for the quota-aware lane: threshold boundaries
(14.9/15/15.1), both-below preference, fetch-failure fail-open, cooldown
elapsed-vs-not hysteresis, quota-reset recovery (never for auth),
explicit fail-closed in both directions, bounded double-failure block,
failure-reason classification, metrics emission, and worker env key
stripping. Based on PR #15 (fix/hermes-result-decomposition-reliability).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on PR #15 (fix/hermes-result-decomposition-reliability); stacked
on the decomposed cli_lane modules.
- cli_lane_quota: soft-exclude a provider from NEW cli-auto work below
the remaining-quota threshold (both-below prefers more remaining;
fetch failure fails open with a metric).
- cli_lane_health: lane now writes provider health (G7) with classified
failure reasons splitting the capacity conflation (quota/auth/
rate-limit/transport) and cooldown hysteresis; re-admission only on
full cooldown expiry, passed quota reset, or fresh success (G4).
- cli_lane_routing: capacity-limited health now excludes a provider
(G3); cooldown/reset-aware re-admission.
- cli_lane_failover: explicit cli-codex-*/cli-claude-* assignees fail
closed as transient instead of switching providers (G5); fallback
depth stays bounded at two hosted providers (G1) with effort
preserved; Switchyard outages block transient, not capability (G9).
- cli_lane_metrics: route-decision/fallback counters, quota and
soft-exclusion gauges, pod-local scrape server (G6).
- cli_lane_provider: worker env drops ANTHROPIC_API_KEY, CLAUDE_API_KEY,
OPENAI_API_KEY, API_SERVER_KEY so no metered path exists (G10).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand sudoers User_Alias chains so aliases, wildcards, netgroups, and
undefined names cannot smuggle authority to the Hermes account, and
require polkit grants to scope through exact literal identity
comparisons: computed strings, bracket lookups, subject aliasing,
operator-built values, and unconditional or wildcard grants fail closed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inflate every receive-pack object under strict pack, size, and checksum
bounds, resolve deltas against in-pack bases only, and scan the real
decompressed payloads for runtime-token forms, private keys, SSH key
material, and known provider token formats. Thin packs are rejected so
no pushed content escapes the scan, and upstream Git exchanges now run
under one absolute stream deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run every Gitea API and broker control exchange inside a killable helper
process whose connect, send, and read share one absolute wall-clock
deadline, and add a watchdog that force-closes streaming connections at
expiry. Redirects are rejected before authentication headers can move.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the reimplemented gobwas/glob matcher with exact-literal rule
matching that fails closed on any special or malformed pattern, accepts
legacy zero priorities, and rejects duplicate primary-branch rules.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The coverage check read only line-rate, so a file could pass with weak
branch coverage. Load both Cobertura rates and fail any tracked file
below the 95% floor on either metric, failing closed when branch
evidence is absent. Prove enforcement end to end with a synthetic
fully-lined but branch-weak file failing run_profile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exercise the mailu sync retry, attribute, and skip branches, the
listener non-object JSON path, and the hygiene conftest skip; drop the
unreachable inverted-range clamp in the semgrep report (the line helper
already floors the end line) and pin that behavior with a test. Exclude
the mailu __main__ guards from measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cover the uncovered branches in the image release script (redirect
handler, policy-read credential and size bounds, count and shape
rejections, EOF manifest scan) and in the Harbor immutability helper
(scope-failure verification, absent robot duration, stale-read retries,
created-rule ID binding, bounded startup retries). Exclude the
__main__ guard like the sibling release script.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sticky-block gate added in the previous commit classifies a task from
the `created` event payload that upstream `create_task` writes. That
producer is code we do not own, so trusting it silently was the gap: if
upstream renamed the key, dropped it, or stopped deriving it from
`initial_status`, the image would still build and ship a consumer that
mis-classifies every task it reads.
Anchor the producer contract at build time, before the regression suite
runs, with three assert-only preconditions: the `initial_status="blocked"`
park resolves `task_status` to `"blocked"`, every non-park creation
resolves it to something else, and the `created` event carries that same
variable under `"status"`. None of them rewrite the producer.
Textual anchors cannot see dataflow, so add the runtime net the reviewer
asked for. The suite now drives the real API: create + claim an ordinary
task, trip the circuit breaker once at failure_limit=1 so it parks with a
`gave_up` event (leaving its own `created` event as the most recent
create/block/unblock row), then recompute at failure_limit=2 and require
promotion to ready. That case is red under an unconditional-true created
predicate and red under producer drift that labels every created event
blocked, while the explicit block/unblock, dependency-promotion and
circuit-breaker-at-current-limit cases stay green. Non-blocked and
malformed created payloads are pinned as controls, and the gate now
rejects non-dict payloads rather than trusting `.get`.
Also make the live placement correction durable: titan-04 is cordoned
after repeated kernel undervoltage and kubelet failure and titan-19 was
probe/Longhorn unstable under worker load, so both join the hard NotIn
list; titan-05 is healthy but sits at 3592m/3600m requested CPU, so the
main hermes container gives back 50m (350m -> 300m) to schedule there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three narrowly scoped Hermes reliability fixes backed by live evidence
from the Cassandra/titan-iac proof run.
Worker concurrency. Three simultaneous direct CLI workers on the 4-core
hermes-agent node drove load to ~45 and made the hermes and oauth2-proxy
containers fail their probes, leaving the pod 8/10 Ready; two workers
stayed at 10/10. Cap HERMES_CLI_LANE_CONCURRENCY at 2 and lower the
cli-lane-runner CPU limit from 3 to 2 so the dashboard and auth sidecars
keep a guaranteed share of the node. Requests are unchanged: the pod
still asks for 745m total, so placement does not move.
Service links. Kubernetes injects a service-link variable pair for every
service in the namespace, and hermes-claude-broker produces
HERMES_CLAUDE_BROKER_PORT=tcp://10.43.31.76:9006 — a value the broker
parses as an int. That contaminated worker and test environments even
though the deployment already addresses every service by DNS name. Set
enableServiceLinks: false on the hermes-agent pod spec.
Blocked-task scheduling. create_task(initial_status="blocked") records a
created event carrying status=blocked but never a blocked event, while
_has_sticky_block() only inspects blocked/unblocked events. recompute_ready()
considers blocked tasks, so an explicitly parked task with no incomplete
parent auto-promoted on the next dispatcher cycle. Teach _has_sticky_block()
to also recognize a created event whose payload status is blocked, which
covers tasks created before this image patch without adding a persisted
field. Dependency-driven promotion and the circuit-breaker failure-limit
guard are untouched; unblock_task() still releases either kind of block.
hermes-kanban-blocked-regression.py runs against the real upstream
kanban_db API during the image build, so the build fails if any of these
semantics regress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
