titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

hermes: add fenced three-node execution pool #18

Manually merged
atlas-reconciler merged 0 commits from feature/hermes-distributed-worker-pool into main 2026-08-18 21:27:59 +00:00
Conversation 0 Commits - Files Changed -
hermes-automation commented 2026-08-17 10:00:02 +00:00
Member
Copy Link

Summary

  • keep the existing hermes-agent Deployment at one Recreate replica as the sole Hermes Kanban/SQLite and hermes-agent-home owner
  • add three ordinal-stable StatefulSet workers with separate 30Gi RWO workspaces, required hostname anti-affinity/topology spread, accelerator preference, rpi5 fallback, and all unhealthy/titan-22/titan-24 exclusions
  • add a bounded HMAC assignment/result/heartbeat ledger binding board, task, run, ordinal, attempt, delivery, and payload digest; coordinator-only claim/finalize remains fenced by expected_run_id
  • keep provider OAuth in memory with exact-run durable sessions; expose no Kubernetes identity, pool signing key, Vault projection, or Gitea credential to the model-facing container
  • route Git through an ordinal SCM boundary using assignment-bound branches and model-created bundles; no token-bearing Git runs in a model-controlled checkout
  • stream sanitized activity, route, heartbeat, node/ordinal, provider sessions, and final evidence into the existing Kanban UI

Validation

Exact pushed SHA: 20002527512235b29054f3636b02418168373009

  • configured unit suite: 397 passed (build/junit-hermes-pool-final.xml)
  • focused final pool/lane/runtime/router suite: 150 passed
  • strict Ruff (new pool sources/tests), compileall, sh -n, ShellCheck, and git diff --check: passed
  • production changed-file detect-secrets: 0 findings
  • Hermes and Vault: Kustomize render, Flux build dry-run, kubectl client dry-run, and server-side dry-run passed
  • no image publish, Flux reconcile, deployment, workload mutation, or live three-node claim performed

Adversarial coverage

Executable tests cover simultaneous claims; duplicate assignment/heartbeat/result; delivery reuse; stale/replacement runs; coordinator/worker restart; heartbeat loss; node replacement; malformed/oversized payloads; traversal/symlinks/FIFOs; missing credentials; cross-worker PVC and RBAC isolation; exact-run provider sessions; bounded HTTP concurrency; three-node scheduling; sanitized visible activity; and conservative retention GC.

Draft PR compatibility

Fresh remote heads were inspected and conflict-free synthetic merge trees were rendered and tested:

  • #14 3cd8a17e772978fb554d88dd1c59777786be3991 -> tree b4cad7237d741cf4be3242978a2857280e4c8183, 61 passed
  • #15 7609cd77e87a64e332b2f0d670c1d7a4f62b3d9e -> tree e815363f71275b3a6206dda5a50f6e04e1141a0d, 62 passed
  • #16 5ab0eba619779b3dfed2eb09d5c02f0a9c92dd0a -> tree b35b5f066c74b645f355c8ba4c9ee8396dad7613, 62 passed
  • #17 48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c -> tree 52f5aac3d8656430633ffbef1d2e73030315fb01, 62 passed

The implementation uses additive Kustomize/config/script surfaces and does not import the draft implementations.

Known repository baseline

The repository-wide local quality profile remains red only for four pre-existing test files already above the 500-line hygiene limit. Docs, Ruff, unit, and coverage portions pass; every new pool production file is below 500 lines and the primary adversarial test file is exactly 500 lines.

Human review and rollout are required. This PR intentionally does not claim live scheduling proof.

## Summary - keep the existing `hermes-agent` Deployment at one Recreate replica as the sole Hermes Kanban/SQLite and `hermes-agent-home` owner - add three ordinal-stable StatefulSet workers with separate 30Gi RWO workspaces, required hostname anti-affinity/topology spread, accelerator preference, rpi5 fallback, and all unhealthy/titan-22/titan-24 exclusions - add a bounded HMAC assignment/result/heartbeat ledger binding board, task, run, ordinal, attempt, delivery, and payload digest; coordinator-only claim/finalize remains fenced by `expected_run_id` - keep provider OAuth in memory with exact-run durable sessions; expose no Kubernetes identity, pool signing key, Vault projection, or Gitea credential to the model-facing container - route Git through an ordinal SCM boundary using assignment-bound branches and model-created bundles; no token-bearing Git runs in a model-controlled checkout - stream sanitized activity, route, heartbeat, node/ordinal, provider sessions, and final evidence into the existing Kanban UI ## Validation Exact pushed SHA: `20002527512235b29054f3636b02418168373009` - configured unit suite: **397 passed** (`build/junit-hermes-pool-final.xml`) - focused final pool/lane/runtime/router suite: **150 passed** - strict Ruff (new pool sources/tests), `compileall`, `sh -n`, ShellCheck, and `git diff --check`: passed - production changed-file `detect-secrets`: **0 findings** - Hermes and Vault: Kustomize render, Flux build dry-run, kubectl client dry-run, and server-side dry-run passed - no image publish, Flux reconcile, deployment, workload mutation, or live three-node claim performed ## Adversarial coverage Executable tests cover simultaneous claims; duplicate assignment/heartbeat/result; delivery reuse; stale/replacement runs; coordinator/worker restart; heartbeat loss; node replacement; malformed/oversized payloads; traversal/symlinks/FIFOs; missing credentials; cross-worker PVC and RBAC isolation; exact-run provider sessions; bounded HTTP concurrency; three-node scheduling; sanitized visible activity; and conservative retention GC. ## Draft PR compatibility Fresh remote heads were inspected and conflict-free synthetic merge trees were rendered and tested: - #14 `3cd8a17e772978fb554d88dd1c59777786be3991` -> tree `b4cad7237d741cf4be3242978a2857280e4c8183`, **61 passed** - #15 `7609cd77e87a64e332b2f0d670c1d7a4f62b3d9e` -> tree `e815363f71275b3a6206dda5a50f6e04e1141a0d`, **62 passed** - #16 `5ab0eba619779b3dfed2eb09d5c02f0a9c92dd0a` -> tree `b35b5f066c74b645f355c8ba4c9ee8396dad7613`, **62 passed** - #17 `48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c` -> tree `52f5aac3d8656430633ffbef1d2e73030315fb01`, **62 passed** The implementation uses additive Kustomize/config/script surfaces and does not import the draft implementations. ## Known repository baseline The repository-wide local quality profile remains red only for four pre-existing test files already above the 500-line hygiene limit. Docs, Ruff, unit, and coverage portions pass; every new pool production file is below 500 lines and the primary adversarial test file is exactly 500 lines. Human review and rollout are required. This PR intentionally does not claim live scheduling proof.
hermes-automation added 1 commit 2026-08-17 10:00:03 +00:00
hermes-automation changed title from hermes: add fenced three-node execution pool to WIP: hermes: add fenced three-node execution pool 2026-08-17 10:00:42 +00:00
hermes-automation added 29 commits 2026-08-17 13:52:10 +00:00 
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.

The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.

Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.

Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.

Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.

The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.

docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.

Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
# Conflicts:
#	scripts/tests/test_dashboards_render_jobs.py
#	testing/tests/test_hermes_chat_images.py
#	testing/tests/test_hermes_chat_provider_auth.py
#	testing/tests/test_hermes_chat_quality.py
#	testing/tests/test_hermes_chat_support.py
#	testing/tests/test_hermes_cli_lanes.py
#	testing/tests/test_hermes_coordinator_routing.py
#	testing/tests/test_hermes_coordinator_support.py
# Conflicts:
#	testing/tests/test_hermes_chat_quality.py
#	testing/tests/test_hermes_cli_lanes.py
# Conflicts:
#	testing/quality_contract.json
# Conflicts:
#	clusters/atlas/flux-system/applications/hermes/kustomization.yaml
#	dockerfiles/Dockerfile.hermes-agent.dockerignore
#	services/hermes/agent-deployment.yaml
#	services/hermes/scripts/stage_runtime_access.py
#	services/vault/scripts/vault_k8s_auth_configure.sh
#	testing/quality_contract.json
#	testing/tests/test_hermes_runtime_access.py
# Conflicts:
#	testing/quality_contract.json
#	testing/quality_coverage.py
#	testing/tests/conftest.py
#	testing/tests/test_hermes_auto_router.py
#	testing/tests/test_quality_coverage_helpers.py
hermes-automation force-pushed feature/hermes-distributed-worker-pool from 689bcb6e48 to 54156f839e 2026-08-17 16:50:31 +00:00 Compare
bstein added 14 commits 2026-08-18 05:02:39 +00:00 
The local goal judge scored every worker report against "did the reviewed
implementation reach a shippable state". A read-only reviewer that returned a
completed BLOCK verdict with findings was therefore resumed turn after turn with
an instruction to repair code it was forbidden to touch (observed live on
t_dbdcd739), burning subscription capacity and risking an unbounded loop.

Completion is now judged against the action the card assigned:

* Cards declare their role explicitly with Hermes-Task-Role / Hermes-Expected-
  Output metadata. Pre-contract cards fall back to a narrow inference that needs
  a read-only scope, a requested verdict, no requested mutation deliverable, and
  a report that changed no files.
* Role resolution reads only the card itself. Prior attempts, parent results,
  cross-task history and comments appended to the worker context can no longer
  reassign the role.
* Review, audit and diagnostic cards finalize deterministically on a truthful
  SHIP or BLOCK verdict with evidence, and fail closed on a missing,
  unrecognized or self-contradictory verdict, on a BLOCK without findings, and
  on a verdict without evidence. Every rejection reason carries the read-only
  guard, so a resumed review is never told to edit the reviewed implementation.
* Implementation cards keep the fail-closed model judge unchanged, including the
  unfinished-work heuristic and judge-unavailable rejection.
* All judge reasons are bounded, single-line and secret-redacted before they
  reach Kanban metadata, comments and continuation prompts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Repairs the blockers from the independent review of the previous head. The
role-aware verdict contract was correct but reachable only through one call
site and only on cards written with real newlines, so most real review cards
never used it.

* The role-blind call is no longer a weaker classifier that can reject before
  the role-aware judge runs. Without a card there is no defensible
  role-dependent judgement, so `unfinished_result_reason()` applies only the
  card-independent checks. That closes the short-circuit at every call site,
  including the one PR15 moves to `cli_lane_execution`, and it makes a
  journalled terminal record accepted under one version of these semantics
  re-validate under any other instead of being quarantined into a re-dispatch
  of an already-accepted task.
* The lane resolves the role from the card and passes it, and records it in
  the Kanban metadata. The verdict contract binds only where the lane can buy
  another turn: in single-shot mode a rejection discards the worker's real
  result, so review cards keep the relaxation without gaining any rejection
  single-shot mode did not already have.
* Card scope expands the literal \n escapes the board stores in one-line
  bodies, so explicit `Hermes-Task-Role` / `Hermes-Expected-Output` directives
  are honoured on the 6 of 78 live cards that carry no real newline, and the
  read-only, verdict and mutation heuristics stop being cut apart by them.
* Card scope now ends at the first non-card H2 and at the runner's controller
  evidence, which is emitted under its own heading. Goal-controller rejection
  history can no longer sit inside the card, and an upstream heading rename
  fails closed instead of admitting history into role resolution.
* The inference recognises the SHIP/BLOCK-shaped deliverables real cards
  actually use: 21 of 78 live cards resolve to review, up from 10, with no
  implementation card misclassified. Cards asking for a findings list rather
  than a verdict deliberately stay on the model judge, since the verdict is
  the review contract's only gate.
* A declared review role no longer outranks mutation evidence: a report that
  changed files falls back to the implementation regime.
* Judge reasons go through the agent runtime's canonical redactor, extended
  for the two shapes it deliberately passes through and this lane handles -
  `scheme://user:secret@host` and a credential named in prose - while a 40-hex
  commit SHA survives as evidence.

Regressions cover the recovered t_dbdcd739 incident, a verbatim snapshot of
every live card on three boards with a hand-labelled expected role, the
upgrade and single-shot properties against the previous gate, the upstream
context-heading contract, and the end-to-end `execute_claim` shape that used
to burn every goal turn.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add kanban routing keys (provider_quota_min_remaining_percent: 15,
provider_capacity_cooldown_seconds: 300, provider_auth_cooldown_seconds:
3600), a lane-metrics port/Service on 9011 with service-annotation
scraping, monitoring ingress for the new port, and the lane's quota
metrics URL env. Based on PR #15 (fix/hermes-result-decomposition-
reliability); stacked because this work rides on the decomposed lane.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on PR #15 (fix/hermes-result-decomposition-reliability); stacked
on the decomposed cli_lane modules.

- cli_lane_quota: soft-exclude a provider from NEW cli-auto work below
  the remaining-quota threshold (both-below prefers more remaining;
  fetch failure fails open with a metric).
- cli_lane_health: lane now writes provider health (G7) with classified
  failure reasons splitting the capacity conflation (quota/auth/
  rate-limit/transport) and cooldown hysteresis; re-admission only on
  full cooldown expiry, passed quota reset, or fresh success (G4).
- cli_lane_routing: capacity-limited health now excludes a provider
  (G3); cooldown/reset-aware re-admission.
- cli_lane_failover: explicit cli-codex-*/cli-claude-* assignees fail
  closed as transient instead of switching providers (G5); fallback
  depth stays bounded at two hosted providers (G1) with effort
  preserved; Switchyard outages block transient, not capability (G9).
- cli_lane_metrics: route-decision/fallback counters, quota and
  soft-exclusion gauges, pod-local scrape server (G6).
- cli_lane_provider: worker env drops ANTHROPIC_API_KEY, CLAUDE_API_KEY,
  OPENAI_API_KEY, API_SERVER_KEY so no metered path exists (G10).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deterministic coverage for the quota-aware lane: threshold boundaries
(14.9/15/15.1), both-below preference, fetch-failure fail-open, cooldown
elapsed-vs-not hysteresis, quota-reset recovery (never for auth),
explicit fail-closed in both directions, bounded double-failure block,
failure-reason classification, metrics emission, and worker env key
stripping. Based on PR #15 (fix/hermes-result-decomposition-reliability).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CAPACITY_PATTERN (the gate that sets result.capacity_failure) lacked the
bare unauthorized/forbidden/401/403 signals that classify_capacity_failure
already recognizes, so an auth blip surfacing only as "403 Forbidden"
was blocked as capability instead of transient: no auto failover, no
health cooldown. Add 401|403|unauthorized|forbidden to the gate so it
matches the classifier; reason classification still distinguishes auth
from quota/rate-limit/transport.

Tests: an auto card failing with only "403 Forbidden" now fails over to
the other provider, records an auth cooldown (authenticated:false), and
classifies the fallback reason as auth in metrics; a manual card with the
same failure still fails closed as transient. Router-outage tests moved to
test_hermes_cli_router_outage.py to keep both files <500 LOC.

Based on PR #15 (fix/hermes-result-decomposition-reliability).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile two independent test/gate reorganizations:
- Gate/semgrep/mailu: keep main's #16 dual-metric implementation.
- quality_contract.json: union #16 image-builder + #14 scm/node + #15 cli_lane.
- agent-deployment.yaml: keep #14 gitea removal + #16 image-build-token + #15 probe.
- Test splits: main's chat/coordinator/agent organization is authoritative;
  drop #15's redundant competing splits and #14's stale cli-lane duplicates;
  keep #15's cli-lane decomposition suite and port the execution-safety test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
#	services/hermes/networkpolicy.yaml
#	testing/quality_contract.json
#	testing/tests/test_hermes_agent_security.py
# Conflicts:
#	services/hermes/scripts/cli_lane_runner.py
# Conflicts:
#	scripts/tests/test_dashboards_render_atlas_drilldowns.py
#	scripts/tests/test_dashboards_render_jobs.py
#	services/hermes/scm-common/scripts/scm_broker.py
#	services/hermes/scripts/cli_lane_dispatch.py
#	services/hermes/scripts/cli_lane_execution.py
#	testing/quality_contract.json
#	testing/tests/test_hermes_agent_access.py
#	testing/tests/test_hermes_agent_security.py
#	testing/tests/test_hermes_chat_config.py
#	testing/tests/test_hermes_chat_images.py
#	testing/tests/test_hermes_chat_provider_auth.py
#	testing/tests/test_hermes_chat_quality.py
#	testing/tests/test_hermes_chat_support.py
#	testing/tests/test_hermes_chat_voice.py
#	testing/tests/test_hermes_cli_finalization_edges.py
#	testing/tests/test_hermes_cli_foundation_coverage.py
#	testing/tests/test_hermes_cli_lanes_configuration.py
#	testing/tests/test_hermes_cli_recovery_edges.py
#	testing/tests/test_hermes_cli_retention_edges.py
#	testing/tests/test_hermes_coordinator.py
#	testing/tests/test_hermes_coordinator_boards.py
#	testing/tests/test_hermes_coordinator_support.py
Post-merge fixes after rebasing the distributed worker pool onto the
review-goal-semantics train tip:

- Pool SCM submission tests target the train's relocated receive-pack
  scanner: FEATURE_REF_RE now lives in receive_pack_scan, bodies are
  built via the shared _receive_command helper (valid pack), and the
  update-rejection assertion matches the train's message.
- Take the train's canonical test_hermes_scm_broker,
  test_hermes_cli_dispatch_runtime and test_hermes_cli_execution_edges,
  which exercise the train's broker/dispatch/execution behavior.
- Runtime staging tests patch os.fchown alongside os.chown so the
  UID-10000 _write_secret path passes under a non-root gate runner
  (production ownership behavior unchanged).
- Split the jenkins build-evidence contracts out of
  test_hermes_runtime_access into test_hermes_runtime_evidence to keep
  both files under the 500-LOC hygiene ceiling after the merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
atlas-reconciler changed title from WIP: hermes: add fenced three-node execution pool to hermes: add fenced three-node execution pool 2026-08-18 21:26:49 +00:00
atlas-reconciler manually merged commit 1c4ed3af93 into main 2026-08-18 21:27:59 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#18
No description provided.
Delete Branch "feature/hermes-distributed-worker-pool"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?