47 lines
17 KiB
JSON
47 lines
17 KiB
JSON
{
|
|
"status": "completed",
|
|
"summary": "Independent read-only xhigh release/security review of PR #18 (titan/atlas-iac, head 20002527512235b29054f3636b02418168373009, base origin/main d8f2d818b9a552ea6c2d7fe86554be829bd5ffff which contains merged PR #13 ab346f55). Verdict: BLOCK. Five P0 defects are executably reproducible, four of which fire on the coordinator's first tick: (1) Coordinator.finalize() compares an int Kanban current_run_id to a str run_id so every accepted result is discarded while the worker is told accepted=True, and the resulting PK collision makes reconcile() throw every cycle; (2) the workspace_path gate force-blocks every live Kanban task including the two currently running ones; (3) the branch regex rejects the established wt/<task-id> convention; (4) Task has no repo_url so every board's tasks are pushed to titan-iac; (5) the pool ConfigMap does not ship the 14 modules PR #15 splits cli_lane_runner into, so coordinator and all three workers fail at import after a clean textual merge. P1 findings cover control-plane node placement, capacity/affinity steering onto the routing model and Flux controllers, model-container access to its own signing/SCM boundaries, worker poisoning of the shared Vault credential store, and bypass of PR #14's SCM authority boundary. Architecture that is sound (coordinator-only Kanban/SQLite, per-ordinal RWO PVCs, no shared mutable git, zero worker RBAC, worker-to-worker isolation, HMAC envelope binding) is called out explicitly. Two observed failures were confirmed pre-existing and unrelated to the PR. The review worktree was restored to review/hermes-distributed-worker-pool @ 0dd6ea0f and is clean including ignored paths.",
|
|
"changed_files": [],
|
|
"tests_run": [
|
|
"Focused pool suite at PR head: pytest testing/tests/test_hermes_execution_pool.py test_hermes_execution_pool_assignment.py test_hermes_runtime_access.py test_hermes_auto_router.py -> 82 passed",
|
|
"Full suite at PR head (PYTHONPATH=.): pytest testing/tests -> 1 failed, 332 passed (failure pre-existing and environment-sensitive)",
|
|
"Repo quality gate: python -m testing.quality_gate --profile local -> 1 failed, 396 passed (same pre-existing failure); ruff check on all changed .py -> All checks passed",
|
|
"compileall on all changed .py -> OK; sh -n services/hermes/scripts/execution_pool_askpass.sh -> OK; git diff --check ab346f55..20002527 -> OK",
|
|
"Adversarial repro repro_finalize_stale.py -> worker ack accepted=True, complete_task=[] block_task=[], store state=stale had_result=True, all 3 ordinals freed, reconcile() raises ProtocolError: conflicting duplicate assignment",
|
|
"Adversarial repro repro_workspace_block.py against live titan-iac board row (mode=ro&immutable=1) -> dispatch() and reconcile() both emit kanban block_task; control case with workspace_path cleared still blocks on branch policy",
|
|
"Adversarial repro repro_configmap_import.py over synthetic merges -> merge-main/pr14/pr16 import rc=0; merge-pr15 rc=1 ModuleNotFoundError: No module named 'cli_lane_board' for both execution_pool_coordinator and execution_pool_worker",
|
|
"Synthetic merges via git merge-tree --write-tree of PR18 head against origin/main d8f2d818 and PR14 3cd8a17e772978fb554d88dd1c59777786be3991, PR15 7609cd77e87a64e332b2f0d670c1d7a4f62b3d9e, PR16 5ab0eba619779b3dfed2eb09d5c02f0a9c92dd0a, PR17 48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c, PR19 8f005458282269ba5c07941814e4237f5d4cf3ac -> all CLEAN; each materialized tree ran the focused pool suite (81-82 passed) and kustomize build services/hermes (OK)",
|
|
"kustomize build services/hermes and services/vault -> OK; flux build kustomization hermes --dry-run -> exit 0",
|
|
"kubectl apply --dry-run=client -> only the pre-existing >256KiB hermes-coordinator annotation error; kubectl apply --dry-run=server --server-side --field-manager=kustomize-controller -> clean (Flux path)",
|
|
"Read-only live cluster checks: k3s v1.33.3, node labels/taints, apps.kubernetes.io/pod-index availability, astreae StorageClass (Longhorn, Immediate, diskSelector=astreae, 24.6TiB free vs 270GiB needed), per-node free CPU/memory feasibility for the 1150m/2304Mi worker pod, vault pod labels, existing hermes NetworkPolicies, RBAC bindings in the render, image digest drift"
|
|
],
|
|
"artifacts": [
|
|
"/opt/data/workspace/evidence/t_dbdcd739/RUN.md",
|
|
"/opt/data/workspace/evidence/t_dbdcd739/repro_finalize_stale.py",
|
|
"/opt/data/workspace/evidence/t_dbdcd739/repro_workspace_block.py",
|
|
"/opt/data/workspace/evidence/t_dbdcd739/repro_configmap_import.py"
|
|
],
|
|
"findings": [
|
|
"P0-1 BLOCKER - services/hermes/scripts/execution_pool_coordinator.py:248: finalize() evaluates `_task_value(task, \"current_run_id\") != binding[\"run_id\"]` without coercion. hermes_cli/kanban_db.py:853 and the schema at :1108 declare current_run_id as INTEGER, and live rows confirm typeof=integer (titan-iac t_dbdcd739 -> 23), while binding['run_id'] is a str produced by dispatch() at :386 and stored in a TEXT column. int 23 != str '23' is always True, so every result is marked stale: complete_task/block_task are never called, the Kanban task stays 'running' forever, and the worker receives accepted=True and writes terminal_at. Repro repro_finalize_stale.py output: ack {'accepted': True, 'duplicate': False}; complete_task []; block_task []; store state=stale had_result=True; free ordinals [0,1,2]. Follow-on: the row survives under PK (board,task_id,run_id) with a different payload digest, so reconcile() at :358 raises ProtocolError 'conflicting duplicate assignment' (execution_pool_protocol.py:307) on every 5s cycle, aborting the entire reconcile pass for all boards. Zero test coverage: the only tests touching Coordinator.finalize stub it (testing/tests/test_hermes_execution_pool_assignment.py:71 `pool.finalize = called.append`), and grep for current_run_id in both new test files returns nothing.",
|
|
"P0-2 BLOCKER - services/hermes/scripts/execution_pool_coordinator.py:65-69: resolve_scm raises RuntimeError whenever task.workspace_path is non-empty; dispatch() (:396-406) and reconcile() (:347-356) convert that into kanban_db.block_task(kind='capability'). A read-only census of /opt/data/kanban/boards/*/kanban.db shows every task on titan-iac, cassandra and soteria carries workspace_path. On first coordinator tick this force-blocks the two currently running titan-iac tasks (t_6496c271 and t_dbdcd739, this review) and the two ready cli tasks (t_83252d81, t_87cd9076). Repro repro_workspace_block.py cases [1] and [2].",
|
|
"P0-3 BLOCKER - services/hermes/scripts/execution_pool_coordinator.py:40,77: BRANCH = ^(?:feature|fix|chore|docs|test|refactor)/... rejects the branch names Hermes itself writes. hermes_cli/kanban_db.py:5447, :7275 and :7367 default worktree tasks to f'wt/{task.id}', and live rows carry wt/t_1e95ea6d as well as review/... names (t_dbdcd739 -> review/hermes-distributed-worker-pool). Repro repro_workspace_block.py case [3]: with workspace_path cleared the same task still blocks with 'task feature branch is outside the SCM branch policy'.",
|
|
"P0-4 BLOCKER - services/hermes/scripts/execution_pool_coordinator.py:70,79-81: remote and base_branch come from _task_value(task, 'repo_url') / 'base_branch', but the hermes_cli.kanban_db.Task dataclass (kanban_db.py:838-916) defines neither field, so getattr always returns the DEFAULT_REPO https://scm.bstein.dev/titan/atlas-iac.git. Every task on every board is therefore cloned from and pushed to titan-iac. The metis scratch task t_8569b7b9 - the one live task that survives P0-2 - would run against a titan-iac checkout and push to feature/hermes-t_8569b7b9 there.",
|
|
"P0-5 BLOCKER - services/hermes/kustomization.yaml:64-78 is runtime-incompatible with open PR #14... correction, PR #15 fix/hermes-result-decomposition-reliability head 7609cd77e87a64e332b2f0d670c1d7a4f62b3d9e, despite a clean textual merge. PR15 decomposes cli_lane_runner.py into 14 sibling modules (cli_lane_board, cli_lane_config, cli_lane_dispatch, cli_lane_evidence, cli_lane_execution, cli_lane_files, cli_lane_finalization, cli_lane_prompt, cli_lane_provider, cli_lane_quarantine, cli_lane_records, cli_lane_recovery, cli_lane_retention, cli_lane_routing), while the hermes-execution-pool ConfigMap ships only cli_lane_goal.py and cli_lane_runner.py. Proof repro_configmap_import.py: merge-main/pr14/pr16 import rc=0, merge-pr15 rc=1 ModuleNotFoundError: No module named 'cli_lane_board' for both execution_pool_coordinator and execution_pool_worker -> coordinator plus all three workers CrashLoopBackOff.",
|
|
"P1 - services/hermes/execution-worker-statefulset.yaml:87-90: the second nodeSelectorTerm requires only arch=arm64, hardware=rpi5 and a hostname NotIn list, dropping the node-role.kubernetes.io/worker In ['true'] requirement that agent-deployment.yaml:109-111 uses. Measured against live capacity: 8 nodes satisfy the affinity, only 4 can fit the 1150m CPU / 2304Mi pod (titan-20 1270m free, titan-21 1520m, titan-0a 3730m, titan-0c 3630m), and titan-0a/titan-0c are k3s control-plane/master. podAntiAffinity forces 3 distinct hosts, so one worker necessarily lands on a control-plane node while running codex --dangerously-bypass-approvals-and-sandbox / claude --dangerously-skip-permissions (cli_lane_runner.py:594, :631).",
|
|
"P1 - services/hermes/execution-worker-statefulset.yaml:91-99: preferredDuringScheduling weight 100 on node-role.kubernetes.io/accelerator steers two of three workers onto titan-20 (which runs ai/ollama, the Jetson routing model behind Switchyard AUTO) and titan-21 (which runs flux-system source-controller, image-automation-controller, image-reflector-controller and notification-controller). titan-20 retains only ~120m CPU slack after placement while the worker container's limits are 5 CPU / 6Gi, so the pool contends directly with the routing model and the GitOps control plane.",
|
|
"P1 - services/hermes/scripts/execution_pool_client.py:142 and execution_pool_scm.py:442 bind 0.0.0.0 (required for kubelet httpGet probes). The model container holds no key but shares the pod network namespace, so it can POST 127.0.0.1:9009 {'operation':'poll'} to obtain a freshly coordinator-signed assignment envelope and replay it to 127.0.0.1:9008 /v1/scm {'operation':'submit'} and to 9009 {'operation':'result'}. Authority stays bounded to its own board/task/run/ordinal by ClientBoundary.forward (client.py:84) and _binding (scm.py:123), and worker-to-worker reach is blocked by ingress: [], but it bypasses the worker's own gates: the uncommitted/untracked check (execution_pool_worker.py:332-337) and the commits-ahead check (:342).",
|
|
"P1 - The model container has read-write access to /runtime-access/claude/.credentials.json and /runtime-access/codex/auth.json (execution-worker-statefulset.yaml:206), and the credential-sync sidecar CAS-writes whatever it finds there into kv/data/atlas/hermes/agent-tokens (sync_runtime_credentials.py:105-116) using role hermes-execution-credential-sync, which services/vault/scripts/vault_k8s_auth_configure.sh:228-229 grants create/update/read on that path. That secret backs the coordinator's own credentials and is the derivation input for the pool HMAC key (printf \"hermes-execution-pool-v1:%s\" .Data.data.agent_api_key | sha256sum). Separately, all three workers plus the agent now refresh the same subscription OAuth documents independently; provider-side refresh-token rotation makes this a lost-update/lockout race that CAS cannot resolve.",
|
|
"P1 - Open PR #14 feature/hermes-safe-gitea-pr-client (head 3cd8a17e772978fb554d88dd1c59777786be3991) deletes services/hermes/scripts/gitea_api.py and gitea_askpass.sh and introduces a dedicated hermes-scm-broker (own namespace, NetworkPolicy, gitea_api_policy.py, DLP tests) to bound Atlas PR authority. PR18 concurrently stages the raw kv/data/atlas/hermes/developer-gitea token into three new pods (stage_runtime_access.py:192), authenticates git with it (execution_pool_askpass.sh), calls the Gitea REST API directly (execution_pool_scm.py:355-373) and opens the egress path (execution-worker-networkpolicy.yaml:37-42). Verified on the merged tree: merge-pr14 still stages the raw token, so the broker is bypassed for the highest-volume automated PR path and the token blast radius widens by three pods on three more nodes.",
|
|
"P2 - services/hermes/scripts/execution_pool_protocol.py:227-229,337,385: lease_seconds, lease_until and last_heartbeat are written and never read anywhere in the pool. There is no lease-expiry reaper, so a worker that dies mid-assignment holds its ordinal until Kanban's own DEFAULT_CLAIM_TTL (7 days, cli_lane_runner.py:40) causes a divergence that reconcile() can observe.",
|
|
"P2 - services/hermes/scripts/execution_pool_worker.py:407: main() swallows OSError/RuntimeError/ValueError, sleeps 10s and re-polls the identical assignment; attempt is never incremented and Kanban is never informed. Reachable today: cli_lane_runner.parse_assignee (:152) raises ValueError for cli-codex-sol-xhigh, a live assignee on two titan-iac tasks, so such an assignment spins forever holding its ordinal. The legacy path blocked the task instead (cli_lane_runner.py:832-848).",
|
|
"P2 - services/hermes/kustomization.yaml:77-78 sets disableNameSuffixHash: true on hermes-execution-pool, and neither execution-worker-statefulset.yaml nor execution-coordinator-patch.yaml carries a config-revision annotation (agent-deployment.yaml:28 uses ai.bstein.dev/config-rev for exactly this). Pool code changes therefore never trigger a rollout; coordinator and workers pick up new code only on unrelated restarts, producing mixed-version windows against a strict contract that has no negotiation (execution_pool_protocol.py:177 rejects any envelope whose field set differs from version 1).",
|
|
"P2 - services/hermes/execution-worker-networkpolicy.yaml:33-36 allows worker egress to hermes-model-gate:8080, but services/hermes/networkpolicy.yaml:26-41 (hermes-model-gate-ingress) does not list hermes-execution-worker and nothing in the worker path calls model-gate (select_route targets Switchyard only). The rule is dead and asymmetric; remove it or complete the ingress side.",
|
|
"P2 - services/hermes/execution-worker-statefulset.yaml:303 mounts the whole runtime-access volume read-write into credential-sync, whereas the agent mounts only the claude and codex subPaths read-only (agent-deployment.yaml:951-952). Minor hardening regression relative to the established pattern.",
|
|
"PRE-EXISTING, not caused by PR18 - testing/tests/test_hermes_coordinator.py::test_cassandra_sync_repairs_origin_without_token fails wherever /runtime-access/gitea-token exists, because the test never patches hermes_coordinator._gitea_token_available (hermes_coordinator.py:148-155). Neither the module nor the test is in the PR diff and it fails standalone. This accounts for the 1 failure in both the full suite (332 passed) and the quality gate (396 passed), versus the parent task's claim of 397 passed in an environment without that file.",
|
|
"PRE-EXISTING, not caused by PR18 - kubectl apply --dry-run=client rejects ConfigMap hermes-coordinator with 'metadata.annotations: Too long: may not be more than 262144 bytes'. Measured generator input is 341438 bytes at base ab346f55 and 343841 at head, so the threshold was already exceeded before this PR. Flux uses server-side apply and the SSA dry-run with field-manager kustomize-controller is clean.",
|
|
"NOTE - ruff format --check would reformat 10 of the changed files, but testing/quality_gate.py:228 runs only ruff check, so this is unenforced style drift rather than a gate failure.",
|
|
"NOTE - git/cluster image drift: both origin/main and PR18 pin hermes-agent@sha256:81970563e542f0720773e72297810b3a844b83e381e278f25c0916c78d930107, while the live hermes-agent pod runs sha256:37ebf720c783ae908a602916ffccf88d43d205a157957f5dc4b487867aee45e7. Pre-existing drift, but it means the new workers and the currently running coordinator would execute different image builds until Flux recreates the agent. The pinned digest is cached only on titan-11/15/19, none of the four candidate worker nodes, so first scheduling incurs a cold pull.",
|
|
"VERIFIED SOUND (no defect) - coordinator-only Kanban/SQLite ownership (workers mount no /opt/data; hermes-agent is replicas:1 with strategy Recreate); per-ordinal RWO 30Gi Longhorn PVCs via volumeClaimTemplates with ample astreae capacity (24624Gi schedulable free vs 270Gi needed across 87 existing volumes); no shared mutable git (per-run clone under /workspace/runs/<board>/<task>/<run>); zero Kubernetes RBAC for the hermes-execution-worker service account (no RoleBinding or ClusterRoleBinding in the render, automountServiceAccountToken: false, Vault-audience token projected only into credential-sync); worker-to-worker isolation via ingress: [] with kubelet probes proven working under the identical pattern on hermes-chat-sandbox; HMAC envelope binding to board/task/run/worker ordinal/attempt/payload digest with expiry, clock-skew bounds, delivery-id replay guard and a partial unique index enforcing one live assignment per ordinal; result idempotency by digest; heartbeat lease loss terminates the provider process (cli_lane_runner.py:46-50); no node-SSH material staged to workers (stage_runtime_access.py:179-204); Switchyard is the only routing path; heartbeat activity plus a deduplicated terminal batch reach the Kanban worker log for agent UI visibility."
|
|
],
|
|
"blockers": []
|
|
} |