hermes: judge completion against the assigned task role #22
Loading…
x
Reference in New Issue
Block a user
No description provided.
Delete Branch "feature/hermes-review-goal-semantics"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
The local goal judge scored every worker report against "did the reviewed implementation reach a shippable state". On live task
t_dbdcd739, a read-only independent release reviewer returnedstatus=completedwith aBLOCKverdict and twenty reproducible findings, and the judge resumed it turn after turn with an instruction to repair code it was explicitly forbidden to touch. The card burned four continuation turns on subscription capacity before a human stopped it.Reproduction
repro_review_goal_loop.pyreplays the exactt_dbdcd739worker context and result against the livehermes-model-gatejudge.Before (three consecutive runs of the same input):
Turns 1-2 match the reason recorded verbatim in the live Kanban comment thread. Turn 3 shows the verdict is also non-deterministic for identical input.
After:
That exact context and result are now a committed fixture (
testing/tests/data/hermes_t_dbdcd739_*), replayed in CI with the model judge stubbed to raise.Change
Completion is judged against the action the card assigned.
Hermes-Task-Role: review|implementationorHermes-Expected-Output: verdict. Directives are read from real newlines and from the literal\nescapes the board stores in one-line bodies - 6 of the 78 live cards carry no real newline at all, including the card that authored this change. Two conflicting directives fall back to the stricter implementation regime.reviewonly when the card declares a read-only scope, asks for a SHIP/BLOCK-shaped verdict, requests no mutation deliverable, and the report changed no files. Measured on a verbatim snapshot of every live card on three boards: 21 of 78 resolve to review, up from 10, with no implementation card misclassified.##heading and at the runner's goal-controller evidence, which is now emitted under its own heading. Prior attempts, parent results, cross-task history, comments and the controller's own rejection history can no longer reassign the role. This was a real defect: without itt_dbdcd739resolved toimplementationbecause a parent task's summary contained the phrase "the exact pushed SHA".SHIP/BLOCKverdict and evidence finalizes with no model call at all. Findings that make the reviewed artifact unfit to ship are the deliverable, not a task blocker.BLOCKwith no findings; a verdict with no findings/tests/artifacts; a summary too short to carry a rationale; blockers alongsidecompleted; a non-completedstatus. Every rejection reason is prefixed with the read-only guard, so a resumed review is never told to edit the reviewed implementation.review, even on a card that declaresHermes-Task-Role: review. It falls back to the model judge.unfinished_result_reason, and recordstask_role/task_role_sourcein the Kanban metadata.d8f2d818shipped.redact_sensitive_text, extended for the two shapes it deliberately passes through and this lane actually handles:scheme://user:secret@hostclone URLs and a credential named in prose. Redaction is anchored on surrounding syntax, so a 40-hex commit SHA - indistinguishable from a Gitea PAT in isolation - survives as review evidence. Reasons stay single-line, control-character-free and capped at 600 chars, and nothing mutates the reported result.Deliberate decisions, stated explicitly
Hermes-Task-Roleif deterministic finalization is wanted.role=Nonetherefore keeps the unfinished-work heuristic - PR #15's integrity check still works - and skips it only for a report that satisfies the whole review contract and changed no files. That is exactly the set the role-aware lane accepts as a review, so a record accepted under one version re-validates under every other instead of being quarantined into a re-dispatch of an already-accepted task.judge_goal_completionnow also rejects a non-completedstatus before reaching the model. Unreachable from the lane, which only calls it inside astatus == 'completed'branch, but it is a behaviour change for direct callers.Verification
PYTHONPATH=. pytest testing/tests -q-> 467 passed, 2 failed. Both failures are the pre-existingtest_hermes_auto_router/test_hermes_coordinatorfailures that reproduce identically on maind8f2d818.docs ok, smell ok, hygiene failed, unit failed, coverage okon both, with the identical four pre-existing >500 LOC files (none touched here) and the identical two unit failures.cli_lane_goal.pyat 100% statement and branch coverage (173 stmts, 66 branches), measured both with and without the agent runtime onPYTHONPATHso the canonical-redactor branch and its fallback are each exercised.F,B,SIM,C4,UP) clean on every lint path.make renderexit 0;kustomize build services/hermesexit 0;kubectl apply --server-side --field-manager=kustomize-controller --dry-run=server-> 113 objects, 0 errors.testing/tests/data/hermes_kanban_card_corpus.jsonlis a verbatim snapshot of all 78 cards on the live titan-iac, cassandra and soteria boards with a hand-labelled expected role per card, asserted one card at a time.PR #15 merge
git merge-treeagainst PR #15 head73fefbb5conflicts onservices/hermes/scripts/cli_lane_runner.py, because PR #15 reduces that file to a re-export shim and movesexecute_claimintocli_lane_execution.py. Resolution: take PR #15'scli_lane_runner.pyand port this PR's call-site change intocli_lane_execution.py- resolve the role withcli_lane_goal.task_role(context, structured), passrole=tounfinished_result_reason, and emit the controller evidence undercli_lane_goal.CONTROLLER_EVIDENCE_HEADING.Both resolutions were built and tested:
test_hermes_auto_router, twotest_hermes_cli_evidence_edgesretirement tests,test_hermes_coordinator_boards). No new failure.test_hermes_cli_review_roles.pythat assert the call-site behaviour. The merge fails loudly rather than silently regressing. Correctness does not depend on the port - thet_dbdcd739incident is fixed either way, because a role-blind call can no longer reject a complete review deliverable before the role-aware judge runs. What the port restores is the resolved-role metadata and the controller-evidence heading.Notes for the reviewer
test_hermes_cli_evidence_edgesretirement failures in the merged tree reproduce on PR #15's head alone and are not caused by this merge.hermes-coordinatorConfigMap exceeds the 256KBlast-applied-configurationannotation limit under a legacy client-sidekubectl apply. This reproduces identically on main and is pre-existing; Flux's kustomize-controller uses server-side apply, which dry-runs clean./runtime-access/gitea-tokenand echoed its value into their session output. Rotating that Gitea PAT would be prudent. Nothing in this change reads that file.Evidence:
/opt/data/workspace/evidence/t_6da029e0/🤖 Generated with Claude Code
hermes: finalize completed review verdicts in the goal judgeto WIP: hermes: finalize completed review verdicts in the goal judgeWIP: hermes: finalize completed review verdicts in the goal judgeto WIP: hermes: judge completion against the assigned task roleWIP: hermes: judge completion against the assigned task roleto hermes: judge completion against the assigned task role