titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs: record multi-pass suite acceptance and progress results

Browse Source
This commit is contained in:
jenkins 2026-09-29 14:28:05 -05:00
parent 9c327c6fd0
commit 0382ac463d
2 changed files with 14758 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14714
docs/evidence/hermes_suite_multipass_20260929.json Normal file
View File

File diff suppressed because it is too large Load Diff

50
docs/hermes_suite_multipass.md
View File

@ -188,12 +188,13 @@ fix. Wait for active jobs before a worker restart because completed results are
held in memory. No Vault credential or ingress/routing changes are required. held in memory. No Vault credential or ingress/routing changes are required.
The policy implementation is commit `7af4f4f2`; the approved 20-minute / USD 10 The policy implementation is commit `7af4f4f2`; the approved 20-minute / USD 10
estimate guard and progress reporting are commit `15c05cb7`. To restore the prior estimate guard and progress reporting are commit `15c05cb7`. Required-alias
single-pass implementation after active jobs finish, revert both through the normal structured output is commit `9c327c6f`. To restore the prior
single-pass implementation after active jobs finish, revert these through the normal
Git deployment branch, preserving the earlier six-turn diagnostics fix: Git deployment branch, preserving the earlier six-turn diagnostics fix:
```bash ```bash
git revert --no-commit 15c05cb7 7af4f4f2 git revert --no-commit 9c327c6f 15c05cb7 7af4f4f2
git commit -m "hermes: restore prior suite grouping policy" git commit -m "hermes: restore prior suite grouping policy"
git push origin HEAD:main git push origin HEAD:main
flux reconcile kustomization hermes --namespace flux-system --with-source flux reconcile kustomization hermes --namespace flux-system --with-source
@ -218,12 +219,22 @@ This verifies that LAN test path; it does not replace a WSL connectivity test.
| 14 | 9,847 | 61.805 | 3 / 6 | 9 | 9 | 4 / 4 | 18,216 / 5,794 | | 14 | 9,847 | 61.805 | 3 / 6 | 9 | 9 | 4 / 4 | 18,216 / 5,794 |
| 75 | 55,984 | 274.620 | 5 / 12 | 9 | 21 | 3 / 3 | 174,942 / 28,950 | | 75 | 55,984 | 274.620 | 5 / 12 | 9 | 21 | 3 / 3 | 174,942 / 28,950 |
| 38 balanced fixture | 23,871 | 231.337 | 5 / 15 | 4 | 10 | 0 / 0 | 139,789 / 24,681 | | 38 balanced fixture | 23,871 | 231.337 | 5 / 15 | 4 | 10 | 0 / 0 | 139,789 / 24,681 |
| 363, current required-alias schema | 273,763 | 842.777 | 5 / 12 | 9 | 75 | 3 / 3 | 983,491 / 91,932 |
| 363, old limits | 273,761 | 747.513 | 3 completed; pass 4 failed | Not finalized | No result | Not finalized | 629,141 / 64,424 | | 363, old limits | 273,761 | 747.513 | 3 completed; pass 4 failed | Not finalized | No result | Not finalized | 629,141 / 64,424 |
Token totals include all completed CLI turns across the job, not unique source Token totals include all completed CLI turns across the job, not unique source
size. The old-limit 363 result includes reported failed-invocation usage and was size. The old-limit 363 result includes reported failed-invocation usage and was
never returned as completed. CLI estimated costs were 0.23593, 1.59846, 1.31597, never returned as completed. The earlier 14/75/38 and old-limit 363 CLI cost
and 5.62765 respectively; these are not subscription bills. estimates were 0.23593, 1.59846, 1.31597, and 5.62765 respectively; these are not subscription bills.
The current 363-case run reported a CLI cost estimate of 7.215755 and a complete
response envelope of 122,115 bytes. Its five passes took 80.962, 98.247, 350.114,
148.165, and 165.211 seconds, with 2/2/4/2/2 CLI turns. Six coherent 60-case
families became twelve five-case tasks each, plus three genuine singletons.
The 135 authorized status samples covered all five stages and recorded 59 changes
in CLI output activity. No compaction or truncation was reported. No terminal rate-limit
error or gateway job retry was reported; structured-output CLI turns are recorded
separately and do not imply provider-network retries.
For every completed run above, natural-family pair precision and recall were For every completed run above, natural-family pair precision and recall were
1.0 against the independently defined implementation patterns, exact alias 1.0 against the independently defined implementation patterns, exact alias
@ -240,7 +251,7 @@ It yielded natural sizes 6/7/11/14 and final sizes 3+3 / 4+3 / 4+4+3 / 5+5+4.
Thirty-eight authorized status samples showed five stages, changing heartbeats, Thirty-eight authorized status samples showed five stages, changing heartbeats,
and nineteen CLI activity changes without exposing event bodies. and nineteen CLI activity changes without exposing event bodies.
The actual 14/75/38 independent proposals agreed on membership. Disagreement The actual 14/75/38/363 independent proposals agreed on membership. Disagreement
reconciliation, genuine semantic subdivision, reversal of an unjustified split, reconciliation, genuine semantic subdivision, reversal of an unjustified split,
and naming-collision rejection are separately tested with controlled mocked and naming-collision rejection are separately tested with controlled mocked
responses; do not describe those as observed live-provider disagreements. responses; do not describe those as observed live-provider disagreements.
@ -274,3 +285,30 @@ in CLI estimated-cost accounting. The previous validator did not retain counts o
missing, repeated, or unknown assignments, and transient output was cleaned; the missing, repeated, or unknown assignments, and transient output was cleaned; the
specific mismatch cannot be recovered. No partial partition was accepted. The new specific mismatch cannot be recovered. No partial partition was accepted. The new
required-alias schema and count-only failure diagnostics address this failure mode. required-alias schema and count-only failure diagnostics address this failure mode.
The updated native CLI loopback test deliberately omitted one required alias from
its first 14-case structured output. The installed CLI rejected that tool payload
and requested a correction, then the complete job passed. This demonstrates actual
schema-repair behavior within the existing six-turn headroom. The updated
14/75/363 transport runs made 4/5/5 mock provider requests across 3/5/5 model passes;
all transmitted the full source and system instructions. No hosted inference was
used for this transport/repair test.
The 14/75 and balanced-38 live measurements preceded the required-alias internal
serialization change. The final serialization was exercised by native loopback at
all three public sizes and by the successful hosted 363-case five-pass job. The
engineering objective, model, effort, cap, and public result shape stayed the same.
The complete suite is supported directly for the tested 273,763-byte request.
This is evidence for this fixture, not a guarantee that every 363-case request
will fit the same time/output allowance. There is no remaining blocker for starting
a new laptop attempt with the approved Claude credential and adequate job limits.
The result still requires the client's independent coverage audit and engineering
review. No real job was run by these checks.
Full synthetic results, review summaries, safe per-pass diagnostics, input/response
sizes, source-field length distributions, failed attempts, and native transport
measurements are in [the acceptance evidence](evidence/hermes_suite_multipass_20260929.json).
Routine planner logs contained only allowed operational fields, the completed
job's SQLite metadata contained no result/review/case content, and temporary CLI
job directories were empty after completion.