atlas-iac/docs/hermes_full_handoff_acceptance.md

Hermes full-handoff acceptance harness and release runbook

scripts/ops/hermes_handoff_acceptance.py decides whether the Hermes platform handoff is fit to release. It is read-only by default, it classifies every check PASS / FAIL / NOT_RUN / NOT_APPLICABLE, and any mandatory FAIL or NOT_RUN makes the whole run NO_GO. Absence of evidence is never rounded up to evidence of absence: a probe that could not run, a catalog entry whose evidence no longer exists, an expired deadline, or a harness bug all land in NOT_RUN and stop the release.

scripts/ops/hermes_handoff_acceptance.py \
  --context atlas-operator \
  --node-count 21 \
  --expected-suspension flux-system/bstein-dev-home-migrations \
  --output build/hermes-handoff-acceptance.json

Exit status is 0 for GO and 1 for NO_GO. The JSON report goes to --output (or stdout); a short human summary goes to stderr unless --json-only is passed.

Why two vantages

Every authority claim is checked from two places that cannot cover for each other:

Vantage	What it is	What it proves
operator	An external, read-only kubeconfig	Desired state: bindings, manifests, workloads, Flux health, forge state
self	kubectl exec into the running hermes-agent container	Enforced state: what the agent's own service-account token can actually do
switchyard, node, chat	kubectl exec into the Switchyard, node-hardening, and chat pods	Evidence that only exists inside those workloads

Two rules keep this honest:

• The harness cannot impersonate from outside. --as, --as-group, and --as-uid are rejected by policy for any operator-side command. They are reachable only as the inner command of a kubectl exec, where the identity is testing its own refusal to impersonate rather than borrowing authority. kubectl auth can-i --as is therefore not available as a substitute for a real self-probe.
• The two vantages must be different principals. Before any check runs, the harness asks each vantage kubectl auth whoami and compares. If the operator and the in-pod probe authenticate as the same principal, the dual-vantage evidence is a restatement rather than a corroboration, and the run is NO_GO with the operator and in-pod vantages authenticate as the same principal. Run the harness from a genuinely external kubeconfig — not from inside the Hermes owner pod.

Deny checks additionally require a real refused request, not only an authorization review. kubectl auth can-i reports what RBAC says; only an attempt proves the API server enforces it. Mutating attempts go through --dry-run=server: the API server authorises the request and then discards it, so a refusal is genuine and nothing is written either way.

Safety properties

These hold structurally, in hermes_handoff_policy.py, so a catalog mistake is a policy error before a run rather than a mutation or a leak during one.

• No mutation in the default mode. Only read subcommands are reachable; apply, create, delete, patch, replace, and scale require a dry-run flag. git push and every non-GET forge method are refused outside the armed mode. There is deliberately no live TokenRequest probe — a successful one would mint a real credential.
• No credential is ever read. vault, sops, age, gpg, openssl, curl, and wget are not on the binary allowlist. Secrets may only be read with -o name. Environment probes list variable names and never values. Provider health files are read through a frozen template that projects an allow-listed set of non-secret status fields. The node account's lock state is reported without reading its password hash; its authorized keys are counted, never printed.
• Shell is frozen. sh -c is reachable only through the reviewed, parameter-validated templates in SHELL_TEMPLATES. Parameters may not contain shell metacharacters.
• Output is bounded and screened. Each capture is truncated to a byte budget before it is screened, so a large dump cannot outrun the screen. Bulk evidence (the routing-log tail, the Kanban listing) is parsed but replaced with [output not recorded] in the artifact. The screen preserves JSON quoting, so screened structured output is still parsable.
• The report is re-checked before it is published. unscreened_fields walks the rendered payload; if anything still looks credential-shaped the report is replaced with a NO_GO stub naming the offending pointers, and the file that would have leaked is never written.

What the catalog asserts

The catalog is declarative (hermes_handoff_catalog.py plus the four hermes_handoff_checks_*.py modules) so a reviewer reads what is asserted, not how it is plumbed. 71 checks in 14 groups:

Group	Assertions
baseline	origin/main descends from the merged worker-isolation baseline; every dependency pull request is merged into main; the exact open pull-request heads are recorded
identity	Codex authenticates as codex-chatgpt-subscription; Claude as firstParty / claude.ai / claude-code-cli-subscription; both evidence files are fresh; no provider API-key variable exists in the running process or in the manifest
routing	Switchyard has a ready replica; the routing log covers both providers, the route and worker lanes, medium/high/xhigh effort, and at least one real fallback; the newest record is fresh; both providers record a measured latency
scopes	Chat, agent, and triage each declare their own HERMES_AUTO_ROUTER_PROFILE, on distinct hostnames
access-denied	No cluster-admin binding for the agent; Secrets, service-account tokens, impersonation, workload mutation, arbitrary pod exec, attach, port-forward, and control-plane writes are all refused
access-allowed	get/list/watch on core objects, pod logs, and Flux/Helm status still work; the operator and in-pod namespace inventories agree
gitops	No unexpected Kustomization or HelmRelease suspension, and nothing unhealthy
forge	No forge credential in the worker environment or home; the forge identity is not an administrator; repository authority is pull/push only; an administrative route is live-refused; clone/fetch still works
nodes	The dedicated hermes-agent account is locked, has the expected uid/gid/home/shell, joins no supplementary group, has no sudoers fragment, holds exactly one authorized key; K3s credentials, K3s state, kubelet state, and the container runtime are root-only; the audit covers every node
build	The image builder service account mounts no token; the build pod is unprivileged with exactly CHOWN, FOWNER, DAC_OVERRIDE, SETGID, SETUID; the Harbor immutable-tag job succeeded; the running image is digest-pinned
reliability	The finalization/replay/session patches are running in the pod; the regression suites are present on main
pool	Three ready workers, on distinct nodes, with distinct volumes, mounting no API token and none of the coordinator's claims
surfaces	Kanban activity and durable sessions are visible and render without blank rows; the agent and chat session patches are deployed; Telegram topic state is on durable storage
ephemeral	Feature-branch push, draft pull request, protected-branch refusal, and verified cleanup — NOT_RUN unless armed

Site configuration

A few checks are emitted only when the operator declares what to expect, because a names_present assertion over an empty expectation would pass on any workload at all — which reads as coverage without being any:

Flag	Enables
--pool-worker-env NAME (repeatable)	pool.assignment-safety-knobs-are-configured
--chat-config-revision REV	surfaces.chat-runs-the-telegram-topic-revision
--expect-telegram-sessions with --chat-ordinal N	surfaces.chat-telegram-sessions-are-continuous

--node-count must be the real node count; the default of 1 makes the fleet-coverage check vacuous.

Catalog drift is a NO_GO, not a pass

If a workload, field, or manifest is renamed, the affected checks report NOT_RUN because the evidence they name is no longer there. That is the intended behaviour: the release stops until either the platform or the catalog is corrected. Do not "fix" a NOT_RUN by deleting the check.

Ephemeral mutation mode

Proving that a worker can still open a draft pull request means actually opening one. That is the only part of the harness that writes anything, and it is a separate mode:

scripts/ops/hermes_handoff_acceptance.py \
  --context atlas-operator \
  --arm-ephemeral-push \
  --confirm "ARM EPHEMERAL HERMES HANDOFF PUSH" \
  --ephemeral-token acceptance-20260817a \
  --output build/hermes-handoff-acceptance-armed.json

• The confirmation phrase must match exactly.
• --ephemeral-token is 8–64 lowercase alphanumeric/dash characters and must be unique per run; the ref is always ephemeral/hermes-handoff-acceptance/<token>.
• Preflight runs before any network call: wrong phrase, wrong repository, malformed token, or a protected push target all abort with arming refused and a NO_GO report. main, master, HEAD, develop, release, production, stable, trunk, prod, and default are refused by name (case- and refs/heads/-insensitive), and anything that is not an ephemeral acceptance ref is refused by pattern.
• The guard is exercised on every armed run by ephemeral.protected-branch-refusal, rather than trusting that it was checked once.
• The push target is guarded; the pull-request base is not. Opening a draft pull request against main writes nothing to main. The draft title always starts with WIP: .
• Cleanup closes the pull request, deletes the ref, and then re-reads both to confirm they are gone. ephemeral.cleanup-verified is mandatory: a half-removed branch fails the run rather than passing it. Cleanup runs even when an earlier step failed.
• git push --force, --force-with-lease, --mirror, --all, and --tags are refused even in armed mode.

A default run reports all four ephemeral checks NOT_RUN and non-mandatory, so a read-only sweep can still reach GO.

Release order

Baseline, already merged and deployed

PR #13 (hermes: harden worker isolation and blocked-task semantics) merged as ab346f55509d584e457fe26cf90be3078f7a375c. Every check below assumes it is present; baseline.origin-main-descends-merged-work asserts it. Override with --baseline-commit only when a later merge legitimately replaces it.

Pending work, observed 2026-08-17T10:07Z

These heads move. The harness re-fetches them at run time via baseline.pending-pull-requests-recorded; the table is the snapshot this document was written against, not an input.

PR	Branch	Head at observation	State	Delivers
#14	feature/hermes-safe-gitea-pr-client	3cd8a17e772978fb554d88dd1c59777786be3991	draft	Bounded forge authority, observer RBAC (removes hermes-agent-cluster-admin), node account hardening
#15	fix/hermes-result-decomposition-reliability	7609cd77e87a64e332b2f0d670c1d7a4f62b3d9e	draft	Result preservation, duplicate-fanout prevention, execution-safety patch
#16	feature/hermes-unprivileged-image-builder	5ab0eba619779b3dfed2eb09d5c02f0a9c92dd0a	ready	Daemonless Kaniko build lane, tokenless builder, Harbor immutable-tag rule
#17	feature/hermes-three-lane-placement	48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c	draft	Three concurrent CLI lanes on the larger ARM64 nodes
#18	feature/hermes-distributed-worker-pool	20002527512235b29054f3636b02418168373009	draft	Three-node fenced execution pool with coordinator-only state ownership

Base at observation: 30259b526be5b2e226620aeebd9fb5f57dc88b7d.

Merge order and why

1. #14 first. It is the only change that reduces authority: it deletes services/hermes/agent-rbac.yaml (the cluster-admin binding) and replaces it with the namespaced/cluster observer roles, and it hardens the node account. Merging it last would mean every earlier acceptance run measured a cluster-admin agent. Every access-denied and nodes check depends on it.
2. #16 second. The image lane must exist before anything that needs a new agent image is rolled out. It also touches testing/quality_contract.json, so merging it early keeps later rebases small. build.* depends on it.
3. #15 third. It changes dockerfiles/Dockerfile.hermes-agent, which #16 also touches — take #16's shape and rebase #15 onto it. reliability.* depends on it.
4. #17 fourth. Placement only; it edits services/hermes/agent-deployment.yaml and testing/tests/test_hermes_cli_lanes.py, both of which #14, #15, and #18 also touch. Merging it after them avoids resolving the same conflict twice.
5. #18 last. The pool depends on the bounded forge client from #14, the execution-safety patch from #15, and the agent image from #16. pool.* depends on it.

Overlapping files to expect (rebase, do not merge blind): services/hermes/agent-deployment.yaml (#12, #14, #16, #17, #18), services/hermes/kustomization.yaml (#14, #16, #18), services/hermes/scripts/stage_runtime_access.py (#14, #16, #18), services/vault/scripts/vault_k8s_auth_configure.sh (#14, #16, #18), dockerfiles/Dockerfile.hermes-agent (#15, #16), testing/tests/test_hermes_cli_lanes.py (#14, #15, #17, #18), testing/tests/test_hermes_runtime_access.py (#16, #18).

PR #12 (AI usage exporter) is independent and can merge at any point; it touches services/hermes/agent-deployment.yaml, so land it before #17 if convenient.

Image build and Flux rollout

1. Merge in the order above; observe the default-branch Jenkins build to a terminal result after each merge (/opt/coordinator/jenkins_build_evidence.py titan-iac --branch main --wait).
2. After #16 is merged, run the hermes-agent-image pipeline. It publishes an immutable git-<sha>-build-<n> tag and emits build/hermes-agent-release/hermes-image-update.patch.
3. Apply that patch to services/hermes/kustomization.yaml on a branch, review, and merge. The digest is what pins the deployment; build.deployed-image-is-digest-pinned asserts it.
4. Let Flux reconcile. Confirm with flux get kustomizations -A that hermes and hermes-chat are Ready, then wait for the rollout to settle.
5. Re-run the harness. Do not run it mid-rollout: gitops.* tolerates an in-flight reconcile (Reconciling=True with Ready=False) but a half-rolled deployment will legitimately fail the identity, routing, and surfaces groups.

Rollback points

After	Roll back by	Restores
#14 merge	Revert the merge commit; Flux restores hermes-agent-cluster-admin	Cluster-admin agent, previous node account state
#16 merge	Revert the merge; the previously pinned digest is unchanged in kustomization.yaml	Previous build lane; published immutable tags are not deleted
Image digest bump	Revert the kustomization.yaml commit to the prior digest	Previous agent image; Harbor immutability means the old tag is still exactly what it was
#15 merge	Revert the merge; the init-container patch disappears on the next rollout	Previous finalization/replay behaviour
#17 merge	Revert the merge	Previous lane concurrency and node placement
#18 merge	Revert the merge; the StatefulSet and its PVCs are removed by Flux pruning	Single-coordinator execution

Every rollback is a Git revert reconciled by Flux. Do not roll back by direct kubectl edit; the next reconcile would undo it.

Go / no-go checklist

Run in order. Stop at the first item that does not hold.

1. origin/main is ab346f55509d584e457fe26cf90be3078f7a375c or a descendant.
2. PRs #14, #15, #16, #17, #18 are all merged into main, in that order, each with a green default-branch build.
3. The agent image has been built by the #16 lane, is pinned by digest in services/hermes/kustomization.yaml, and Flux has finished reconciling.
4. The harness is run from an external operator kubeconfig, not from inside the Hermes owner pod, with --node-count set to the real node count and every deliberate Flux suspension declared via --expected-suspension.
5. The default read-only run reports GO — no mandatory FAIL, no mandatory NOT_RUN, no harness_errors.
6. The armed run reports PASS for all four ephemeral.* checks, including ephemeral.cleanup-verified, and the ephemeral ref and pull request are confirmed gone.
7. The JSON report is archived with the exact origin/main commit and the recorded pull-request heads from baseline.pending-pull-requests-recorded.

Only then is the handoff releasable.

Known limits

Stated so nobody mistakes a proxy for the thing itself.

• Merge, approve, and close authority is asserted, not exercised. The harness never attempts a merge. It proves the forge identity holds permissions.admin: false and is live-refused on an administrative route; the route-level allowlist is covered by PR #14's own regression suite.
• Attach and port-forward have no side-effect-free live attempt. Their authorization reviews ride alongside a real, refused control-plane write in the same check, and pod exec — the one subresource with a harmless real attempt — has its own check.
• TokenRequest is asserted by review. A successful live attempt would mint a real credential, so the check pairs the serviceaccounts/token review with a refused serviceaccount create in the same RBAC family.
• Branch protection is not read. The harness identity is not a repository administrator, so GET /branch_protections is refused — which is itself recorded as evidence. Protected-push refusal is guaranteed by the harness's own preflight, exercised on every armed run, not by reading the forge's config.
• Stale/duplicate assignment safety is asserted as deployed configuration. The behaviour itself is covered by the pool's regression suite; the harness checks that the workers carry the configuration and that the coordinator keeps sole ownership of its claims.
• Telegram session continuity is per tenant and depends on there having been Telegram traffic, so it is off by default. The always-on check is that topic and tenant state live on a persistent claim rather than in the pod.
• The routing window is a tail, 600 records by default. A property that did not occur inside that window is reported as missing rather than absent; widen the window or wait for traffic rather than removing the assertion.

Tests

python3 -m pytest testing/tests/ -k handoff
python3 -m coverage run --source=scripts/ops --include="*/hermes_handoff_*.py" \
  -m pytest testing/tests/ -k handoff && python3 -m coverage report

The suite covers parsing, redaction, timeouts and deadlines, partial failure, zero state, dual-vantage mismatch, mandatory skips, catalog structure, and ephemeral arming and cleanup — including the cases where cleanup cannot be verified.