atlas-iac/docs/hermes_full_handoff_acceptance.md

# Hermes full-handoff acceptance and release runbook

`scripts/ops/hermes_handoff_acceptance.py` is a fail-closed release gate. It
contains 63 checks in 15 groups. An empty catalog, an unknown result status, a
mandatory `FAIL` or `NOT_RUN`, a harness error, or incomplete evidence makes the
decision `NO_GO`. The four ephemeral checks are non-mandatory `NOT_RUN` in the
default mode and are replaced with mandatory results only in armed mode.

This runbook describes evidence collection. It does not authorize a merge,
publish, deployment, Flux reconcile, credential read, or live cluster mutation.
Human review of PR #19 and of the resulting report is required.

## Required release inputs

Every value that can go stale must be supplied explicitly. Resolve `main`, PR
heads, the image digest, build source SHA, deployment revision, and chat config
revision from their authoritative read-only sources immediately before the run.
Do not copy the snapshot in the dependency section into a later run.

```sh
scripts/ops/hermes_handoff_acceptance.py \
  --context atlas-operator \
  --remote-main-sha "$REMOTE_MAIN_SHA" \
  --reviewed-head-sha "$REVIEWED_PR19_HEAD" \
  --agent-image "$AGENT_GIT_SHA_BUILD_TAG_AT_SHA256" \
  --build-sha "$IMAGE_BUILD_SOURCE_SHA" \
  --deployment-revision "$RUNNING_DEPLOYMENT_REVISION" \
  --chat-config-revision "$RUNNING_CHAT_CONFIG_REVISION" \
  --dependency-head "12=$PR12_HEAD" \
  --dependency-head "14=$PR14_HEAD" \
  --dependency-head "15=$PR15_HEAD" \
  --dependency-head "16=$PR16_HEAD" \
  --dependency-head "17=$PR17_HEAD" \
  --dependency-head "18=$PR18_HEAD" \
  --dependency-head "20=$PR20_HEAD" \
  --output build/hermes-handoff-acceptance.json
```

The fixed topology is three pool workers on three distinct nodes. `--node-count`
and `--pool-replicas` cannot lower that requirement. Pool node/ordinal assignment,
Telegram session continuity, and the chat revision are mandatory. Deliberate Flux
suspensions must be named with repeatable `--expected-suspension` arguments.

Exit status is `0` only for `GO`, `1` for `NO_GO`, and `2` for invalid execution
bounds. JSON is written to `--output` or stdout; the human summary goes to stderr
unless `--json-only` is passed.

Operator prerequisites for the forge evidence:

- The SCM broker (`hermes-scm-broker.hermes-scm.svc:9081`) must be healthy;
  every forge read is brokered and a dead broker makes all broker-backed
  mandatory checks `NOT_RUN`.
- No forge token anywhere on the harness path: the harness, the agent pod, and
  the operator never read `/vault/secrets/gitea-token` — only the broker holds
  it. There is no askpass helper to install.
- The agent pod must mount the current `hermes-scm-boundary-v2` ConfigMap:
  the harness attests `/opt/scm/gitea_api.py` against the digest of the merged
  `services/hermes/scm-common/scripts/gitea_api.py`, so a stale mount (or a
  stale broker rollout) fails loudly instead of answering.
- Operator-vantage broker reads need the same pinned client (plus its sibling
  modules) at `/opt/scm` on the operator host, with the broker origin
  resolvable; `git ls-remote origin refs/heads/main` must work from the
  operator checkout with repo-local access, since global Git config is nulled.

## Structural safety boundary

The catalog is validated before a `Runner` exists. Invalid inputs, an empty or
unsafe catalog, and an invalid armed preflight therefore create no subprocess.
The command policy is checked both before and after executable resolution.

Default mode permits a narrow read grammar:

- Kubernetes reads use `name` or reviewed JSONPath projections. Secret resources,
  raw API reads, full JSON/YAML, `describe`, `cluster-info dump`, credential-bearing
  projections, `auth reconcile`, config operations, all mutations, and every
  dry-run mutation trick are rejected before spawn.
- Git is read-only. `fetch`, configuration/helper overrides, alternate worktrees,
  external diff helpers, and push are rejected.
- Forge evidence is collected through the credential-isolated SCM broker
  client that ships in the agent pod at `/opt/scm/gitea_api.py` (ConfigMap
  `hermes-scm-boundary-v2`, generated from
  `services/hermes/scm-common/scripts/gitea_api.py`). Its only admitted
  grammar is `read <api-path>`; bare HTTP methods (`GET`, `POST`, ...) are not
  part of the deployed client's grammar and are refused before spawn. Paths
  are pinned to `titan/atlas-iac` on an exact segment boundary (plus the bare
  `/api/v1/user` identity read), so a look-alike repository such as
  `titan/atlas-iac-evil` is refused, and any relative segment — including a
  percent-encoded `%2e%2e` — is rejected before the request is built. The
  broker holds the only forge token; the harness, the pod, and the operator
  environment never read `/vault/secrets/gitea-token` and no Git askpass
  helper exists or is configured.
- Shell commands must be rendered from frozen templates, use fixed executable
  paths, and cannot name service-account, Vault, runtime-access, SSH, or other
  credential roots.
- Impersonation is structurally impossible. `--as`, `--as-group`, and `--as-uid`
  are refused for every argv, in every mode, from every vantage, and the inner
  command of a `kubectl exec` is re-checked under the same rule rather than being
  granted an exemption. `validate_catalog` refuses an impersonating step from any
  vantage, not only the operator. A future audited self-probe that genuinely needs
  impersonation is a reviewed code change, not a configuration flag.
- The binary allowlist is exactly the set with pinned release digests:
  `kubectl`, `git`, `sh`, `hermes`, and the Gitea client. `flux` and `helm` are
  not on it; Flux and Helm status is read through `kubectl` instead, so no
  allowlisted binary can be admitted that the executor would refuse to attest.
- `PATH` is replaced with a fixed path. Every outer executable is resolved to an
  allowed real path, checked for unsafe write mode, and compared with a pinned
  SHA-256 digest. A caller-authored `kubectl`, `git`, or helper cannot fabricate
  evidence.

The executor uses a new process group, drains stdout and stderr concurrently,
and retains at most the configured bytes while the child is running. A monotonic
absolute timeout terminates and reaps the group, including descendants that keep
pipes open. Timeout, deadline, and output-byte values reject booleans,
zero/negative values, non-finite numbers, and values outside fixed bounds. There
is no concurrency knob: `run_catalog` is strictly sequential and says so rather
than accepting a bound it would ignore.
Truncation discards the affected stream and makes every evaluator `NOT_RUN`.
Bulk-evidence steps have their bytes replaced before they are recorded, but they
keep the resolved executable path and its attested SHA-256, so withholding output
never withholds the provenance of the binary that produced it.

Credential screening runs before truncation. Command, output, error, identity,
reason, evidence, JSON, and human-summary fields are screened recursively,
including mapping keys and short/numeric sensitive values. The complete rendered
report is screened again; a remaining credential shape is replaced by a bounded
`NO_GO` stub rather than being written. No credential prefix is retained at a
truncation boundary.

## Vantages and evidence

The operator must use an external read-only kubeconfig. The self probe runs in
the actual Hermes agent pod. Both identities are obtained with
`kubectl auth whoami`; equal or missing principals make the run `NO_GO`. Required
Switchyard and chat/Telegram pod vantages must also resolve. Node coverage is
derived from all daemonset pod node names, not one selected pod.

Dangerous Kubernetes permissions are checked only with side-effect-free
`kubectl auth can-i` reviews. The default catalog never constructs a Secret,
TokenRequest, mutation, attach, port-forward, or arbitrary exec request.

Forge evidence rides the broker path. The SELF-vantage forge checks prove the
*broker's* identity — the only forge identity the platform can exercise — is
not an administrator (`read /api/v1/user`), holds push-scoped rather than
administrative repository authority, and is refused an administrative route:
the broker's read allowlist rejects `branch_protections`, and only the
client's real refusal lines ("SCM broker request failed with HTTP 400/403" and
"Forgejo request rejected or unavailable; no credential was disclosed") count
as denial evidence; generic `404`, `not found`, and `no route` text are not.
Because every other broker-backed mandatory check must pass in the same run, a
dead broker reads as `NOT_RUN` across the catalog and cannot masquerade as a
refusal. Operator-vantage broker reads (pending PRs, dependency PRs, the
reviewed PR) additionally require `/opt/scm/gitea_api.py` with its sibling
modules present on the operator host at the pinned digest and the broker
origin `hermes-scm-broker.hermes-scm.svc` resolvable from it; the `remote-main`
step needs `git ls-remote origin refs/heads/main` to work from the operator
checkout using repo-local access (the harness nulls global Git config and
disables prompts).

An absence is only evidence when the probe observed something. Both name rules
resolve their step through one guard, so a step that exits `0` with no lines — or
with only separators — is `NOT_RUN`, never `PASS`. This is reachable in practice
two ways, and both are pinned by regressions: a POSIX pipeline reports the status
of its *last* stage, so a missing upstream binary in the frozen `env_names`
template exits `0` with no bytes; and `kubectl -o jsonpath` exits `0` when the
structure it walks has drifted, whether a parent field, a leaf field, or a
container filter. Because all five absence checks are mandatory, silence costs
the release its `GO`. The pool claim projection emits one `<volume>=<claim>` line
per template volume so a volume without a PVC still counts as an observation.

The strict evaluators reject duplicate-key JSON, raw control characters,
`NaN`/infinity, boolean/number equality coercion, future timestamps, stale
provider evidence, and every malformed routing or Flux row. Routing freshness is
required separately for each provider, lane, and effort. Flux objects must be
non-suspended, current at `observedGeneration`, and explicitly `Ready=True`;
unknown or reconciling state is not healthy.

`release.exact-lineage-is-running` atomically binds:

- fixed remote `main` SHA and local `origin/main`;
- merged PR #19 — `state=closed` with `merged=true`, base `main`, the existing
  feature ref, and the exact reviewed head — with that head additionally proven
  an ancestor of the release `main` by `git merge-base --is-ancestor`. A PR
  that is still open, closed without merging, or carrying a different head is a
  `FAIL`: after integration, only the merged terminal state proves the reviewed
  code is what runs;
- the exact `:git-<source-SHA>-build-<n>@sha256:<digest>` image identity,
  with the source SHA equal to the release `main` SHA;
- current Deployment generation and revision;
- the one active ReplicaSet revision/hash; every Ready pod image and imageID;
- the Flux-applied `main` revision.

The pool evaluator requires exactly ordinals 0, 1, and 2, all Ready, on three
distinct nodes and claims, with service-account token automount disabled. The
chat checks always require durable Telegram sessions and the exact running chat
configuration revision.

## Armed ephemeral mode

Armed mode is the only write path and runs only after the complete default
catalog reports `GO`.

**Armed mode is currently deferred.** The deployed broker client speaks only
`read` and `create-draft`: it has no close-PR or delete-branch operation, the
armed `git push` has no credential path (no askpass exists and the broker git
remote is not part of the harness), and the policy's armed mutation windows
therefore describe probes the live platform cannot execute. Run armed mode
only after the ephemeral probes are rebuilt on the broker's bounded mutation
surface; until then the four ephemeral checks stay non-mandatory `NOT_RUN` in
the default mode and an armed run fails closed at the client.

```sh
scripts/ops/hermes_handoff_acceptance.py \
  <all required release inputs above> \
  --arm-ephemeral-push \
  --confirm "ARM EPHEMERAL HERMES HANDOFF PUSH" \
  --ephemeral-token acceptance-20260817a \
  --output build/hermes-handoff-acceptance-armed.json
```

Before any runner, network request, or cluster vantage is created, preflight
binds the working directory to the existing linked PR #19 worktree, the exact
feature branch and reviewed local HEAD, the fixed HTTPS Atlas origin, base
`main`, confirmation phrase, and one unique ephemeral ref. The command policy is
then pinned to that exact ref. It allows only its push, one exact draft-create
payload, cleanup of the exactly discovered PR number, and deletion of that ref.

The run reserves at least 300 seconds (up to 6000) for cleanup. It verifies the
pushed SHA, then identifies the pull request from two independent sources: the
create response itself, and the pull index read page by page to its last page
(50 per page, 20 pages maximum) rather than a single truncated first page. Either
source alone names a pull request this run definitely made, because preflight
proved the ephemeral ref did not exist. Every number found from either source is
registered and closed, so a failed or unusable index can no longer leave an
orphan open. A `GO` still requires exactly one open draft with the expected base,
head ref, base SHA, and head SHA, and the two sources must agree.

Cleanup requires successful PR close, successful branch deletion, a successful
empty `ls-remote`, and a strict re-read of the same closed PR. Malformed or
partial cleanup is `FAIL`; it is never inferred from an error. When anything is
uncertain the `draft-pull-request` evidence carries `residue_ref` and an exact
`manual_cleanup` action list — closing every numbered pull request and deleting
the ephemeral branch with the operator's own forge credentials (the broker
client exposes no close or delete operation), and, when no number could be
determined at all, the exact ref to sweep the index for by hand.

Force, force-with-lease, fan-out, protected refs, other repositories, other PR
numbers, arbitrary PATCH payloads, and repository administration remain denied.

## Dependency, merge, and conflict graph

Authoritative snapshot observed 2026-08-17 after refreshing Gitea and Git against
remote `main` `d8f2d818b9a552ea6c2d7fe86554be829bd5ffff`:

| PR | Head SHA | Draft | Base | Role |
| --- | --- | --- | --- | --- |
| #12 | `ed43dbaa7f094e1841aa760862e80015deb9d80c` | no | `main` | AI usage exporter repair |
| #14 | `8c6e3acdacb223f68b1716f0a74bd543e18890e6` | yes | `main` | bounded forge client and least-privilege RBAC |
| #15 | `73fefbb5d92ef07371a9a997f22d848acd2d06ed` | yes | `main` | result-decomposition reliability |
| #16 | `e8c26ecf85a0c802d378fffffb98b7df43b78242` | no | `main` | unprivileged image builder |
| #17 | `48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c` | yes | `main` | three-lane placement |
| #18 | `689bcb6e48414854447988ba174143a307d308b9` | yes | `main` | distributed three-worker pool |
| #19 | `c808baff40a12a90f43c2e5e39b94d139f88ea29` | yes | `main` | head this repair replaces on the same branch |
| #20 | `a242dcc786576ae1a18000cb4941836ff610dff2` | yes | `main` | CLI process-group reaping |

The repaired #19 SHA cannot be embedded in its own commit without changing that
SHA. Resolve it from Gitea after push and require it as `--reviewed-head-sha`;
the PR handoff records the exact value. All eight heads in this table, including
the repaired #19, merge cleanly with `main` individually. Pairwise
`git merge-tree --write-tree` over all 28 combinations found conflicts for:
#12+#17, #12+#18, #12+#19, #14+#15, #14+#16, #14+#17, #14+#19, #14+#20, #15+#16,
#15+#17, #15+#19, #16+#19, #16+#20, #17+#18, and #17+#20. Gitea's
`mergeable=false` on a draft is not treated as conflict evidence.

This graph differs from the previous snapshot in exactly two ways. #12 and #18
both advanced, which adds #12+#18, #17+#18, and #12+#19. And #19 no longer
duplicates the repo-wide hygiene legacy-exception mechanism that #18 also
carries, which makes #18+#19 clean. Recompute before every integration step;
nothing here is durable.

Use this dependency order, refreshing every head and re-running merge-tree at
each step:

1. #14 first so the release never restores or prolongs cluster-admin authority.
2. #16, then #15, resolving their Dockerfile and quality-contract changes
   additively. Never take one PR's quality contract wholesale.
3. #20 after #15; it depends on the CLI lifecycle work and must be rebased across
   the current #14/#16 contract and helper changes.
4. #12 before #17, resolving their Deployment overlap once.
5. #17, then #18, after the authority, image, reliability, and reaping work.
6. #19 last as the integration/acceptance change, only after #12/#14-#18/#20 are
   merged and their exact supplied heads are ancestors of refreshed `main`.

An exact post-repair merge-tree check reports #19 clean with current `main`, #17,
#18, and #20, and conflicting with #12, #14, #15, and #16 on these files:

| Pair | Overlapping files |
| --- | --- |
| #12+#19 | `testing/tests/test_hermes_auto_router.py` |
| #14+#19 | `testing/quality_contract.json`, `testing/quality_coverage.py`, `testing/tests/test_quality_coverage_helpers.py` |
| #15+#19 | `testing/tests/conftest.py` |
| #16+#19 | `testing/quality_contract.json` |

Every one of those overlaps is additive on the #19 side. Resolve the contract by
union — take every PR's managed modules, lint paths, tracked files, and branch
tracked files, and never take one PR's quality contract wholesale. #19 adds two
fixture-isolation fixtures to `conftest.py` and two lines to the auto-router test;
keep both sides. Preserve branch coverage, every tracked module, fixture
isolation, and all desired-state changes during conflict resolution.

Recompute this graph from the Gitea API immediately before integration; do not
infer conflict status from Gitea's draft `mergeable` field.

The #16/#19 integration must keep both the immutable digest and the source tag
(`newTag: git-<main-SHA>-build-<n>` plus the digest) in rendered image identity.
Digest-only rendering loses the attested build-source link and therefore
correctly remains `NO_GO` under the exact-lineage evaluator.

## Safe rollback guidance

Every rollback is a reviewed forward fix or selective Git revert followed by
the normal Flux workflow. Never restore `hermes-agent-cluster-admin`, even when
reverting unrelated portions of #14. If a rollback would widen RBAC, stop and
write a forward least-privilege fix instead.

Reverting #18 removes the StatefulSet desired state, but its
`volumeClaimTemplates` PVCs are retained by the Kubernetes default because no
delete retention policy is set. Do not claim Flux pruning deletes those PVCs.
Any PVC deletion is a separate, explicit, human-reviewed data-destruction task
with exact targets and recovery implications.

Image rollback means a reviewed commit restoring a previously attested digest;
never retag an immutable image. Do not use direct `kubectl` edits as a durable
rollback.

## Quality and release evidence

The canonical quality contract explicitly manages, lints, and tracks all 16
`scripts/ops/hermes_handoff_*.py` modules. It enforces at least 95% line and
branch coverage per tracked file and the 500-physical-line source cap. Contract
tests derive the module set so a future file cannot silently escape the gate.
This change is purely additive to the contract: it registers its own modules and
changes no repo-wide hygiene semantics.

The gate's `hygiene` step is red on `main` and stays red here. Exactly four
pre-existing test files exceed the 500-line cap —
`scripts/tests/test_dashboards_render_atlas.py` (516),
`testing/tests/test_hermes_chat_quality.py` (2242),
`testing/tests/test_hermes_cli_lanes.py` (1971), and
`testing/tests/test_hermes_coordinator.py` (510). All four are byte-identical at
merge-base `30259b52` and at current `origin/main`, none is touched by this
change, and the same four are the only issues a pristine `main` reports. Splitting
them, or grandfathering them, is a repo-wide contract decision that belongs to the
canonical contract change carried by PR #14/#15, not to this acceptance harness.
Every other gate step — docs, smell, unit, coverage — is green, and every file
this change adds or edits is under the cap.

```sh
python3 -m testing.quality_gate --profile local --build-dir build
python3 -m testing.quality_handoff_mutation
python3 -m ruff check --select F,B,SIM,C4,UP --ignore B017,UP015,UP035 \
  scripts/ops/hermes_handoff_*.py testing/quality_coverage.py \
  testing/quality_handoff_mutation.py testing/tests/test_hermes_handoff_*.py
python3 -m ruff format --check \
  scripts/ops/hermes_handoff_*.py testing/quality_coverage.py \
  testing/quality_handoff_mutation.py testing/tests/test_hermes_handoff_*.py
python3 -m compileall -q scripts/ops testing
kustomize build services/hermes | kubectl apply --dry-run=client --validate=false -f -
git diff --check origin/main...HEAD
```

The format scope names the files this change owns. A wider `testing/quality_*.py`
glob picks up `testing/quality_gate.py`, which is unmodified here and not
format-clean, so the documented command would fail for a reason this change did
not cause. `ruff format` is evidence, not a gate step: the canonical gate runs
`ruff check` only.

The deterministic mutation gate must kill every policy, redaction, verdict,
numeric, truncation, provenance, and cleanup mutant, including one that restores
the zero-evidence pass, one that restores the bare-prefix repository pin, one
that stops re-checking a `kubectl exec` inner command, and one that drops binary
attestation from an unrecorded step. The adversarial suite includes large
concurrent stdout/stderr, malformed JSON and bytes, zero state, partial failure,
absolute timeout, descendant-held pipes, malformed cleanup, ambiguous creates,
and no-subprocess/no-mutation spies. Render/client dry-run, exact conflict checks,
and bounded diff/full-tree secret scans must also pass.

The provider-health and Gitea-token-sensitive tests use hermetic fixtures. Their
result must be identical whether synthetic runtime files exist or are absent;
tests must never read the mounted production token.

## Release decision

The handoff remains `NO_GO` until all dependency heads are merged, the exact
lineage check passes from an external operator, both default and armed reports
are complete, the branch and PR cleanup is verified, the canonical and
adversarial gates are green, and a human has reviewed PR #19 plus the artifacts.
The harness never merges, deploys, reconciles, publishes, or approves anything.