atlas-iac/docs/hermes_full_handoff_acceptance.md

# Hermes full-handoff acceptance harness and release runbook

`scripts/ops/hermes_handoff_acceptance.py` decides whether the Hermes platform
handoff is fit to release. It is read-only by default, it classifies every check
`PASS` / `FAIL` / `NOT_RUN` / `NOT_APPLICABLE`, and **any mandatory `FAIL` or
`NOT_RUN` makes the whole run `NO_GO`**. Absence of evidence is never rounded up
to evidence of absence: a probe that could not run, a catalog entry whose
evidence no longer exists, an expired deadline, or a harness bug all land in
`NOT_RUN` and stop the release.

```
scripts/ops/hermes_handoff_acceptance.py \
  --context atlas-operator \
  --node-count 21 \
  --expected-suspension flux-system/bstein-dev-home-migrations \
  --output build/hermes-handoff-acceptance.json
```

Exit status is `0` for `GO` and `1` for `NO_GO`. The JSON report goes to
`--output` (or stdout); a short human summary goes to stderr unless
`--json-only` is passed.

## Why two vantages

Every authority claim is checked from two places that cannot cover for each
other:

| Vantage | What it is | What it proves |
| --- | --- | --- |
| `operator` | An external, read-only kubeconfig | Desired state: bindings, manifests, workloads, Flux health, forge state |
| `self` | `kubectl exec` into the running `hermes-agent` container | Enforced state: what the agent's own service-account token can actually do |
| `switchyard`, `node`, `chat` | `kubectl exec` into the Switchyard, node-hardening, and chat pods | Evidence that only exists inside those workloads |

Two rules keep this honest:

* **The harness cannot impersonate from outside.** `--as`, `--as-group`, and
  `--as-uid` are rejected by policy for any operator-side command. They are
  reachable only as the *inner* command of a `kubectl exec`, where the identity
  is testing its own refusal to impersonate rather than borrowing authority.
  `kubectl auth can-i --as` is therefore not available as a substitute for a
  real self-probe.
* **The two vantages must be different principals.** Before any check runs, the
  harness asks each vantage `kubectl auth whoami` and compares. If the operator
  and the in-pod probe authenticate as the same principal, the dual-vantage
  evidence is a restatement rather than a corroboration, and the run is `NO_GO`
  with `the operator and in-pod vantages authenticate as the same principal`.
  Run the harness from a genuinely external kubeconfig — not from inside the
  Hermes owner pod.

Deny checks additionally require a **real refused request**, not only an
authorization review. `kubectl auth can-i` reports what RBAC says; only an
attempt proves the API server enforces it. Mutating attempts go through
`--dry-run=server`: the API server authorises the request and then discards it,
so a refusal is genuine and nothing is written either way.

## Safety properties

These hold structurally, in `hermes_handoff_policy.py`, so a catalog mistake is
a policy error before a run rather than a mutation or a leak during one.

* **No mutation in the default mode.** Only read subcommands are reachable;
  `apply`, `create`, `delete`, `patch`, `replace`, and `scale` require a dry-run
  flag. `git push` and every non-`GET` forge method are refused outside the
  armed mode. There is deliberately no live TokenRequest probe — a successful
  one would mint a real credential.
* **No credential is ever read.** `vault`, `sops`, `age`, `gpg`, `openssl`,
  `curl`, and `wget` are not on the binary allowlist. Secrets may only be read
  with `-o name`. Environment probes list variable *names* and never values.
  Provider health files are read through a frozen template that projects an
  allow-listed set of non-secret status fields. The node account's lock state is
  reported without reading its password hash; its authorized keys are counted,
  never printed.
* **Shell is frozen.** `sh -c` is reachable only through the reviewed,
  parameter-validated templates in `SHELL_TEMPLATES`. Parameters may not contain
  shell metacharacters.
* **Output is bounded and screened.** Each capture is truncated to a byte budget
  *before* it is screened, so a large dump cannot outrun the screen. Bulk
  evidence (the routing-log tail, the Kanban listing) is parsed but replaced with
  `[output not recorded]` in the artifact. The screen preserves JSON quoting, so
  screened structured output is still parsable.
* **The report is re-checked before it is published.** `unscreened_fields` walks
  the rendered payload; if anything still looks credential-shaped the report is
  replaced with a `NO_GO` stub naming the offending pointers, and the file that
  would have leaked is never written.

## What the catalog asserts

The catalog is declarative (`hermes_handoff_catalog.py` plus the four
`hermes_handoff_checks_*.py` modules) so a reviewer reads what is asserted, not
how it is plumbed. 71 checks in 14 groups:

| Group | Assertions |
| --- | --- |
| `baseline` | `origin/main` descends from the merged worker-isolation baseline; every dependency pull request is merged into `main`; the exact open pull-request heads are recorded |
| `identity` | Codex authenticates as `codex-chatgpt-subscription`; Claude as `firstParty` / `claude.ai` / `claude-code-cli-subscription`; both evidence files are fresh; no provider API-key variable exists in the running process **or** in the manifest |
| `routing` | Switchyard has a ready replica; the routing log covers both providers, the `route` and `worker` lanes, medium/high/xhigh effort, and at least one real fallback; the newest record is fresh; both providers record a measured latency |
| `scopes` | Chat, agent, and triage each declare their own `HERMES_AUTO_ROUTER_PROFILE`, on distinct hostnames |
| `access-denied` | No cluster-admin binding for the agent; Secrets, service-account tokens, impersonation, workload mutation, arbitrary pod exec, attach, port-forward, and control-plane writes are all refused |
| `access-allowed` | get/list/watch on core objects, pod logs, and Flux/Helm status still work; the operator and in-pod namespace inventories agree |
| `gitops` | No unexpected Kustomization or HelmRelease suspension, and nothing unhealthy |
| `forge` | No forge credential in the worker environment or home; the forge identity is not an administrator; repository authority is pull/push only; an administrative route is live-refused; clone/fetch still works |
| `nodes` | The dedicated `hermes-agent` account is locked, has the expected uid/gid/home/shell, joins no supplementary group, has no sudoers fragment, holds exactly one authorized key; K3s credentials, K3s state, kubelet state, and the container runtime are root-only; the audit covers every node |
| `build` | The image builder service account mounts no token; the build pod is unprivileged with exactly `CHOWN, FOWNER, DAC_OVERRIDE, SETGID, SETUID`; the Harbor immutable-tag job succeeded; the running image is digest-pinned |
| `reliability` | The finalization/replay/session patches are running in the pod; the regression suites are present on `main` |
| `pool` | Three ready workers, on distinct nodes, with distinct volumes, mounting no API token and none of the coordinator's claims |
| `surfaces` | Kanban activity and durable sessions are visible and render without blank rows; the agent and chat session patches are deployed; Telegram topic state is on durable storage |
| `ephemeral` | Feature-branch push, draft pull request, protected-branch refusal, and verified cleanup — `NOT_RUN` unless armed |

### Site configuration

A few checks are emitted only when the operator declares what to expect,
because a `names_present` assertion over an empty expectation would pass on any
workload at all — which reads as coverage without being any:

| Flag | Enables |
| --- | --- |
| `--pool-worker-env NAME` (repeatable) | `pool.assignment-safety-knobs-are-configured` |
| `--chat-config-revision REV` | `surfaces.chat-runs-the-telegram-topic-revision` |
| `--expect-telegram-sessions` with `--chat-ordinal N` | `surfaces.chat-telegram-sessions-are-continuous` |

`--node-count` must be the real node count; the default of `1` makes the
fleet-coverage check vacuous.

### Catalog drift is a NO_GO, not a pass

If a workload, field, or manifest is renamed, the affected checks report
`NOT_RUN` because the evidence they name is no longer there. That is the
intended behaviour: the release stops until either the platform or the catalog
is corrected. Do not "fix" a `NOT_RUN` by deleting the check.

## Ephemeral mutation mode

Proving that a worker can still open a draft pull request means actually opening
one. That is the only part of the harness that writes anything, and it is a
separate mode:

```
scripts/ops/hermes_handoff_acceptance.py \
  --context atlas-operator \
  --arm-ephemeral-push \
  --confirm "ARM EPHEMERAL HERMES HANDOFF PUSH" \
  --ephemeral-token acceptance-20260817a \
  --output build/hermes-handoff-acceptance-armed.json
```

* The confirmation phrase must match exactly.
* `--ephemeral-token` is 8–64 lowercase alphanumeric/dash characters and must be
  unique per run; the ref is always
  `ephemeral/hermes-handoff-acceptance/<token>`.
* **Preflight runs before any network call**: wrong phrase, wrong repository,
  malformed token, or a protected push target all abort with `arming refused`
  and a `NO_GO` report. `main`, `master`, `HEAD`, `develop`, `release`,
  `production`, `stable`, `trunk`, `prod`, and `default` are refused by name
  (case- and `refs/heads/`-insensitive), and anything that is not an ephemeral
  acceptance ref is refused by pattern.
* The guard is **exercised on every armed run** by
  `ephemeral.protected-branch-refusal`, rather than trusting that it was checked
  once.
* The push target is guarded; the pull-request *base* is not. Opening a draft
  pull request against `main` writes nothing to `main`. The draft title always
  starts with `WIP: `.
* Cleanup closes the pull request, deletes the ref, and then **re-reads both** to
  confirm they are gone. `ephemeral.cleanup-verified` is mandatory: a half-removed
  branch fails the run rather than passing it. Cleanup runs even when an earlier
  step failed.
* `git push --force`, `--force-with-lease`, `--mirror`, `--all`, and `--tags`
  are refused even in armed mode.

A default run reports all four ephemeral checks `NOT_RUN` and non-mandatory, so
a read-only sweep can still reach `GO`.

## Release order

### Baseline, already merged and deployed

PR #13 (`hermes: harden worker isolation and blocked-task semantics`) merged as
`ab346f55509d584e457fe26cf90be3078f7a375c`. Every check below assumes it is
present; `baseline.origin-main-descends-merged-work` asserts it. Override with
`--baseline-commit` only when a later merge legitimately replaces it.

### Pending work, observed 2026-08-17T10:07Z

These heads move. The harness re-fetches them at run time via
`baseline.pending-pull-requests-recorded`; the table is the snapshot this
document was written against, not an input.

| PR | Branch | Head at observation | State | Delivers |
| --- | --- | --- | --- | --- |
| #14 | `feature/hermes-safe-gitea-pr-client` | `3cd8a17e772978fb554d88dd1c59777786be3991` | draft | Bounded forge authority, observer RBAC (removes `hermes-agent-cluster-admin`), node account hardening |
| #15 | `fix/hermes-result-decomposition-reliability` | `7609cd77e87a64e332b2f0d670c1d7a4f62b3d9e` | draft | Result preservation, duplicate-fanout prevention, execution-safety patch |
| #16 | `feature/hermes-unprivileged-image-builder` | `5ab0eba619779b3dfed2eb09d5c02f0a9c92dd0a` | ready | Daemonless Kaniko build lane, tokenless builder, Harbor immutable-tag rule |
| #17 | `feature/hermes-three-lane-placement` | `48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c` | draft | Three concurrent CLI lanes on the larger ARM64 nodes |
| #18 | `feature/hermes-distributed-worker-pool` | `20002527512235b29054f3636b02418168373009` | draft | Three-node fenced execution pool with coordinator-only state ownership |

Base at observation: `30259b526be5b2e226620aeebd9fb5f57dc88b7d`.

### Merge order and why

1. **#14 first.** It is the only change that *reduces* authority: it deletes
   `services/hermes/agent-rbac.yaml` (the `cluster-admin` binding) and replaces
   it with the namespaced/cluster observer roles, and it hardens the node
   account. Merging it last would mean every earlier acceptance run measured a
   cluster-admin agent. Every `access-denied` and `nodes` check depends on it.
2. **#16 second.** The image lane must exist before anything that needs a new
   agent image is rolled out. It also touches `testing/quality_contract.json`,
   so merging it early keeps later rebases small. `build.*` depends on it.
3. **#15 third.** It changes `dockerfiles/Dockerfile.hermes-agent`, which #16
   also touches — take #16's shape and rebase #15 onto it. `reliability.*`
   depends on it.
4. **#17 fourth.** Placement only; it edits `services/hermes/agent-deployment.yaml`
   and `testing/tests/test_hermes_cli_lanes.py`, both of which #14, #15, and #18
   also touch. Merging it after them avoids resolving the same conflict twice.
5. **#18 last.** The pool depends on the bounded forge client from #14, the
   execution-safety patch from #15, and the agent image from #16. `pool.*`
   depends on it.

Overlapping files to expect (rebase, do not merge blind): `services/hermes/agent-deployment.yaml`
(#12, #14, #16, #17, #18), `services/hermes/kustomization.yaml` (#14, #16, #18),
`services/hermes/scripts/stage_runtime_access.py` (#14, #16, #18),
`services/vault/scripts/vault_k8s_auth_configure.sh` (#14, #16, #18),
`dockerfiles/Dockerfile.hermes-agent` (#15, #16),
`testing/tests/test_hermes_cli_lanes.py` (#14, #15, #17, #18),
`testing/tests/test_hermes_runtime_access.py` (#16, #18).

PR #12 (AI usage exporter) is independent and can merge at any point; it touches
`services/hermes/agent-deployment.yaml`, so land it before #17 if convenient.

### Image build and Flux rollout

1. Merge in the order above; observe the default-branch Jenkins build to a
   terminal result after each merge
   (`/opt/coordinator/jenkins_build_evidence.py titan-iac --branch main --wait`).
2. After #16 is merged, run the `hermes-agent-image` pipeline. It publishes an
   immutable `git-<sha>-build-<n>` tag and emits
   `build/hermes-agent-release/hermes-image-update.patch`.
3. Apply that patch to `services/hermes/kustomization.yaml` on a branch, review,
   and merge. The digest is what pins the deployment;
   `build.deployed-image-is-digest-pinned` asserts it.
4. Let Flux reconcile. Confirm with `flux get kustomizations -A` that `hermes`
   and `hermes-chat` are `Ready`, then wait for the rollout to settle.
5. Re-run the harness. Do not run it mid-rollout: `gitops.*` tolerates an
   in-flight reconcile (`Reconciling=True` with `Ready=False`) but a half-rolled
   deployment will legitimately fail the `identity`, `routing`, and `surfaces`
   groups.

### Rollback points

| After | Roll back by | Restores |
| --- | --- | --- |
| #14 merge | Revert the merge commit; Flux restores `hermes-agent-cluster-admin` | Cluster-admin agent, previous node account state |
| #16 merge | Revert the merge; the previously pinned digest is unchanged in `kustomization.yaml` | Previous build lane; published immutable tags are not deleted |
| Image digest bump | Revert the `kustomization.yaml` commit to the prior digest | Previous agent image; Harbor immutability means the old tag is still exactly what it was |
| #15 merge | Revert the merge; the init-container patch disappears on the next rollout | Previous finalization/replay behaviour |
| #17 merge | Revert the merge | Previous lane concurrency and node placement |
| #18 merge | Revert the merge; the StatefulSet and its PVCs are removed by Flux pruning | Single-coordinator execution |

Every rollback is a Git revert reconciled by Flux. Do not roll back by direct
`kubectl` edit; the next reconcile would undo it.

## Go / no-go checklist

Run in order. Stop at the first item that does not hold.

1. `origin/main` is `ab346f55509d584e457fe26cf90be3078f7a375c` or a descendant.
2. PRs #14, #15, #16, #17, #18 are all merged into `main`, in that order, each
   with a green default-branch build.
3. The agent image has been built by the #16 lane, is pinned by digest in
   `services/hermes/kustomization.yaml`, and Flux has finished reconciling.
4. The harness is run from an **external operator kubeconfig**, not from inside
   the Hermes owner pod, with `--node-count` set to the real node count and every
   deliberate Flux suspension declared via `--expected-suspension`.
5. The default read-only run reports `GO` — no mandatory `FAIL`, no mandatory
   `NOT_RUN`, no `harness_errors`.
6. The armed run reports `PASS` for all four `ephemeral.*` checks, including
   `ephemeral.cleanup-verified`, and the ephemeral ref and pull request are
   confirmed gone.
7. The JSON report is archived with the exact `origin/main` commit and the
   recorded pull-request heads from `baseline.pending-pull-requests-recorded`.

Only then is the handoff releasable.

## Known limits

Stated so nobody mistakes a proxy for the thing itself.

* **Merge, approve, and close authority is asserted, not exercised.** The
  harness never attempts a merge. It proves the forge identity holds
  `permissions.admin: false` and is live-refused on an administrative route;
  the route-level allowlist is covered by PR #14's own regression suite.
* **Attach and port-forward have no side-effect-free live attempt.** Their
  authorization reviews ride alongside a real, refused control-plane write in the
  same check, and pod exec — the one subresource with a harmless real attempt —
  has its own check.
* **TokenRequest is asserted by review.** A successful live attempt would mint a
  real credential, so the check pairs the `serviceaccounts/token` review with a
  refused `serviceaccount` create in the same RBAC family.
* **Branch protection is not read.** The harness identity is not a repository
  administrator, so `GET /branch_protections` is refused — which is itself
  recorded as evidence. Protected-push refusal is guaranteed by the harness's own
  preflight, exercised on every armed run, not by reading the forge's config.
* **Stale/duplicate assignment safety is asserted as deployed configuration.**
  The behaviour itself is covered by the pool's regression suite; the harness
  checks that the workers carry the configuration and that the coordinator keeps
  sole ownership of its claims.
* **Telegram session continuity is per tenant** and depends on there having been
  Telegram traffic, so it is off by default. The always-on check is that topic
  and tenant state live on a persistent claim rather than in the pod.
* **The routing window is a tail**, 600 records by default. A property that did
  not occur inside that window is reported as missing rather than absent; widen
  the window or wait for traffic rather than removing the assertion.

## Tests

```
python3 -m pytest testing/tests/ -k handoff
python3 -m coverage run --source=scripts/ops --include="*/hermes_handoff_*.py" \
  -m pytest testing/tests/ -k handoff && python3 -m coverage report
```

The suite covers parsing, redaction, timeouts and deadlines, partial failure,
zero state, dual-vantage mismatch, mandatory skips, catalog structure, and
ephemeral arming and cleanup — including the cases where cleanup cannot be
verified.