titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity
atlas-iac/docs/hermes_full_handoff_acceptance.md

14 KiB
Raw Blame History

Hermes full-handoff acceptance and release runbook

scripts/ops/hermes_handoff_acceptance.py is a fail-closed release gate. It contains 63 checks in 15 groups. An empty catalog, an unknown result status, a mandatory FAIL or NOT_RUN, a harness error, or incomplete evidence makes the decision NO_GO. The four ephemeral checks are non-mandatory NOT_RUN in the default mode and are replaced with mandatory results only in armed mode.

This runbook describes evidence collection. It does not authorize a merge, publish, deployment, Flux reconcile, credential read, or live cluster mutation. Human review of PR #19 and of the resulting report is required.

Required release inputs

Every value that can go stale must be supplied explicitly. Resolve main, PR heads, the image digest, build source SHA, deployment revision, and chat config revision from their authoritative read-only sources immediately before the run. Do not copy the snapshot in the dependency section into a later run.

scripts/ops/hermes_handoff_acceptance.py \
  --context atlas-operator \
  --remote-main-sha "$REMOTE_MAIN_SHA" \
  --reviewed-head-sha "$REVIEWED_PR19_HEAD" \
  --agent-image "$AGENT_GIT_SHA_BUILD_TAG_AT_SHA256" \
  --build-sha "$IMAGE_BUILD_SOURCE_SHA" \
  --deployment-revision "$RUNNING_DEPLOYMENT_REVISION" \
  --chat-config-revision "$RUNNING_CHAT_CONFIG_REVISION" \
  --dependency-head "12=$PR12_HEAD" \
  --dependency-head "14=$PR14_HEAD" \
  --dependency-head "15=$PR15_HEAD" \
  --dependency-head "16=$PR16_HEAD" \
  --dependency-head "17=$PR17_HEAD" \
  --dependency-head "18=$PR18_HEAD" \
  --dependency-head "20=$PR20_HEAD" \
  --output build/hermes-handoff-acceptance.json

The fixed topology is three pool workers on three distinct nodes. --node-count and --pool-replicas cannot lower that requirement. Pool node/ordinal assignment, Telegram session continuity, and the chat revision are mandatory. Deliberate Flux suspensions must be named with repeatable --expected-suspension arguments.

Exit status is 0 only for GO, 1 for NO_GO, and 2 for invalid execution bounds. JSON is written to --output or stdout; the human summary goes to stderr unless --json-only is passed.

Structural safety boundary

The catalog is validated before a Runner exists. Invalid inputs, an empty or unsafe catalog, and an invalid armed preflight therefore create no subprocess. The command policy is checked both before and after executable resolution.

Default mode permits a narrow read grammar:

  • Kubernetes reads use name or reviewed JSONPath projections. Secret resources, raw API reads, full JSON/YAML, describe, cluster-info dump, credential-bearing projections, auth reconcile, config operations, all mutations, and every dry-run mutation trick are rejected before spawn.
  • Git is read-only. fetch, configuration/helper overrides, alternate worktrees, external diff helpers, and push are rejected. Helm values are not readable.
  • Shell commands must be rendered from frozen templates, use fixed executable paths, and cannot name service-account, Vault, runtime-access, SSH, or other credential roots.
  • Outer impersonation is rejected. A reviewed inner command can use impersonation only after an explicit kubectl exec -- boundary; it cannot substitute for the independent self identity.
  • PATH is replaced with a fixed path. Every outer executable is resolved to an allowed real path, checked for unsafe write mode, and compared with a pinned SHA-256 digest. A caller-authored kubectl, git, or helper cannot fabricate evidence.

The executor uses a new process group, drains stdout and stderr concurrently, and retains at most the configured bytes while the child is running. A monotonic absolute timeout terminates and reaps the group, including descendants that keep pipes open. Timeout, deadline, output, and concurrency values reject booleans, zero/negative values, non-finite numbers, and values outside fixed bounds. Truncation discards the affected stream and makes every evaluator NOT_RUN.

Credential screening runs before truncation. Command, output, error, identity, reason, evidence, JSON, and human-summary fields are screened recursively, including mapping keys and short/numeric sensitive values. The complete rendered report is screened again; a remaining credential shape is replaced by a bounded NO_GO stub rather than being written. No credential prefix is retained at a truncation boundary.

Vantages and evidence

The operator must use an external read-only kubeconfig. The self probe runs in the actual Hermes agent pod. Both identities are obtained with kubectl auth whoami; equal or missing principals make the run NO_GO. Required Switchyard and chat/Telegram pod vantages must also resolve. Node coverage is derived from all daemonset pod node names, not one selected pod.

Dangerous Kubernetes permissions are checked only with side-effect-free kubectl auth can-i reviews. The default catalog never constructs a Secret, TokenRequest, mutation, attach, port-forward, or arbitrary exec request. The one forge administrative denial check performs a read-only GET and accepts only an explicit authorization refusal; generic 404, not found, and no route text are not denial evidence.

The strict evaluators reject duplicate-key JSON, raw control characters, NaN/infinity, boolean/number equality coercion, future timestamps, stale provider evidence, and every malformed routing or Flux row. Routing freshness is required separately for each provider, lane, and effort. Flux objects must be non-suspended, current at observedGeneration, and explicitly Ready=True; unknown or reconciling state is not healthy.

release.exact-lineage-is-running atomically binds:

  • fixed remote main SHA and local origin/main;
  • open draft PR #19, base main, existing feature ref, and exact reviewed head;
  • the exact :git-<source-SHA>-build-<n>@sha256:<digest> image identity, with the source SHA equal to the release main SHA;
  • current Deployment generation and revision;
  • the one active ReplicaSet revision/hash; every Ready pod image and imageID;
  • the Flux-applied main revision.

The pool evaluator requires exactly ordinals 0, 1, and 2, all Ready, on three distinct nodes and claims, with service-account token automount disabled. The chat checks always require durable Telegram sessions and the exact running chat configuration revision.

Armed ephemeral mode

Armed mode is the only write path and runs only after the complete default catalog reports GO.

scripts/ops/hermes_handoff_acceptance.py \
  <all required release inputs above> \
  --arm-ephemeral-push \
  --confirm "ARM EPHEMERAL HERMES HANDOFF PUSH" \
  --ephemeral-token acceptance-20260817a \
  --output build/hermes-handoff-acceptance-armed.json

Before any runner, network request, or cluster vantage is created, preflight binds the working directory to the existing linked PR #19 worktree, the exact feature branch and reviewed local HEAD, the fixed HTTPS Atlas origin, base main, confirmation phrase, and one unique ephemeral ref. The command policy is then pinned to that exact ref. It allows only its push, one exact draft-create payload, cleanup of the exactly discovered PR number, and deletion of that ref.

The run reserves at least 300 seconds (up to 6000) for cleanup. It verifies the pushed SHA, always discovers the PR after create (including ambiguous create responses), and requires exactly one open draft with the expected base, head ref, base SHA, and head SHA. Cleanup requires successful PR close, successful branch deletion, a successful empty ls-remote, and a strict re-read of the same closed PR. Malformed or partial cleanup is FAIL; it is never inferred from an error.

Force, force-with-lease, fan-out, protected refs, other repositories, other PR numbers, arbitrary PATCH payloads, and repository administration remain denied.

Dependency, merge, and conflict graph

Authoritative snapshot observed 2026-08-17 after refreshing Gitea and Git against remote main d8f2d818b9a552ea6c2d7fe86554be829bd5ffff:

PR Head SHA Draft Base Role
#12 465d386da5237f011cb408f1a3d4b7d0c6d8ffae no main AI usage exporter repair
#14 8c6e3acdacb223f68b1716f0a74bd543e18890e6 yes main bounded forge client and least-privilege RBAC
#15 73fefbb5d92ef07371a9a997f22d848acd2d06ed yes main result-decomposition reliability
#16 e8c26ecf85a0c802d378fffffb98b7df43b78242 no main unprivileged image builder
#17 48cbe13ee50ce3fcb07cea3fe8d088cfed349e0c yes main three-lane placement
#18 20002527512235b29054f3636b02418168373009 yes main distributed three-worker pool
#19 8f005458282269ba5c07941814e4237f5d4cf3ac yes main reviewed blocked head; this repair replaces it on the same branch
#20 a242dcc786576ae1a18000cb4941836ff610dff2 yes main CLI process-group reaping

The repaired #19 SHA cannot be embedded in its own commit without changing that SHA. Resolve it from Gitea after push and require it as --reviewed-head-sha; the PR handoff records the exact value. All eight snapshot heads merge cleanly with the snapshot main individually. Pairwise git merge-tree --write-tree found conflicts only for: #12+#17, #14+#15, #14+#16, #14+#17, #14+#20, #15+#16, #15+#17, #16+#20, and #17+#20. Gitea's mergeable=false on a draft is not treated as conflict evidence.

Use this dependency order, refreshing every head and re-running merge-tree at each step:

  1. #14 first so the release never restores or prolongs cluster-admin authority.
  2. #16, then #15, resolving their Dockerfile and quality-contract changes additively. Never take one PR's quality contract wholesale.
  3. #20 after #15; it depends on the CLI lifecycle work and must be rebased across the current #14/#16 contract and helper changes.
  4. #12 before #17, resolving their Deployment overlap once.
  5. #17, then #18, after the authority, image, reliability, and reaping work.
  6. #19 last as the integration/acceptance change, only after #12/#14-#18/#20 are merged and their exact supplied heads are ancestors of refreshed main.

#19 overlaps testing/quality_contract.json with #14/#15/#16, testing/quality_coverage.py and its helper tests with #14, test_hermes_auto_router.py with #18, and existing Kustomize content with #12/#17/#18. Preserve branch coverage, every tracked module, fixture isolation, and all desired-state changes during conflict resolution.

The #16/#19 integration must keep both the immutable digest and the source tag (newTag: git-<main-SHA>-build-<n> plus the digest) in rendered image identity. Digest-only rendering loses the attested build-source link and therefore correctly remains NO_GO under the exact-lineage evaluator.

Safe rollback guidance

Every rollback is a reviewed forward fix or selective Git revert followed by the normal Flux workflow. Never restore hermes-agent-cluster-admin, even when reverting unrelated portions of #14. If a rollback would widen RBAC, stop and write a forward least-privilege fix instead.

Reverting #18 removes the StatefulSet desired state, but its volumeClaimTemplates PVCs are retained by the Kubernetes default because no delete retention policy is set. Do not claim Flux pruning deletes those PVCs. Any PVC deletion is a separate, explicit, human-reviewed data-destruction task with exact targets and recovery implications.

Image rollback means a reviewed commit restoring a previously attested digest; never retag an immutable image. Do not use direct kubectl edits as a durable rollback.

Quality and release evidence

The canonical quality contract explicitly manages, lints, and tracks all 15 scripts/ops/hermes_handoff_*.py modules. It enforces at least 95% line and branch coverage per tracked file and the 500-physical-line source cap. Contract tests derive the module set so a future file cannot silently escape the gate. Four pre-existing over-cap test files are temporarily locked to their exact line count and SHA-256; any byte edit invalidates the exception and must split the file below 500 lines. PR #14 supplies the intended additive test splits.

python3 -m testing.quality_gate --profile local --build-dir build
python3 -m testing.quality_handoff_mutation
python3 -m ruff check --select F,B,SIM,C4,UP --ignore B017,UP015,UP035 \
  scripts/ops/hermes_handoff_*.py testing/quality_*.py testing/tests/test_hermes_handoff_*.py
python3 -m ruff format --check \
  scripts/ops/hermes_handoff_*.py testing/quality_*.py testing/tests/test_hermes_handoff_*.py
python3 -m compileall -q scripts/ops testing
kustomize build services/hermes | kubectl apply --dry-run=client --validate=false -f -
git diff --check origin/main...HEAD

The deterministic mutation gate must kill every policy, redaction, verdict, numeric, truncation, and cleanup mutant. The adversarial suite includes large concurrent stdout/stderr, malformed JSON and bytes, zero state, partial failure, absolute timeout, descendant-held pipes, malformed cleanup, ambiguous creates, and no-subprocess/no-mutation spies. Render/client dry-run, exact conflict checks, and bounded diff/full-tree secret scans must also pass.

The provider-health and Gitea-token-sensitive tests use hermetic fixtures. Their result must be identical whether synthetic runtime files exist or are absent; tests must never read the mounted production token.

Release decision

The handoff remains NO_GO until all dependency heads are merged, the exact lineage check passes from an external operator, both default and armed reports are complete, the branch and PR cleanup is verified, the canonical and adversarial gates are green, and a human has reviewed PR #19 plus the artifacts. The harness never merges, deploys, reconciles, publishes, or approves anything.