hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
#!/usr/bin/env python3
|
|
|
|
|
"""Acceptance checks for the release baseline, provider identity, and routing.
|
|
|
|
|
|
|
|
|
|
These are the facts a handoff rests on before any authority question is asked:
|
|
|
|
|
that the tree under test is the merged one, that both providers authenticate
|
|
|
|
|
through the subscriptions they are supposed to, that Switchyard actually decided
|
|
|
|
|
the routes it claims to, and that chat, agent, and triage are still three
|
|
|
|
|
distinct scopes rather than one surface wearing three hostnames.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
from hermes_handoff_catalog import (
|
|
|
|
|
CLAUDE_HEALTH,
|
|
|
|
|
CLAUDE_HEALTH_FIELDS,
|
|
|
|
|
CODEX_HEALTH,
|
|
|
|
|
CODEX_HEALTH_FIELDS,
|
|
|
|
|
ENV_NAME_PROJECTION,
|
|
|
|
|
OPERATOR,
|
|
|
|
|
PROVIDER_API_KEY_NAMES,
|
|
|
|
|
ROUTING_LOG,
|
|
|
|
|
SELF,
|
|
|
|
|
SWITCHYARD,
|
|
|
|
|
Targets,
|
|
|
|
|
check,
|
|
|
|
|
step,
|
|
|
|
|
)
|
|
|
|
|
from hermes_handoff_model import CheckSpec
|
2026-08-17 13:11:34 +00:00
|
|
|
from hermes_handoff_policy import GITEA_CLIENT, projection, shell
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def _baseline(targets: Targets) -> list[CheckSpec]:
|
|
|
|
|
api = f"/api/v1/repos/{targets.repo}"
|
2026-08-17 13:11:34 +00:00
|
|
|
dependency_heads = dict(targets.dependency_heads)
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
checks = [
|
2026-08-17 13:11:34 +00:00
|
|
|
check(
|
|
|
|
|
"baseline.local-origin-main-is-current",
|
|
|
|
|
"The local origin/main ref equals the fixed remote-main SHA",
|
|
|
|
|
"baseline",
|
|
|
|
|
"stdout_matches",
|
|
|
|
|
[step("main", OPERATOR, "git", "rev-parse", f"{targets.remote}/main")],
|
|
|
|
|
{"step": "main", "equals": targets.remote_main_sha},
|
|
|
|
|
rationale="The ancestry check is meaningful only after the local tracking ref is bound to the remote observation.",
|
|
|
|
|
),
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
check(
|
|
|
|
|
"baseline.origin-main-descends-merged-work",
|
|
|
|
|
"origin/main descends from the merged Hermes isolation baseline",
|
|
|
|
|
"baseline",
|
|
|
|
|
"allowed",
|
|
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"ancestor",
|
|
|
|
|
OPERATOR,
|
|
|
|
|
"git",
|
|
|
|
|
"merge-base",
|
|
|
|
|
"--is-ancestor",
|
|
|
|
|
targets.baseline_commit,
|
|
|
|
|
f"{targets.remote}/main",
|
|
|
|
|
)
|
|
|
|
|
],
|
|
|
|
|
rationale="Everything downstream assumes the merged worker-isolation baseline is present.",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"baseline.pending-pull-requests-recorded",
|
|
|
|
|
"Exact open pull-request heads are recorded, not assumed",
|
|
|
|
|
"baseline",
|
|
|
|
|
"json_record",
|
2026-08-17 13:11:34 +00:00
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"open",
|
|
|
|
|
OPERATOR,
|
|
|
|
|
GITEA_CLIENT,
|
hermes: source handoff forge evidence through the scm broker
The acceptance harness pinned a forge client that has never existed in any
commit or pod (/opt/coordinator/gitea_api.py, digest f0943db4..., GIT/POST
grammar, an askpass helper). Every Gitea-backed check was therefore
unrunnable as merged. Point the harness at the credential-isolated SCM
broker client that actually ships in the agent pod.
- policy: GITEA_CLIENT=/opt/scm/gitea_api.py; trust /opt/scm/ instead of
the phantom /opt/coordinator/; admit the client's real grammar
(`read <api-path>`, exactly one path) with the same atlas/titan-iac pin
and dot-segment rejection; bare HTTP methods are refused in every mode.
The armed POST/PATCH/DELETE windows remain but are documented as
deferred: the deployed client cannot execute them.
- exec: pin the client digest to the sha256 of
services/hermes/scm-common/scripts/gitea_api.py — the exact file the
hermes-scm-boundary-v2 ConfigMap mounts at /opt/scm/gitea_api.py — so
the pin is derivable from merged source and equal to the deployed
client. gitea_api.py gains a narrow /api/v1/user identity read in
_authorize_read (see below), so the pin is the NEW source hash
76efd16dedbeb74425b12fbbdbfaa391854771292077e0463bf22706855ae6dc.
Drop the dangling GIT_ASKPASS (no helper exists; broker git needs
none) and swap /opt/coordinator for /opt/scm in SAFE_PATH.
- checks: all forge/baseline/lineage probes use (client, "read", path).
The SELF-vantage identity checks now truthfully assert the *broker's*
forge identity (the only one the platform can exercise) is not an
administrator and holds push-scoped, non-administrative repository
authority; the administrative-route check asserts the broker read
allowlist's live refusal of branch_protections. The remote-main step
keeps `origin` (the broker remote exists only in pool workspaces and
the broker origin is cluster-local); its rationale now tells the
operator to ensure origin fetchability.
- gitea_api.py/_authorize_read: allow exactly `/api/v1/user` (no query,
no sibling routes) as operation "identity" so the harness can prove
the broker identity is not an administrator. The broker imports the
same module, so one reviewed edit covers both sides of the boundary.
- rules: DENIAL_MARKERS now match the client's real refusal lines
("SCM broker request failed with HTTP 400/403" and the no-credential
rejection) and drop "gitea api returned http 403", which the client
never emits; a broker 404 is deliberately not denial evidence.
- ephemeral: index/verification reads use the real grammar; manual
cleanup guidance now says close/delete require operator forge
credentials (the client exposes no mutation besides create-draft);
armed mode is documented as deferred until the probes are rebuilt on
the broker's bounded mutation surface.
- docs: broker vantage/evidence section, operator prerequisites (broker
healthy, no /vault/secrets/gitea-token anywhere on the harness path,
current ConfigMap mount, operator-side client + origin fetchability),
armed-mode deferral.
- tests: read-grammar accepted / GET refused in every mode, /opt/scm
attestation pin proven equal to the merged source digest, real
denial-marker matching, /api/v1/user identity route bounds; the
repository-pin mutant probe speaks the new grammar. Full handoff +
gitea + broker families pass (952 tests), mutation gate 13/13, per-file
line+branch coverage >=95%, all touched sources within the 500-line cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 17:43:13 -03:00
|
|
|
"read",
|
2026-08-17 13:11:34 +00:00
|
|
|
f"{api}/pulls?state=open&limit=50",
|
|
|
|
|
)
|
|
|
|
|
],
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
{
|
|
|
|
|
"step": "open",
|
|
|
|
|
"record": {
|
|
|
|
|
"numbers": "[].number",
|
|
|
|
|
"head_refs": "[].head.ref",
|
|
|
|
|
"head_shas": "[].head.sha",
|
|
|
|
|
"base_shas": "[].base.sha",
|
|
|
|
|
"drafts": "[].draft",
|
|
|
|
|
"mergeable": "[].mergeable",
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
rationale="The runbook has to name the heads it was verified against; stale heads are how a release verifies the wrong tree.",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"baseline.operator-checkout-clean",
|
|
|
|
|
"The operator checkout has no uncommitted drift",
|
|
|
|
|
"baseline",
|
|
|
|
|
"stdout_matches",
|
|
|
|
|
[step("status", OPERATOR, "git", "status", "--porcelain")],
|
|
|
|
|
{"step": "status", "equals": ""},
|
|
|
|
|
mandatory=False,
|
|
|
|
|
rationale="Advisory: a dirty checkout does not invalidate cluster evidence, but it does muddy what was compared.",
|
|
|
|
|
),
|
|
|
|
|
]
|
|
|
|
|
checks += [
|
|
|
|
|
check(
|
|
|
|
|
f"baseline.dependency-pr-{number}-merged",
|
|
|
|
|
f"Pull request #{number} is merged into main",
|
|
|
|
|
"baseline",
|
|
|
|
|
"json_field",
|
hermes: source handoff forge evidence through the scm broker
The acceptance harness pinned a forge client that has never existed in any
commit or pod (/opt/coordinator/gitea_api.py, digest f0943db4..., GIT/POST
grammar, an askpass helper). Every Gitea-backed check was therefore
unrunnable as merged. Point the harness at the credential-isolated SCM
broker client that actually ships in the agent pod.
- policy: GITEA_CLIENT=/opt/scm/gitea_api.py; trust /opt/scm/ instead of
the phantom /opt/coordinator/; admit the client's real grammar
(`read <api-path>`, exactly one path) with the same atlas/titan-iac pin
and dot-segment rejection; bare HTTP methods are refused in every mode.
The armed POST/PATCH/DELETE windows remain but are documented as
deferred: the deployed client cannot execute them.
- exec: pin the client digest to the sha256 of
services/hermes/scm-common/scripts/gitea_api.py — the exact file the
hermes-scm-boundary-v2 ConfigMap mounts at /opt/scm/gitea_api.py — so
the pin is derivable from merged source and equal to the deployed
client. gitea_api.py gains a narrow /api/v1/user identity read in
_authorize_read (see below), so the pin is the NEW source hash
76efd16dedbeb74425b12fbbdbfaa391854771292077e0463bf22706855ae6dc.
Drop the dangling GIT_ASKPASS (no helper exists; broker git needs
none) and swap /opt/coordinator for /opt/scm in SAFE_PATH.
- checks: all forge/baseline/lineage probes use (client, "read", path).
The SELF-vantage identity checks now truthfully assert the *broker's*
forge identity (the only one the platform can exercise) is not an
administrator and holds push-scoped, non-administrative repository
authority; the administrative-route check asserts the broker read
allowlist's live refusal of branch_protections. The remote-main step
keeps `origin` (the broker remote exists only in pool workspaces and
the broker origin is cluster-local); its rationale now tells the
operator to ensure origin fetchability.
- gitea_api.py/_authorize_read: allow exactly `/api/v1/user` (no query,
no sibling routes) as operation "identity" so the harness can prove
the broker identity is not an administrator. The broker imports the
same module, so one reviewed edit covers both sides of the boundary.
- rules: DENIAL_MARKERS now match the client's real refusal lines
("SCM broker request failed with HTTP 400/403" and the no-credential
rejection) and drop "gitea api returned http 403", which the client
never emits; a broker 404 is deliberately not denial evidence.
- ephemeral: index/verification reads use the real grammar; manual
cleanup guidance now says close/delete require operator forge
credentials (the client exposes no mutation besides create-draft);
armed mode is documented as deferred until the probes are rebuilt on
the broker's bounded mutation surface.
- docs: broker vantage/evidence section, operator prerequisites (broker
healthy, no /vault/secrets/gitea-token anywhere on the harness path,
current ConfigMap mount, operator-side client + origin fetchability),
armed-mode deferral.
- tests: read-grammar accepted / GET refused in every mode, /opt/scm
attestation pin proven equal to the merged source digest, real
denial-marker matching, /api/v1/user identity route bounds; the
repository-pin mutant probe speaks the new grammar. Full handoff +
gitea + broker families pass (952 tests), mutation gate 13/13, per-file
line+branch coverage >=95%, all touched sources within the 500-line cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 17:43:13 -03:00
|
|
|
[step("pr", OPERATOR, GITEA_CLIENT, "read", f"{api}/pulls/{number}")],
|
2026-08-17 13:11:34 +00:00
|
|
|
{
|
|
|
|
|
"step": "pr",
|
|
|
|
|
"fields": {
|
|
|
|
|
"merged": True,
|
|
|
|
|
"base.ref": "main",
|
|
|
|
|
"head.sha": dependency_heads.get(number, ""),
|
|
|
|
|
},
|
|
|
|
|
},
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
rationale="This harness certifies the post-merge platform; an unmerged dependency means it is measuring something else.",
|
|
|
|
|
)
|
|
|
|
|
for number in targets.dependency_pull_requests
|
|
|
|
|
]
|
|
|
|
|
return checks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _identity(targets: Targets) -> list[CheckSpec]:
|
|
|
|
|
codex = shell("json_fields", path=CODEX_HEALTH, fields=CODEX_HEALTH_FIELDS)
|
|
|
|
|
claude = shell("json_fields", path=CLAUDE_HEALTH, fields=CLAUDE_HEALTH_FIELDS)
|
|
|
|
|
agent = f"deploy/{targets.agent_deployment}"
|
|
|
|
|
return [
|
|
|
|
|
check(
|
|
|
|
|
"identity.codex-is-chatgpt-subscription",
|
|
|
|
|
"Codex authenticates through the ChatGPT subscription, not an API key",
|
|
|
|
|
"identity",
|
|
|
|
|
"json_field",
|
|
|
|
|
[step("health", SELF, *codex)],
|
|
|
|
|
{
|
|
|
|
|
"step": "health",
|
|
|
|
|
"fields": {
|
|
|
|
|
"authenticated": True,
|
|
|
|
|
"transport": "codex-chatgpt-subscription",
|
|
|
|
|
"state": "available",
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
rationale="Only allow-listed status fields are read; the credential itself is never opened.",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"identity.codex-evidence-is-fresh",
|
|
|
|
|
"Codex authentication evidence is current",
|
|
|
|
|
"identity",
|
|
|
|
|
"json_recent",
|
|
|
|
|
[step("health", SELF, *codex)],
|
|
|
|
|
{
|
|
|
|
|
"step": "health",
|
|
|
|
|
"now": targets.now,
|
|
|
|
|
"fields": {"checked_at": targets.max_evidence_age_seconds},
|
|
|
|
|
},
|
|
|
|
|
rationale="Stale provider evidence describes a state the release no longer has.",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"identity.claude-is-firstparty-subscription",
|
|
|
|
|
"Claude authenticates as a claude.ai first-party subscription",
|
|
|
|
|
"identity",
|
|
|
|
|
"json_field",
|
|
|
|
|
[step("health", SELF, *claude)],
|
|
|
|
|
{
|
|
|
|
|
"step": "health",
|
|
|
|
|
"fields": {
|
|
|
|
|
"authenticated": True,
|
|
|
|
|
"api_provider": "firstParty",
|
|
|
|
|
"auth_method": "claude.ai",
|
|
|
|
|
"transport": "claude-code-cli-subscription",
|
|
|
|
|
"state": "available",
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"identity.claude-evidence-is-fresh",
|
|
|
|
|
"Claude authentication evidence is current",
|
|
|
|
|
"identity",
|
|
|
|
|
"json_recent",
|
|
|
|
|
[step("health", SELF, *claude)],
|
|
|
|
|
{
|
|
|
|
|
"step": "health",
|
|
|
|
|
"now": targets.now,
|
|
|
|
|
"fields": {"checked_at": targets.max_evidence_age_seconds},
|
|
|
|
|
},
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"identity.no-provider-api-key-in-pod",
|
|
|
|
|
"No provider API-key variable is set in the running agent process",
|
|
|
|
|
"identity",
|
|
|
|
|
"names_absent",
|
|
|
|
|
[step("env", SELF, *shell("env_names"))],
|
|
|
|
|
{
|
|
|
|
|
"step": "env",
|
|
|
|
|
"names": PROVIDER_API_KEY_NAMES,
|
|
|
|
|
"contains": ("API_KEY",),
|
|
|
|
|
},
|
|
|
|
|
rationale="Names only. The probe lists variable names and never their values.",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"identity.no-provider-api-key-in-manifest",
|
|
|
|
|
"No provider API-key variable is declared on the agent workload",
|
|
|
|
|
"identity",
|
|
|
|
|
"names_absent",
|
|
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"env",
|
|
|
|
|
OPERATOR,
|
|
|
|
|
"kubectl",
|
|
|
|
|
"--namespace",
|
|
|
|
|
targets.namespace,
|
|
|
|
|
"get",
|
|
|
|
|
agent,
|
|
|
|
|
"-o",
|
|
|
|
|
ENV_NAME_PROJECTION,
|
|
|
|
|
)
|
|
|
|
|
],
|
|
|
|
|
{"step": "env", "names": PROVIDER_API_KEY_NAMES, "contains": ("API_KEY",)},
|
|
|
|
|
rationale="The operator counterpart to the in-pod probe: desired state and running state are asserted separately.",
|
|
|
|
|
),
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _routing(targets: Targets) -> list[CheckSpec]:
|
|
|
|
|
switchyard = f"deploy/{targets.switchyard_deployment}"
|
|
|
|
|
tail = shell("tail_lines", path=ROUTING_LOG, limit=str(targets.routing_tail_lines))
|
|
|
|
|
routing_step = step(
|
|
|
|
|
"routing", SWITCHYARD, *tail, record=False, max_bytes=targets.routing_tail_bytes
|
|
|
|
|
)
|
|
|
|
|
return [
|
|
|
|
|
check(
|
|
|
|
|
"routing.switchyard-is-available",
|
|
|
|
|
"Switchyard has a ready replica serving every model-call boundary",
|
|
|
|
|
"routing",
|
2026-08-17 13:11:34 +00:00
|
|
|
"lines_match",
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"deployment",
|
|
|
|
|
OPERATOR,
|
|
|
|
|
"kubectl",
|
|
|
|
|
"--namespace",
|
|
|
|
|
targets.namespace,
|
|
|
|
|
"get",
|
|
|
|
|
switchyard,
|
|
|
|
|
"-o",
|
2026-08-17 13:11:34 +00:00
|
|
|
projection("jsonpath={.status.readyReplicas}"),
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
)
|
|
|
|
|
],
|
2026-08-17 13:11:34 +00:00
|
|
|
{"step": "deployment", "pattern": r"^[1-9][0-9]*$", "minimum": 1},
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"routing.provider-effort-and-fallback-evidence",
|
|
|
|
|
"Routing evidence covers both providers, all graded efforts, and a real fallback",
|
|
|
|
|
"routing",
|
|
|
|
|
"routing_evidence",
|
|
|
|
|
[routing_step],
|
|
|
|
|
{
|
|
|
|
|
"step": "routing",
|
|
|
|
|
"providers": ("codex", "claude"),
|
|
|
|
|
"efforts": ("medium", "high", "xhigh"),
|
|
|
|
|
"lanes": ("route", "worker"),
|
|
|
|
|
"require_fallback_evidence": True,
|
|
|
|
|
"max_age_seconds": targets.max_evidence_age_seconds,
|
|
|
|
|
"now": targets.now,
|
|
|
|
|
},
|
|
|
|
|
rationale="The routing log records the decided route per boundary; the tail is parsed here and never pasted into the report.",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"routing.codex-latency-recorded",
|
|
|
|
|
"Codex boundary latency is measured, not assumed",
|
|
|
|
|
"routing",
|
|
|
|
|
"json_numeric",
|
2026-08-17 13:11:34 +00:00
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"health",
|
|
|
|
|
SELF,
|
|
|
|
|
*shell(
|
|
|
|
|
"json_fields", path=CODEX_HEALTH, fields=CODEX_HEALTH_FIELDS
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
],
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
{"step": "health", "fields": {"latency_ms": {"min": 1, "max": 600_000}}},
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"routing.claude-latency-recorded",
|
|
|
|
|
"Claude boundary latency is measured, not assumed",
|
|
|
|
|
"routing",
|
|
|
|
|
"json_numeric",
|
2026-08-17 13:11:34 +00:00
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"health",
|
|
|
|
|
SELF,
|
|
|
|
|
*shell(
|
|
|
|
|
"json_fields", path=CLAUDE_HEALTH, fields=CLAUDE_HEALTH_FIELDS
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
],
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
{"step": "health", "fields": {"latency_ms": {"min": 1, "max": 600_000}}},
|
|
|
|
|
),
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _scope_profile(
|
|
|
|
|
targets: Targets, name: str, workload: str, container: str, expected: str
|
|
|
|
|
) -> CheckSpec:
|
2026-08-17 13:11:34 +00:00
|
|
|
profile_projection = projection(
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
f'jsonpath={{.spec.template.spec.containers[?(@.name=="{container}")]'
|
|
|
|
|
'.env[?(@.name=="HERMES_AUTO_ROUTER_PROFILE")].value}'
|
|
|
|
|
)
|
|
|
|
|
return check(
|
|
|
|
|
f"scopes.{name}-profile-is-distinct",
|
|
|
|
|
f"The {name} surface runs under its own routing profile",
|
|
|
|
|
"scopes",
|
|
|
|
|
"stdout_matches",
|
|
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"profile",
|
|
|
|
|
OPERATOR,
|
|
|
|
|
"kubectl",
|
|
|
|
|
"--namespace",
|
|
|
|
|
targets.namespace,
|
|
|
|
|
"get",
|
|
|
|
|
workload,
|
|
|
|
|
"-o",
|
2026-08-17 13:11:34 +00:00
|
|
|
profile_projection,
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
)
|
|
|
|
|
],
|
|
|
|
|
{"step": "profile", "equals": expected},
|
|
|
|
|
rationale="Chat, agent, and triage must not collapse into one scope; each states its own profile.",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _scopes(targets: Targets) -> list[CheckSpec]:
|
|
|
|
|
return [
|
|
|
|
|
_scope_profile(
|
2026-08-17 13:11:34 +00:00
|
|
|
targets,
|
|
|
|
|
"agent",
|
|
|
|
|
f"deploy/{targets.agent_deployment}",
|
|
|
|
|
targets.agent_container,
|
|
|
|
|
"agent",
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
),
|
|
|
|
|
_scope_profile(
|
|
|
|
|
targets,
|
|
|
|
|
"chat",
|
|
|
|
|
f"statefulset/{targets.chat_statefulset}",
|
|
|
|
|
targets.chat_container,
|
|
|
|
|
"chat",
|
|
|
|
|
),
|
|
|
|
|
_scope_profile(
|
|
|
|
|
targets,
|
|
|
|
|
"triage",
|
|
|
|
|
f"deploy/{targets.triage_deployment}",
|
|
|
|
|
targets.triage_container,
|
|
|
|
|
"triage",
|
|
|
|
|
),
|
|
|
|
|
check(
|
|
|
|
|
"scopes.surfaces-have-distinct-hosts",
|
|
|
|
|
"Chat, agent, and triage are published on distinct hostnames",
|
|
|
|
|
"scopes",
|
|
|
|
|
"distinct_count",
|
|
|
|
|
[
|
|
|
|
|
step(
|
|
|
|
|
"hosts",
|
|
|
|
|
OPERATOR,
|
|
|
|
|
"kubectl",
|
|
|
|
|
"--namespace",
|
|
|
|
|
targets.namespace,
|
|
|
|
|
"get",
|
|
|
|
|
"ingress",
|
|
|
|
|
"-o",
|
2026-08-17 13:11:34 +00:00
|
|
|
projection(
|
|
|
|
|
'jsonpath={range .items[*]}{range .spec.rules[*]}{.host}{"\\n"}{end}{end}'
|
|
|
|
|
),
|
hermes: add a fail-closed full-handoff acceptance harness
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
2026-08-17 10:14:17 +00:00
|
|
|
)
|
|
|
|
|
],
|
|
|
|
|
{"step": "hosts", "minimum": 3},
|
|
|
|
|
),
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def platform_checks(targets: Targets) -> list[CheckSpec]:
|
|
|
|
|
"""Return the baseline, identity, routing, and scope checks."""
|
2026-08-17 13:11:34 +00:00
|
|
|
return [
|
|
|
|
|
*_baseline(targets),
|
|
|
|
|
*_identity(targets),
|
|
|
|
|
*_routing(targets),
|
|
|
|
|
*_scopes(targets),
|
|
|
|
|
]
|