Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.
The rename is additive, matching the shape #38 restored for chat and triage.
CoreDNS, both agent Ingresses and the hermes-sites certificate now serve BOTH
names, so merging this cannot take away the endpoint the operator uses to
reach the coordinator. Retiring agent.hermes.bstein.dev is a separate,
separately scheduled change. No redirect middleware is added.
What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak agent proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default
What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
rollback only needs the oauth2-proxy --redirect-url reverted and does not
require re-running the ensure job
The agent client passes its legacy origin through the optional fourth argument
#38 added to ensure_proxy_client, so no second mechanism is introduced. The
immutable ensure Job goes -11 -> -12 because #38 already consumed -11 and that
run has completed; without a further bump this change would never be applied.
Login on the new host fails until the -12 Job completes.
Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.
#38's public-host continuity test now covers the agent proxy's dual origins
rather than asserting the agent surface was untouched by the rename.
Knowledge catalogs and diagrams regenerated with `make knowledge`.
The 2026-08-18 metrics-storage outage exposed two defects in the
availability pipeline that distorted the figure in opposite directions at
once.
The Overview panel fell back to a live one-hour Traefik ratio whenever the
yearly rollup sample went stale for 48h, and rendered it under the same
"365d" title. When the rollup stopped publishing on 2026-08-18 the panel
quietly swapped a 365-day measurement for a 60-minute one and read 99.74%
instead of the recorded 99.95%. The fallback is removed: a stale rollup now
renders no value, and a new atlas-availability-rollup-stale alert pages at
26h, well before the panel goes blank at 48h.
The yearly ratio also silently excluded the 34-hour telemetry gap, because
missing days contribute zero requests and zero failures. Absent data was
read as "nothing happened" — had Atlas genuinely been down in that window,
the figure would still have said 99.95%. Availability keeps its
measured-days-only definition, which is correct, but coverage is now
published alongside it and shown in a new panel, so a telemetry gap lowers
disclosed coverage instead of vanishing. The title reads "365d window" to
stop implying 365 days of data exist; request-v4 begins 2026-05-01.
The rollup job reported healthy runs across a day and a half of lost
publishes: a read-only VictoriaMetrics accepts an import and discards it.
It now reads each sample back and fails loudly when the write did not
survive.
Not addressed here: availability is still measured from inside the platform
via Traefik counters, so it cannot distinguish "Atlas down" from "telemetry
down", and misses failures that never reach Traefik (DNS, TLS, node dead).
An external synthetic prober is the real fix and needs a hosting decision.
The acceptance harness pinned a forge client that has never existed in any
commit or pod (/opt/coordinator/gitea_api.py, digest f0943db4..., GIT/POST
grammar, an askpass helper). Every Gitea-backed check was therefore
unrunnable as merged. Point the harness at the credential-isolated SCM
broker client that actually ships in the agent pod.
- policy: GITEA_CLIENT=/opt/scm/gitea_api.py; trust /opt/scm/ instead of
the phantom /opt/coordinator/; admit the client's real grammar
(`read <api-path>`, exactly one path) with the same atlas/titan-iac pin
and dot-segment rejection; bare HTTP methods are refused in every mode.
The armed POST/PATCH/DELETE windows remain but are documented as
deferred: the deployed client cannot execute them.
- exec: pin the client digest to the sha256 of
services/hermes/scm-common/scripts/gitea_api.py — the exact file the
hermes-scm-boundary-v2 ConfigMap mounts at /opt/scm/gitea_api.py — so
the pin is derivable from merged source and equal to the deployed
client. gitea_api.py gains a narrow /api/v1/user identity read in
_authorize_read (see below), so the pin is the NEW source hash
76efd16dedbeb74425b12fbbdbfaa391854771292077e0463bf22706855ae6dc.
Drop the dangling GIT_ASKPASS (no helper exists; broker git needs
none) and swap /opt/coordinator for /opt/scm in SAFE_PATH.
- checks: all forge/baseline/lineage probes use (client, "read", path).
The SELF-vantage identity checks now truthfully assert the *broker's*
forge identity (the only one the platform can exercise) is not an
administrator and holds push-scoped, non-administrative repository
authority; the administrative-route check asserts the broker read
allowlist's live refusal of branch_protections. The remote-main step
keeps `origin` (the broker remote exists only in pool workspaces and
the broker origin is cluster-local); its rationale now tells the
operator to ensure origin fetchability.
- gitea_api.py/_authorize_read: allow exactly `/api/v1/user` (no query,
no sibling routes) as operation "identity" so the harness can prove
the broker identity is not an administrator. The broker imports the
same module, so one reviewed edit covers both sides of the boundary.
- rules: DENIAL_MARKERS now match the client's real refusal lines
("SCM broker request failed with HTTP 400/403" and the no-credential
rejection) and drop "gitea api returned http 403", which the client
never emits; a broker 404 is deliberately not denial evidence.
- ephemeral: index/verification reads use the real grammar; manual
cleanup guidance now says close/delete require operator forge
credentials (the client exposes no mutation besides create-draft);
armed mode is documented as deferred until the probes are rebuilt on
the broker's bounded mutation surface.
- docs: broker vantage/evidence section, operator prerequisites (broker
healthy, no /vault/secrets/gitea-token anywhere on the harness path,
current ConfigMap mount, operator-side client + origin fetchability),
armed-mode deferral.
- tests: read-grammar accepted / GET refused in every mode, /opt/scm
attestation pin proven equal to the merged source digest, real
denial-marker matching, /api/v1/user identity route bounds; the
repository-pin mutant probe speaks the new grammar. Full handoff +
gitea + broker families pass (952 tests), mutation gate 13/13, per-file
line+branch coverage >=95%, all touched sources within the 500-line cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reviewed PR stack is now merged into main, so an open draft PR #19 is
no longer proof that the reviewed code is what runs. The mandatory
release.exact-lineage-is-running check now requires the merged terminal
state instead: PR #19 closed with merged=true, base main, the existing
feature ref, and the exact reviewed head, plus a new merge-ancestry step
that proves the reviewed head is an ancestor of the pinned origin/main
via git merge-base --is-ancestor. An open PR, a PR closed without
merging, a mismatched head, or a head that is not a proven ancestor of
main still fails closed; an undecidable ancestry probe is NOT_RUN. The
recorded pre-merge base SHA is no longer compared against current main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exercise the mailu sync retry, attribute, and skip branches, the
listener non-object JSON path, and the hygiene conftest skip; drop the
unreachable inverted-range clamp in the semgrep report (the line helper
already floors the end line) and pin that behavior with a test. Exclude
the mailu __main__ guards from measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The distributed worker pool stacks on exactly two open pull requests, in this
order: PR 14 supplies the broker-only SCM boundary the mediators route through
and the hermes-scm-boundary-v2 ConfigMap they mount, and PR 15 supplies the
cli_lane_* decomposition -- including canonical_run_id and the eligibility
predicate on claim_ready -- that the coordinator depends on. Neither can be
dropped without breaking a fixed P0 boundary, so both are carried here as
prerequisites and this branch must not merge before them.
PR 16 (agent image release lane) and PR 19 (full-handoff acceptance harness)
are NOT prerequisites and are deliberately absent, so reviewing this branch no
longer means approving them.
PR 14 and PR 15 conflict with each other in nine paths. Each is resolved to the
resolution already reviewed on this branch at 4d4cf1bd.
evaluate_names_absent returned PASS when its step exited 0 with no output,
so five mandatory checks - the ones asserting that provider API keys, forge
credentials, a cluster-admin binding, and shared coordinator state are
absent - could report a pass on no evidence and turn a NO_GO into a GO.
Both name rules now resolve their step through one guard in _line_step, so
zero observations are NOT_RUN. Regressions pin all five real catalog specs
plus both reachable silence paths: a POSIX pipeline whose status comes from
its last stage, and a drifted kubectl -o jsonpath. The pool claim projection
emits one <volume>=<claim> line per template volume so a volume without a
PVC still counts as an observation rather than reading as drift.
Also closes the review's reachable hardening and evidence defects:
- pin Gitea paths to atlas/titan-iac on an exact segment boundary and
reject relative segments, including percent-encoded ones
- forbid impersonation structurally in every mode and vantage; the inner
command of kubectl exec is re-checked rather than exempted, and
validate_catalog no longer guards only the operator vantage
- drop flux and helm from the binary allowlist; they had no pinned release
digest, so no allowlisted binary can now be admitted that the executor
would refuse to attest
- remove the inert --concurrency and --expect-telegram-sessions flags and
the dead concurrency bound; Telegram continuity stays mandatory
- read the ephemeral pull index page by page, treat the create response as
an authoritative source for the pull number, close every number either
source names, and surface residue_ref plus exact manual_cleanup commands
when creation is uncertain
- keep executable_path and executable_sha256 on unrecorded bulk-evidence
steps so withholding bytes never withholds binary attestation
- revert the repo-wide hygiene legacy-exception mechanism; the contract
change here is purely additive and the four pre-existing over-cap files
are left to the canonical contract change in PR #14/#15
- correct the runbook ruff format scope so the documented command passes
Split hermes_handoff_arming.py out of hermes_handoff_ephemeral.py to keep
both modules under the 500-line cap. All 16 handoff modules hold at least
95% line and branch coverage; the mutation gate is 13/13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
The cassandra pipelineJob had no trigger, so Gitea's notifyCommit
webhook found no matching job and pushes only built when started
manually. Poll every 5 minutes like lesavka/typhon so pushes build
within one poll cycle and the webhook race disappears.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bugs, both visible in a single demo transcript.
The stage memory was one shared set cleared whenever the incident changed. A
ten-minute window routinely holds two incidents - the build just pushed and
the one before it - so they wiped each other's progress and every stage
reprinted on every poll, forever. Memory is now per incident, and the monitor
follows one incident at a time: --incident pins an exact id, --filter matches
a substring, and otherwise the newest wins, which is what someone who just
triggered a build wants.
Worse: the diagnosis panel read the newest diagnosis in the whole table, not
the one for the incident on screen. During a code-demo run it displayed
ananke/249 - a different service, a different failure - directly beneath the
heading naming this incident. Showing an unrelated answer is worse than
showing nothing, because nothing about it looks wrong.
It is now scoped to the incident and reads both event types, since the code
path records a code_proposal rather than a diagnosis - which is why nothing
matched and the fallback took over. There is no fallback any more: an incident
with no recorded run says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both demos went quiet for up to a minute between the build turning red and
the monitor reacting, because Ariadne's tick is on cron. The scripts now run
that tick immediately over the pod's own loopback - nothing exposed outside
the cluster - and print what it saw, so the pause becomes a visible step
rather than dead air.
Falls back to silence rather than failure: if the request does not land the
scheduler still picks the build up within the minute, which is exactly the
old behaviour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wait_for_build slept before its first poll, so a build that had already
finished still bought a full interval of silence. On stage that reads as the
script having missed the result - the build goes red in Jenkins and the
terminal sits there. Polling first and sleeping after removes it entirely: a
finished build now returns in under a second, measured.
The interval drops from ten seconds to three for the same reason. The wait is
dead air in front of an audience and a Jenkins status read is cheap. The
budget is now expressed in seconds rather than poll counts, so shortening the
interval does not silently shorten the timeout.
This is only the script's own latency. Ariadne's autotriage cron is * * * * *,
so an incident still takes up to a minute to appear after a build fails, and
no amount of polling here changes that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The run tab said a defect was seeded and then moved on. The audience had to
take on trust both that a change was made and that the diagnosis Hermes
produced two minutes later actually matched it. Printing the diff first closes
that gap: they watch the wrong line go in, then watch a model find the same
line without being told where it is.
Placed before the commit rather than after the push because this is the only
moment in the demo where a human changes any code, and it is worth being seen
as such. The short sha follows the push so the commit on screen can be matched
to the one Jenkins builds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Hermes stage is where the demo claims a model made the call. It showed the
stored JSON and named the run id, which asks the audience to take the rest on
trust. It now prints the console link for that exact run, so the claim can be
opened on screen instead of described.
Same route the pull request links to, so following either lands on the same
page.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hermes_code_demo.sh run crashed before it did anything, with a JSON traceback
pointing at the parser rather than the cause. Jenkins tree selectors use square
brackets; this curl build treats them as glob metacharacters and declines to
send the request, so the body came back empty and json.load reported column 1.
Encoded, and an empty body now says which job and which Jenkins rather than
raising from inside the parser. The crash was harmless as crashes go - it
happened before the defect was seeded, so nothing was pushed - but it happened
at the exact moment a demo starts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One script with a 'fixture' and a 'code' subcommand meant the wrong subcommand
was always one word away, in front of an audience, with different blast radii
behind each: the triage demo touches only a ConfigMap, the code demo pushes to
a repository and deletes issues. Those should not share a command line.
Each demo is now its own driver with the same five verbs - monitor, reset,
preflight, run, status - so knowing one teaches the other. What they genuinely
share (credentials, Jenkins access, the tick reader, the lab-wide preflight
checks) moved to hermes_demo_lib.sh rather than being duplicated, because the
reason to split was clarity at the command line, not two copies of the same
helper drifting apart.
Each reset now covers only its own demo. The triage reset no longer reaches
into a Gitea repository it never writes to, and the code reset owns the
repository cleanup entirely.
The credentials file is now hermes_demo.env since both read it; the old
hermes_triage_demo.env is still sourced as a fallback so a filled-in file
keeps working, and both names stay git-ignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scripts said "flow chart" while the artifact the audience is following
has a name. Telling a room to follow along on the flow chart and then handing
them something titled Test Automation Diagram makes them hunt for the match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The code demo seeds its defect by pushing to master, and the fix only lands if
someone merges the pull request - which, by design, nobody does during a demo.
So master stays broken, and reset only printed "revert it before demoing"
while leaving it that way. The second run of the day then aborted on "defect
already present" before anything started.
Reset now reverts it on master. The substitution is anchored so it cannot
match an already-correct divisor, which keeps reset idempotent: running it on
a healthy repository changes nothing rather than corrupting the file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
