Introduce worker.bstein.dev as the canonical hostname for the owner-only
Hermes coordinator, previously agent.hermes.bstein.dev.
The old host is deliberately kept alive. CoreDNS, both agent Ingresses and
the hermes-sites certificate now serve BOTH names, so merging this cannot
take away the endpoint the operator uses to reach the coordinator. Retiring
agent.hermes.bstein.dev is a separate, separately scheduled change. No
redirect middleware is added.
What switches to the new host:
- HERMES_DASHBOARD_PUBLIC_URL and the oauth2-proxy --redirect-url
- the Keycloak hermes-agent-proxy rootUrl
- operator docs, skills, the ZAP baseline target and the triage monitor default
What stays dual-homed until retirement:
- CoreDNS hosts entry, both agent Ingress rules, certificate SANs
- API_SERVER_CORS_ORIGINS (now a comma-separated pair)
- the Keycloak redirect URIs, web origins and post-logout origins, so a
rollback only needs the oauth2-proxy --redirect-url reverted and does not
require re-running the ensure job
ensure_proxy_client now takes optional trailing legacy URLs; called with a
single URL it emits the identical payload as before, so the chat and triage
clients are unaffected. The ensure Job is bumped to -11 so Flux re-runs it;
login on the new host fails until that Job completes.
Because the session and CSRF cookies use the __Host- prefix they are bound to
one origin, so a fresh login must start on worker.bstein.dev and existing
sessions do not carry over -- re-login is required after rollout.
Knowledge catalogs and diagrams regenerated with `make knowledge`.
The acceptance harness pinned a forge client that has never existed in any
commit or pod (/opt/coordinator/gitea_api.py, digest f0943db4..., GIT/POST
grammar, an askpass helper). Every Gitea-backed check was therefore
unrunnable as merged. Point the harness at the credential-isolated SCM
broker client that actually ships in the agent pod.
- policy: GITEA_CLIENT=/opt/scm/gitea_api.py; trust /opt/scm/ instead of
the phantom /opt/coordinator/; admit the client's real grammar
(`read <api-path>`, exactly one path) with the same atlas/titan-iac pin
and dot-segment rejection; bare HTTP methods are refused in every mode.
The armed POST/PATCH/DELETE windows remain but are documented as
deferred: the deployed client cannot execute them.
- exec: pin the client digest to the sha256 of
services/hermes/scm-common/scripts/gitea_api.py — the exact file the
hermes-scm-boundary-v2 ConfigMap mounts at /opt/scm/gitea_api.py — so
the pin is derivable from merged source and equal to the deployed
client. gitea_api.py gains a narrow /api/v1/user identity read in
_authorize_read (see below), so the pin is the NEW source hash
76efd16dedbeb74425b12fbbdbfaa391854771292077e0463bf22706855ae6dc.
Drop the dangling GIT_ASKPASS (no helper exists; broker git needs
none) and swap /opt/coordinator for /opt/scm in SAFE_PATH.
- checks: all forge/baseline/lineage probes use (client, "read", path).
The SELF-vantage identity checks now truthfully assert the *broker's*
forge identity (the only one the platform can exercise) is not an
administrator and holds push-scoped, non-administrative repository
authority; the administrative-route check asserts the broker read
allowlist's live refusal of branch_protections. The remote-main step
keeps `origin` (the broker remote exists only in pool workspaces and
the broker origin is cluster-local); its rationale now tells the
operator to ensure origin fetchability.
- gitea_api.py/_authorize_read: allow exactly `/api/v1/user` (no query,
no sibling routes) as operation "identity" so the harness can prove
the broker identity is not an administrator. The broker imports the
same module, so one reviewed edit covers both sides of the boundary.
- rules: DENIAL_MARKERS now match the client's real refusal lines
("SCM broker request failed with HTTP 400/403" and the no-credential
rejection) and drop "gitea api returned http 403", which the client
never emits; a broker 404 is deliberately not denial evidence.
- ephemeral: index/verification reads use the real grammar; manual
cleanup guidance now says close/delete require operator forge
credentials (the client exposes no mutation besides create-draft);
armed mode is documented as deferred until the probes are rebuilt on
the broker's bounded mutation surface.
- docs: broker vantage/evidence section, operator prerequisites (broker
healthy, no /vault/secrets/gitea-token anywhere on the harness path,
current ConfigMap mount, operator-side client + origin fetchability),
armed-mode deferral.
- tests: read-grammar accepted / GET refused in every mode, /opt/scm
attestation pin proven equal to the merged source digest, real
denial-marker matching, /api/v1/user identity route bounds; the
repository-pin mutant probe speaks the new grammar. Full handoff +
gitea + broker families pass (952 tests), mutation gate 13/13, per-file
line+branch coverage >=95%, all touched sources within the 500-line cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reviewed PR stack is now merged into main, so an open draft PR #19 is
no longer proof that the reviewed code is what runs. The mandatory
release.exact-lineage-is-running check now requires the merged terminal
state instead: PR #19 closed with merged=true, base main, the existing
feature ref, and the exact reviewed head, plus a new merge-ancestry step
that proves the reviewed head is an ancestor of the pinned origin/main
via git merge-base --is-ancestor. An open PR, a PR closed without
merging, a mismatched head, or a head that is not a proven ancestor of
main still fails closed; an undecidable ancestry probe is NOT_RUN. The
recorded pre-merge base SHA is no longer compared against current main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exercise the mailu sync retry, attribute, and skip branches, the
listener non-object JSON path, and the hygiene conftest skip; drop the
unreachable inverted-range clamp in the semgrep report (the line helper
already floors the end line) and pin that behavior with a test. Exclude
the mailu __main__ guards from measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fenced worker Pods claim Hermes Kanban runs through a coordinator that
owns every state transition, with per-ordinal HMAC authority, a mediated
broker-only SCM path, and durable per-ordinal workspaces.
Content is the reviewed head of PR #18 (689bcb6e) with PR 16's and PR 19's
contributions removed: they were merged in only to validate co-existence and are
not prerequisites, so this branch no longer carries them as ancestors. Only PR 14
and PR 15 remain, because the broker boundary and the cli_lane_* decomposition
are load-bearing for two of the fixed P0 boundaries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The distributed worker pool stacks on exactly two open pull requests, in this
order: PR 14 supplies the broker-only SCM boundary the mediators route through
and the hermes-scm-boundary-v2 ConfigMap they mount, and PR 15 supplies the
cli_lane_* decomposition -- including canonical_run_id and the eligibility
predicate on claim_ready -- that the coordinator depends on. Neither can be
dropped without breaking a fixed P0 boundary, so both are carried here as
prerequisites and this branch must not merge before them.
PR 16 (agent image release lane) and PR 19 (full-handoff acceptance harness)
are NOT prerequisites and are deliberately absent, so reviewing this branch no
longer means approving them.
PR 14 and PR 15 conflict with each other in nine paths. Each is resolved to the
resolution already reviewed on this branch at 4d4cf1bd.
evaluate_names_absent returned PASS when its step exited 0 with no output,
so five mandatory checks - the ones asserting that provider API keys, forge
credentials, a cluster-admin binding, and shared coordinator state are
absent - could report a pass on no evidence and turn a NO_GO into a GO.
Both name rules now resolve their step through one guard in _line_step, so
zero observations are NOT_RUN. Regressions pin all five real catalog specs
plus both reachable silence paths: a POSIX pipeline whose status comes from
its last stage, and a drifted kubectl -o jsonpath. The pool claim projection
emits one <volume>=<claim> line per template volume so a volume without a
PVC still counts as an observation rather than reading as drift.
Also closes the review's reachable hardening and evidence defects:
- pin Gitea paths to atlas/titan-iac on an exact segment boundary and
reject relative segments, including percent-encoded ones
- forbid impersonation structurally in every mode and vantage; the inner
command of kubectl exec is re-checked rather than exempted, and
validate_catalog no longer guards only the operator vantage
- drop flux and helm from the binary allowlist; they had no pinned release
digest, so no allowlisted binary can now be admitted that the executor
would refuse to attest
- remove the inert --concurrency and --expect-telegram-sessions flags and
the dead concurrency bound; Telegram continuity stays mandatory
- read the ephemeral pull index page by page, treat the create response as
an authoritative source for the pull number, close every number either
source names, and surface residue_ref plus exact manual_cleanup commands
when creation is uncertain
- keep executable_path and executable_sha256 on unrecorded bulk-evidence
steps so withholding bytes never withholds binary attestation
- revert the repo-wide hygiene legacy-exception mechanism; the contract
change here is purely additive and the four pre-existing over-cap files
are left to the canonical contract change in PR #14/#15
- correct the runbook ruff format scope so the documented command passes
Split hermes_handoff_arming.py out of hermes_handoff_ephemeral.py to keep
both modules under the 500-line cap. All 16 handoff modules hold at least
95% line and branch coverage; the mutation gate is 13/13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Decides whether the Hermes platform handoff is fit to release, and refuses
to round an absence of evidence up to a pass.
The harness is read-only by default and classifies 71 checks PASS / FAIL /
NOT_RUN / NOT_APPLICABLE. Any mandatory FAIL or NOT_RUN is NO_GO, and so is a
harness-level problem: an unreachable vantage, a catalog entry whose evidence
no longer exists, an expired deadline, or an evaluator that raised.
Evidence comes from two vantages that cannot cover for each other: an external
read-only operator kubeconfig, and the Hermes agent probing itself from inside
its own pod. Before any check runs, the harness asks each vantage who it is and
stops if they are the same principal, because dual-vantage evidence from one
identity is a restatement rather than a corroboration. `--as` is rejected for
every operator-side command and reachable only as the inner command of a
`kubectl exec`, so impersonation can never stand in for a real self-probe. A
deny check needs a live refused request, not only an authorization review.
Two safety properties are structural rather than conventional, enforced where
an argv becomes a subprocess: the default mode mutates nothing (mutating verbs
require a server dry run; there is deliberately no live TokenRequest probe,
because a successful one would mint a real credential), and no probe can pull a
credential value into a report (no vault/sops/curl, secrets readable only with
-o name, environment probes list names, shell only through frozen reviewed
templates). Captures are bounded before they are screened, and the rendered
report is re-screened before it is written.
Mutation lives behind a separate arming flag with an exact confirmation phrase,
a caller-supplied unique ref, a preflight that refuses a protected push target
before any network call, and a cleanup whose verification is itself mandatory.
A default run reports those four checks NOT_RUN.
The catalog is declarative so a reviewer reads what is asserted rather than how
it is plumbed, and so structural properties can be proven over every entry
before a run. Catalog drift surfaces as NOT_RUN, which stops the release.
docs/hermes_full_handoff_acceptance.md carries the merge order for PRs #14-#18
on top of the merged #13 baseline, the image build and Flux rollout, the
rollback point for each step, the go/no-go checklist, and the limits that are
asserted rather than exercised.
Validation: 295 handoff tests pass with 100% line coverage on all 15 new
modules; the full unit suite is 647 passed with two failures that reproduce
unchanged on origin/main; Ruff, py_compile, kustomize render, and a diff
credential screen are clean; a live read-only run against Atlas returns NO_GO
for the pre-merge cluster with no unscreened fields in the report.
The cassandra pipelineJob had no trigger, so Gitea's notifyCommit
webhook found no matching job and pushes only built when started
manually. Poll every 5 minutes like lesavka/typhon so pushes build
within one poll cycle and the webhook race disappears.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bugs, both visible in a single demo transcript.
The stage memory was one shared set cleared whenever the incident changed. A
ten-minute window routinely holds two incidents - the build just pushed and
the one before it - so they wiped each other's progress and every stage
reprinted on every poll, forever. Memory is now per incident, and the monitor
follows one incident at a time: --incident pins an exact id, --filter matches
a substring, and otherwise the newest wins, which is what someone who just
triggered a build wants.
Worse: the diagnosis panel read the newest diagnosis in the whole table, not
the one for the incident on screen. During a code-demo run it displayed
ananke/249 - a different service, a different failure - directly beneath the
heading naming this incident. Showing an unrelated answer is worse than
showing nothing, because nothing about it looks wrong.
It is now scoped to the incident and reads both event types, since the code
path records a code_proposal rather than a diagnosis - which is why nothing
matched and the fallback took over. There is no fallback any more: an incident
with no recorded run says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both demos went quiet for up to a minute between the build turning red and
the monitor reacting, because Ariadne's tick is on cron. The scripts now run
that tick immediately over the pod's own loopback - nothing exposed outside
the cluster - and print what it saw, so the pause becomes a visible step
rather than dead air.
Falls back to silence rather than failure: if the request does not land the
scheduler still picks the build up within the minute, which is exactly the
old behaviour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wait_for_build slept before its first poll, so a build that had already
finished still bought a full interval of silence. On stage that reads as the
script having missed the result - the build goes red in Jenkins and the
terminal sits there. Polling first and sleeping after removes it entirely: a
finished build now returns in under a second, measured.
The interval drops from ten seconds to three for the same reason. The wait is
dead air in front of an audience and a Jenkins status read is cheap. The
budget is now expressed in seconds rather than poll counts, so shortening the
interval does not silently shorten the timeout.
This is only the script's own latency. Ariadne's autotriage cron is * * * * *,
so an incident still takes up to a minute to appear after a build fails, and
no amount of polling here changes that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The run tab said a defect was seeded and then moved on. The audience had to
take on trust both that a change was made and that the diagnosis Hermes
produced two minutes later actually matched it. Printing the diff first closes
that gap: they watch the wrong line go in, then watch a model find the same
line without being told where it is.
Placed before the commit rather than after the push because this is the only
moment in the demo where a human changes any code, and it is worth being seen
as such. The short sha follows the push so the commit on screen can be matched
to the one Jenkins builds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Hermes stage is where the demo claims a model made the call. It showed the
stored JSON and named the run id, which asks the audience to take the rest on
trust. It now prints the console link for that exact run, so the claim can be
opened on screen instead of described.
Same route the pull request links to, so following either lands on the same
page.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hermes_code_demo.sh run crashed before it did anything, with a JSON traceback
pointing at the parser rather than the cause. Jenkins tree selectors use square
brackets; this curl build treats them as glob metacharacters and declines to
send the request, so the body came back empty and json.load reported column 1.
Encoded, and an empty body now says which job and which Jenkins rather than
raising from inside the parser. The crash was harmless as crashes go - it
happened before the defect was seeded, so nothing was pushed - but it happened
at the exact moment a demo starts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One script with a 'fixture' and a 'code' subcommand meant the wrong subcommand
was always one word away, in front of an audience, with different blast radii
behind each: the triage demo touches only a ConfigMap, the code demo pushes to
a repository and deletes issues. Those should not share a command line.
Each demo is now its own driver with the same five verbs - monitor, reset,
preflight, run, status - so knowing one teaches the other. What they genuinely
share (credentials, Jenkins access, the tick reader, the lab-wide preflight
checks) moved to hermes_demo_lib.sh rather than being duplicated, because the
reason to split was clarity at the command line, not two copies of the same
helper drifting apart.
Each reset now covers only its own demo. The triage reset no longer reaches
into a Gitea repository it never writes to, and the code reset owns the
repository cleanup entirely.
The credentials file is now hermes_demo.env since both read it; the old
hermes_triage_demo.env is still sourced as a fallback so a filled-in file
keeps working, and both names stay git-ignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scripts said "flow chart" while the artifact the audience is following
has a name. Telling a room to follow along on the flow chart and then handing
them something titled Test Automation Diagram makes them hunt for the match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The code demo seeds its defect by pushing to master, and the fix only lands if
someone merges the pull request - which, by design, nobody does during a demo.
So master stays broken, and reset only printed "revert it before demoing"
while leaving it that way. The second run of the day then aborted on "defect
already present" before anything started.
Reset now reverts it on master. The substitution is anchored so it cannot
match an already-correct divisor, which keeps reset idempotent: running it on
a healthy repository changes nothing rather than corrupting the file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The diagnosis dump was clipped at 16 lines, which was already tight and now
truncates the stored outcome once a suggested remediation is present - the
audience would see the JSON cut off mid-object at exactly the moment the
interesting field appears.
Also says what that field means when it shows up, since a proposal is easy to
misread as something Ariadne is about to do. It is inert: no gate reads it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three categories now appear in the deployment next to the action
allowlist, and the monitor prints both at the policy gate so the difference is
visible during a demo rather than asserted: two ids Ariadne may execute on its
own authority, three categories it may only ask Hermes to propose a patch for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Several stages assumed the fixture job. On a code-demo run they would have
read the fixture ConfigMap, which is unrelated to that job, and asserted that
a field changing from unhealthy to healthy was the repair - flatly false when
the outcome is a pull request. The route stage would also have claimed the
source-proposal branch was not taken during the run that takes it.
The stages now branch on the job: the route explains that a source fix is not
a registered action and so takes the proposal branch; the response points at
the pull request recorded on the incident; and the verify stage says the
branch build validates the proposal and that nothing merges without a human
whatever it reports.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gitea wedged at 2073Mi against a 2Gi limit: its API stopped answering even on
its own loopback, and the repeated SSH LoginGraceTime drops in its log were
starvation symptoms rather than a separate fault. Raised to 3Gi.
The reset command's Gitea calls had no --max-time, so a slow service became an
indefinite hang with no output - the script appeared frozen after 'clearing
bstein/hermes-code-demo'. They now fail after 25 seconds and say so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five corrections to how the monitor explains itself, plus code-demo support.
The authorization is now described as what it is: a match between an action
Ariadne already has code to perform, an action id Hermes is permitted to
request, and the action Hermes actually recommended. Hermes cannot add to that
registry, and the recommendation proceeds only because it names something
already in it. The editorial line about pull requests is gone.
The repair now reads as a change rather than a value: the field said unhealthy
when the build failed and says healthy because Ariadne just patched it.
The rebuild stage says plainly that the operational branch is complete and
that neither the human-required response nor the optional source proposal is
entered, rather than leaving the reader to infer it from silence.
Inspectable outputs states that on the chart it is the 'records and artifacts'
edge out of the whole response box, not out of one branch - every path ends
there. The layout makes it look like it belongs to the proposal path alone.
follows hermes-code-demo, which takes the proposal branch: the
stages now read as a patch proposal and a pull request instead of being
mislabelled an escalation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bundle stage printed the job allowlist, which says nothing about what was
actually sent. It now samples the bundle itself, rebuilt with the same
collector Ariadne used - durable, because a finished build's console does not
change - and trimmed hard, since the point is to show what each source
contributes rather than to reprint it:
jenkins.console_failures : 1 region(s), truncated=False
| ERROR: Demo fixture check failed for incident hermes-triage-demo/34
jenkins.failed_tests : 0
log_evidence.records : 35 from OpenSearch kube-*
| [jenkins] Using /home/jenkins/agent/remoting as a remoting work directory
The diagnosis now also carries run_id and run_seconds, so the run can be
opened in the Hermes dashboard to see the prompt it was given and every tool
call it made - the request, not just the answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The detection stage read the fixture ConfigMap, which is a live value. Every
stage describes a moment that has passed, and by the time detection is
narrated the repair has already run - so it printed 'healthy' and implied the
fixture had never failed, contradicting the premise of the whole run. Replay
made it certain rather than occasional.
It now prints the incident's recorded state changes, which stay true
afterwards:
21:48:03 detected {result: FAILURE, ...}
21:48:18 diagnosed {classification: known_demo_fixture_failure, ...}
21:48:18 repairing {action: repair_demo_fixture}
21:48:18 awaiting_rebuild {repair: configmap_patch, ...}
21:49:03 resolved {resolved_by_build: 35}
The fixture read stays on the response stage, where healthy is the point.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Hermes stage proved only that a pod was running, when the substantive
evidence is the JSON Hermes returned; it now reads that from the audit trail,
so classification, confidence and requested action are on screen. The policy
gates stage never showed the verdict it was describing; authorized and
authorize_reason now appear with it.
The chart paths were paraphrases. They are now the node labels from
mermaid/TestAutomation.mmd verbatim - Response check, Scoped repair guard,
Action authorizer, Action registry, Scoped ConfigMap repair, Action result -
so a line on screen can be found on the diagram. The checklist carries them too.
The Optional source proposal branch was silently absent rather than explained.
The route stage now states it was not taken and why: a fixture repair is an
operational action, and source fixes are not actions at all.
Also widened the log window and its timeout; the fetch occasionally exceeded
25s and dropped a poll.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It read only the newest tick from a --tail=400 window. Ariadne now emits
roughly two hundred lines per tick, so that window held about two ticks, and a
step that lasted a single tick - the repair, most importantly - vanished if a
poll landed after it. Build 30 was triaged, repaired and resolved correctly
while the monitor printed nothing at all.
The window is now time-based, and every tick inside it is replayed rather than
just the last. mark() was already idempotent, so replaying is safe and the
monitor recovers stages it was not running for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An audience watching a dashboard has to take the result on trust. Watching
kubectl run against the cluster and reading the raw answer is the difference
between a demonstration and an assertion, so every read is now printed before
it runs and its output shown beneath.
Each stage also names the service UI worth looking at when something changes
there: the Jenkins job as the build goes red and again when the rebuild starts
on its own, the Hermes dashboard as the agent run appears, the Grafana triage
panels and the filed issues at the end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An audience following mermaid/TestAutomation.mmd needs to know which edge off
the route diamond was taken, which was the one thing the output left them to
infer. It now quotes the chart's own labels - authorized action, human
required, and the optional code path - and the docstring states where the
reporting is coarser than the diagram.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A rerun should start from nothing. DEMO_REPOS is the entire blast radius and
is deliberately explicit: a real service's issues are genuine triage records,
and clearing them to tidy a demo would destroy the evidence the system exists
to produce.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The resolving tick remains the newest one until another incident opens, so the
completed checklist reprinted on every poll.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reset restores only what the demo itself creates: the fixture ConfigMap, the
demo repository's open repair pull requests and their branches. Real service
repositories are deliberately untouched - those issues are genuine triage
records, and deleting them to tidy a demo would destroy the evidence the
system exists to produce.
monitor streams the flow in the stages of mermaid/TestAutomation.mmd, so a
second terminal can be followed against the chart: Detect and gather, Hermes
analysis, Ariadne policy gates, Ariadne response, Inspectable outputs. Each
stage prints the evidence that moved it, since the claim the chart makes is
that every step is answerable from data rather than asserted. Read-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
