titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

hermes: restore legacy chat/triage hosts alongside the renamed ones #38

Merged
bstein merged 3 commits from fix/hermes-restore-legacy-chat-triage-hosts-v2 into main 2026-08-21 10:17:49 +00:00
Conversation 1 Commits 3 Files Changed 6 +207 -7
hermes-automation commented 2026-08-21 08:58:25 +00:00
Member
Copy Link

Supersedes #37 and #36 - please close both in favour of this one. Same manifest changes as #37, plus a regression test. (The SCM broker only allows new-branch creation, never branch updates, so adding the test required a new branch and PR rather than a push to #37.)

Why

Merged PR #34 renamed the chat/triage public hosts in place instead of adding the new names. It removed chat.hermes.bstein.dev and triage.hermes.bstein.dev from the certificate SANs, the hermes-sites Ingress rules and the CoreDNS overrides in one change. Verified live on 2026-08-21:

  • Legacy hosts: HTTP 404 with Traefik's default self-signed certificate (no Ingress rule matches), and no longer resolvable in cluster CoreDNS.
  • Renamed hosts: TLS and routing healthy (302 to Keycloak, oauth2-proxy /ping = 200), but Keycloak answers the auth redirect with HTTP 400 "Invalid parameter", because hermes-chat-proxy and hermes-triage-proxy still hold the pre-rename callback URLs and the immutable ensure Job never reran.

Net effect: chat and triage have no working hostname at all. The post-merge runbook's rollback stance for a failed OIDC step is to reinstate the known-good old DNS/ingress/cert/OIDC targets rather than remove anything.

What this does

Makes the rename additive instead of destructive:

  • services/hermes/agent-certificate.yaml - legacy names back on hermes-sites-tls (5 SANs; agent.hermes.bstein.dev preserved).
  • services/hermes/agent-ingress.yaml - legacy host rules restored against the same oauth2-proxy-hermes-chat / -triage backends, plus both names on the TLS block.
  • infrastructure/core/coredns-custom.yaml - both legacy overrides restored for in-cluster resolution.
  • services/keycloak/scripts/hermes_access_oidc_ensure.sh - ensure_proxy_client takes an optional legacy origin and registers old and new callback URLs, web origins and post-logout origins together. rootUrl stays on the canonical new host.
  • services/keycloak/bootstrap-jobs/hermes-access-oidc-client-job.yaml - Job bumped to ensure-11 so Flux recreates the immutable Job.
  • testing/tests/test_hermes_public_host_continuity.py - new regression test.

Legacy hosts are kept serving rather than redirected: oauth2-proxy cookies are host-bound, so a redirect would silently drop live sessions. Retiring them stays a separate, explicit change.

Validation

End-to-end run of the actual ensure script against a mock Keycloak admin API and mock Vault (only the three bootstrap lines repointed - the vault env source, KC_URL and the service-account token path; ensure_proxy_client itself verbatim). Script exits 0 and records:

  • hermes-chat-proxy - redirectUris [chat.bstein.dev/oauth2/callback, chat.hermes.bstein.dev/oauth2/callback], webOrigins both, rootUrl https://chat.bstein.dev, post-logout "new##legacy"
  • hermes-triage-proxy - same shape for the triage pair
  • hermes-agent-proxy and hermes-automation - payloads byte-identical to the same run against main, so the untouched clients cannot regress

16 assertions pass, including PKCE S256 and confidential-client flags preserved, and no client secret or admin password echoed into any client payload. A control run of main's script reproduces the root cause: it registers only the new host.

The new test pins the invariant that a public host is either served by all four layers (certificate SAN, CoreDNS, Ingress rule, Keycloak registration) or by none. Against the pre-fix tree it fails 9 times across both legacy hosts and all four layers; against this branch the suite is green (19 passed). Also checked: kubectl kustomize services/hermes renders 5 SANs and 4 host rules; the two unrelated failures in test_hermes_agent_layout.py (CPU headroom / node capacity) reproduce identically on pristine main and are not caused by this change.

Expected after merge

Flux reconciles, cert-manager reissues hermes-sites-tls covering all 5 names, and the ensure-11 Job registers both origins per client. All four hostnames then serve 302 to Keycloak and reach the login form instead of the 400.

Operator note: users must log in again on the renamed hosts because oauth2-proxy cookies are host-bound; sessions on the legacy hosts are preserved by this change.

Supersedes #37 and #36 - please close both in favour of this one. Same manifest changes as #37, plus a regression test. (The SCM broker only allows new-branch creation, never branch updates, so adding the test required a new branch and PR rather than a push to #37.) ## Why Merged PR #34 renamed the chat/triage public hosts in place instead of adding the new names. It removed chat.hermes.bstein.dev and triage.hermes.bstein.dev from the certificate SANs, the hermes-sites Ingress rules and the CoreDNS overrides in one change. Verified live on 2026-08-21: - Legacy hosts: HTTP 404 with Traefik's default self-signed certificate (no Ingress rule matches), and no longer resolvable in cluster CoreDNS. - Renamed hosts: TLS and routing healthy (302 to Keycloak, oauth2-proxy /ping = 200), but Keycloak answers the auth redirect with HTTP 400 "Invalid parameter", because hermes-chat-proxy and hermes-triage-proxy still hold the pre-rename callback URLs and the immutable ensure Job never reran. Net effect: chat and triage have no working hostname at all. The post-merge runbook's rollback stance for a failed OIDC step is to reinstate the known-good old DNS/ingress/cert/OIDC targets rather than remove anything. ## What this does Makes the rename additive instead of destructive: - services/hermes/agent-certificate.yaml - legacy names back on hermes-sites-tls (5 SANs; agent.hermes.bstein.dev preserved). - services/hermes/agent-ingress.yaml - legacy host rules restored against the same oauth2-proxy-hermes-chat / -triage backends, plus both names on the TLS block. - infrastructure/core/coredns-custom.yaml - both legacy overrides restored for in-cluster resolution. - services/keycloak/scripts/hermes_access_oidc_ensure.sh - ensure_proxy_client takes an optional legacy origin and registers old and new callback URLs, web origins and post-logout origins together. rootUrl stays on the canonical new host. - services/keycloak/bootstrap-jobs/hermes-access-oidc-client-job.yaml - Job bumped to ensure-11 so Flux recreates the immutable Job. - testing/tests/test_hermes_public_host_continuity.py - new regression test. Legacy hosts are kept serving rather than redirected: oauth2-proxy cookies are host-bound, so a redirect would silently drop live sessions. Retiring them stays a separate, explicit change. ## Validation End-to-end run of the actual ensure script against a mock Keycloak admin API and mock Vault (only the three bootstrap lines repointed - the vault env source, KC_URL and the service-account token path; ensure_proxy_client itself verbatim). Script exits 0 and records: - hermes-chat-proxy - redirectUris [chat.bstein.dev/oauth2/callback, chat.hermes.bstein.dev/oauth2/callback], webOrigins both, rootUrl https://chat.bstein.dev, post-logout "new##legacy" - hermes-triage-proxy - same shape for the triage pair - hermes-agent-proxy and hermes-automation - payloads byte-identical to the same run against main, so the untouched clients cannot regress 16 assertions pass, including PKCE S256 and confidential-client flags preserved, and no client secret or admin password echoed into any client payload. A control run of main's script reproduces the root cause: it registers only the new host. The new test pins the invariant that a public host is either served by all four layers (certificate SAN, CoreDNS, Ingress rule, Keycloak registration) or by none. Against the pre-fix tree it fails 9 times across both legacy hosts and all four layers; against this branch the suite is green (19 passed). Also checked: kubectl kustomize services/hermes renders 5 SANs and 4 host rules; the two unrelated failures in test_hermes_agent_layout.py (CPU headroom / node capacity) reproduce identically on pristine main and are not caused by this change. ## Expected after merge Flux reconciles, cert-manager reissues hermes-sites-tls covering all 5 names, and the ensure-11 Job registers both origins per client. All four hostnames then serve 302 to Keycloak and reach the login form instead of the 400. Operator note: users must log in again on the renamed hosts because oauth2-proxy cookies are host-bound; sessions on the legacy hosts are preserved by this change.
hermes-automation added 2 commits 2026-08-21 08:58:26 +00:00 
PR #34 renamed the public chat/triage hosts in place rather than adding
the new names, so chat.hermes.bstein.dev and triage.hermes.bstein.dev
were dropped from the certificate SANs, the hermes-sites Ingress rules
and the CoreDNS overrides at once. Both legacy hosts now answer 404 with
Traefik's default self-signed certificate, and the renamed hosts cannot
complete a login because the Keycloak clients still carry the old
redirect URIs, so chat and triage are unreachable on every hostname.

Make the rename additive, which is the rollback path the post-merge
runbook asks for when the OIDC step fails:

- put the legacy names back on hermes-sites-tls and on the Ingress,
  pointing at the same oauth2-proxy backends
- restore both CoreDNS host overrides for in-cluster resolution
- teach ensure_proxy_client to register an optional legacy origin, so
  hermes-chat-proxy and hermes-triage-proxy accept the old and new
  redirect URIs, web origins and post-logout origins at the same time
  while rootUrl stays on the canonical new host
- bump the immutable ensure Job so Flux reruns the script

Serving both names is deliberate: oauth2-proxy cookies are host-bound,
so redirecting the legacy hosts would silently drop live sessions.
Retiring them stays a separate, explicit change.

Supersedes #36, which only bumped the Job and would have left the
legacy hosts dark.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The #34 rename dropped the legacy chat/triage names from the certificate
SANs, the hermes-sites Ingress, the CoreDNS overrides and the Keycloak
ensure script at the same time, so nothing failed loudly: DNS and TLS
still looked healthy while the legacy hosts served 404 and the renamed
hosts could not finish a login.

Pin the invariant that makes that silent: a public host is either served
by all four layers or by none. The table of hosts is the contract, so
retiring a name stays a deliberate edit rather than a side effect.

Verified to catch the regression: against the pre-fix tree these fail for
both legacy hosts on all four layers (9 failures); against this branch
the suite is green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bstein approved these changes 2026-08-21 10:17:21 +00:00
bstein changed title from WIP: hermes: restore legacy chat/triage hosts alongside the renamed ones to hermes: restore legacy chat/triage hosts alongside the renamed ones 2026-08-21 10:17:27 +00:00
bstein added 1 commit 2026-08-21 10:17:28 +00:00
bstein merged commit 5f9c600f6e into main 2026-08-21 10:17:49 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
bstein
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#38
No description provided.
Delete Branch "fix/hermes-restore-legacy-chat-triage-hosts-v2"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?