titan/atlas-iac
3
0
Fork 0
Code Issues Pull Requests 9 Actions Packages Projects Releases Wiki Activity

WIP: hermes: restore legacy chat/triage hosts alongside the renamed ones #40

Closed
hermes-automation wants to merge 1 commits from fix/hermes-restore-legacy-chat-triage-hosts-v3 into main
pull from: fix/hermes-restore-legacy-chat-triage-hosts-v3
merge into: titan:main
Conversation 0 Commits 1 Files Changed 2 +31 -2
hermes-automation commented 2026-08-21 09:10:40 +00:00
Member
Copy Link

Supersedes #38, #37 and #36 - please close all three in favour of this one. (The SCM broker only allows new-branch creation, never branch updates, so each revision needs a fresh branch and PR.)

Same content as #38 plus one further fix found while reviewing #38 against the oauth2-proxy source.

Why

Merged PR #34 renamed the chat/triage public hosts in place instead of adding the new names, removing chat.hermes.bstein.dev and triage.hermes.bstein.dev from the certificate SANs, the hermes-sites Ingress rules and the CoreDNS overrides in one change. Verified live on 2026-08-21:

  • Legacy hosts: HTTP 404 with Traefik's default self-signed certificate, and no longer resolvable in cluster CoreDNS.
  • Renamed hosts: TLS and routing healthy (302 to Keycloak, oauth2-proxy /ping = 200), but Keycloak answers the auth redirect with HTTP 400 "Invalid parameter", because hermes-chat-proxy and hermes-triage-proxy still hold the pre-rename callback URLs.

Chat and triage therefore have no working hostname at all. The runbook's rollback stance for a failed OIDC step is to reinstate the known-good old DNS/ingress/cert/OIDC targets rather than remove anything.

Root cause of the stale Keycloak state: services/keycloak/kustomization.yaml sets generatorOptions.disableNameSuffixHash: true, so the hermes-access-oidc-script ConfigMap keeps a stable name. #34 updated the script in place, but Jobs are immutable, so nothing recreated hermes-access-oidc-client-ensure-10 and the old callbacks stayed registered. Bumping the Job name is the only mechanism that forces the rerun.

What this does

  • services/hermes/agent-certificate.yaml - legacy names back on hermes-sites-tls (5 SANs; agent.hermes.bstein.dev preserved).
  • services/hermes/agent-ingress.yaml - legacy host rules restored against the same backends, plus both names on the TLS block.
  • infrastructure/core/coredns-custom.yaml - both legacy overrides restored.
  • services/keycloak/scripts/hermes_access_oidc_ensure.sh - ensure_proxy_client takes an optional legacy origin and registers old and new callback URLs, web origins and post-logout origins together; rootUrl stays canonical.
  • services/keycloak/bootstrap-jobs/hermes-access-oidc-client-job.yaml - Job bumped to ensure-11.
  • services/hermes/oauth2-proxy.yaml - NEW IN THIS REVISION: drop the pinned host from the chat and triage callback URLs.
  • testing/tests/test_hermes_public_host_continuity.py - regression tests.

The additional fix

Restoring the hosts is not sufficient on its own. Both proxies pinned --redirect-url to the renamed host, and oauth2-proxy returns that value verbatim when it carries a host - getOAuthRedirectURI short-circuits on redirectURL.Host != "" (v7.15.3 oauthproxy.go:1113-1135). A login started on a legacy host would send the browser to the canonical host's callback, while the CSRF cookie stays behind: it is issued with the __Host- prefix, which forbids a Domain attribute and pins it to one origin. Confirmed live - the current 302 sets "_Host-hermes_chat_csrf; Path=/; Secure; SameSite=Lax" with no Domain. The callback would land without it and fail as "unable to find a valid CSRF token".

Removing the host makes oauth2-proxy derive the callback from the request host. For the renamed hosts the derived value is byte-identical to the pinned one, so their behaviour is unchanged. Derivation reads X-Forwarded-Host only behind a trusted reverse proxy (pkg/requests/util/util.go GetRequestHost), and both deployments already set --reverse-proxy=true and --trusted-proxy-ip=10.42.0.0/16. Keycloak still matches the result against its registered redirect URIs. Startup is unaffected: parseURL only errors on a genuine parse failure, and the insecure-default warning needs cookie-secure and reverse-proxy both off. The agent proxy keeps its pinned callback because it serves a single host.

Validation

  • End-to-end run of the real ensure script against a mock Keycloak admin API and mock Vault (only the vault env source, KC_URL and service-account token path repointed; ensure_proxy_client verbatim): exits 0 and registers both callbacks, both web origins, canonical rootUrl and "##"-joined post-logout origins for each renamed client. The agent and automation client payloads are byte-identical to the same run against main, so untouched clients cannot regress. 16 assertions pass, including PKCE S256 and confidential-client flags preserved and no secret echoed into any payload. A control run of main's script reproduces the single-host root cause.
  • Regression tests: 21 pass on this branch. Against the pre-fix tree the host-continuity tests fail 9 times across both legacy hosts and all four layers; re-pinning a callback host alone makes the new derivation test fail. Both were checked deliberately.
  • kubectl kustomize services/hermes renders 5 SANs, 4 host rules, and --redirect-url=/oauth2/callback on both multi-host proxies.
  • Flux delivery path audited: infrastructure/core, services/hermes and services/keycloak are each reconciled (10m, prune: true) and every changed file is in its kustomization's resource list; hermes dependsOn keycloak and core. The CoreDNS custom ConfigMap applies without a restart, evidenced by the renamed hosts resolving in-cluster today from that file alone.
  • The two failures in test_hermes_agent_layout.py (CPU headroom, node capacity) reproduce identically on pristine main and are unrelated.

Expected after merge

Flux reconciles, cert-manager reissues hermes-sites-tls covering all 5 names, and the ensure-11 Job registers both origins per client. All four hostnames then serve 302 to Keycloak and reach the login form instead of the 400.

Operator note: users must log in again on the renamed hosts because oauth2-proxy cookies are host-bound. Sessions on the legacy hosts are preserved.

Not verified from CI: the oauth2-proxy derivation path is proven by source reading and rendered-manifest checks, not by an live login, since that needs interactive credentials. Worth watching on the first login after merge; the fallback is to re-pin the callback host, which returns the legacy hosts to existing-session-only behaviour without affecting the renamed hosts.

Supersedes #38, #37 and #36 - please close all three in favour of this one. (The SCM broker only allows new-branch creation, never branch updates, so each revision needs a fresh branch and PR.) Same content as #38 plus one further fix found while reviewing #38 against the oauth2-proxy source. ## Why Merged PR #34 renamed the chat/triage public hosts in place instead of adding the new names, removing chat.hermes.bstein.dev and triage.hermes.bstein.dev from the certificate SANs, the hermes-sites Ingress rules and the CoreDNS overrides in one change. Verified live on 2026-08-21: - Legacy hosts: HTTP 404 with Traefik's default self-signed certificate, and no longer resolvable in cluster CoreDNS. - Renamed hosts: TLS and routing healthy (302 to Keycloak, oauth2-proxy /ping = 200), but Keycloak answers the auth redirect with HTTP 400 "Invalid parameter", because hermes-chat-proxy and hermes-triage-proxy still hold the pre-rename callback URLs. Chat and triage therefore have no working hostname at all. The runbook's rollback stance for a failed OIDC step is to reinstate the known-good old DNS/ingress/cert/OIDC targets rather than remove anything. Root cause of the stale Keycloak state: services/keycloak/kustomization.yaml sets generatorOptions.disableNameSuffixHash: true, so the hermes-access-oidc-script ConfigMap keeps a stable name. #34 updated the script in place, but Jobs are immutable, so nothing recreated hermes-access-oidc-client-ensure-10 and the old callbacks stayed registered. Bumping the Job name is the only mechanism that forces the rerun. ## What this does - services/hermes/agent-certificate.yaml - legacy names back on hermes-sites-tls (5 SANs; agent.hermes.bstein.dev preserved). - services/hermes/agent-ingress.yaml - legacy host rules restored against the same backends, plus both names on the TLS block. - infrastructure/core/coredns-custom.yaml - both legacy overrides restored. - services/keycloak/scripts/hermes_access_oidc_ensure.sh - ensure_proxy_client takes an optional legacy origin and registers old and new callback URLs, web origins and post-logout origins together; rootUrl stays canonical. - services/keycloak/bootstrap-jobs/hermes-access-oidc-client-job.yaml - Job bumped to ensure-11. - services/hermes/oauth2-proxy.yaml - NEW IN THIS REVISION: drop the pinned host from the chat and triage callback URLs. - testing/tests/test_hermes_public_host_continuity.py - regression tests. ## The additional fix Restoring the hosts is not sufficient on its own. Both proxies pinned --redirect-url to the renamed host, and oauth2-proxy returns that value verbatim when it carries a host - getOAuthRedirectURI short-circuits on redirectURL.Host != "" (v7.15.3 oauthproxy.go:1113-1135). A login started on a legacy host would send the browser to the canonical host's callback, while the CSRF cookie stays behind: it is issued with the __Host- prefix, which forbids a Domain attribute and pins it to one origin. Confirmed live - the current 302 sets "__Host-hermes_chat_<nonce>_csrf; Path=/; Secure; SameSite=Lax" with no Domain. The callback would land without it and fail as "unable to find a valid CSRF token". Removing the host makes oauth2-proxy derive the callback from the request host. For the renamed hosts the derived value is byte-identical to the pinned one, so their behaviour is unchanged. Derivation reads X-Forwarded-Host only behind a trusted reverse proxy (pkg/requests/util/util.go GetRequestHost), and both deployments already set --reverse-proxy=true and --trusted-proxy-ip=10.42.0.0/16. Keycloak still matches the result against its registered redirect URIs. Startup is unaffected: parseURL only errors on a genuine parse failure, and the insecure-default warning needs cookie-secure and reverse-proxy both off. The agent proxy keeps its pinned callback because it serves a single host. ## Validation - End-to-end run of the real ensure script against a mock Keycloak admin API and mock Vault (only the vault env source, KC_URL and service-account token path repointed; ensure_proxy_client verbatim): exits 0 and registers both callbacks, both web origins, canonical rootUrl and "##"-joined post-logout origins for each renamed client. The agent and automation client payloads are byte-identical to the same run against main, so untouched clients cannot regress. 16 assertions pass, including PKCE S256 and confidential-client flags preserved and no secret echoed into any payload. A control run of main's script reproduces the single-host root cause. - Regression tests: 21 pass on this branch. Against the pre-fix tree the host-continuity tests fail 9 times across both legacy hosts and all four layers; re-pinning a callback host alone makes the new derivation test fail. Both were checked deliberately. - kubectl kustomize services/hermes renders 5 SANs, 4 host rules, and --redirect-url=/oauth2/callback on both multi-host proxies. - Flux delivery path audited: infrastructure/core, services/hermes and services/keycloak are each reconciled (10m, prune: true) and every changed file is in its kustomization's resource list; hermes dependsOn keycloak and core. The CoreDNS custom ConfigMap applies without a restart, evidenced by the renamed hosts resolving in-cluster today from that file alone. - The two failures in test_hermes_agent_layout.py (CPU headroom, node capacity) reproduce identically on pristine main and are unrelated. ## Expected after merge Flux reconciles, cert-manager reissues hermes-sites-tls covering all 5 names, and the ensure-11 Job registers both origins per client. All four hostnames then serve 302 to Keycloak and reach the login form instead of the 400. Operator note: users must log in again on the renamed hosts because oauth2-proxy cookies are host-bound. Sessions on the legacy hosts are preserved. Not verified from CI: the oauth2-proxy derivation path is proven by source reading and rendered-manifest checks, not by an live login, since that needs interactive credentials. Worth watching on the first login after merge; the fallback is to re-pin the callback host, which returns the legacy hosts to existing-session-only behaviour without affecting the renamed hosts.
hermes-automation added 3 commits 2026-08-21 09:10:41 +00:00 
PR #34 renamed the public chat/triage hosts in place rather than adding
the new names, so chat.hermes.bstein.dev and triage.hermes.bstein.dev
were dropped from the certificate SANs, the hermes-sites Ingress rules
and the CoreDNS overrides at once. Both legacy hosts now answer 404 with
Traefik's default self-signed certificate, and the renamed hosts cannot
complete a login because the Keycloak clients still carry the old
redirect URIs, so chat and triage are unreachable on every hostname.

Make the rename additive, which is the rollback path the post-merge
runbook asks for when the OIDC step fails:

- put the legacy names back on hermes-sites-tls and on the Ingress,
  pointing at the same oauth2-proxy backends
- restore both CoreDNS host overrides for in-cluster resolution
- teach ensure_proxy_client to register an optional legacy origin, so
  hermes-chat-proxy and hermes-triage-proxy accept the old and new
  redirect URIs, web origins and post-logout origins at the same time
  while rootUrl stays on the canonical new host
- bump the immutable ensure Job so Flux reruns the script

Serving both names is deliberate: oauth2-proxy cookies are host-bound,
so redirecting the legacy hosts would silently drop live sessions.
Retiring them stays a separate, explicit change.

Supersedes #36, which only bumped the Job and would have left the
legacy hosts dark.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The #34 rename dropped the legacy chat/triage names from the certificate
SANs, the hermes-sites Ingress, the CoreDNS overrides and the Keycloak
ensure script at the same time, so nothing failed loudly: DNS and TLS
still looked healthy while the legacy hosts served 404 and the renamed
hosts could not finish a login.

Pin the invariant that makes that silent: a public host is either served
by all four layers or by none. The table of hosts is the contract, so
retiring a name stays a deliberate edit rather than a side effect.

Verified to catch the regression: against the pre-fix tree these fail for
both legacy hosts on all four layers (9 failures); against this branch
the suite is green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Restoring the legacy chat/triage hosts is not enough on its own: both
proxies pinned --redirect-url to the renamed host, and oauth2-proxy
returns that value verbatim whenever it carries a host
(getOAuthRedirectURI short-circuits on redirectURL.Host != ""). A login
started on a legacy host would therefore send the browser to the
canonical host's callback, while the CSRF cookie stays behind: it is
issued with the __Host- prefix, which forbids a Domain attribute and
pins it to the exact origin. The callback lands without it and fails as
"unable to find a valid CSRF token".

Drop the host from both callback URLs so oauth2-proxy builds them from
the request host instead. For the renamed hosts the derived value is
byte-identical to the pinned one, so their behaviour is unchanged; the
legacy hosts now complete a login on the host the user actually visited.
Derivation reads X-Forwarded-Host only behind a trusted reverse proxy,
which both deployments already declare, and Keycloak still matches the
result against the redirect URIs registered by the ensure script.

The agent proxy keeps its pinned callback: it serves one host.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bstein closed this pull request 2026-08-22 20:22:43 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
atlas-reconciler bstein hermes-automation
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: titan/atlas-iac#40
No description provided.
Delete Branch "fix/hermes-restore-legacy-chat-triage-hosts-v3"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?