bstein/titan-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
titan-iac/services/cassandra/NOTES.md

7.9 KiB
Raw Blame History

Cassandra Infrastructure Contract

This stack is staged as a parallel replacement for Veles. Do not remove or mutate Veles data while Cassandra is being validated.

Cluster Contract

  • Namespace: cassandra
  • Hostname: https://cassandra.bstein.dev
  • Namespace: cassandra; no alternate alpha namespace is used.
  • Backend service: cassandra-backend.cassandra.svc.cluster.local:80
  • Frontend service: cassandra-frontend.cassandra.svc.cluster.local:80
  • Postgres service: cassandra-postgres.cassandra.svc.cluster.local:5432
  • Artifact PVC: cassandra-artifacts, mounted at /data/cassandra-artifacts
  • Storage classes: cassandra-db, cassandra-artifacts
  • Images:
    • registry.bstein.dev/cassandra/cassandra-backend
    • registry.bstein.dev/cassandra/cassandra-frontend
    • registry.bstein.dev/cassandra/cassandra-sim-worker
  • Backend http container port: 8796
  • Frontend http container port: 8080
  • Generator and retention workloads start disabled so copied data is not consumed or pruned before the migration is verified.

Auth Contract

Cassandra owns authorization in the app. The cassandra Ingress does not use oauth2-proxy or Traefik forward-auth, so no ingress/auth layer should strip OIDC token claims. The app should validate tokens from https://sso.bstein.dev/realms/cassandra and expect stable sub, email, preferred_username, groups, and realm_access.roles claims. Do not scale Cassandra for real user traffic until native OIDC login/session flow is implemented and smoke-tested.

The Keycloak realm setup creates both groups and realm roles named alpha and admin. Members of the alpha group receive the alpha realm role; members of admin receive both alpha and admin. Built-in/meta strategies can stay universal, while runs and user-created strategies should remain user-scoped in the Cassandra database.

Runtime Env

Cassandra should consume:

  • CASSANDRA_PUBLIC_BASE_URL=https://cassandra.bstein.dev
  • CASSANDRA_OIDC_ISSUER=https://sso.bstein.dev/realms/cassandra
  • CASSANDRA_OIDC_CLIENT_ID=cassandra-web
  • CASSANDRA_OIDC_REQUIRED_GROUPS=alpha,cassandra-tester,admin
  • CASSANDRA_OIDC_GROUPS_CLAIM=groups
  • CASSANDRA_OIDC_ROLES_CLAIM=realm_access.roles
  • DATABASE_URL from kv/data/atlas/cassandra/cassandra-db
  • CASSANDRA_SESSION_SECRET from kv/data/atlas/cassandra/app-secrets
  • CASSANDRA_BYOK_ENCRYPTION_KEY from kv/data/atlas/cassandra/app-secrets

User OpenAI API keys must stay in the Cassandra database encrypted with CASSANDRA_BYOK_ENCRYPTION_KEY; do not store per-user BYOK secrets in Vault.

Backend runtime secrets are synced from Vault by cassandra-vault into the generated Kubernetes Secret cassandra-runtime-secrets; no secret values are committed. The backend consumes that secret with envFrom.

Artifact Contract

cassandra-artifacts is an RWO Longhorn PVC mounted into backend pods at /data/cassandra-artifacts. Backend pods own artifact writes and serving. Simulation Jobs should not mount or write directly to this PVC unless they are explicitly scheduled on Oceanus with the Cassandra toleration and the app has chosen a same-node direct-write model. Queue-mediated upload/copy through the backend remains the safer default until the app contract settles.

Backend, simulation workers, and retention/cleanup workers must run on Oceanus/titan-23 when they need artifact access. Frontend pods must not mount cassandra-artifacts.

Simulation Jobs

The backend service account can create, watch, and delete Jobs only inside the cassandra namespace. Simulation pods should use service account cassandra-sim, set automountServiceAccountToken: false, and use:

priorityClassName: cassandra-sim
nodeSelector:
  cassandra.bstein.dev/node-pool: oceanus
tolerations:
  - key: veles.bstein.dev/simulation
    operator: Equal
    value: "true"
    effect: NoSchedule

The toleration intentionally references the existing Veles taint while Veles and Cassandra run side by side. The node carries Cassandra labels for selection, but adding a Cassandra taint waits until Veles no longer needs the same node.

Retention/cleanup Jobs that touch artifacts should use the same node selector and toleration. If they do not need Kubernetes API access, use cassandra-sim; otherwise keep control-plane actions in the backend/controller and run artifact cleanup through a no-token worker.

Staged Operator Steps

  1. Join titan-23/Oceanus to Atlas as a worker.
  2. Use Metis with titan-23 in METIS_FLASH_HOSTS; the existing node secret placeholder uses 192.168.22.23.
  3. Confirm the node normalizer applies the Cassandra labels and taint.
  4. Add Oceanus Longhorn disks at paths tagged by the Longhorn tag ensure job.
  5. Let Vault policy reconciliation run, then unsuspend cassandra-secrets-ensure-2.
  6. Unsuspend cassandra-realm-ensure-1 in services/keycloak to create the realm/client secret, groups, and roles.
  7. Create the Harbor cassandra project or robot access before image automation is enabled in production.
  8. Let cassandra-postgres come up on the new cassandra-db PVC.
  9. Unsuspend cassandra-db-copy-from-veles-1 once to copy Veles Postgres into Cassandra and run table/count/hash verification against the target.
  10. Inspect new Cassandra Longhorn volumes and confirm each two-replica volume has one titan-23 local replica plus one Asteria replica. If Longhorn places replicas differently, fix placement before live use.
  11. Copy artifacts from veles-artifacts to cassandra-artifacts and verify file counts/bytes before enabling retention.
  12. Smoke-test https://cassandra.bstein.dev, only then enable generator and retention workers.
  13. Redirect veles.bstein.dev only after Cassandra is validated. Do not delete Veles resources until explicitly approved.

Copy Commands

The DB copy job is suspended by default. After cassandra-postgres is ready:

kubectl -n cassandra patch job cassandra-db-copy-from-veles-1 \
  --type=merge -p '{"spec":{"suspend":false}}'
kubectl -n cassandra logs job/cassandra-db-copy-from-veles-1 -f

The artifact copy is also opt-in. Start the read-only Veles exporter, run the Cassandra importer, inspect its verification summary, then scale the exporter back down:

kubectl -n veles scale deploy/veles-artifact-export-for-cassandra --replicas=1
kubectl -n veles rollout status deploy/veles-artifact-export-for-cassandra
kubectl -n cassandra patch job cassandra-artifact-copy-from-veles-1 \
  --type=merge -p '{"spec":{"suspend":false}}'
kubectl -n cassandra logs job/cassandra-artifact-copy-from-veles-1 -f
kubectl -n veles scale deploy/veles-artifact-export-for-cassandra --replicas=0

The artifact importer writes verification details under /data/cassandra-artifacts/.cassandra-artifact-migration/. If the Veles artifact PVC changes during the copy, verification can fail correctly; rerun after quiescing artifact writers or after a final delta-copy window.

Assumptions

  • cassandra-artifacts is RWO for alpha; simulation workers should either run on Oceanus with the backend or stream logs to the backend, which owns writes.
  • Cassandra storage classes request two Longhorn replicas with best-effort data locality. This is intended to keep one replica local to titan-23 and one replica on an Asteria disk, but actual placement must be inspected after PVC provisioning.
  • Longhorn default backup target is s3://atlas-soteria@us-west-004/ with credential secret longhorn-backup-b2. As of the migration prep, remote backups are failing because the object-store cap is exceeded; local snapshots still work, but remote backup protection is not healthy until that cap is resolved.
  • The Jenkins Cassandra job should publish: registry.bstein.dev/cassandra/cassandra-backend, registry.bstein.dev/cassandra/cassandra-frontend, registry.bstein.dev/cassandra/cassandra-sim-worker, and registry.bstein.dev/cassandra/cassandra-generator-worker.