# Cassandra Infrastructure Contract

This stack is staged as a parallel replacement for Veles. Do not remove or
mutate Veles data while Cassandra is being validated.

## Cluster Contract

- Namespace: `cassandra`
- Hostname: `https://cassandra.bstein.dev`
- Namespace: `cassandra`; no alternate alpha namespace is used.
- Backend service: `cassandra-backend.cassandra.svc.cluster.local:80`
- Frontend service: `cassandra-frontend.cassandra.svc.cluster.local:80`
- Postgres service: `cassandra-postgres.cassandra.svc.cluster.local:5432`
- Artifact PVC: `cassandra-artifacts`, mounted at `/data/cassandra-artifacts`
- Storage classes: `cassandra-db`, `cassandra-artifacts`
- Images:
  - `registry.bstein.dev/cassandra/cassandra-backend`
  - `registry.bstein.dev/cassandra/cassandra-frontend`
  - `registry.bstein.dev/cassandra/cassandra-sim-worker`
- Backend `http` container port: `8796`
- Frontend `http` container port: `8080`
- Generator and retention workloads start disabled so copied data is not
  consumed or pruned before the migration is verified.

## Auth Contract

Cassandra owns authorization in the app. The `cassandra` Ingress does not use oauth2-proxy or Traefik forward-auth, so no ingress/auth layer should strip OIDC token claims. The app should validate tokens from `https://sso.bstein.dev/realms/cassandra` and expect stable `sub`, `email`, `preferred_username`, `groups`, and `realm_access.roles` claims. Do not scale Cassandra for real user traffic until native OIDC login/session flow is implemented and smoke-tested.

The Keycloak realm setup creates both groups and realm roles named `alpha` and `admin`. Members of the `alpha` group receive the `alpha` realm role; members of `admin` receive both `alpha` and `admin`. Built-in/meta strategies can stay universal, while runs and user-created strategies should remain user-scoped in the Cassandra database.

## Runtime Env

Cassandra should consume:

- `CASSANDRA_PUBLIC_BASE_URL=https://cassandra.bstein.dev`
- `CASSANDRA_OIDC_ISSUER=https://sso.bstein.dev/realms/cassandra`
- `CASSANDRA_OIDC_CLIENT_ID=cassandra-web`
- `CASSANDRA_OIDC_REQUIRED_GROUPS=alpha,cassandra-tester,admin`
- `CASSANDRA_OIDC_GROUPS_CLAIM=groups`
- `CASSANDRA_OIDC_ROLES_CLAIM=realm_access.roles`
- `DATABASE_URL` from `kv/data/atlas/cassandra/cassandra-db`
- `CASSANDRA_SESSION_SECRET` from `kv/data/atlas/cassandra/app-secrets`
- `CASSANDRA_BYOK_ENCRYPTION_KEY` from `kv/data/atlas/cassandra/app-secrets`

User OpenAI API keys must stay in the Cassandra database encrypted with
`CASSANDRA_BYOK_ENCRYPTION_KEY`; do not store per-user BYOK secrets in Vault.

Backend runtime secrets are synced from Vault by `cassandra-vault` into the generated Kubernetes Secret `cassandra-runtime-secrets`; no secret values are committed. The backend consumes that secret with `envFrom`.

## Artifact Contract

`cassandra-artifacts` is an RWO Longhorn PVC mounted into backend pods at `/data/cassandra-artifacts`. Backend pods own artifact writes and serving. Simulation Jobs should not mount or write directly to this PVC unless they are explicitly scheduled on Oceanus with the Cassandra toleration and the app has chosen a same-node direct-write model. Queue-mediated upload/copy through the backend remains the safer default until the app contract settles.

Backend, simulation workers, and retention/cleanup workers must run on Oceanus/titan-23 when they need artifact access. Frontend pods must not mount `cassandra-artifacts`.

## Simulation Jobs

The backend service account can create, watch, and delete Jobs only inside the `cassandra` namespace. Simulation pods should use service account `cassandra-sim`, set `automountServiceAccountToken: false`, and use:

```yaml
priorityClassName: cassandra-sim
nodeSelector:
  cassandra.bstein.dev/node-pool: oceanus
tolerations:
  - key: veles.bstein.dev/simulation
    operator: Equal
    value: "true"
    effect: NoSchedule
```

The toleration intentionally references the existing Veles taint while Veles and
Cassandra run side by side. The node carries Cassandra labels for selection, but
adding a Cassandra taint waits until Veles no longer needs the same node.

Retention/cleanup Jobs that touch artifacts should use the same node selector and toleration. If they do not need Kubernetes API access, use `cassandra-sim`; otherwise keep control-plane actions in the backend/controller and run artifact cleanup through a no-token worker.

## Staged Operator Steps

1. Join `titan-23`/Oceanus to Atlas as a worker.
2. Use Metis with `titan-23` in `METIS_FLASH_HOSTS`; the existing node secret placeholder uses `192.168.22.23`.
3. Confirm the node normalizer applies the Cassandra labels and taint.
4. Add Oceanus Longhorn disks at paths tagged by the Longhorn tag ensure job.
5. Let Vault policy reconciliation run, then unsuspend `cassandra-secrets-ensure-2`.
6. Unsuspend `cassandra-realm-ensure-1` in `services/keycloak` to create the realm/client secret, groups, and roles.
7. Create the Harbor `cassandra` project or robot access before image automation is enabled in production.
8. Let `cassandra-postgres` come up on the new `cassandra-db` PVC.
9. Unsuspend `cassandra-db-copy-from-veles-1` once to copy Veles Postgres into
   Cassandra and run table/count/hash verification against the target.
10. Inspect new Cassandra Longhorn volumes and confirm each two-replica volume
    has one titan-23 local replica plus one Asteria replica. If Longhorn places
    replicas differently, fix placement before live use.
11. Copy artifacts from `veles-artifacts` to `cassandra-artifacts` and verify
    file counts/bytes before enabling retention.
12. Smoke-test `https://cassandra.bstein.dev`, only then enable generator and
    retention workers.
13. Redirect `veles.bstein.dev` only after Cassandra is validated. Do not delete
    Veles resources until explicitly approved.

## Copy Commands

The DB copy job is suspended by default. After `cassandra-postgres` is ready:

```bash
kubectl -n cassandra patch job cassandra-db-copy-from-veles-1 \
  --type=merge -p '{"spec":{"suspend":false}}'
kubectl -n cassandra logs job/cassandra-db-copy-from-veles-1 -f
```

The artifact copy is also opt-in. Start the read-only Veles exporter, run the
Cassandra importer, inspect its verification summary, then scale the exporter
back down:

```bash
kubectl -n veles scale deploy/veles-artifact-export-for-cassandra --replicas=1
kubectl -n veles rollout status deploy/veles-artifact-export-for-cassandra
kubectl -n cassandra patch job cassandra-artifact-copy-from-veles-1 \
  --type=merge -p '{"spec":{"suspend":false}}'
kubectl -n cassandra logs job/cassandra-artifact-copy-from-veles-1 -f
kubectl -n veles scale deploy/veles-artifact-export-for-cassandra --replicas=0
```

The artifact importer writes verification details under
`/data/cassandra-artifacts/.cassandra-artifact-migration/`. If the Veles
artifact PVC changes during the copy, verification can fail correctly; rerun
after quiescing artifact writers or after a final delta-copy window.

## Assumptions

- `cassandra-artifacts` is RWO for alpha; simulation workers should either run
  on Oceanus with the backend or stream logs to the backend, which owns writes.
- Cassandra storage classes request two Longhorn replicas with `best-effort`
  data locality. This is intended to keep one replica local to titan-23 and one
  replica on an Asteria disk, but actual placement must be inspected after PVC
  provisioning.
- Longhorn default backup target is `s3://atlas-soteria@us-west-004/` with
  credential secret `longhorn-backup-b2`. As of the migration prep, remote
  backups are failing because the object-store cap is exceeded; local snapshots
  still work, but remote backup protection is not healthy until that cap is
  resolved.
- The Jenkins Cassandra job should publish:
  `registry.bstein.dev/cassandra/cassandra-backend`,
  `registry.bstein.dev/cassandra/cassandra-frontend`,
  `registry.bstein.dev/cassandra/cassandra-sim-worker`, and
  `registry.bstein.dev/cassandra/cassandra-generator-worker`.