80 lines
4.7 KiB
Markdown
80 lines
4.7 KiB
Markdown
# Veles Infrastructure Contract
|
|
|
|
This stack is staged for Flux and intentionally starts the app deployments at `replicas: 0` until images and the app-side runtime contract are ready.
|
|
|
|
## Cluster Contract
|
|
|
|
- Namespace: `veles`
|
|
- Hostname: `https://veles.bstein.dev`
|
|
- Namespace: `veles`; no alternate alpha namespace is used.
|
|
- Backend service: `veles-backend.veles.svc.cluster.local:80`
|
|
- Frontend service: `veles-frontend.veles.svc.cluster.local:80`
|
|
- Postgres service: `veles-postgres.veles.svc.cluster.local:5432`
|
|
- Artifact PVC: `veles-artifacts`, mounted at `/data/veles-artifacts`
|
|
- Storage classes: `veles-oceanus-db`, `veles-oceanus-artifacts`
|
|
- Images:
|
|
- `registry.bstein.dev/veles/veles-backend`
|
|
- `registry.bstein.dev/veles/veles-frontend`
|
|
- `registry.bstein.dev/veles/veles-sim-worker`
|
|
- Backend/frontend deployments are placeholders and remain scaled to `0` until final image layout, container ports, and health endpoints are confirmed. Services route to a named `http` target port so the numeric container port can change without changing Ingress.
|
|
|
|
## Auth Contract
|
|
|
|
Veles owns authorization in the app. The `veles` Ingress does not use oauth2-proxy or Traefik forward-auth, so no ingress/auth layer should strip OIDC token claims. The app should validate tokens from `https://sso.bstein.dev/realms/veles` and expect stable `sub`, `email`, `preferred_username`, `groups`, and `realm_access.roles` claims.
|
|
|
|
The Keycloak realm setup creates both groups and realm roles named `alpha` and `admin`. Members of the `alpha` group receive the `alpha` realm role; members of `admin` receive both `alpha` and `admin`. Built-in/meta strategies can stay universal, while runs and user-created strategies should remain user-scoped in the Veles database.
|
|
|
|
## Runtime Env
|
|
|
|
Veles should consume:
|
|
|
|
- `VELES_PUBLIC_BASE_URL=https://veles.bstein.dev`
|
|
- `VELES_OIDC_ISSUER=https://sso.bstein.dev/realms/veles`
|
|
- `VELES_OIDC_CLIENT_ID=veles-web`
|
|
- `VELES_OIDC_REQUIRED_GROUPS=alpha,admin`
|
|
- `VELES_OIDC_GROUPS_CLAIM=groups`
|
|
- `VELES_OIDC_ROLES_CLAIM=realm_access.roles`
|
|
- `DATABASE_URL` from `kv/data/atlas/veles/veles-db`
|
|
- `VELES_SESSION_SECRET` from `kv/data/atlas/veles/app-secrets`
|
|
- `VELES_BYOK_ENCRYPTION_KEY` from `kv/data/atlas/veles/app-secrets`
|
|
|
|
User OpenAI API keys must stay in the Veles database encrypted with `VELES_BYOK_ENCRYPTION_KEY`; do not store per-user BYOK secrets in Vault.
|
|
|
|
Backend runtime secrets are synced from Vault by `veles-vault` into the generated Kubernetes Secret `veles-runtime-secrets`; no secret values are committed. The backend consumes that secret with `envFrom`.
|
|
|
|
## Artifact Contract
|
|
|
|
`veles-artifacts` is an RWO Longhorn PVC mounted into backend pods at `/data/veles-artifacts`. Backend pods own artifact writes and serving. Simulation Jobs should not mount or write directly to this PVC unless they are explicitly scheduled on Oceanus with the Veles toleration and the app has chosen a same-node direct-write model. Queue-mediated upload/copy through the backend remains the safer default until the app contract settles.
|
|
|
|
## Simulation Jobs
|
|
|
|
The backend service account can create, watch, and delete Jobs only inside the `veles` namespace. Simulation pods should use service account `veles-sim`, set `automountServiceAccountToken: false`, and use:
|
|
|
|
```yaml
|
|
priorityClassName: veles-sim
|
|
nodeSelector:
|
|
veles.bstein.dev/simulation: "true"
|
|
tolerations:
|
|
- key: veles.bstein.dev/simulation
|
|
operator: Equal
|
|
value: "true"
|
|
effect: NoSchedule
|
|
```
|
|
|
|
## Staged Operator Steps
|
|
|
|
1. Join `titan-23`/Oceanus to Atlas as a worker.
|
|
2. Use Metis with `titan-23` in `METIS_FLASH_HOSTS`; the existing node secret placeholder uses `192.168.22.23`.
|
|
3. Confirm the node normalizer applies the Veles labels and taint.
|
|
4. Add Oceanus Longhorn disks at paths tagged by the Longhorn tag ensure job.
|
|
5. Let Vault policy reconciliation run, then unsuspend `veles-secrets-ensure-2`.
|
|
6. Unsuspend `veles-realm-ensure-4` in `services/keycloak` to create the realm/client secret, groups, and roles.
|
|
7. Create the Harbor `veles` project or robot access before image automation is enabled in production.
|
|
8. Scale `veles-postgres`, then backend/frontend once app images exist.
|
|
|
|
## Assumptions
|
|
|
|
- `veles-oceanus-artifacts` is RWO for alpha; simulation workers should either run on Oceanus with the backend or stream logs to the backend, which owns writes.
|
|
- Longhorn default backup target is `s3://atlas-soteria@us-west-004/` with credential secret `longhorn-backup-b2`; the live `BackupTarget/default` currently reports available. Postgres and artifact volumes have Longhorn recurring snapshot and backup jobs attached by their StorageClasses. This is not a substitute for a tested restore drill.
|
|
- The Jenkins job skeleton points at the Veles repo but stays disabled until that repo provides a Jenkinsfile.
|