1b29b56f50
Detection only ever considered builds that finished, so a build wedged on a network call was invisible: no incident, no issue, no alert, while it held one of the five Jenkins agent slots. Observed on metis build 272, which sat on apt-get update for 75 minutes and starved the triage demo of an agent with nothing anywhere saying so. A build still running past ARIADNE_HERMES_HUNG_BUILD_MINUTES (default 45) now files an issue and lands human_required. No model is consulted: the console is still being written, so a root cause would be invented. Escalates once per build, and a cap of zero disables the check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ariadne
Ariadne is the Atlas admin and account automation service.
It sits behind the portal and handles the jobs that are annoying or risky to do by hand: approving access, syncing account state, rotating service passwords, cleaning stale Kubernetes work, checking platform health, and keeping a few service integrations lined up.
How it works
Ariadne is a FastAPI service with a small scheduler. It talks to Keycloak, Vault, Mailu, Nextcloud, Wger, Firefly, Jenkins, Metis, Kubernetes, and a few Atlas-specific services through focused adapters under ariadne/services/.
The API is split between admin routes, account self-service routes, internal event hooks, and Prometheus metrics. Background jobs store run history in the Ariadne database so failures can be inspected later instead of vanishing into logs.
The following are notes for future Brad.
Bring-up dependencies
Ariadne needs:
- Kubernetes API, service DNS, and Ariadne's service account/RBAC
- the Ariadne database, plus the portal database if portal/account sync is enabled
- Vault or the Kubernetes secrets that Vault normally feeds it
- Keycloak/OIDC, because auth and profile sync assume it exists
- ingress/proxy plumbing if humans are going to use it through the portal
- the services for whatever jobs are enabled: Mailu, Nextcloud, Vaultwarden, Wger, Firefly, Jenkins, Metis, OpenSearch, and the comms/game-mode pieces
It can start before every integration is perfect, but the matching scheduled jobs will fail or no-op until their service is actually alive. In a total bring-up, wait for storage, Flux, Postgres, Vault, Keycloak, and ingress first. Afterwards Ariadne becomes useful glue.
Useful routes:
GET /healthGET /metricsGET /api/admin/cluster/statePOST /api/admin/access/requests/{username}/approvePOST /api/account/mailu/rotatePOST /api/account/wger/resetPOST /api/account/firefly/resetPOST /events
Development
python -m pytest
ruff check .
Most runtime behavior is configured through environment variables in ariadne/settings.py. Service-specific logic is in the small adapter modules; ariadne/app.py is focused on request flow and task orchestration.