---
name: triage-titan-test-failures
description: Diagnose Titan CI test failures and environment regressions from Ariadne evidence, Jenkins artifacts, Flux state, Kubernetes health, Pushgateway quality metrics, and Grafana context. Use for failed or flaky in-scope suites, suspected cluster-caused test failures, release-quality questions, or requests for a supervised read-only triage summary and repo-side next steps.
---

# Triage Titan Test Failures

Follow the established Titan evidence path. Keep the investigation read-only and distinguish observed facts from inference.

## Collect the canonical evidence

1. Read the latest Ariadne diagnosis:

   ```sh
   curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/latest"
   ```

2. Always read the deterministic bundle, even when the diagnosis looks complete:

   ```sh
   curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/latest"
   ```

3. Ask for human approval before triggering a fresh collection or diagnosis. Use these only after approval:

   ```sh
   curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/collect"
   curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/run"
   ```

Record the timestamps and status from both responses. Treat the diagnosis as stale when it predates the deterministic bundle, reports `unavailable`, or lacks concrete evidence. In any of those cases, continue the investigation from the deterministic bundle; do not stop at model availability.

Treat Ariadne's bundle as the evidence source of truth. A local-model diagnosis may be unavailable while Wolf owns titan-24; continue from the stored bundle using the active Hermes fallback model.

## Atlas facts and guardrails

- Ariadne runs as the `ariadne` Deployment and Service in the `maintenance` namespace.
- Hermes local inference runs through `hermes-model-gate` in the `hermes` namespace. Ollama is the `hermes-ollama` Deployment and Service in that namespace.
- OpenClaw is not part of this inference path. Never suggest an `openclaw` resource unless a read-only query first proves one exists.
- Custom runtime namespaces do not always match suite names. In particular,
  Soteria, Metis, and Ariadne run in `maintenance`; Pegasus runs in `jellyfin`;
  Atlasbot runs in `comms`; Data Prepper runs in `logging`; and Lesavka is a
  desktop-hosted application. Read the service-health skill's
  `references/service-map.md` before claiming that a suite has no runtime.
- A connection error recorded inside an older diagnosis proves only that the model request failed at that timestamp. It does not prove the service is currently down.
- Never invent a namespace, workload, container, port, URL, log line, metric, commit, or pod condition. Verify a target with a read-only query before presenting an exact follow-up command; otherwise state what must be discovered first.
- HTTP POST collection and diagnosis endpoints are state-changing operations. Put them only under `Approval required`; never describe them as read-only or include them in the read-only command list.

## Narrow the failure

Work in this order:

1. Confirm the failed suite and build are in the canonical scope: `ananke`, `ariadne`, `atlasbot`, `bstein_home`, `data_prepper`, `lesavka`, `metis`, `pegasus`, `soteria`, or `titan_iac`.
2. Identify the first failed gate in the enforced order: `style`, `loc`, `coverage`, `tests`, `gate_glue`, `sonarqube`, `supply_chain`.
3. Correlate the build timestamp with retained Jenkins logs/artifacts, recent Git commits, and Flux revisions.
4. Check whether Kubernetes health, node pressure, image pulls, storage, DNS, or a shared dependency explains the failure better than a repo regression.
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry. A zero-valued series is not a failure.
6. Keep running Jenkins builds in an in-progress collection. Do not count them as failed until Jenkins reports a terminal failure state.
7. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.

For recent Git context, map the canonical suite to its Gitea repository and
read the latest commits. The current mappings are:

- `ananke` -> `bstein/ananke`
- `ariadne` -> `bstein/Ariadne`
- `atlasbot` -> `bstein/atlasbot`
- `bstein_home` -> `bstein/bstein-dev-home`
- `data_prepper` -> `bstein/titan-iac`
- `lesavka` -> `bstein/lesavka`
- `metis` -> `bstein/metis`
- `pegasus` -> `bstein/pegasus`
- `soteria` -> `bstein/soteria`
- `titan_iac` -> `bstein/titan-iac`

Use the read-only Gitea API and compare commit timestamps to the failing build:

```sh
curl -fsS "$GITEA_BASE_URL/api/v1/repos/<owner>/<repo>/commits?limit=10"
```

For quality telemetry, query VictoriaMetrics directly when the bundle is stale
or a specific label needs confirmation. URL-encode PromQL rather than manually
escaping it:

```sh
curl -G -fsS --data-urlencode \
  'query=platform_quality_gate_runs_total{suite="<suite>"}' \
  "$VICTORIA_METRICS_URL/api/v1/query"
curl -G -fsS --data-urlencode \
  'query=<suite>_quality_gate_checks_total' \
  "$VICTORIA_METRICS_URL/api/v1/query"
```

For image-pull failures, inspect the pod event, its ServiceAccount metadata,
and Flux image objects without reading referenced Secret values:

```sh
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> get serviceaccount <name> -o yaml
kubectl get imagerepositories,imagepolicies,imageupdateautomations -A
```

If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.

Use read-only commands such as `kubectl get`, `kubectl describe`, `kubectl logs`, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.

## Produce the triage result

Return these sections:

- `Finding`: one sentence naming the most likely failure class.
- `Confidence`: low, medium, or high, with the reason.
- `Evidence`: the smallest set of concrete timestamps, build IDs, artifact paths, commits, metrics, pods, nodes, or Flux revisions that support the finding.
- `Likely cause`: explain the causal chain and label inference as inference.
- `Blast radius`: affected suites, services, branches, or environments.
- `Next checks`: ordered read-only checks with exact commands or URLs.
- `Repo-side fix`: the smallest Flux/IaC or application change, or `none yet` when evidence is insufficient.
- `Approval required`: call out every step that would modify files, infrastructure, credentials, test environments, or external systems.

Never present a proposed fix as applied. Prefer a concise evidence-backed answer over a general log summary.