---
name: triage-titan-test-failures
description: Diagnose Titan CI test failures and environment regressions from Ariadne evidence, Jenkins artifacts, Flux state, Kubernetes health, Pushgateway quality metrics, and Grafana context. Use for failed or flaky in-scope suites, suspected cluster-caused test failures, release-quality questions, or requests for a supervised read-only triage summary and repo-side next steps.
---

# Triage Titan Test Failures

Follow the established Titan evidence path. Keep the investigation read-only and distinguish observed facts from inference.

## Collect the canonical evidence

1. Read the latest Ariadne diagnosis:

   ```sh
   curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/latest"
   ```

2. Always read the deterministic bundle, even when the diagnosis looks complete:

   ```sh
   curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/latest"
   ```

3. Ask for human approval before triggering a fresh collection or diagnosis. Use these only after approval:

   ```sh
   curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/collect"
   curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/run"
   ```

Record the timestamps and status from both responses. Treat the diagnosis as stale when it predates the deterministic bundle, reports `unavailable`, or lacks concrete evidence. In any of those cases, continue the investigation from the deterministic bundle; do not stop at model availability.

Treat Ariadne's bundle as the evidence source of truth. A local-model diagnosis may be unavailable while Wolf owns titan-24; continue from the stored bundle using the active Hermes fallback model.

## Atlas facts and guardrails

- Ariadne runs as the `ariadne` Deployment and Service in the `maintenance` namespace.
- Hermes local inference runs through `hermes-model-gate` in the `hermes` namespace. Ollama is the `hermes-ollama` Deployment and Service in that namespace.
- OpenClaw is not part of this inference path. Never suggest an `openclaw` resource unless a read-only query first proves one exists.
- A connection error recorded inside an older diagnosis proves only that the model request failed at that timestamp. It does not prove the service is currently down.
- Never invent a namespace, workload, container, port, URL, log line, metric, commit, or pod condition. Verify a target with a read-only query before presenting an exact follow-up command; otherwise state what must be discovered first.
- HTTP POST collection and diagnosis endpoints are state-changing operations. Put them only under `Approval required`; never describe them as read-only or include them in the read-only command list.

## Narrow the failure

Work in this order:

1. Confirm the failed suite and build are in the canonical scope: `ananke`, `ariadne`, `atlasbot`, `bstein_home`, `data_prepper`, `metis`, `pegasus`, `soteria`, or `titan_iac`.
2. Identify the first failed gate in the enforced order: `style`, `loc`, `coverage`, `tests`, `gate_glue`, `sonarqube`, `supply_chain`.
3. Correlate the build timestamp with retained Jenkins logs/artifacts, recent Git commits, and Flux revisions.
4. Check whether Kubernetes health, node pressure, image pulls, storage, DNS, or a shared dependency explains the failure better than a repo regression.
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry.
6. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.

If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.

Use read-only commands such as `kubectl get`, `kubectl describe`, `kubectl logs`, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.

## Produce the triage result

Return these sections:

- `Finding`: one sentence naming the most likely failure class.
- `Confidence`: low, medium, or high, with the reason.
- `Evidence`: the smallest set of concrete timestamps, build IDs, artifact paths, commits, metrics, pods, nodes, or Flux revisions that support the finding.
- `Likely cause`: explain the causal chain and label inference as inference.
- `Blast radius`: affected suites, services, branches, or environments.
- `Next checks`: ordered read-only checks with exact commands or URLs.
- `Repo-side fix`: the smallest Flux/IaC or application change, or `none yet` when evidence is insufficient.
- `Approval required`: call out every step that would modify files, infrastructure, credentials, test environments, or external systems.

Never present a proposed fix as applied. Prefer a concise evidence-backed answer over a general log summary.
