114 lines
6.2 KiB
Markdown
114 lines
6.2 KiB
Markdown
---
|
|
name: triage-titan-test-failures
|
|
description: Diagnose Titan CI test failures and environment regressions from Ariadne evidence, Jenkins artifacts, Flux state, Kubernetes health, Pushgateway quality metrics, and Grafana context. Use for failed or flaky in-scope suites, suspected cluster-caused test failures, release-quality questions, or requests for a supervised read-only triage summary and repo-side next steps.
|
|
---
|
|
|
|
# Triage Titan Test Failures
|
|
|
|
Follow the established Titan evidence path. Keep the investigation read-only and distinguish observed facts from inference.
|
|
|
|
## Collect the canonical evidence
|
|
|
|
1. Read the latest Ariadne diagnosis:
|
|
|
|
```sh
|
|
curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/latest"
|
|
```
|
|
|
|
2. Always read the deterministic bundle, even when the diagnosis looks complete:
|
|
|
|
```sh
|
|
curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/latest"
|
|
```
|
|
|
|
3. Ask for human approval before triggering a fresh collection or diagnosis. Use these only after approval:
|
|
|
|
```sh
|
|
curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/collect"
|
|
curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/run"
|
|
```
|
|
|
|
Record the timestamps and status from both responses. Treat the diagnosis as stale when it predates the deterministic bundle, reports `unavailable`, or lacks concrete evidence. In any of those cases, continue the investigation from the deterministic bundle; do not stop at model availability.
|
|
|
|
Treat Ariadne's bundle as the evidence source of truth. A local-model diagnosis may be unavailable while Wolf owns titan-24; continue from the stored bundle using the active Hermes fallback model.
|
|
|
|
## Atlas facts and guardrails
|
|
|
|
- Ariadne runs as the `ariadne` Deployment and Service in the `maintenance` namespace.
|
|
- Hermes local inference runs through `hermes-model-gate` in the `hermes` namespace. Ollama is the `hermes-ollama` Deployment and Service in that namespace.
|
|
- OpenClaw is not part of this inference path. Never suggest an `openclaw` resource unless a read-only query first proves one exists.
|
|
- A connection error recorded inside an older diagnosis proves only that the model request failed at that timestamp. It does not prove the service is currently down.
|
|
- Never invent a namespace, workload, container, port, URL, log line, metric, commit, or pod condition. Verify a target with a read-only query before presenting an exact follow-up command; otherwise state what must be discovered first.
|
|
- HTTP POST collection and diagnosis endpoints are state-changing operations. Put them only under `Approval required`; never describe them as read-only or include them in the read-only command list.
|
|
|
|
## Narrow the failure
|
|
|
|
Work in this order:
|
|
|
|
1. Confirm the failed suite and build are in the canonical scope: `ananke`, `ariadne`, `atlasbot`, `bstein_home`, `data_prepper`, `metis`, `pegasus`, `soteria`, or `titan_iac`.
|
|
2. Identify the first failed gate in the enforced order: `style`, `loc`, `coverage`, `tests`, `gate_glue`, `sonarqube`, `supply_chain`.
|
|
3. Correlate the build timestamp with retained Jenkins logs/artifacts, recent Git commits, and Flux revisions.
|
|
4. Check whether Kubernetes health, node pressure, image pulls, storage, DNS, or a shared dependency explains the failure better than a repo regression.
|
|
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry.
|
|
6. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.
|
|
|
|
For recent Git context, map the canonical suite to its Gitea repository and
|
|
read the latest commits. The current mappings are:
|
|
|
|
- `ananke` -> `bstein/ananke`
|
|
- `ariadne` -> `bstein/Ariadne`
|
|
- `atlasbot` -> `bstein/atlasbot`
|
|
- `bstein_home` -> `bstein/bstein-dev-home`
|
|
- `data_prepper` -> `bstein/titan-iac`
|
|
- `metis` -> `bstein/metis`
|
|
- `pegasus` -> `bstein/pegasus`
|
|
- `soteria` -> `bstein/soteria`
|
|
- `titan_iac` -> `bstein/titan-iac`
|
|
|
|
Use the read-only Gitea API and compare commit timestamps to the failing build:
|
|
|
|
```sh
|
|
curl -fsS "$GITEA_BASE_URL/api/v1/repos/<owner>/<repo>/commits?limit=10"
|
|
```
|
|
|
|
For quality telemetry, query VictoriaMetrics directly when the bundle is stale
|
|
or a specific label needs confirmation. URL-encode PromQL rather than manually
|
|
escaping it:
|
|
|
|
```sh
|
|
curl -G -fsS --data-urlencode \
|
|
'query=platform_quality_gate_runs_total{suite="<suite>"}' \
|
|
"$VICTORIA_METRICS_URL/api/v1/query"
|
|
curl -G -fsS --data-urlencode \
|
|
'query=<suite>_quality_gate_checks_total' \
|
|
"$VICTORIA_METRICS_URL/api/v1/query"
|
|
```
|
|
|
|
For image-pull failures, inspect the pod event, its ServiceAccount metadata,
|
|
and Flux image objects without reading referenced Secret values:
|
|
|
|
```sh
|
|
kubectl -n <namespace> describe pod <pod>
|
|
kubectl -n <namespace> get serviceaccount <name> -o yaml
|
|
kubectl get imagerepositories,imagepolicies,imageupdateautomations -A
|
|
```
|
|
|
|
If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.
|
|
|
|
Use read-only commands such as `kubectl get`, `kubectl describe`, `kubectl logs`, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.
|
|
|
|
## Produce the triage result
|
|
|
|
Return these sections:
|
|
|
|
- `Finding`: one sentence naming the most likely failure class.
|
|
- `Confidence`: low, medium, or high, with the reason.
|
|
- `Evidence`: the smallest set of concrete timestamps, build IDs, artifact paths, commits, metrics, pods, nodes, or Flux revisions that support the finding.
|
|
- `Likely cause`: explain the causal chain and label inference as inference.
|
|
- `Blast radius`: affected suites, services, branches, or environments.
|
|
- `Next checks`: ordered read-only checks with exact commands or URLs.
|
|
- `Repo-side fix`: the smallest Flux/IaC or application change, or `none yet` when evidence is insufficient.
|
|
- `Approval required`: call out every step that would modify files, infrastructure, credentials, test environments, or external systems.
|
|
|
|
Never present a proposed fix as applied. Prefer a concise evidence-backed answer over a general log summary.
|