titan-iac/services/hermes/skills/triage-titan-test-failures/SKILL.md

name, description

name	description
triage-titan-test-failures	Diagnose Titan CI test failures and environment regressions from Ariadne evidence, Jenkins artifacts, Flux state, Kubernetes health, Pushgateway quality metrics, and Grafana context. Use for failed or flaky in-scope suites, suspected cluster-caused test failures, release-quality questions, or requests for a supervised read-only triage summary and repo-side next steps.

Triage Titan Test Failures

Follow the established Titan evidence path. Keep the investigation read-only and distinguish observed facts from inference.

Collect the canonical evidence

1. Read the latest Ariadne diagnosis:

   curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/latest"
   

2. Always read the deterministic bundle, even when the diagnosis looks complete:

   curl -fsS "$ARIADNE_BASE_URL/api/internal/testing/triage/latest"
   

3. Ask for human approval before triggering a fresh collection or diagnosis. Use these only after approval:

   curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/collect"
   curl -fsS -X POST "$ARIADNE_BASE_URL/api/internal/testing/triage/diagnosis/run"
   

Record the timestamps and status from both responses. Treat the diagnosis as stale when it predates the deterministic bundle, reports unavailable, or lacks concrete evidence. In any of those cases, continue the investigation from the deterministic bundle; do not stop at model availability.

Treat Ariadne's bundle as the evidence source of truth. A local-model diagnosis may be unavailable while Wolf owns titan-24; continue from the stored bundle using the active Hermes fallback model.

Atlas facts and guardrails

• Ariadne runs as the ariadne Deployment and Service in the maintenance namespace.
• Hermes local inference runs through hermes-model-gate in the hermes namespace. Ollama is the hermes-ollama Deployment and Service in that namespace.
• OpenClaw is not part of this inference path. Never suggest an openclaw resource unless a read-only query first proves one exists.
• Custom runtime namespaces do not always match suite names. In particular, Soteria, Metis, and Ariadne run in maintenance; Pegasus runs in jellyfin; Atlasbot runs in comms; Data Prepper runs in logging; and Lesavka is a desktop-hosted application. Read the service-health skill's references/service-map.md before claiming that a suite has no runtime.
• A connection error recorded inside an older diagnosis proves only that the model request failed at that timestamp. It does not prove the service is currently down.
• Never invent a namespace, workload, container, port, URL, log line, metric, commit, or pod condition. Verify a target with a read-only query before presenting an exact follow-up command; otherwise state what must be discovered first.
• HTTP POST collection and diagnosis endpoints are state-changing operations. Put them only under Approval required; never describe them as read-only or include them in the read-only command list.

Narrow the failure

Work in this order:

1. Confirm the failed suite and build are in the canonical scope: ananke, ariadne, atlasbot, bstein_home, data_prepper, lesavka, metis, pegasus, soteria, or titan_iac.
2. Identify the first failed gate in the enforced order: style, loc, coverage, tests, gate_glue, sonarqube, supply_chain.
3. Correlate the build timestamp with retained Jenkins logs/artifacts, recent Git commits, and Flux revisions.
4. Check whether Kubernetes health, node pressure, image pulls, storage, DNS, or a shared dependency explains the failure better than a repo regression.
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry. A zero-valued series is not a failure.
6. Keep running Jenkins builds in an in-progress collection. Do not count them as failed until Jenkins reports a terminal failure state.
7. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.

For recent Git context, map the canonical suite to its Gitea repository and read the latest commits. The current mappings are:

• ananke -> bstein/ananke
• ariadne -> bstein/Ariadne
• atlasbot -> bstein/atlasbot
• bstein_home -> bstein/bstein-dev-home
• data_prepper -> bstein/titan-iac
• lesavka -> bstein/lesavka
• metis -> bstein/metis
• pegasus -> bstein/pegasus
• soteria -> bstein/soteria
• titan_iac -> bstein/titan-iac

Use the read-only Gitea API and compare commit timestamps to the failing build:

curl -fsS "$GITEA_BASE_URL/api/v1/repos/<owner>/<repo>/commits?limit=10"

For quality telemetry, query VictoriaMetrics directly when the bundle is stale or a specific label needs confirmation. URL-encode PromQL rather than manually escaping it:

curl -G -fsS --data-urlencode \
  'query=platform_quality_gate_runs_total{suite="<suite>"}' \
  "$VICTORIA_METRICS_URL/api/v1/query"
curl -G -fsS --data-urlencode \
  'query=<suite>_quality_gate_checks_total' \
  "$VICTORIA_METRICS_URL/api/v1/query"

For image-pull failures, inspect the pod event, its ServiceAccount metadata, and Flux image objects without reading referenced Secret values:

kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> get serviceaccount <name> -o yaml
kubectl get imagerepositories,imagepolicies,imageupdateautomations -A

If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.

Use read-only commands such as kubectl get, kubectl describe, kubectl logs, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.

Produce the triage result

Return these sections:

• Finding: one sentence naming the most likely failure class.
• Confidence: low, medium, or high, with the reason.
• Evidence: the smallest set of concrete timestamps, build IDs, artifact paths, commits, metrics, pods, nodes, or Flux revisions that support the finding.
• Likely cause: explain the causal chain and label inference as inference.
• Blast radius: affected suites, services, branches, or environments.
• Next checks: ordered read-only checks with exact commands or URLs.
• Repo-side fix: the smallest Flux/IaC or application change, or none yet when evidence is insufficient.
• Approval required: call out every step that would modify files, infrastructure, credentials, test environments, or external systems.

Never present a proposed fix as applied. Prefer a concise evidence-backed answer over a general log summary.