ee7b243106
All checks were successful
Tests / Declarative: Post Actions passed: 1257
A build whose agent never started is a distinct failure from one that lost a connection mid-run: retrying can work, but when the pool is already full of stuck pods the retry queues behind them and fails identically. Clearing first is what makes the retry worth making. The clear is Ariadne's existing scheduled pod cleanup, which only removes pods that have already succeeded or failed, so nothing running is touched. This is the failure behind lesavka's open issue and behind two stalled demo runs tonight. Critically, all three remediations are now described in the triage prompt. They were wired in Ariadne but absent from what Hermes is told, so Hermes could never have requested them - the allowlist would have advertised capability that could not fire. A test now asserts every allowlisted action id appears in the prompt, so the two cannot drift apart again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
68 lines
5.2 KiB
Python
68 lines
5.2 KiB
Python
"""Frozen triage prompt sent to Hermes for every Jenkins incident.
|
|
|
|
The prompt lives in its own module so its wording is reviewed as a unit and
|
|
so changing it never competes for room with the orchestrator's logic.
|
|
|
|
The fixture-specific rules are appended only for the fixture job. Stating them
|
|
to every job made Hermes reason about them out loud, and that reasoning was
|
|
published verbatim into issues and pull requests on real service repositories,
|
|
which read as though the whole system existed to serve a demonstration.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from typing import Any
|
|
|
|
|
|
DEMO_JOB = "hermes-triage-demo"
|
|
|
|
_BASE = """Use $triage-titan-test-failures.
|
|
Analyze incident __INCIDENT_ID__ for the Jenkins job __JOB__.
|
|
Treat the attached Ariadne bundle as the source of truth.
|
|
Identify the first enforced failure.
|
|
jenkins.failed_tests carries the structured test results when the build published any; when it is populated it names the failing test, its class, and its assertion, and it is better evidence than console text.
|
|
jenkins.first_failed_stage names the pipeline stage that failed when the build reported stages.
|
|
The jenkins.console_failures array holds excerpts around detected failure markers in chronological order; the earliest region usually contains the first enforced failure, and jenkins.console_tail is the end of the build which often only shows downstream noise.
|
|
Distinguish facts from inference.
|
|
Return ONLY a single JSON object with exactly these keys and no others:
|
|
{"incident_id": "<must equal __INCIDENT_ID__>", "classification": "<short snake_case name for what actually failed>", "confidence": <0..1>, "facts": [{"statement": "...", "source": "jenkins|opensearch|victoriametrics|kubernetes|flux|gitea", "reference": "..."}], "inferences": ["..."], "first_failed_gate": "<string>", "requested_action": <object or null>, "human_required": <bool>, "reason": "<string>"}
|
|
You are diagnosing only; you do not execute anything. Ariadne separately validates and executes any requested action under its own authorization policy, and will refuse anything its own reading of the evidence does not support.
|
|
Write the reason and the inferences for an engineer who maintains this service and who has no knowledge of how this triage system is configured. Describe the failure and what a fix would involve. Do not discuss classifications, actions, policies, or which of them are permitted; those are Ariadne's concern and are meaningless in the service's issue tracker.
|
|
Three classifications have a predefined remediation. Use one only when the evidence plainly shows that failure; otherwise leave requested_action null.
|
|
Use transient_infra_failure with requested_action {"type": "run_ariadne_job", "id": "retry_transient_infra"} when the evidence shows an infrastructure, connectivity, or registry error unrelated to the repository's code or tests (DNS resolution failure, connection refused, reset, or timed out, TLS handshake failure, image pull failure, or a 5xx from a registry or SCM host), because re-running the same commit is then the whole remediation.
|
|
Use workspace_storage_exhausted with requested_action {"type": "run_ariadne_job", "id": "reclaim_workspace_storage"} when the evidence shows the build ran out of disk on its workspace volume (no space left on device, disk quota exceeded) while writing under the agent workspace. Do not classify this as transient_infra_failure: a plain rebuild lands on the same full volume, so the remediation must reclaim the stale workspace storage first.
|
|
Use jenkins_agent_provisioning_failure with requested_action {"type": "run_ariadne_job", "id": "clear_stuck_agent_pods"} when the evidence shows the build never got an agent (all nodes of a label offline, an agent pod stuck ContainerCreating or Pending, or an error in provisioning) rather than failing once it was running.
|
|
Otherwise leave requested_action null.
|
|
Set human_required to true when decisive evidence is missing or the failure needs a judgement only a maintainer can make; otherwise set it false and say plainly what you believe is wrong.
|
|
Do not perform mutations."""
|
|
|
|
_FIXTURE_RULES = """
|
|
This job is the fixture job. Use classification known_demo_fixture_failure with requested_action {"type": "run_ariadne_job", "id": "repair_demo_fixture"} when the evidence matches the fixture unhealthy signature, and set human_required false, because the predefined repair is then the whole remediation."""
|
|
|
|
_BUNDLE_SUFFIX = """
|
|
|
|
Bundle:
|
|
__BUNDLE__"""
|
|
|
|
|
|
def build_prompt(incident_id: str, job: str, bundle: dict[str, Any]) -> str:
|
|
"""Render the frozen triage prompt for one incident.
|
|
|
|
Inputs: the incident id, the Jenkins job the incident belongs to, and the
|
|
evidence bundle. Outputs: the prompt string sent to Hermes.
|
|
|
|
The fixture rules are appended only for the fixture job, so a real service
|
|
is never told about them and cannot repeat them into its own issues. The
|
|
demo classification therefore cannot be produced elsewhere by construction
|
|
rather than by instruction.
|
|
"""
|
|
|
|
template = _BASE + (_FIXTURE_RULES if job == DEMO_JOB else "") + _BUNDLE_SUFFIX
|
|
compact = json.dumps(bundle, separators=(",", ":"), ensure_ascii=True)
|
|
return (
|
|
template.replace("__INCIDENT_ID__", incident_id)
|
|
.replace("__JOB__", job)
|
|
.replace("__BUNDLE__", compact)
|
|
)
|