# Hermes Investigative Triage Demonstration

- Date: 2026-08-05
- Run ID: run_d2f61902c00044948a3912a94e895ab9
- Duration: 128 seconds, status completed
- Usage: {"input_tokens": 200961, "output_tokens": 4370, "total_tokens": 205331}
- Mode: supervised, read-only; approvals `smart` with mutation deny-list; zero mutations, zero denied approvals
- Tool audit (from the run's SSE event stream): 8 skill loads (triage-titan-test-failures pack: orchestrator, jenkins-retained-evidence, platform-quality-metrics, kubernetes-readonly-failure-classifier, flux-git-change-correlation, ...), 4 read-only terminal batches (Ariadne internal API sweep, kubectl logs/jobs/configmap in hermes-triage-demo, VictoriaMetrics queries incl. last_over_time fallback, Flux/Gitea revision correlation)

Unlike the automated loop (where Ariadne supplies a sanitized bundle), this run
was given ONLY an incident ID and gathered all evidence itself — the operator's
triage path executed by the agent. Notable behaviors: honest dead-end reporting
(Jenkins 403 -> pivot to Kubernetes-retained logs; Ariadne audit endpoints 404),
instant-query lookback miss -> switched to last_over_time, causal chain
reconstruction across five sources, and an unprompted, legitimate hardening
suggestion (add a build label to ariadne_hermes_triage_action_total).

## Prompt

```text
Use $triage-titan-test-failures.
Investigate Jenkins incident hermes-triage-demo/10. No evidence bundle is attached this time: gather the evidence yourself with your read-only tools, following the operator's standard triage path:
1. Ariadne is the source of truth: query its internal API (base URL in ARIADNE_BASE_URL, e.g. GET /api/internal/testing/triage/latest and any internal audit endpoints you can reach) for records about this incident.
2. Quality metrics: query VictoriaMetrics (VICTORIA_METRICS_URL) for the series ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo",build="10"} and ariadne_hermes_triage_action_total, and note what they show.
3. Service health: use read-only kubectl to inspect the hermes-triage-demo namespace (jobs, pods, recent events) and confirm what ran there.
4. Change correlation: check Flux state (kubectl get kustomizations -n flux-system; the flux-system GitRepository revision) for recent changes relevant to this incident.
Then return a concise operator-style report with EXACTLY these sections:
FACTS (each with source and reference), INFERENCES, LIKELY CAUSE, WHAT HAPPENED (was it repaired, by what mechanism, final state), NEXT CHECKS (what a human should verify next), and REPO-SIDE FIXES (if any are warranted).
Read-only throughout. Do not mutate anything. Do not use kubectl apply/delete/patch/scale.
```

## Full agent report

```text
FACTS
Ariadne diagnosis/latest, generated 2026-08-05T21:31:06Z, references only the broad testing bundle and does not identify hermes-triage-demo/10. Ariadne’s incident/audit candidate endpoints returned HTTP 404.
Kubernetes logs, hermes-triage-demo/hermes-demo-test-10-dgvpj: build 10 emitted incident_id="hermes-triage-demo/10", classification_hint="demo_fixture_unhealthy", and “expected fixture state healthy; found unhealthy.”
Kubernetes Job status: hermes-demo-test-10 was created 2026-08-05T20:59:47Z and failed; its pod ran on titan-06 and had no restarts or image-pull events.
Kubernetes logs, hermes-demo-repair-10-nkrzz: repair job patched hermes-triage-demo-fixture and reported “fixture state reset to healthy”; the Job completed successfully at 2026-08-05T21:01:06Z.
Kubernetes logs/status, hermes-demo-test-11-hv5mh: follow-up test completed at 2026-08-05T21:01:35Z and reported “fixture state healthy.”
Kubernetes ConfigMap hermes-triage-demo-fixture: current data.state is healthy.
VictoriaMetrics, last_over_time(ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo",build="10"}[24h]): sample at 2026-08-05T21:41:01Z has resolved=1 and every other lifecycle state, including failed and human_required, at 0. The instantaneous selector returned no result because its samples were outside the default instant-query lookback.
VictoriaMetrics, ariadne_hermes_triage_action_total over 24h: repair_demo_fixture has requested=1, accepted=1, success=1 for instance 10.42.19.143; no build label exists on this counter, so it is correlated by the incident instance and timeline rather than proven per-build.
Flux Kustomization hermes-triage-demo: Ready=True at revision main@sha1:aa36b06b2bc43eda21b36e54a7d312e55d27c370. Flux source GitRepository is Ready=True at that same revision.
Gitea commit 49aab8da3298ebfc1b9edab5ce2e6938ad033a23, “feat(hermes-triage): enable autoremediation for the demo loop,” predates this incident and modifies services/maintenance/apps/ariadne-deployment.yaml. The current revision aa36b06 is an unrelated maintenance image update.
Direct Jenkins GET for /job/hermes-triage-demo/10 API metadata and console returned HTTP 403. The Kubernetes Job logs are therefore the retained direct execution evidence used here.

INFERENCES
The failure was an intentional demo-fixture health assertion failure, not an image, scheduling, node, or storage failure. The failed test, repair, and successful follow-up all ran in sequence; this is a strong causal chain.
The repair was automatically accepted under the previously deployed allowlisted autoremediation path. The repair Job itself performed the ConfigMap patch; Flux did not repair the fixture during this incident.
The unrelated cluster concerns in the Ariadne bundle, including Longhorn probe failures and pending Veles pods, have no evidenced link to this ConfigMap-based test path.

LIKELY CAUSE
The fixture was deliberately or otherwise pre-seeded unhealthy before Jenkins incident hermes-triage-demo/10. The test correctly failed on that condition. Confidence: high.

WHAT HAPPENED
Build 10 failed at 20:59:47Z because the fixture state was unhealthy. The deployed autoremediation mechanism accepted repair_demo_fixture, ran hermes-demo-repair-10, and patched only hermes-triage-demo-fixture to healthy. Test 11 then passed at 21:01:35Z. Final state is repaired/resolved: ConfigMap state=healthy, incident metric resolved=1, repair success counter=1, and the demo Flux Kustomization is Ready at aa36b06.

NEXT CHECKS
Human should verify the current deployed revision and readiness after the next Flux interval:
kubectl -n flux-system get kustomization hermes-triage-demo
kubectl -n flux-system describe gitrepository flux-system

Human should retain the build-10 Jenkins evidence if access is available:
curl -fsS https://ci.bstein.dev/job/hermes-triage-demo/10/api/json
curl -fsS https://ci.bstein.dev/job/hermes-triage-demo/10/consoleText
These currently require Jenkins authentication (HTTP 403).

Human should confirm that the resolved gauge remains the only asserted lifecycle state:
curl -G -fsS --data-urlencode 'query=last_over_time(ariadne_hermes_triage_incident{jenkins_job="hermes-triage-demo",build="10"}[24h])' "$VICTORIA_METRICS_URL/api/v1/query"

REPO-SIDE FIXES
None warranted for this incident. The observed behavior matches the committed demo design: an isolated ConfigMap fixture, one allowlisted repair action, a scoped repair ServiceAccount, and one repair per incident. A future hardening change, if desired, would be to add a build label to ariadne_hermes_triage_action_total so repair counters can be directly attributed to a specific incident rather than correlated by instance and time.
```
