feat(hermes): make cluster triage immediately usable

This commit is contained in:
jenkins 2026-08-02 17:26:58 -03:00
parent fa21a760ef
commit fcb6fd0171
5 changed files with 86 additions and 4 deletions

View File

@ -110,6 +110,10 @@ data:
read-only state. Turn repeated successful triage paths into reusable
skills or memory when the pattern is stable.
When Brad says "triage", "triage now", "check the latest failures", or
"what is broken", begin the triage-titan-test-failures workflow
immediately. Do not make him choose a tool, page, or evidence source first.
Stay Flux-first. Do not mutate the cluster directly. Explain evidence,
recommend the smallest repo-side change, and name the exact verification
commands a human should run after Flux reconciles.
@ -119,6 +123,11 @@ data:
You are Hermes running inside the Titan Kubernetes cluster as a read-only
testing and operations triage assistant.
Treat short requests such as `triage`, `triage now`, `check the latest
failures`, and `what is broken` as instructions to load and run the
`triage-titan-test-failures` skill immediately. Ask only when approval is
required for a state-changing evidence refresh.
Ariadne owns deterministic evidence collection and local diagnosis. Start
every testing triage by reading:
@ -152,3 +161,28 @@ data:
Do not run mutating commands such as `kubectl apply`, `delete`, `scale`,
`patch`, `cordon`, `uncordon`, `drain`, or `rollout restart`. Do not read
Kubernetes Secret values. Draft repo changes or operator steps instead.
START-HERE.md: |
# Hermes on Atlas: start here
Use Chat for a new investigation or Sessions to resume a previous one.
The shortest useful prompts are:
- `Triage the latest test failure.`
- `What is broken in the cluster right now?`
- `Triage <suite> build <number> and save the report.`
Hermes will use Ariadne evidence, Jenkins log excerpts, quality metrics,
recent Git changes, Flux state, Grafana context, and Kubernetes read-only
state. It will separate facts, inference, and unknowns and will ask before
triggering a fresh evidence collection.
Two real examples are already available under Sessions:
1. `Proof 1 - Soteria 269 SonarQube triage`
2. `Proof 2 - Veles registry outage triage`
Their saved reports are in Files under `triage-proof/`.
Hermes may inspect workload and delivery metadata, logs, metrics, and
events. Its Kubernetes identity cannot read Secret values or mutate cluster
resources. Apply fixes through the titan-iac GitOps workflow after review.

View File

@ -24,7 +24,7 @@ spec:
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
ai.bstein.dev/role: testing-triage
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
ai.bstein.dev/config-rev: "20260802-session-stability"
ai.bstein.dev/config-rev: "20260802-triage-ready"
spec:
serviceAccountName: hermes-triage
automountServiceAccountToken: true
@ -82,10 +82,11 @@ spec:
- -c
- |
set -eu
mkdir -p /opt/data/workspace /opt/data/home/.local/bin /opt/data/logs
mkdir -p /opt/data/workspace/triage-proof /opt/data/home/.local/bin /opt/data/logs
cp /config/config.yaml /opt/data/config.yaml
cp /config/SOUL.md /opt/data/SOUL.md
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
touch /opt/data/.env
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"

View File

@ -22,6 +22,7 @@ rules:
- pods
- pods/log
- replicationcontrollers
- serviceaccounts
- services
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
@ -54,6 +55,12 @@ rules:
- gitrepositories
- helmrepositories
verbs: ["get", "list", "watch"]
- apiGroups: ["image.toolkit.fluxcd.io"]
resources:
- imagepolicies
- imagerepositories
- imageupdateautomations
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
@ -67,4 +74,3 @@ subjects:
- kind: ServiceAccount
name: hermes-triage
namespace: hermes

View File

@ -52,6 +52,47 @@ Work in this order:
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry.
6. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.
For recent Git context, map the canonical suite to its Gitea repository and
read the latest commits. The current mappings are:
- `ananke` -> `bstein/ananke`
- `ariadne` -> `bstein/Ariadne`
- `atlasbot` -> `bstein/atlasbot`
- `bstein_home` -> `bstein/bstein-dev-home`
- `data_prepper` -> `bstein/titan-iac`
- `metis` -> `bstein/metis`
- `pegasus` -> `bstein/pegasus`
- `soteria` -> `bstein/soteria`
- `titan_iac` -> `bstein/titan-iac`
Use the read-only Gitea API and compare commit timestamps to the failing build:
```sh
curl -fsS "$GITEA_BASE_URL/api/v1/repos/<owner>/<repo>/commits?limit=10"
```
For quality telemetry, query VictoriaMetrics directly when the bundle is stale
or a specific label needs confirmation. URL-encode PromQL rather than manually
escaping it:
```sh
curl -G -fsS --data-urlencode \
'query=platform_quality_gate_runs_total{suite="<suite>"}' \
"$VICTORIA_METRICS_URL/api/v1/query"
curl -G -fsS --data-urlencode \
'query=<suite>_quality_gate_checks_total' \
"$VICTORIA_METRICS_URL/api/v1/query"
```
For image-pull failures, inspect the pod event, its ServiceAccount metadata,
and Flux image objects without reading referenced Secret values:
```sh
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> get serviceaccount <name> -o yaml
kubectl get imagerepositories,imagepolicies,imageupdateautomations -A
```
If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.
Use read-only commands such as `kubectl get`, `kubectl describe`, `kubectl logs`, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.

View File

@ -431,7 +431,7 @@ spec:
- name: ARIADNE_VM_URL
value: http://victoria-metrics-single-server.monitoring.svc.cluster.local:8428
- name: ARIADNE_CLUSTER_STATE_VM_TIMEOUT_SEC
value: "5"
value: "30"
- name: ARIADNE_ALERTMANAGER_URL
value: http://alertmanager.monitoring.svc.cluster.local
- name: OPENSEARCH_URL