feat(hermes): make cluster triage immediately usable
This commit is contained in:
parent
fa21a760ef
commit
fcb6fd0171
@ -110,6 +110,10 @@ data:
|
|||||||
read-only state. Turn repeated successful triage paths into reusable
|
read-only state. Turn repeated successful triage paths into reusable
|
||||||
skills or memory when the pattern is stable.
|
skills or memory when the pattern is stable.
|
||||||
|
|
||||||
|
When Brad says "triage", "triage now", "check the latest failures", or
|
||||||
|
"what is broken", begin the triage-titan-test-failures workflow
|
||||||
|
immediately. Do not make him choose a tool, page, or evidence source first.
|
||||||
|
|
||||||
Stay Flux-first. Do not mutate the cluster directly. Explain evidence,
|
Stay Flux-first. Do not mutate the cluster directly. Explain evidence,
|
||||||
recommend the smallest repo-side change, and name the exact verification
|
recommend the smallest repo-side change, and name the exact verification
|
||||||
commands a human should run after Flux reconciles.
|
commands a human should run after Flux reconciles.
|
||||||
@ -119,6 +123,11 @@ data:
|
|||||||
You are Hermes running inside the Titan Kubernetes cluster as a read-only
|
You are Hermes running inside the Titan Kubernetes cluster as a read-only
|
||||||
testing and operations triage assistant.
|
testing and operations triage assistant.
|
||||||
|
|
||||||
|
Treat short requests such as `triage`, `triage now`, `check the latest
|
||||||
|
failures`, and `what is broken` as instructions to load and run the
|
||||||
|
`triage-titan-test-failures` skill immediately. Ask only when approval is
|
||||||
|
required for a state-changing evidence refresh.
|
||||||
|
|
||||||
Ariadne owns deterministic evidence collection and local diagnosis. Start
|
Ariadne owns deterministic evidence collection and local diagnosis. Start
|
||||||
every testing triage by reading:
|
every testing triage by reading:
|
||||||
|
|
||||||
@ -152,3 +161,28 @@ data:
|
|||||||
Do not run mutating commands such as `kubectl apply`, `delete`, `scale`,
|
Do not run mutating commands such as `kubectl apply`, `delete`, `scale`,
|
||||||
`patch`, `cordon`, `uncordon`, `drain`, or `rollout restart`. Do not read
|
`patch`, `cordon`, `uncordon`, `drain`, or `rollout restart`. Do not read
|
||||||
Kubernetes Secret values. Draft repo changes or operator steps instead.
|
Kubernetes Secret values. Draft repo changes or operator steps instead.
|
||||||
|
START-HERE.md: |
|
||||||
|
# Hermes on Atlas: start here
|
||||||
|
|
||||||
|
Use Chat for a new investigation or Sessions to resume a previous one.
|
||||||
|
The shortest useful prompts are:
|
||||||
|
|
||||||
|
- `Triage the latest test failure.`
|
||||||
|
- `What is broken in the cluster right now?`
|
||||||
|
- `Triage <suite> build <number> and save the report.`
|
||||||
|
|
||||||
|
Hermes will use Ariadne evidence, Jenkins log excerpts, quality metrics,
|
||||||
|
recent Git changes, Flux state, Grafana context, and Kubernetes read-only
|
||||||
|
state. It will separate facts, inference, and unknowns and will ask before
|
||||||
|
triggering a fresh evidence collection.
|
||||||
|
|
||||||
|
Two real examples are already available under Sessions:
|
||||||
|
|
||||||
|
1. `Proof 1 - Soteria 269 SonarQube triage`
|
||||||
|
2. `Proof 2 - Veles registry outage triage`
|
||||||
|
|
||||||
|
Their saved reports are in Files under `triage-proof/`.
|
||||||
|
|
||||||
|
Hermes may inspect workload and delivery metadata, logs, metrics, and
|
||||||
|
events. Its Kubernetes identity cannot read Secret values or mutate cluster
|
||||||
|
resources. Apply fixes through the titan-iac GitOps workflow after review.
|
||||||
|
|||||||
@ -24,7 +24,7 @@ spec:
|
|||||||
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
|
ai.bstein.dev/model: openai-codex/gpt-5.6-terra with local gpt-oss:20b fallback
|
||||||
ai.bstein.dev/role: testing-triage
|
ai.bstein.dev/role: testing-triage
|
||||||
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
|
ai.bstein.dev/placement: arm64 gateway lane (rpi5 preferred)
|
||||||
ai.bstein.dev/config-rev: "20260802-session-stability"
|
ai.bstein.dev/config-rev: "20260802-triage-ready"
|
||||||
spec:
|
spec:
|
||||||
serviceAccountName: hermes-triage
|
serviceAccountName: hermes-triage
|
||||||
automountServiceAccountToken: true
|
automountServiceAccountToken: true
|
||||||
@ -82,10 +82,11 @@ spec:
|
|||||||
- -c
|
- -c
|
||||||
- |
|
- |
|
||||||
set -eu
|
set -eu
|
||||||
mkdir -p /opt/data/workspace /opt/data/home/.local/bin /opt/data/logs
|
mkdir -p /opt/data/workspace/triage-proof /opt/data/home/.local/bin /opt/data/logs
|
||||||
cp /config/config.yaml /opt/data/config.yaml
|
cp /config/config.yaml /opt/data/config.yaml
|
||||||
cp /config/SOUL.md /opt/data/SOUL.md
|
cp /config/SOUL.md /opt/data/SOUL.md
|
||||||
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
|
cp /config/AGENTS.md /opt/data/workspace/AGENTS.md
|
||||||
|
cp /config/START-HERE.md /opt/data/workspace/START-HERE.md
|
||||||
touch /opt/data/.env
|
touch /opt/data/.env
|
||||||
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
|
if ! grep -q '^API_SERVER_KEY=' /opt/data/.env; then
|
||||||
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
api_key="$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n')"
|
||||||
|
|||||||
@ -22,6 +22,7 @@ rules:
|
|||||||
- pods
|
- pods
|
||||||
- pods/log
|
- pods/log
|
||||||
- replicationcontrollers
|
- replicationcontrollers
|
||||||
|
- serviceaccounts
|
||||||
- services
|
- services
|
||||||
verbs: ["get", "list", "watch"]
|
verbs: ["get", "list", "watch"]
|
||||||
- apiGroups: ["apps"]
|
- apiGroups: ["apps"]
|
||||||
@ -54,6 +55,12 @@ rules:
|
|||||||
- gitrepositories
|
- gitrepositories
|
||||||
- helmrepositories
|
- helmrepositories
|
||||||
verbs: ["get", "list", "watch"]
|
verbs: ["get", "list", "watch"]
|
||||||
|
- apiGroups: ["image.toolkit.fluxcd.io"]
|
||||||
|
resources:
|
||||||
|
- imagepolicies
|
||||||
|
- imagerepositories
|
||||||
|
- imageupdateautomations
|
||||||
|
verbs: ["get", "list", "watch"]
|
||||||
---
|
---
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
kind: ClusterRoleBinding
|
kind: ClusterRoleBinding
|
||||||
@ -67,4 +74,3 @@ subjects:
|
|||||||
- kind: ServiceAccount
|
- kind: ServiceAccount
|
||||||
name: hermes-triage
|
name: hermes-triage
|
||||||
namespace: hermes
|
namespace: hermes
|
||||||
|
|
||||||
|
|||||||
@ -52,6 +52,47 @@ Work in this order:
|
|||||||
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry.
|
5. Check Pushgateway and Grafana evidence for branch gaps, stale metrics, aliases, or missing zero-state telemetry.
|
||||||
6. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.
|
6. State unknowns explicitly. Never invent a log line, metric, commit, pod condition, or root cause.
|
||||||
|
|
||||||
|
For recent Git context, map the canonical suite to its Gitea repository and
|
||||||
|
read the latest commits. The current mappings are:
|
||||||
|
|
||||||
|
- `ananke` -> `bstein/ananke`
|
||||||
|
- `ariadne` -> `bstein/Ariadne`
|
||||||
|
- `atlasbot` -> `bstein/atlasbot`
|
||||||
|
- `bstein_home` -> `bstein/bstein-dev-home`
|
||||||
|
- `data_prepper` -> `bstein/titan-iac`
|
||||||
|
- `metis` -> `bstein/metis`
|
||||||
|
- `pegasus` -> `bstein/pegasus`
|
||||||
|
- `soteria` -> `bstein/soteria`
|
||||||
|
- `titan_iac` -> `bstein/titan-iac`
|
||||||
|
|
||||||
|
Use the read-only Gitea API and compare commit timestamps to the failing build:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -fsS "$GITEA_BASE_URL/api/v1/repos/<owner>/<repo>/commits?limit=10"
|
||||||
|
```
|
||||||
|
|
||||||
|
For quality telemetry, query VictoriaMetrics directly when the bundle is stale
|
||||||
|
or a specific label needs confirmation. URL-encode PromQL rather than manually
|
||||||
|
escaping it:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -G -fsS --data-urlencode \
|
||||||
|
'query=platform_quality_gate_runs_total{suite="<suite>"}' \
|
||||||
|
"$VICTORIA_METRICS_URL/api/v1/query"
|
||||||
|
curl -G -fsS --data-urlencode \
|
||||||
|
'query=<suite>_quality_gate_checks_total' \
|
||||||
|
"$VICTORIA_METRICS_URL/api/v1/query"
|
||||||
|
```
|
||||||
|
|
||||||
|
For image-pull failures, inspect the pod event, its ServiceAccount metadata,
|
||||||
|
and Flux image objects without reading referenced Secret values:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
kubectl -n <namespace> describe pod <pod>
|
||||||
|
kubectl -n <namespace> get serviceaccount <name> -o yaml
|
||||||
|
kubectl get imagerepositories,imagepolicies,imageupdateautomations -A
|
||||||
|
```
|
||||||
|
|
||||||
If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.
|
If the latest diagnosis is unavailable, the minimum acceptable result still analyzes the deterministic bundle's failed suites, build/check evidence, freshness, and environment observations. Model unavailability is context, not the triage finding, unless the user's question is specifically about model health.
|
||||||
|
|
||||||
Use read-only commands such as `kubectl get`, `kubectl describe`, `kubectl logs`, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.
|
Use read-only commands such as `kubectl get`, `kubectl describe`, `kubectl logs`, and HTTP GET requests. Do not read Secret values or run mutating Kubernetes, Flux, Vault, Jenkins, or Git commands.
|
||||||
|
|||||||
@ -431,7 +431,7 @@ spec:
|
|||||||
- name: ARIADNE_VM_URL
|
- name: ARIADNE_VM_URL
|
||||||
value: http://victoria-metrics-single-server.monitoring.svc.cluster.local:8428
|
value: http://victoria-metrics-single-server.monitoring.svc.cluster.local:8428
|
||||||
- name: ARIADNE_CLUSTER_STATE_VM_TIMEOUT_SEC
|
- name: ARIADNE_CLUSTER_STATE_VM_TIMEOUT_SEC
|
||||||
value: "5"
|
value: "30"
|
||||||
- name: ARIADNE_ALERTMANAGER_URL
|
- name: ARIADNE_ALERTMANAGER_URL
|
||||||
value: http://alertmanager.monitoring.svc.cluster.local
|
value: http://alertmanager.monitoring.svc.cluster.local
|
||||||
- name: OPENSEARCH_URL
|
- name: OPENSEARCH_URL
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user