# Hermes Automated Triage: 24-Hour Delivery Plan

- Status: planning only
- Created: 2026-08-05
- Hard delivery limit: 24 elapsed hours from execution authorization
- Priority: deadline first, then the most complete safe automation that fits

## Executive decision

The first delivery will automate one narrow, real, repeatable failure-to-repair path. It will not attempt general autonomous coding before that path works end to end.

The target demonstration is:

```text
Controlled test failure
  -> Jenkins build fails
  -> Ariadne detects the terminal failure automatically
  -> Ariadne gathers Jenkins evidence and bounded OpenSearch logs
  -> Ariadne automatically invokes the Hermes Agent
  -> Hermes selects its Titan test-triage skills
  -> Hermes returns a structured diagnosis and requested action
  -> Ariadne validates the request against an exact allowlist
  -> Ariadne creates one predefined repair Job
  -> Ariadne triggers one Jenkins rebuild
  -> the rebuild passes and the incident resolves
```

A second path will demonstrate safe escalation:

```text
Unsupported, ambiguous or unsafe failure
  -> Hermes returns human_required
  -> Ariadne publishes a triage metric
  -> VictoriaMetrics and vmalert evaluate it
  -> Grafana and Alertmanager provide the human-facing signal
```

At H+24 work stops. Lower-priority functionality is removed or deferred rather than extending the deadline.

## Definition of done

The delivery is fully successful when all of the following are demonstrated:

- A deliberately seeded test failure causes a Jenkins build to fail.
- Ariadne detects that exact terminal build once, preferably within two minutes.
- Ariadne assigns a stable incident ID based on the Jenkins job and build number.
- The diagnosis bundle includes the first failed stage, retained Jenkins evidence and provenance-bearing OpenSearch excerpts.
- Ariadne invokes the Hermes Agent without a human prompt.
- The invocation explicitly selects the Titan test-triage skill.
- Hermes returns schema-valid facts, inferences, classification, confidence and action request.
- Ariadne accepts only the predefined demo action and executes it once.
- A predefined repair Job corrects only the isolated demo fixture.
- Ariadne triggers one rebuild with failure seeding disabled.
- The rebuilt job passes and Ariadne records the incident as resolved.
- An unsupported classification produces a human-required metric and no mutation.
- No secret values are stored in evidence, model prompts, responses or logs.
- No path can automatically write to `main`, reconcile Flux, or execute model-supplied commands.
- A single Flux-managed setting can disable all automatic actions immediately.

Target demo timings are:

- Failure detection: less than two minutes.
- Evidence collection and Hermes response: less than three minutes.
- Repair launch and rebuild request: less than two minutes.
- Complete failure-to-green loop: less than ten minutes, excluding image pulls during a cold cluster start.

## Current system context

This section records the facts needed to execute the plan without relying on prior conversation.

### Jenkins and the homegrown suites

Jenkins runs the homegrown repositories on ephemeral Kubernetes agent pods. The active quality scope includes Ariadne, Metis, Ananke, Atlasbot, Pegasus, Soteria, titan-iac, bstein-dev-home, data-prepper and Lesavka, with additional build/test jobs for projects such as Arcanagon, Typhon and Veles.

The normalized quality evidence includes, as applicable:

```text
build
  -> style/docs
  -> LOC/naming
  -> coverage
  -> tests
  -> gate glue
  -> SonarQube
  -> Semgrep
  -> Trivy/supply-chain evidence
  -> gate enforcement
  -> image build and Harbor push
```

Jenkins retains build metadata, console output, stage results, JUnit, coverage and quality reports. These retained artifacts are the authoritative source for the exact failed test or gate.

### Logging and OpenSearch

OpenSearch does not collect logs by itself. Fluent Bit tails Kubernetes container logs and writes them directly into `kube-*` indices. It also writes node journal records into `journald-*` indices.

The relevant path is:

```text
Kubernetes pod stdout/stderr
  -> Fluent Bit DaemonSet
  -> OpenSearch kube-* indices
```

Jenkins-retained console logs and OpenSearch pod logs are separate evidence paths. Jenkins agent command output must not be assumed to exist in CRI container logs. The demo must use Jenkins artifacts for the exact assertion and must ensure that a dedicated demo test-runner or related workload writes a correlated incident message to pod stdout/stderr for OpenSearch.

Data Prepper is not part of the ordinary log path. It receives OpenTelemetry traces, transforms them and writes trace/service-map indices into OpenSearch. It is irrelevant to the Jenkins log integration except as another separately tested workload.

Relevant IaC sources include:

- `services/logging/fluent-bit-helmrelease.yaml`
- `services/logging/opensearch-observability-objects.yaml`
- `services/logging/Jenkinsfile.data-prepper`

### Metrics and human-facing status

Jenkins publishers and runtime probes send normalized quality metrics to `platform-quality-gateway`. VictoriaMetrics scrapes Prometheus-compatible endpoints, vmalert evaluates rules, Grafana displays the Atlas Testing dashboards, and Alertmanager owns notification delivery.

The intended human-escalation path is therefore:

```text
Ariadne triage metric
  -> VictoriaMetrics
  -> vmalert
  -> Alertmanager notification and Grafana status
```

Hermes does not send an alert "to Grafana." Grafana visualizes the metric and alert state.

Relevant IaC sources include:

- `services/monitoring/platform-quality-gateway-deployment.yaml`
- `services/monitoring/platform-quality-suite-probe-cronjob.yaml`
- `services/monitoring/vmalert-atlas-availability.yaml`
- `services/monitoring/dashboards/atlas-testing.json`
- `services/quality/sonarqube-exporter-deployment.yaml`

### Ariadne today

Ariadne is the deterministic automation and evidence-collection service in the `maintenance` namespace. Its current deployment has access to:

- Jenkins API credentials and `JENKINS_BASE_URL`.
- VictoriaMetrics through `ARIADNE_VM_URL`.
- OpenSearch through `OPENSEARCH_URL`.
- Kubernetes through its service account.
- The local model through `ARIADNE_TESTING_TRIAGE_MODEL_URL`.

The deployment schedules Jenkins build-weather collection every ten minutes and testing triage every fifteen minutes. Those intervals are too slow for a smooth demo and should be reduced only for the allowlisted demo path.

Ariadne's current OpenSearch integration manages index retention. It does not currently retrieve historical log content for testing diagnosis.

Ariadne currently points model-assisted testing triage at the Hermes Ollama service. Calling Ollama uses the same Qwen model but bypasses the Hermes Agent, its tool loop, skills, memory and approval model. This is not equivalent to invoking Hermes.

Ariadne already has patterns for launching a platform quality probe Job and for performing Jenkins workspace maintenance. These patterns should be reused for the predefined demo repair rather than introducing a general job-execution system.

Relevant IaC source:

- `services/maintenance/apps/ariadne-deployment.yaml`

Relevant application repository:

- Gitea: `bstein/Ariadne`
- Local checkout currently at `/home/brad/Development/Ariadne`

### Hermes today

Hermes runs as a single read-only triage agent in the `hermes` namespace. Its current configuration provides:

- Hermes Agent API on port `8642`.
- Hermes dashboard on port `9119`.
- Local Qwen 2.5 `7b-instruct-q4_0` inference through Hermes Ollama.
- Ariadne, Jenkins, VictoriaMetrics, Gitea and Grafana base URLs.
- `kubectl` installed as a read-only cluster inspection tool.
- A persistent home PVC containing memories and runtime-installed skills.

Hermes does not currently receive `OPENSEARCH_URL`. In the desired design it does not need direct OpenSearch access: Ariadne will provide bounded, sanitized excerpts in the diagnosis bundle.

Hermes's Flux-managed instructions say to start with Ariadne as the source of truth, then use Jenkins, quality metrics, Flux, Grafana and Kubernetes read-only evidence. Its RBAC allows reads of workloads, events, pod logs, services, storage objects and Flux resources. It cannot read Kubernetes Secret values.

Hermes explicitly denies or lacks permission for actions such as:

- `kubectl apply`, delete, patch, scale, cordon, drain or rollout restart.
- Flux suspend, resume or reconcile.
- Vault secret reads.
- Kubernetes Secret reads.
- Jenkins rebuilds.
- Git writes.

This boundary remains in place. Hermes will request actions; Ariadne will validate and execute exact predefined actions.

Relevant IaC sources include:

- `services/hermes/deployment.yaml`
- `services/hermes/configmap.yaml`
- `services/hermes/rbac.yaml`

### Hermes's current Atlas skills

A read-only inventory of the live Hermes PVC on 2026-08-05 showed the Atlas triage entry skills:

- `triage-titan-test-failures`
- `triage-atlas-service-health`
- `tune-atlas-alerts`
- `master-hermes-on-atlas`

The test-triage skill pack includes:

- `titan-triage-orchestrator`
- `jenkins-retained-evidence`
- `platform-quality-metrics`
- `kubernetes-readonly-failure-classifier`
- `flux-git-change-correlation`
- `grafana-metric-provenance`
- `triage-evidence-reporting`
- `approved-triage-actions`
- `soteria-backup-health`

The orchestrator already defines the needed evidence order: Ariadne first, then the smallest relevant specialist, followed by a fact-versus-inference report. The demo invocation should explicitly request `$triage-titan-test-failures`; a new general triage framework is unnecessary.

These skills currently live on the runtime PVC and are not all declared in `titan-iac`. The demo should not depend on mutating the live PVC manually. Any essential prompt or output contract must be supplied by Ariadne or added through Flux-managed Hermes configuration.

### Current gaps

The desired closed loop does not exist today because:

- Ariadne does not query OpenSearch for triage log evidence.
- Ariadne calls Ollama rather than the Hermes Agent API.
- There is no shared Ariadne-to-Hermes API authentication contract.
- There is no schema for a Hermes triage response.
- There is no allowlisted Hermes-to-Ariadne action request.
- Ariadne does not automatically execute a Hermes-selected repair.
- There is no incident-level idempotency preventing duplicate repair attempts.
- There is no dedicated human-required triage metric and alert.
- The current architecture chart compresses Hermes's internal trigger, routing and decision steps into a single box.

## Worktree and source-control safety

The existing working directories must not be used as clean implementation bases.

As of 2026-08-05:

- `/home/brad/Development/titan-iac` has extensive unrelated modified, deleted and untracked user work, including an ongoing repository-layout migration and untracked Mermaid files.
- `/home/brad/Development/Ariadne` is dirty on branch `codex/ariadne-metrics-hotfix`.

Execution must:

1. Inspect the actual revisions deployed by Flux and Jenkins.
2. Fetch without altering either existing worktree.
3. Create dedicated clean Git worktrees for the demo branches.
4. Never reset, clean, stash, rebase or overwrite either existing dirty worktree.
5. Keep Ariadne application changes and titan-iac/Flux changes in separately reviewable commits.
6. Bring over only the intentional planning/diagram change when appropriate.

Suggested isolated paths are:

```text
/home/brad/Development/_worktrees/ariadne-hermes-triage-demo
/home/brad/Development/_worktrees/titan-iac-hermes-triage-demo
```

The implementation branch point must be the current authoritative primary/deployed revision, not an assumed local branch.

## Deadline-driven scope

### Required scope

- One dedicated demo Jenkins job.
- One deterministic failure signature.
- One isolated demo fixture.
- One bounded OpenSearch search contract.
- One automatic Hermes invocation.
- One structured decision contract.
- One exact repair action.
- One rebuild attempt.
- One human-required metric and alert path.
- One complete source-level Mermaid update.

### Explicit non-goals

- General conversational OpenSearch access.
- Arbitrary index selection or unbounded log retrieval.
- General autonomous repository editing.
- Direct commits or merges to `main`.
- Model-generated Kubernetes manifests or commands.
- Automatic Flux reconciliation.
- Automatic SonarQube, Semgrep or Trivy waivers.
- Secret inspection.
- Multiple repair retries.
- Multiple repositories or failure signatures before the primary loop is green.
- A permanent generalized incident-management platform.

## Demo design

### Isolated test fixture

Use a dedicated `hermes-triage-demo` namespace, Jenkins job and fixture. The preferred deterministic design is:

1. A small PVC stores a single state value such as `healthy` or `unhealthy`.
2. The demo is explicitly armed through a Jenkins parameter such as `SEED_FAILURE=true`.
3. A dedicated Kubernetes test-runner Job reads the fixture.
4. When the fixture is unhealthy, it writes a structured incident line to stdout/stderr and exits nonzero.
5. Jenkins waits for the Job, records the test failure and archives a small JUnit result.
6. Fluent Bit forwards the test-runner Job output to OpenSearch.
7. The test-runner Job name and message include the Jenkins job/build incident ID.

Example structured log:

```json
{
  "event": "hermes_demo_test_failure",
  "incident_id": "hermes-triage-demo/42",
  "classification_hint": "demo_fixture_unhealthy",
  "message": "expected fixture state healthy; found unhealthy"
}
```

The Jenkins rebuild must set `SEED_FAILURE=false`, otherwise every rebuild would deliberately recreate the failure.

The namespace isolates the test-runner permissions and prevents a demo repair from touching production workloads.

### Predefined repair Job

Ariadne will contain one exact action mapping:

```text
repair_demo_fixture
  -> create hermes-demo-repair-<incident> Job
  -> mount only the demo fixture PVC
  -> write healthy state
  -> emit a structured repair result
  -> exit
```

Hermes cannot supply the image, command, namespace, PVC, labels or arbitrary parameters. It can request only the action ID.

After the Job succeeds, Ariadne requests one Jenkins rebuild with failure seeding disabled.

### Why a repair Job is the critical path

A source-code test failure generally requires a Git change, but safe autonomous code editing requires repository credentials, checkout isolation, patch generation, tests, branch creation and PR lifecycle handling. Making that the primary demo would threaten the 24-hour limit.

The fixture repair demonstrates a real closed loop:

- Test failure is real.
- Jenkins and OpenSearch evidence are real.
- Hermes classification is real.
- Hermes action selection is real.
- Ariadne authorization and execution are real.
- The repair and green rebuild are real.

Autonomous Gitea PR creation remains a stretch goal that begins only after the required loop is stable.

## Evidence contract

### Incident identity

Use a stable ID:

```text
<jenkins-job>/<build-number>
```

Every log line, bundle, Hermes request, action, repair Job and metric must carry that ID.

### Jenkins evidence

The bundle should include:

- Job and build number.
- Branch and commit when available.
- Build URL.
- Start and end timestamps.
- Terminal result.
- First failed stage.
- Console tail.
- Failed JUnit cases.
- Selected quality reports when relevant.
- Evidence source and retrieval timestamp.

### OpenSearch evidence

The demo query should be intentionally fixed and bounded:

- Indices: `kube-*` only.
- Time range: build start minus five minutes through build end plus five minutes.
- Namespace: `hermes-triage-demo`, plus `jenkins` only if it adds useful context.
- Correlation: exact incident ID first; namespace and time fallback second.
- Size: at most 50 records initially, never more than 100.
- Fields: timestamp, namespace, pod, container, log/message and selected safe labels.
- Timeout: approximately five seconds.
- Response byte cap: small enough to prevent prompt flooding.
- Pagination: none in the demo.

Each excerpt must retain:

- Index or source identifier.
- Timestamp.
- Namespace.
- Pod.
- Container.
- Sanitized message.

Basic sanitization must remove or mask:

- `Authorization` headers.
- Bearer tokens.
- Cookies and session IDs.
- Common password, token and secret assignments.
- Private key blocks.

This is demo-grade sanitization, not a claim of comprehensive data-loss prevention. Queries remain restricted to the isolated namespace wherever possible.

### Ariadne diagnosis bundle

Add a `log_evidence` section without replacing the existing bundle fields:

```json
{
  "incident_id": "hermes-triage-demo/42",
  "generated_at": "...",
  "jenkins": {},
  "quality_metrics": {},
  "runtime_context": {},
  "log_evidence": {
    "query_window": {},
    "records": [],
    "truncated": false,
    "error": null
  }
}
```

Empty or unavailable OpenSearch results must be represented explicitly. They must not block Jenkins-based diagnosis.

## Automatic Hermes invocation

### Required behavior

Ariadne must call the Hermes Agent API, not only the Ollama completion endpoint. The invocation prompt should be stable and small:

```text
Use $triage-titan-test-failures.
Analyze incident <incident-id>.
Treat the attached Ariadne bundle as the source of truth.
Identify the first enforced failure.
Distinguish facts from inference.
Return only the required triage response schema.
Do not perform mutations.
```

The bundle is attached as structured content. Ariadne records the Hermes request ID or session ID for audit and idempotency.

### Authentication

No API key may be committed to Git.

The preferred path is:

1. Generate one internal Hermes API key.
2. Store it in an approved Vault path.
3. Inject it into Hermes as `API_SERVER_KEY`.
4. Inject it into Ariadne as `HERMES_API_KEY`.
5. Restrict network access to the required namespaces where practical.

The exact Hermes Agent API path and authentication behavior must be verified in the first hour. The current Hermes init container generates a key in its persistent `.env` if none exists; the integration must not scrape that file or copy it into Git.

### Response schema

Hermes must return a bounded machine-readable response:

```json
{
  "incident_id": "hermes-triage-demo/42",
  "classification": "known_demo_fixture_failure",
  "confidence": 0.95,
  "facts": [
    {
      "statement": "...",
      "source": "jenkins|opensearch|victoriametrics|kubernetes|flux|gitea",
      "reference": "..."
    }
  ],
  "inferences": [],
  "first_failed_gate": "tests",
  "requested_action": {
    "type": "run_ariadne_job",
    "id": "repair_demo_fixture"
  },
  "human_required": false,
  "reason": ""
}
```

Invalid JSON, unknown fields, mismatched incident IDs, low confidence and unknown actions all become human-required results.

## Detection and deduplication

For the demo, Ariadne should detect only an allowlisted job such as `hermes-triage-demo`. Reducing the global scheduler interval must not cause automatic action against unrelated failures.

Suggested configuration:

```text
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=true
ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST=hermes-triage-demo
ARIADNE_HERMES_AUTOTRIAGE_INTERVAL_SECONDS=60
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false
ARIADNE_HERMES_ALLOWED_ACTIONS=repair_demo_fixture
ARIADNE_HERMES_MIN_CONFIDENCE=0.85
ARIADNE_HERMES_MAX_ACTIONS_PER_INCIDENT=1
```

The remediation flag remains false until the evidence-and-Hermes path passes in observe mode.

Ariadne must persist or otherwise reliably track:

- Incident observed.
- Hermes invocation started/completed.
- Action requested/accepted/rejected.
- Repair Job created/completed.
- Rebuild requested.
- Final build result.

The same Jenkins build must never invoke a second repair after its action record exists.

## Action authorization

Ariadne may accept the demo action only when:

- The Jenkins build is terminal and failed.
- The job is allowlisted.
- The incident ID matches the current record.
- The classification is exactly `known_demo_fixture_failure`.
- The action is exactly `repair_demo_fixture`.
- Confidence is at least the configured threshold.
- The Jenkins/JUnit or log evidence contains the expected failure signature.
- No action has already run for the incident.
- Automatic remediation is enabled.

Hermes must not provide:

- Shell commands.
- Kubernetes YAML.
- Container images.
- Namespaces.
- PVC names.
- Git commands.
- Retry counts.

These remain hardcoded, tested Ariadne behavior.

## Failure-class policy

| Failure class | Automatic behavior in the 24-hour delivery |
|---|---|
| Known demo fixture failure | Run `hermes-demo-repair`, then rebuild once |
| Stale runtime quality telemetry | Optionally run the existing platform quality suite probe |
| Clearly transient demo Jenkins failure | At most one allowlisted rebuild if time permits |
| Ordinary source/test defect | Diagnose and recommend; human required |
| SonarQube failure | Human required; never waive automatically |
| Semgrep failure | Human required; never waive automatically |
| Trivy/supply-chain failure | Human required; never waive automatically |
| Kubernetes or Flux defect | Read-only diagnosis; human-required repo/Flux change |
| Missing or stale decisive evidence | Human required |
| Authentication, Secret or Vault issue | Human required |
| Storage-destructive or hardware issue | Human required |
| Unknown class or low confidence | Human required |

## Triage metrics and escalation

Ariadne should expose a small stable metric set, for example:

```text
ariadne_hermes_triage_incident{job="...",build="...",status="detected|diagnosed|repairing|resolved|human_required"} 1
ariadne_hermes_triage_action_total{action="...",result="accepted|rejected|success|failed"} 1
ariadne_hermes_triage_last_success_timestamp_seconds ...
ariadne_hermes_triage_duration_seconds ...
```

Avoid unbounded labels such as full error messages, commit messages or model text.

The minimum alert is:

```text
human_required incident remains active for a short demo-safe interval
  -> vmalert fires
  -> Alertmanager handles notification
  -> Grafana links to Jenkins and the Ariadne incident report
```

If dashboard changes threaten the deadline, the metric and alert rule take precedence over a polished panel.

## 24-hour execution schedule

### H+0 to H+1: establish authoritative state

- Confirm the deployed Flux revision and Ariadne image/source revision.
- Confirm the Jenkins job definitions and branch behavior.
- Create clean dedicated worktrees without touching existing dirty worktrees.
- Verify the Hermes Agent API route, request format and authentication.
- Verify Ariadne-to-Hermes network reachability.
- Verify Ariadne-to-OpenSearch reachability.
- Verify the maintenance service account can create/watch the intended isolated Job or identify the minimal RBAC change.
- Freeze the exact demo job, namespace, failure signature and repair action.

Gate at H+1:

- No additional repositories or failure classes enter scope.
- If the Hermes Agent API path is still unknown, allocate no more than one additional hour before activating the fallback.

### H+1 to H+4: bounded OpenSearch evidence

- Add the fixed OpenSearch query client to Ariadne.
- Filter by the demo namespace, incident ID and build time window.
- Add response size, count and timeout limits.
- Add basic sanitization.
- Add `log_evidence` to the diagnosis bundle.
- Add unit tests for success, empty result, timeout, malformed payload, truncation and sanitization.
- Perform one live read-only query.

Compromise at H+4:

- Use a single fixed `kube-*` query rather than a general query builder.
- Retain raw chronological excerpts rather than semantic grouping.
- OpenSearch failure becomes explicit missing evidence and does not block Jenkins diagnosis.

### H+4 to H+7: automatic Hermes invocation

- Configure the shared internal API credential through Vault.
- Implement Ariadne's Hermes Agent client.
- Invoke the explicit Titan triage skill with the incident bundle.
- Require and validate structured JSON.
- Add a bounded timeout and one retry.
- Record request/session ID and duration.
- Deduplicate by incident ID.

Compromise at H+7:

- If the Hermes Agent API remains blocked, call the existing Hermes Ollama endpoint with the triage workflow embedded in the prompt.
- Clearly label this fallback as model-assisted Ariadne triage, not full Hermes Agent skill execution.
- Preserve the same response schema so the rest of the loop remains usable.

### H+7 to H+10: decision and authorization contract

- Freeze the response schema.
- Implement incident, class, confidence and action validation.
- Add idempotency and maximum-action checks.
- Add the global action kill switch.
- Reject arbitrary action content.
- Emit requested, accepted and rejected action metrics.
- Test invalid JSON, mismatched incident, low confidence, unknown action and duplicate incident.

Gate at H+10:

- Response schema and action list freeze.
- No additional automatic action type is added.

### H+10 to H+14: isolated repair loop

- Add the isolated namespace, fixture and test-runner resources.
- Add the dedicated Jenkins demo job or pipeline definition.
- Ensure the test-runner emits the incident ID to pod stdout/stderr.
- Add the predefined repair Job builder/executor to Ariadne.
- Wait for repair completion.
- Trigger one Jenkins rebuild with `SEED_FAILURE=false`.
- Record the rebuilt job and final status.
- Stop after one failed repair or rebuild.

Compromise at H+14:

- If repair execution is not repeatable, disable remediation.
- Ship automatic detection, evidence, Hermes diagnosis and human escalation.
- Do not consume remaining time attempting general repair logic.

### H+14 to H+16: human-required signal

- Add incident/action metrics.
- Add one vmalert rule for unresolved human-required incidents.
- Add Jenkins and Ariadne report links where the current dashboard model supports them.
- Confirm the signal clears on a resolved incident.

Compromise at H+16:

- Use the existing Atlas Testing dashboard and direct VictoriaMetrics query if a new panel is not ready.
- Preserve the metric and alert rule over dashboard polish.

### H+16 to H+18: Flux integration and documentation

- Add only required environment, Vault injection, network and RBAC changes.
- Keep the repair Job namespace and permissions isolated.
- Add the automatic-action kill switch to Flux-managed configuration.
- Rewrite `mermaid/TestAutomation.mmd` to show the actual trigger, evidence collection, Agent invocation, skill routing, decision schema, action gate, Ariadne execution and human escalation.
- Update this plan with final deviations and deferred hardening.
- Do not render SVG output unless separately requested.

### H+18 to H+21: validation and first rehearsal

- Run focused and complete relevant Ariadne tests.
- Run repository quality checks required by its Jenkins pipeline.
- Run `kustomize build` for each affected titan-iac kustomization.
- Run client-side dry-run validation without mutating the cluster.
- Deliver through Git and Flux only.
- Seed the controlled failure and observe the full loop.
- Fix only critical-path failures.

Gate at H+21:

- Feature freeze.
- No autonomous-code stretch work starts after this point.

### H+21 to H+23: repeatability rehearsal

Run these scenarios:

1. Known fixture failure -> repair Job -> green rebuild.
2. Unsupported classification -> human-required signal and no repair.
3. Duplicate observation -> no duplicate repair.
4. OpenSearch timeout -> Jenkins-based diagnosis continues.
5. Hermes timeout or invalid response -> human-required signal.
6. Remediation kill switch disabled -> diagnosis only.

Record timings, incident IDs, action count and final Jenkins status.

### H+23 to H+24: hard stop

- Make no new feature changes.
- Confirm the kill switch and rollback path.
- Capture the final working scope and known limitations.
- Ship the highest completed fallback tier.
- Stop at H+24.

## Deadline compromise ladder

When time is at risk, remove functionality in this order:

1. Drop autonomous Gitea/code editing.
2. Limit remediation to `repair_demo_fixture` only.
3. Drop transient rebuild and optional suite-probe actions.
4. Drop additional failure classifications.
5. Replace dynamic OpenSearch construction with one fixed query.
6. Reuse the existing Grafana dashboard instead of adding a polished panel.
7. Fall back from Hermes Agent API to the same Qwen model with the triage skill procedure embedded in the prompt.
8. Disable automatic remediation but keep automatic diagnosis.
9. Preserve detection, evidence and human-required alerting as the minimum shippable system.

The deadline is not extended to preserve a lower-priority feature.

## Delivery tiers at H+24

### Tier A: target

- Automatic detection.
- Jenkins and OpenSearch evidence.
- Hermes Agent skill execution.
- Structured diagnosis.
- One allowlisted repair Job.
- One rebuild.
- Human escalation for unsupported failures.

### Tier B: acceptable compromise

- Automatic detection.
- Jenkins and OpenSearch evidence.
- Hermes Agent skill execution.
- Structured diagnosis and recommendation.
- Human escalation.
- Automatic repair disabled.

### Tier C: minimum shippable

- Automatic detection.
- Jenkins and bounded OpenSearch evidence.
- Ariadne calls Qwen with the triage workflow embedded.
- Structured recommendation.
- Human-required metric/alert.

No tier may silently claim Hermes Agent execution when only Ollama was invoked.

## Testing strategy

### Ariadne unit tests

- OpenSearch query bounds and field selection.
- Correlation by incident ID and time.
- Sanitization and truncation.
- Empty, unavailable and malformed OpenSearch responses.
- Hermes request construction.
- Hermes response schema validation.
- Confidence threshold.
- Action allowlist.
- Incident mismatch rejection.
- Idempotency and maximum action count.
- Repair Job construction from hardcoded values.
- Jenkins rebuild parameter handling.
- Metrics with bounded label cardinality.

### Integration tests

- Mocked Jenkins failure plus mocked OpenSearch evidence.
- Mocked Hermes accepted action.
- Mocked unsupported classification.
- Repair Job success/failure.
- Jenkins rebuild request success/failure.
- Hermes timeout and malformed response.
- OpenSearch timeout while Jenkins evidence remains usable.

### Live read-only preflight

- Query one known `kube-*` time window.
- Confirm the expected Kubernetes field names.
- Confirm Hermes API status and authentication behavior.
- Confirm Jenkins GET access.
- Confirm VictoriaMetrics can query Ariadne metrics.

### End-to-end demonstration

- Arm the demo failure deliberately.
- Confirm one failed Jenkins build.
- Confirm one Ariadne incident.
- Confirm correlated OpenSearch excerpts.
- Confirm the expected Hermes skill and response.
- Confirm one repair Job.
- Confirm one rebuild.
- Confirm green result and resolved metric.
- Repeat once to prove the path is not a one-off race.

## Flux and validation rules

All persistent cluster changes must be represented in Git and reconciled through Flux. No manual `kubectl apply`, patch or ad hoc cluster edit is part of the delivery.

For every affected titan-iac kustomization:

```text
kustomize build <path>
kubectl apply --server-side --dry-run=client -k <path>
flux diff kustomization <name> --path <path>
```

Actual reconciliation occurs only after the intended commits are available to Flux.

Application changes in Ariadne must run the focused tests for new services plus its relevant full test/quality suite. Exact commands should follow the authoritative branch's Jenkinsfile and test configuration rather than assumptions from the currently dirty local checkout.

## Security and safety boundary

- Store all shared credentials in Vault; never Git.
- Restrict automatic triage to an explicit Jenkins job allowlist.
- Restrict OpenSearch to `kube-*`, time bounds and isolated namespaces.
- Sanitize evidence before model use.
- Never send entire indices, artifacts or unrestricted logs to the model.
- Do not add Secret-read permission to Hermes.
- Do not grant Hermes Kubernetes mutation.
- Do not allow model-supplied shell commands or manifests.
- Keep action definitions in tested Ariadne code.
- Permit one action and one rebuild per incident.
- Require human intervention on low confidence or missing decisive evidence.
- Keep a default-off remediation flag until observe mode passes.

## Rollback and kill switches

Immediate behavioral rollback:

```text
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false
```

Further rollback steps are:

1. Disable automatic Hermes invocation while retaining normal Ariadne schedules.
2. Restore the original Jenkins-weather and testing-triage intervals.
3. Disable the demo Jenkins job.
4. Remove or suspend only the isolated demo resources through Git/Flux.
5. Revoke the shared Hermes API key in Vault.
6. Leave ordinary Jenkins, Fluent Bit, OpenSearch, VictoriaMetrics and Hermes read-only behavior untouched.

The repair action must be safe to run twice even though idempotency should prevent the second run.

## Repository touchpoints

Expected Ariadne application changes:

- Bounded OpenSearch evidence client.
- Diagnosis-bundle enrichment.
- Hermes Agent API client.
- Response schema and validation.
- Incident deduplication/state.
- Allowlisted repair action executor.
- Jenkins rebuild request.
- Triage metrics.
- Unit and integration tests.

Expected titan-iac changes:

- Ariadne environment and Vault injection.
- Hermes shared API-key configuration if required.
- Isolated demo namespace/PVC/RBAC/test-runner resources.
- Predefined repair Job permissions.
- Demo Jenkins JCasC/pipeline definition.
- vmalert rule and minimal Grafana integration.
- `mermaid/TestAutomation.mmd` update.

Do not modify unrelated workloads or fold repository-layout migrations into these changes.

## Optional autonomous-code stretch

This begins only if Tier A passes end to end before H+16 and at least eight hours remain.

The only acceptable stretch is a restricted Gitea PR workflow:

```text
Hermes proposes patch
  -> Ariadne validates repository and base branch
  -> dedicated service creates a demo branch/PR
  -> Jenkins validates the branch
  -> human reviews and merges
```

Restrictions:

- One demo repository.
- Dedicated branch prefix.
- No push to `main` or `master`.
- No automatic merge.
- Separate least-privilege Gitea token in Vault.
- Patch size and file-path limits.
- Jenkins validation required.

This stretch is dropped immediately if it threatens rehearsal or the H+24 stop.

## Required architecture-document update

After behavior is known, `mermaid/TestAutomation.mmd` must show these internal steps rather than compressing Hermes into one box:

```text
Trigger
  -> terminal Jenkins failure or human request

Detection
  -> Ariadne allowlisted polling and incident deduplication

Evidence
  -> Jenkins metadata/artifacts
  -> bounded OpenSearch excerpts
  -> VictoriaMetrics
  -> Kubernetes/Flux/Gitea/Grafana context when relevant

Agent invocation
  -> Ariadne calls Hermes Agent API
  -> explicit triage-titan-test-failures skill
  -> local Qwen model through Hermes Ollama

Decision
  -> first enforced failure
  -> facts versus inference
  -> specialist skill routing
  -> confidence and action schema

Authorization
  -> Ariadne allowlist, incident match, confidence, idempotency and kill switch

Response
  -> predefined repair Job and one rebuild
  -> or human-required metric and alert
```

The chart must visually distinguish current behavior, newly implemented demo behavior and deferred capabilities. It should continue to keep Jenkins-retained logs separate from OpenSearch pod logs and should not imply direct Hermes OpenSearch access.

## Final handoff record

At H+24, append a short result section containing:

- Delivered tier.
- Exact commits and deployed revisions.
- Demo job/build IDs.
- Incident ID.
- Evidence query window.
- Hermes request/session ID.
- Requested and executed action.
- Repair Job name.
- Rebuild result.
- Alert result.
- Total end-to-end duration.
- Compromises activated.
- Deferred hardening.
- Kill-switch and rollback verification.

The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis.

---

# Delivery record

- Execution window opened 2026-08-05 ~17:20 UTC.
- Delivered tier: **A, plus the optional autonomous-code stretch.**
- Diagnosis was performed by the **full Hermes Agent** (`POST /v1/runs`,
  skill `$triage-titan-test-failures`) in every recorded case. The Ollama
  fallback described in the plan was never activated.

## What runs today

| Capability | State |
|---|---|
| Detection on 11 Jenkins jobs (2 demo + 9 homegrown) | live |
| Jenkins + bounded OpenSearch evidence | live |
| Hermes Agent diagnosis, schema-validated | live |
| Action registry: `repair_demo_fixture`, `retry_transient_infra` | live |
| Bounded source-patch proposal → branch → pull request | live (demo repo) |
| Human-required metric, vmalert rule, Alertmanager | live |
| Gitea issue filing for human-required incidents | live, proven on 4 real repos |
| Candidate-file selection for real service repos | live, PR opened on `bstein/ariadne` |
| Narrowed alerting (repair failed, or unattended 6h) | live |

## What the fixture and code demos actually prove

Worth stating plainly before demonstrating either one, because the two demos
are not equally representative of the real-service path.

The **fixture demo** is end-to-end real. Ariadne detects a genuine red build,
Hermes diagnoses it from real evidence, the deterministic gates authorize, and
Ariadne executes and verifies via a rebuild. Nothing about it is staged beyond
the fixture itself being deliberately broken.

The **code demo** is real in its hard parts and staged in one: the repository
is real, the failing tests are real, Hermes genuinely locates the defect and
writes an anchored patch, the validator genuinely checks it, and the pull
request is genuinely opened and awaits review. What is staged is *which file*
Hermes is handed - `ARIADNE_HERMES_CODE_CANDIDATE_PATH=src/discount.py` pins
it. File selection, the step that has to work on a real service, is the one
step the demo does not exercise.

On real repositories there is no pin, and selection has to work from evidence
alone. Two defects blocked that and were fixed on 2026-08-06 (see below):
successful tool output crowding the real failure out of the evidence budget,
and the patcher being unable to reach a source file the console never names.

## Measured results

Closed autonomous loop, twice:

```text
armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14
      -> rebuild green 21:01:46 -> resolved 21:02:10   (4m00s)
second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08
```

After converting the fixture repair from a spawned Kubernetes Job to an
in-process API call (Ariadne 0.1.0-402), the automated leg roughly halved:

```text
build red 02:20:03 -> fixture healthy 02:20:28 (25s)
      -> rebuild triggered 02:20:41 -> green and resolved 02:21:07  (1m04s)
```

Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s).

**Autonomous source repair on a real service repository, proven 2026-08-06.**
A one-line defect was seeded in `ariadne/utils/errors.py` (dropping the
`errorMessage` key from the HTTP payload lookup) and pushed to master. Build
409 went red, and Ariadne opened
[bstein/ariadne PR #3](https://scm.bstein.dev/bstein/ariadne/pulls/3) from
branch `hermes-repair/409` with the correct one-line fix:

```diff
-        raw = payload.get("error") or payload.get("message")
+        raw = payload.get("error") or payload.get("message") or payload.get("errorMessage")
```

The stored `hermes_autotriage_code_proposal` event records how it got there:

```text
candidates:  [ariadne/app.py, tests/test_utils.py, ariadne/services/mailu.py,
              ariadne/utils/errors.py, ariadne/utils/http.py, ariadne/utils/passwords.py]
chosen_path: ariadne/utils/errors.py
validated:   true
```

`ariadne/utils/errors.py` appears nowhere in the build console. It is in that
list only because Ariadne read the failing test and followed its imports. No
candidate path was pinned, unlike the `hermes-code-demo` job.

Four rounds were needed, and the failure reasons are the useful part:

| Build | Outcome | Cause |
|---|---|---|
| 404 | `no_candidate_files` | evidence held no failure at all |
| 406 | never triaged | superseded by a newer build 24 s later |
| 408 | `patch_invalid` | only the wrong file was offered |
| 409 | **PR opened** | all three fixes live together |
Hermes diagnosis latency across all runs: 15-21s.

Real homegrown service diagnoses (no action allowlisted, all escalated):

| Incident | Classification | Confidence |
|---|---|---|
| `bstein-dev-home/445` | `sonarqube_node_runtime_missing` | 0.99 |
| `metis/271` | `jenkins_scm_checkout_timeout` | 0.99 |
| `lesavka/582` | `coverage_quality_gate_failure` | 0.94 |
| `soteria/276` | `unknown` | 0.58 (below threshold) |

## Safety behaviour observed in production, not only in tests

- Hermes classified a pipeline-infrastructure failure as the demo fixture
  signature at 0.91 confidence; Ariadne's deterministic evidence gate
  refused the action (`evidence_signature_missing`). The model was wrong and
  the system still did the right thing.
- With the action allowlist emptied, a genuine fixture failure produced
  `human_required` and `HermesTriageHumanRequired` reached FIRING in vmalert
  and active in Alertmanager; after an operator fixed the fixture and the
  job went green, the alert self-cleared through supersession.
- `soteria` returned `unknown` at 0.58 — below the 0.85 threshold and with
  no matching action, so two independent gates blocked it.
- Two defects were found by deliberately seeding a one-line break in a real
  repository (`ariadne`) rather than by reasoning about the code. First, the
  console evidence budget was filled entirely by *successful* tool output:
  build 404 retained six regions spanning lines 259-476 - Trivy setup, Sonar
  banners, and three lines whose only failure signal was the word `coverage` -
  and dropped the pytest failure at line 1487. Markers are now ranked by
  whether they assert a failure or merely name a tool that ran. Second, the
  defective file `ariadne/utils/errors.py` appeared *zero times* in the entire
  174 KB console, because a pytest assertion that fails inside a test names
  only the test; candidate selection returned the wrong file. Ariadne now
  reads the failing test and follows its imports back to the module under
  test, widening what may be read without widening what may be written - the
  patch validator still gates writes on `allowed_path_prefixes`, so a failing
  test can be read for context and never patched, which also forecloses the
  classic bad fix of silencing a test instead of repairing the code.
- `bstein-dev-home/446` was classified `known_demo_fixture_failure` at 0.99,
  even though build 445 of the same job had been classified correctly as
  `sonarqube_node_runtime_missing`. The evidence gate refused it again, so
  nothing ran, but the intermittency is the point: the same evidence produced
  a demo-only classification on one run and the right one on another. The
  triage prompt now interpolates the job under analysis and forbids the demo
  classification and `repair_demo_fixture` on any job other than
  `hermes-triage-demo` (Ariadne 0.1.0-403), removing the judgement call
  rather than asking the model to make it correctly every time.

## Deviations from the plan

- **Vault**: the shared Hermes API key and the Gitea automation token are
  manually created Kubernetes Secrets, not Vault-sourced. `robotuser`
  returned 403 on userpass/ldap/oidc and no token was available. The
  manifests reference the Secrets by name, so the swap is config-only.
- **Fixture storage**: the demo fixture is a ConfigMap, not a PVC. Longhorn
  attach latency and per-node engine availability made a PVC unreliable for
  a fast repeatable demo.
- **JUnit evidence**: RESOLVED 2026-08-06. `junit 1369.v15da_00283f06` and
  `pipeline-stage-view 2.41` are installed and active, with `scm-api` bumped
  724 -> 728 as their `workflow-cps` requires. This was never actually blocked
  on a core upgrade: only the *latest* `junit` (1418) needs core 2.533, while
  1369 needs 2.504.3 and runs on 2.528.3. The earlier record was wrong because
  it read the plugin's latest release rather than the
  `dynamic-stable-2.528.3` update-center channel.

  `testReport` and `wfapi` now return 200 where they returned 404 all night -
  `ariadne` build 414 reports 1183 passing tests through the API. No Ariadne
  change was needed: the evidence collector already fetched
  `testReport/api/json` and had simply always received a 404, so
  `failed_tests` now populates on its own.

  This removes the *cause* of both diagnosis defects fixed the same day. With
  no structured test data, console text was the only evidence, which is why
  the enforced failure could be crowded out of the byte budget and why the
  patcher could not locate a source file the console never named.

  Cost: installing them required restarting the controller, which rescheduled
  it onto a node without the image cached and produced a ~15 minute outage on
  the image pull alone, plus three killed builds. Pre-pull the controller
  image on the worker nodes before any future Jenkins restart; the agent image
  had already been pre-pulled for exactly this reason and the lesson was not
  carried across.
- **Multibranch jobs are not detected**: folder-style jobs expose a null
  `lastBuild` at the job root.
- **OpenSearch namespace** is still pinned to the demo namespace plus
  `jenkins`; real services get their Jenkins agent logs but not their own
  runtime logs.

## Kill switches and rollback

```text
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false   # diagnose only
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false        # fully off
ARIADNE_HERMES_ISSUES_ENABLED=false            # stop filing issues
ARIADNE_HERMES_CODE_ENABLED=false              # stop proposing patches
ARIADNE_HERMES_ALLOWED_ACTIONS=""              # accept no action at all
```

All are Flux-managed in `services/maintenance/apps/ariadne-deployment.yaml`.
The automatic loop can only ever create Jobs named `hermes-demo-repair-*` in
`hermes-triage-demo`, rebuild allowlisted Jenkins jobs, file issues in
explicitly mapped repositories, and push `hermes-repair/*` branches. One
action per incident, ever.

## Verified rollback and safety properties

- Emptying the action allowlist was exercised live and produced diagnosis
  without mutation.
- Restoring it was exercised live and the loop resumed.
- Hermes holds no Git credentials, no Kubernetes write access, and cannot
  read Secrets; its Flux-managed approval deny-list still blocks `kubectl`
  mutations, `flux` mutations and secret reads.
