titan-iac/knowledge/hermes-automated-triage-24h-plan.md

Hermes Automated Triage: 24-Hour Delivery Plan

• Status: planning only
• Created: 2026-08-05
• Hard delivery limit: 24 elapsed hours from execution authorization
• Priority: deadline first, then the most complete safe automation that fits

Executive decision

The first delivery will automate one narrow, real, repeatable failure-to-repair path. It will not attempt general autonomous coding before that path works end to end.

The target demonstration is:

Controlled test failure
  -> Jenkins build fails
  -> Ariadne detects the terminal failure automatically
  -> Ariadne gathers Jenkins evidence and bounded OpenSearch logs
  -> Ariadne automatically invokes the Hermes Agent
  -> Hermes selects its Titan test-triage skills
  -> Hermes returns a structured diagnosis and requested action
  -> Ariadne validates the request against an exact allowlist
  -> Ariadne creates one predefined repair Job
  -> Ariadne triggers one Jenkins rebuild
  -> the rebuild passes and the incident resolves

A second path will demonstrate safe escalation:

Unsupported, ambiguous or unsafe failure
  -> Hermes returns human_required
  -> Ariadne publishes a triage metric
  -> VictoriaMetrics and vmalert evaluate it
  -> Grafana and Alertmanager provide the human-facing signal

At H+24 work stops. Lower-priority functionality is removed or deferred rather than extending the deadline.

Definition of done

The delivery is fully successful when all of the following are demonstrated:

• A deliberately seeded test failure causes a Jenkins build to fail.
• Ariadne detects that exact terminal build once, preferably within two minutes.
• Ariadne assigns a stable incident ID based on the Jenkins job and build number.
• The diagnosis bundle includes the first failed stage, retained Jenkins evidence and provenance-bearing OpenSearch excerpts.
• Ariadne invokes the Hermes Agent without a human prompt.
• The invocation explicitly selects the Titan test-triage skill.
• Hermes returns schema-valid facts, inferences, classification, confidence and action request.
• Ariadne accepts only the predefined demo action and executes it once.
• A predefined repair Job corrects only the isolated demo fixture.
• Ariadne triggers one rebuild with failure seeding disabled.
• The rebuilt job passes and Ariadne records the incident as resolved.
• An unsupported classification produces a human-required metric and no mutation.
• No secret values are stored in evidence, model prompts, responses or logs.
• No path can automatically write to main, reconcile Flux, or execute model-supplied commands.
• A single Flux-managed setting can disable all automatic actions immediately.

Target demo timings are:

• Failure detection: less than two minutes.
• Evidence collection and Hermes response: less than three minutes.
• Repair launch and rebuild request: less than two minutes.
• Complete failure-to-green loop: less than ten minutes, excluding image pulls during a cold cluster start.

Current system context

This section records the facts needed to execute the plan without relying on prior conversation.

Jenkins and the homegrown suites

Jenkins runs the homegrown repositories on ephemeral Kubernetes agent pods. The active quality scope includes Ariadne, Metis, Ananke, Atlasbot, Pegasus, Soteria, titan-iac, bstein-dev-home, data-prepper and Lesavka, with additional build/test jobs for projects such as Arcanagon, Typhon and Veles.

The normalized quality evidence includes, as applicable:

build
  -> style/docs
  -> LOC/naming
  -> coverage
  -> tests
  -> gate glue
  -> SonarQube
  -> Semgrep
  -> Trivy/supply-chain evidence
  -> gate enforcement
  -> image build and Harbor push

Jenkins retains build metadata, console output, stage results, JUnit, coverage and quality reports. These retained artifacts are the authoritative source for the exact failed test or gate.

Logging and OpenSearch

OpenSearch does not collect logs by itself. Fluent Bit tails Kubernetes container logs and writes them directly into kube-* indices. It also writes node journal records into journald-* indices.

The relevant path is:

Kubernetes pod stdout/stderr
  -> Fluent Bit DaemonSet
  -> OpenSearch kube-* indices

Jenkins-retained console logs and OpenSearch pod logs are separate evidence paths. Jenkins agent command output must not be assumed to exist in CRI container logs. The demo must use Jenkins artifacts for the exact assertion and must ensure that a dedicated demo test-runner or related workload writes a correlated incident message to pod stdout/stderr for OpenSearch.

Data Prepper is not part of the ordinary log path. It receives OpenTelemetry traces, transforms them and writes trace/service-map indices into OpenSearch. It is irrelevant to the Jenkins log integration except as another separately tested workload.

Relevant IaC sources include:

• services/logging/fluent-bit-helmrelease.yaml
• services/logging/opensearch-observability-objects.yaml
• services/logging/Jenkinsfile.data-prepper

Metrics and human-facing status

Jenkins publishers and runtime probes send normalized quality metrics to platform-quality-gateway. VictoriaMetrics scrapes Prometheus-compatible endpoints, vmalert evaluates rules, Grafana displays the Atlas Testing dashboards, and Alertmanager owns notification delivery.

The intended human-escalation path is therefore:

Ariadne triage metric
  -> VictoriaMetrics
  -> vmalert
  -> Alertmanager notification and Grafana status

Hermes does not send an alert "to Grafana." Grafana visualizes the metric and alert state.

Relevant IaC sources include:

• services/monitoring/platform-quality-gateway-deployment.yaml
• services/monitoring/platform-quality-suite-probe-cronjob.yaml
• services/monitoring/vmalert-atlas-availability.yaml
• services/monitoring/dashboards/atlas-testing.json
• services/quality/sonarqube-exporter-deployment.yaml

Ariadne today

Ariadne is the deterministic automation and evidence-collection service in the maintenance namespace. Its current deployment has access to:

• Jenkins API credentials and JENKINS_BASE_URL.
• VictoriaMetrics through ARIADNE_VM_URL.
• OpenSearch through OPENSEARCH_URL.
• Kubernetes through its service account.
• The local model through ARIADNE_TESTING_TRIAGE_MODEL_URL.

The deployment schedules Jenkins build-weather collection every ten minutes and testing triage every fifteen minutes. Those intervals are too slow for a smooth demo and should be reduced only for the allowlisted demo path.

Ariadne's current OpenSearch integration manages index retention. It does not currently retrieve historical log content for testing diagnosis.

Ariadne currently points model-assisted testing triage at the Hermes Ollama service. Calling Ollama uses the same Qwen model but bypasses the Hermes Agent, its tool loop, skills, memory and approval model. This is not equivalent to invoking Hermes.

Ariadne already has patterns for launching a platform quality probe Job and for performing Jenkins workspace maintenance. These patterns should be reused for the predefined demo repair rather than introducing a general job-execution system.

Relevant IaC source:

• services/maintenance/apps/ariadne-deployment.yaml

Relevant application repository:

• Gitea: bstein/Ariadne
• Local checkout currently at /home/brad/Development/Ariadne

Hermes today

Hermes runs as a single read-only triage agent in the hermes namespace. Its current configuration provides:

• Hermes Agent API on port 8642.
• Hermes dashboard on port 9119.
• Local Qwen 2.5 7b-instruct-q4_0 inference through Hermes Ollama.
• Ariadne, Jenkins, VictoriaMetrics, Gitea and Grafana base URLs.
• kubectl installed as a read-only cluster inspection tool.
• A persistent home PVC containing memories and runtime-installed skills.

Hermes does not currently receive OPENSEARCH_URL. In the desired design it does not need direct OpenSearch access: Ariadne will provide bounded, sanitized excerpts in the diagnosis bundle.

Hermes's Flux-managed instructions say to start with Ariadne as the source of truth, then use Jenkins, quality metrics, Flux, Grafana and Kubernetes read-only evidence. Its RBAC allows reads of workloads, events, pod logs, services, storage objects and Flux resources. It cannot read Kubernetes Secret values.

Hermes explicitly denies or lacks permission for actions such as:

• kubectl apply, delete, patch, scale, cordon, drain or rollout restart.
• Flux suspend, resume or reconcile.
• Vault secret reads.
• Kubernetes Secret reads.
• Jenkins rebuilds.
• Git writes.

This boundary remains in place. Hermes will request actions; Ariadne will validate and execute exact predefined actions.

Relevant IaC sources include:

• services/hermes/deployment.yaml
• services/hermes/configmap.yaml
• services/hermes/rbac.yaml

Hermes's current Atlas skills

A read-only inventory of the live Hermes PVC on 2026-08-05 showed the Atlas triage entry skills:

• triage-titan-test-failures
• triage-atlas-service-health
• tune-atlas-alerts
• master-hermes-on-atlas

The test-triage skill pack includes:

• titan-triage-orchestrator
• jenkins-retained-evidence
• platform-quality-metrics
• kubernetes-readonly-failure-classifier
• flux-git-change-correlation
• grafana-metric-provenance
• triage-evidence-reporting
• approved-triage-actions
• soteria-backup-health

The orchestrator already defines the needed evidence order: Ariadne first, then the smallest relevant specialist, followed by a fact-versus-inference report. The demo invocation should explicitly request $triage-titan-test-failures; a new general triage framework is unnecessary.

These skills currently live on the runtime PVC and are not all declared in titan-iac. The demo should not depend on mutating the live PVC manually. Any essential prompt or output contract must be supplied by Ariadne or added through Flux-managed Hermes configuration.

Current gaps

The desired closed loop does not exist today because:

• Ariadne does not query OpenSearch for triage log evidence.
• Ariadne calls Ollama rather than the Hermes Agent API.
• There is no shared Ariadne-to-Hermes API authentication contract.
• There is no schema for a Hermes triage response.
• There is no allowlisted Hermes-to-Ariadne action request.
• Ariadne does not automatically execute a Hermes-selected repair.
• There is no incident-level idempotency preventing duplicate repair attempts.
• There is no dedicated human-required triage metric and alert.
• The current architecture chart compresses Hermes's internal trigger, routing and decision steps into a single box.

Worktree and source-control safety

The existing working directories must not be used as clean implementation bases.

As of 2026-08-05:

• /home/brad/Development/titan-iac has extensive unrelated modified, deleted and untracked user work, including an ongoing repository-layout migration and untracked Mermaid files.
• /home/brad/Development/Ariadne is dirty on branch codex/ariadne-metrics-hotfix.

Execution must:

1. Inspect the actual revisions deployed by Flux and Jenkins.
2. Fetch without altering either existing worktree.
3. Create dedicated clean Git worktrees for the demo branches.
4. Never reset, clean, stash, rebase or overwrite either existing dirty worktree.
5. Keep Ariadne application changes and titan-iac/Flux changes in separately reviewable commits.
6. Bring over only the intentional planning/diagram change when appropriate.

Suggested isolated paths are:

/home/brad/Development/_worktrees/ariadne-hermes-triage-demo
/home/brad/Development/_worktrees/titan-iac-hermes-triage-demo

The implementation branch point must be the current authoritative primary/deployed revision, not an assumed local branch.

Deadline-driven scope

Required scope

• One dedicated demo Jenkins job.
• One deterministic failure signature.
• One isolated demo fixture.
• One bounded OpenSearch search contract.
• One automatic Hermes invocation.
• One structured decision contract.
• One exact repair action.
• One rebuild attempt.
• One human-required metric and alert path.
• One complete source-level Mermaid update.

Explicit non-goals

• General conversational OpenSearch access.
• Arbitrary index selection or unbounded log retrieval.
• General autonomous repository editing.
• Direct commits or merges to main.
• Model-generated Kubernetes manifests or commands.
• Automatic Flux reconciliation.
• Automatic SonarQube, Semgrep or Trivy waivers.
• Secret inspection.
• Multiple repair retries.
• Multiple repositories or failure signatures before the primary loop is green.
• A permanent generalized incident-management platform.

Demo design

Isolated test fixture

Use a dedicated hermes-triage-demo namespace, Jenkins job and fixture. The preferred deterministic design is:

1. A small PVC stores a single state value such as healthy or unhealthy.
2. The demo is explicitly armed through a Jenkins parameter such as SEED_FAILURE=true.
3. A dedicated Kubernetes test-runner Job reads the fixture.
4. When the fixture is unhealthy, it writes a structured incident line to stdout/stderr and exits nonzero.
5. Jenkins waits for the Job, records the test failure and archives a small JUnit result.
6. Fluent Bit forwards the test-runner Job output to OpenSearch.
7. The test-runner Job name and message include the Jenkins job/build incident ID.

Example structured log:

{
  "event": "hermes_demo_test_failure",
  "incident_id": "hermes-triage-demo/42",
  "classification_hint": "demo_fixture_unhealthy",
  "message": "expected fixture state healthy; found unhealthy"
}

The Jenkins rebuild must set SEED_FAILURE=false, otherwise every rebuild would deliberately recreate the failure.

The namespace isolates the test-runner permissions and prevents a demo repair from touching production workloads.

Predefined repair Job

Ariadne will contain one exact action mapping:

repair_demo_fixture
  -> create hermes-demo-repair-<incident> Job
  -> mount only the demo fixture PVC
  -> write healthy state
  -> emit a structured repair result
  -> exit

Hermes cannot supply the image, command, namespace, PVC, labels or arbitrary parameters. It can request only the action ID.

After the Job succeeds, Ariadne requests one Jenkins rebuild with failure seeding disabled.

Why a repair Job is the critical path

A source-code test failure generally requires a Git change, but safe autonomous code editing requires repository credentials, checkout isolation, patch generation, tests, branch creation and PR lifecycle handling. Making that the primary demo would threaten the 24-hour limit.

The fixture repair demonstrates a real closed loop:

• Test failure is real.
• Jenkins and OpenSearch evidence are real.
• Hermes classification is real.
• Hermes action selection is real.
• Ariadne authorization and execution are real.
• The repair and green rebuild are real.

Autonomous Gitea PR creation remains a stretch goal that begins only after the required loop is stable.

Evidence contract

Incident identity

Use a stable ID:

<jenkins-job>/<build-number>

Every log line, bundle, Hermes request, action, repair Job and metric must carry that ID.

Jenkins evidence

The bundle should include:

• Job and build number.
• Branch and commit when available.
• Build URL.
• Start and end timestamps.
• Terminal result.
• First failed stage.
• Console tail.
• Failed JUnit cases.
• Selected quality reports when relevant.
• Evidence source and retrieval timestamp.

OpenSearch evidence

The demo query should be intentionally fixed and bounded:

• Indices: kube-* only.
• Time range: build start minus five minutes through build end plus five minutes.
• Namespace: hermes-triage-demo, plus jenkins only if it adds useful context.
• Correlation: exact incident ID first; namespace and time fallback second.
• Size: at most 50 records initially, never more than 100.
• Fields: timestamp, namespace, pod, container, log/message and selected safe labels.
• Timeout: approximately five seconds.
• Response byte cap: small enough to prevent prompt flooding.
• Pagination: none in the demo.

Each excerpt must retain:

• Index or source identifier.
• Timestamp.
• Namespace.
• Pod.
• Container.
• Sanitized message.

Basic sanitization must remove or mask:

• Authorization headers.
• Bearer tokens.
• Cookies and session IDs.
• Common password, token and secret assignments.
• Private key blocks.

This is demo-grade sanitization, not a claim of comprehensive data-loss prevention. Queries remain restricted to the isolated namespace wherever possible.

Ariadne diagnosis bundle

Add a log_evidence section without replacing the existing bundle fields:

{
  "incident_id": "hermes-triage-demo/42",
  "generated_at": "...",
  "jenkins": {},
  "quality_metrics": {},
  "runtime_context": {},
  "log_evidence": {
    "query_window": {},
    "records": [],
    "truncated": false,
    "error": null
  }
}

Empty or unavailable OpenSearch results must be represented explicitly. They must not block Jenkins-based diagnosis.

Automatic Hermes invocation

Required behavior

Ariadne must call the Hermes Agent API, not only the Ollama completion endpoint. The invocation prompt should be stable and small:

Use $triage-titan-test-failures.
Analyze incident <incident-id>.
Treat the attached Ariadne bundle as the source of truth.
Identify the first enforced failure.
Distinguish facts from inference.
Return only the required triage response schema.
Do not perform mutations.

The bundle is attached as structured content. Ariadne records the Hermes request ID or session ID for audit and idempotency.

Authentication

No API key may be committed to Git.

The preferred path is:

1. Generate one internal Hermes API key.
2. Store it in an approved Vault path.
3. Inject it into Hermes as API_SERVER_KEY.
4. Inject it into Ariadne as HERMES_API_KEY.
5. Restrict network access to the required namespaces where practical.

The exact Hermes Agent API path and authentication behavior must be verified in the first hour. The current Hermes init container generates a key in its persistent .env if none exists; the integration must not scrape that file or copy it into Git.

Response schema

Hermes must return a bounded machine-readable response:

{
  "incident_id": "hermes-triage-demo/42",
  "classification": "known_demo_fixture_failure",
  "confidence": 0.95,
  "facts": [
    {
      "statement": "...",
      "source": "jenkins|opensearch|victoriametrics|kubernetes|flux|gitea",
      "reference": "..."
    }
  ],
  "inferences": [],
  "first_failed_gate": "tests",
  "requested_action": {
    "type": "run_ariadne_job",
    "id": "repair_demo_fixture"
  },
  "human_required": false,
  "reason": ""
}

Invalid JSON, unknown fields, mismatched incident IDs, low confidence and unknown actions all become human-required results.

Detection and deduplication

For the demo, Ariadne should detect only an allowlisted job such as hermes-triage-demo. Reducing the global scheduler interval must not cause automatic action against unrelated failures.

Suggested configuration:

ARIADNE_HERMES_AUTOTRIAGE_ENABLED=true
ARIADNE_HERMES_AUTOTRIAGE_JOB_ALLOWLIST=hermes-triage-demo
ARIADNE_HERMES_AUTOTRIAGE_INTERVAL_SECONDS=60
ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false
ARIADNE_HERMES_ALLOWED_ACTIONS=repair_demo_fixture
ARIADNE_HERMES_MIN_CONFIDENCE=0.85
ARIADNE_HERMES_MAX_ACTIONS_PER_INCIDENT=1

The remediation flag remains false until the evidence-and-Hermes path passes in observe mode.

Ariadne must persist or otherwise reliably track:

• Incident observed.
• Hermes invocation started/completed.
• Action requested/accepted/rejected.
• Repair Job created/completed.
• Rebuild requested.
• Final build result.

The same Jenkins build must never invoke a second repair after its action record exists.

Action authorization

Ariadne may accept the demo action only when:

• The Jenkins build is terminal and failed.
• The job is allowlisted.
• The incident ID matches the current record.
• The classification is exactly known_demo_fixture_failure.
• The action is exactly repair_demo_fixture.
• Confidence is at least the configured threshold.
• The Jenkins/JUnit or log evidence contains the expected failure signature.
• No action has already run for the incident.
• Automatic remediation is enabled.

Hermes must not provide:

• Shell commands.
• Kubernetes YAML.
• Container images.
• Namespaces.
• PVC names.
• Git commands.
• Retry counts.

These remain hardcoded, tested Ariadne behavior.

Failure-class policy

Failure class	Automatic behavior in the 24-hour delivery
Known demo fixture failure	Run hermes-demo-repair, then rebuild once
Stale runtime quality telemetry	Optionally run the existing platform quality suite probe
Clearly transient demo Jenkins failure	At most one allowlisted rebuild if time permits
Ordinary source/test defect	Diagnose and recommend; human required
SonarQube failure	Human required; never waive automatically
Semgrep failure	Human required; never waive automatically
Trivy/supply-chain failure	Human required; never waive automatically
Kubernetes or Flux defect	Read-only diagnosis; human-required repo/Flux change
Missing or stale decisive evidence	Human required
Authentication, Secret or Vault issue	Human required
Storage-destructive or hardware issue	Human required
Unknown class or low confidence	Human required

Triage metrics and escalation

Ariadne should expose a small stable metric set, for example:

ariadne_hermes_triage_incident{job="...",build="...",status="detected|diagnosed|repairing|resolved|human_required"} 1
ariadne_hermes_triage_action_total{action="...",result="accepted|rejected|success|failed"} 1
ariadne_hermes_triage_last_success_timestamp_seconds ...
ariadne_hermes_triage_duration_seconds ...

Avoid unbounded labels such as full error messages, commit messages or model text.

The minimum alert is:

human_required incident remains active for a short demo-safe interval
  -> vmalert fires
  -> Alertmanager handles notification
  -> Grafana links to Jenkins and the Ariadne incident report

If dashboard changes threaten the deadline, the metric and alert rule take precedence over a polished panel.

24-hour execution schedule

H+0 to H+1: establish authoritative state

• Confirm the deployed Flux revision and Ariadne image/source revision.
• Confirm the Jenkins job definitions and branch behavior.
• Create clean dedicated worktrees without touching existing dirty worktrees.
• Verify the Hermes Agent API route, request format and authentication.
• Verify Ariadne-to-Hermes network reachability.
• Verify Ariadne-to-OpenSearch reachability.
• Verify the maintenance service account can create/watch the intended isolated Job or identify the minimal RBAC change.
• Freeze the exact demo job, namespace, failure signature and repair action.

Gate at H+1:

• No additional repositories or failure classes enter scope.
• If the Hermes Agent API path is still unknown, allocate no more than one additional hour before activating the fallback.

H+1 to H+4: bounded OpenSearch evidence

• Add the fixed OpenSearch query client to Ariadne.
• Filter by the demo namespace, incident ID and build time window.
• Add response size, count and timeout limits.
• Add basic sanitization.
• Add log_evidence to the diagnosis bundle.
• Add unit tests for success, empty result, timeout, malformed payload, truncation and sanitization.
• Perform one live read-only query.

Compromise at H+4:

• Use a single fixed kube-* query rather than a general query builder.
• Retain raw chronological excerpts rather than semantic grouping.
• OpenSearch failure becomes explicit missing evidence and does not block Jenkins diagnosis.

H+4 to H+7: automatic Hermes invocation

• Configure the shared internal API credential through Vault.
• Implement Ariadne's Hermes Agent client.
• Invoke the explicit Titan triage skill with the incident bundle.
• Require and validate structured JSON.
• Add a bounded timeout and one retry.
• Record request/session ID and duration.
• Deduplicate by incident ID.

Compromise at H+7:

• If the Hermes Agent API remains blocked, call the existing Hermes Ollama endpoint with the triage workflow embedded in the prompt.
• Clearly label this fallback as model-assisted Ariadne triage, not full Hermes Agent skill execution.
• Preserve the same response schema so the rest of the loop remains usable.

H+7 to H+10: decision and authorization contract

• Freeze the response schema.
• Implement incident, class, confidence and action validation.
• Add idempotency and maximum-action checks.
• Add the global action kill switch.
• Reject arbitrary action content.
• Emit requested, accepted and rejected action metrics.
• Test invalid JSON, mismatched incident, low confidence, unknown action and duplicate incident.

Gate at H+10:

• Response schema and action list freeze.
• No additional automatic action type is added.

H+10 to H+14: isolated repair loop

• Add the isolated namespace, fixture and test-runner resources.
• Add the dedicated Jenkins demo job or pipeline definition.
• Ensure the test-runner emits the incident ID to pod stdout/stderr.
• Add the predefined repair Job builder/executor to Ariadne.
• Wait for repair completion.
• Trigger one Jenkins rebuild with SEED_FAILURE=false.
• Record the rebuilt job and final status.
• Stop after one failed repair or rebuild.

Compromise at H+14:

• If repair execution is not repeatable, disable remediation.
• Ship automatic detection, evidence, Hermes diagnosis and human escalation.
• Do not consume remaining time attempting general repair logic.

H+14 to H+16: human-required signal

• Add incident/action metrics.
• Add one vmalert rule for unresolved human-required incidents.
• Add Jenkins and Ariadne report links where the current dashboard model supports them.
• Confirm the signal clears on a resolved incident.

Compromise at H+16:

• Use the existing Atlas Testing dashboard and direct VictoriaMetrics query if a new panel is not ready.
• Preserve the metric and alert rule over dashboard polish.

H+16 to H+18: Flux integration and documentation

• Add only required environment, Vault injection, network and RBAC changes.
• Keep the repair Job namespace and permissions isolated.
• Add the automatic-action kill switch to Flux-managed configuration.
• Rewrite mermaid/TestAutomation.mmd to show the actual trigger, evidence collection, Agent invocation, skill routing, decision schema, action gate, Ariadne execution and human escalation.
• Update this plan with final deviations and deferred hardening.
• Do not render SVG output unless separately requested.

H+18 to H+21: validation and first rehearsal

• Run focused and complete relevant Ariadne tests.
• Run repository quality checks required by its Jenkins pipeline.
• Run kustomize build for each affected titan-iac kustomization.
• Run client-side dry-run validation without mutating the cluster.
• Deliver through Git and Flux only.
• Seed the controlled failure and observe the full loop.
• Fix only critical-path failures.

Gate at H+21:

• Feature freeze.
• No autonomous-code stretch work starts after this point.

H+21 to H+23: repeatability rehearsal

Run these scenarios:

1. Known fixture failure -> repair Job -> green rebuild.
2. Unsupported classification -> human-required signal and no repair.
3. Duplicate observation -> no duplicate repair.
4. OpenSearch timeout -> Jenkins-based diagnosis continues.
5. Hermes timeout or invalid response -> human-required signal.
6. Remediation kill switch disabled -> diagnosis only.

Record timings, incident IDs, action count and final Jenkins status.

H+23 to H+24: hard stop

• Make no new feature changes.
• Confirm the kill switch and rollback path.
• Capture the final working scope and known limitations.
• Ship the highest completed fallback tier.
• Stop at H+24.

Deadline compromise ladder

When time is at risk, remove functionality in this order:

1. Drop autonomous Gitea/code editing.
2. Limit remediation to repair_demo_fixture only.
3. Drop transient rebuild and optional suite-probe actions.
4. Drop additional failure classifications.
5. Replace dynamic OpenSearch construction with one fixed query.
6. Reuse the existing Grafana dashboard instead of adding a polished panel.
7. Fall back from Hermes Agent API to the same Qwen model with the triage skill procedure embedded in the prompt.
8. Disable automatic remediation but keep automatic diagnosis.
9. Preserve detection, evidence and human-required alerting as the minimum shippable system.

The deadline is not extended to preserve a lower-priority feature.

Delivery tiers at H+24

Tier A: target

• Automatic detection.
• Jenkins and OpenSearch evidence.
• Hermes Agent skill execution.
• Structured diagnosis.
• One allowlisted repair Job.
• One rebuild.
• Human escalation for unsupported failures.

Tier B: acceptable compromise

• Automatic detection.
• Jenkins and OpenSearch evidence.
• Hermes Agent skill execution.
• Structured diagnosis and recommendation.
• Human escalation.
• Automatic repair disabled.

Tier C: minimum shippable

• Automatic detection.
• Jenkins and bounded OpenSearch evidence.
• Ariadne calls Qwen with the triage workflow embedded.
• Structured recommendation.
• Human-required metric/alert.

No tier may silently claim Hermes Agent execution when only Ollama was invoked.

Testing strategy

Ariadne unit tests

• OpenSearch query bounds and field selection.
• Correlation by incident ID and time.
• Sanitization and truncation.
• Empty, unavailable and malformed OpenSearch responses.
• Hermes request construction.
• Hermes response schema validation.
• Confidence threshold.
• Action allowlist.
• Incident mismatch rejection.
• Idempotency and maximum action count.
• Repair Job construction from hardcoded values.
• Jenkins rebuild parameter handling.
• Metrics with bounded label cardinality.

Integration tests

• Mocked Jenkins failure plus mocked OpenSearch evidence.
• Mocked Hermes accepted action.
• Mocked unsupported classification.
• Repair Job success/failure.
• Jenkins rebuild request success/failure.
• Hermes timeout and malformed response.
• OpenSearch timeout while Jenkins evidence remains usable.

Live read-only preflight

• Query one known kube-* time window.
• Confirm the expected Kubernetes field names.
• Confirm Hermes API status and authentication behavior.
• Confirm Jenkins GET access.
• Confirm VictoriaMetrics can query Ariadne metrics.

End-to-end demonstration

• Arm the demo failure deliberately.
• Confirm one failed Jenkins build.
• Confirm one Ariadne incident.
• Confirm correlated OpenSearch excerpts.
• Confirm the expected Hermes skill and response.
• Confirm one repair Job.
• Confirm one rebuild.
• Confirm green result and resolved metric.
• Repeat once to prove the path is not a one-off race.

Flux and validation rules

All persistent cluster changes must be represented in Git and reconciled through Flux. No manual kubectl apply, patch or ad hoc cluster edit is part of the delivery.

For every affected titan-iac kustomization:

kustomize build <path>
kubectl apply --server-side --dry-run=client -k <path>
flux diff kustomization <name> --path <path>

Actual reconciliation occurs only after the intended commits are available to Flux.

Application changes in Ariadne must run the focused tests for new services plus its relevant full test/quality suite. Exact commands should follow the authoritative branch's Jenkinsfile and test configuration rather than assumptions from the currently dirty local checkout.

Security and safety boundary

• Store all shared credentials in Vault; never Git.
• Restrict automatic triage to an explicit Jenkins job allowlist.
• Restrict OpenSearch to kube-*, time bounds and isolated namespaces.
• Sanitize evidence before model use.
• Never send entire indices, artifacts or unrestricted logs to the model.
• Do not add Secret-read permission to Hermes.
• Do not grant Hermes Kubernetes mutation.
• Do not allow model-supplied shell commands or manifests.
• Keep action definitions in tested Ariadne code.
• Permit one action and one rebuild per incident.
• Require human intervention on low confidence or missing decisive evidence.
• Keep a default-off remediation flag until observe mode passes.

Rollback and kill switches

Immediate behavioral rollback:

ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false

Further rollback steps are:

1. Disable automatic Hermes invocation while retaining normal Ariadne schedules.
2. Restore the original Jenkins-weather and testing-triage intervals.
3. Disable the demo Jenkins job.
4. Remove or suspend only the isolated demo resources through Git/Flux.
5. Revoke the shared Hermes API key in Vault.
6. Leave ordinary Jenkins, Fluent Bit, OpenSearch, VictoriaMetrics and Hermes read-only behavior untouched.

The repair action must be safe to run twice even though idempotency should prevent the second run.

Repository touchpoints

Expected Ariadne application changes:

• Bounded OpenSearch evidence client.
• Diagnosis-bundle enrichment.
• Hermes Agent API client.
• Response schema and validation.
• Incident deduplication/state.
• Allowlisted repair action executor.
• Jenkins rebuild request.
• Triage metrics.
• Unit and integration tests.

Expected titan-iac changes:

• Ariadne environment and Vault injection.
• Hermes shared API-key configuration if required.
• Isolated demo namespace/PVC/RBAC/test-runner resources.
• Predefined repair Job permissions.
• Demo Jenkins JCasC/pipeline definition.
• vmalert rule and minimal Grafana integration.
• mermaid/TestAutomation.mmd update.

Do not modify unrelated workloads or fold repository-layout migrations into these changes.

Optional autonomous-code stretch

This begins only if Tier A passes end to end before H+16 and at least eight hours remain.

The only acceptable stretch is a restricted Gitea PR workflow:

Hermes proposes patch
  -> Ariadne validates repository and base branch
  -> dedicated service creates a demo branch/PR
  -> Jenkins validates the branch
  -> human reviews and merges

Restrictions:

• One demo repository.
• Dedicated branch prefix.
• No push to main or master.
• No automatic merge.
• Separate least-privilege Gitea token in Vault.
• Patch size and file-path limits.
• Jenkins validation required.

This stretch is dropped immediately if it threatens rehearsal or the H+24 stop.

Required architecture-document update

After behavior is known, mermaid/TestAutomation.mmd must show these internal steps rather than compressing Hermes into one box:

Trigger
  -> terminal Jenkins failure or human request

Detection
  -> Ariadne allowlisted polling and incident deduplication

Evidence
  -> Jenkins metadata/artifacts
  -> bounded OpenSearch excerpts
  -> VictoriaMetrics
  -> Kubernetes/Flux/Gitea/Grafana context when relevant

Agent invocation
  -> Ariadne calls Hermes Agent API
  -> explicit triage-titan-test-failures skill
  -> local Qwen model through Hermes Ollama

Decision
  -> first enforced failure
  -> facts versus inference
  -> specialist skill routing
  -> confidence and action schema

Authorization
  -> Ariadne allowlist, incident match, confidence, idempotency and kill switch

Response
  -> predefined repair Job and one rebuild
  -> or human-required metric and alert

The chart must visually distinguish current behavior, newly implemented demo behavior and deferred capabilities. It should continue to keep Jenkins-retained logs separate from OpenSearch pod logs and should not imply direct Hermes OpenSearch access.

Final handoff record

At H+24, append a short result section containing:

• Delivered tier.
• Exact commits and deployed revisions.
• Demo job/build IDs.
• Incident ID.
• Evidence query window.
• Hermes request/session ID.
• Requested and executed action.
• Repair Job name.
• Rebuild result.
• Alert result.
• Total end-to-end duration.
• Compromises activated.
• Deferred hardening.
• Kill-switch and rollback verification.

The handoff must state plainly whether the full Hermes Agent or the Ollama fallback performed the diagnosis.

---

Delivery record

• Execution window opened 2026-08-05 ~17:20 UTC.
• Delivered tier: A, plus the optional autonomous-code stretch.
• Diagnosis was performed by the full Hermes Agent (POST /v1/runs, skill $triage-titan-test-failures) in every recorded case. The Ollama fallback described in the plan was never activated.

What runs today

Capability	State
Detection on 11 Jenkins jobs (2 demo + 9 homegrown)	live
Jenkins + bounded OpenSearch evidence	live
Hermes Agent diagnosis, schema-validated	live
Action registry: repair_demo_fixture, retry_transient_infra	live
Bounded source-patch proposal → branch → pull request	live (demo repo)
Human-required metric, vmalert rule, Alertmanager	live
Gitea issue filing for human-required incidents	live, proven on 4 real repos
Candidate-file selection for real service repos	live, PR opened on bstein/ariadne
Narrowed alerting (repair failed, or unattended 6h)	live

What the fixture and code demos actually prove

Worth stating plainly before demonstrating either one, because the two demos are not equally representative of the real-service path.

The fixture demo is end-to-end real. Ariadne detects a genuine red build, Hermes diagnoses it from real evidence, the deterministic gates authorize, and Ariadne executes and verifies via a rebuild. Nothing about it is staged beyond the fixture itself being deliberately broken.

The code demo is real in its hard parts and staged in one: the repository is real, the failing tests are real, Hermes genuinely locates the defect and writes an anchored patch, the validator genuinely checks it, and the pull request is genuinely opened and awaits review. What is staged is which file Hermes is handed - ARIADNE_HERMES_CODE_CANDIDATE_PATH=src/discount.py pins it. File selection, the step that has to work on a real service, is the one step the demo does not exercise.

On real repositories there is no pin, and selection has to work from evidence alone. Two defects blocked that and were fixed on 2026-08-06 (see below): successful tool output crowding the real failure out of the evidence budget, and the patcher being unable to reach a source file the console never names.

Measured results

Closed autonomous loop, twice:

armed 20:58:10 -> build red 20:59:59 -> repair Job 21:01:14
      -> rebuild green 21:01:46 -> resolved 21:02:10   (4m00s)
second run: red 21:43:55 -> repair 21:44:39 -> green 21:50:45 -> resolved 21:51:08

After converting the fixture repair from a spawned Kubernetes Job to an in-process API call (Ariadne 0.1.0-402), the automated leg roughly halved:

build red 02:20:03 -> fixture healthy 02:20:28 (25s)
      -> rebuild triggered 02:20:41 -> green and resolved 02:21:07  (1m04s)

Code proposal: red build 22:34:59 -> pull request open 22:35:32 (33s).

Autonomous source repair on a real service repository, proven 2026-08-06. A one-line defect was seeded in ariadne/utils/errors.py (dropping the errorMessage key from the HTTP payload lookup) and pushed to master. Build 409 went red, and Ariadne opened bstein/ariadne PR #3 from branch hermes-repair/409 with the correct one-line fix:

-        raw = payload.get("error") or payload.get("message")
+        raw = payload.get("error") or payload.get("message") or payload.get("errorMessage")

The stored hermes_autotriage_code_proposal event records how it got there:

candidates:  [ariadne/app.py, tests/test_utils.py, ariadne/services/mailu.py,
              ariadne/utils/errors.py, ariadne/utils/http.py, ariadne/utils/passwords.py]
chosen_path: ariadne/utils/errors.py
validated:   true

ariadne/utils/errors.py appears nowhere in the build console. It is in that list only because Ariadne read the failing test and followed its imports. No candidate path was pinned, unlike the hermes-code-demo job.

Four rounds were needed, and the failure reasons are the useful part:

Build	Outcome	Cause
404	no_candidate_files	evidence held no failure at all
406	never triaged	superseded by a newer build 24 s later
408	patch_invalid	only the wrong file was offered
409	PR opened	all three fixes live together
Hermes diagnosis latency across all runs: 15-21s.

Real homegrown service diagnoses (no action allowlisted, all escalated):

Incident	Classification	Confidence
bstein-dev-home/445	sonarqube_node_runtime_missing	0.99
metis/271	jenkins_scm_checkout_timeout	0.99
lesavka/582	coverage_quality_gate_failure	0.94
soteria/276	unknown	0.58 (below threshold)

Safety behaviour observed in production, not only in tests

• Hermes classified a pipeline-infrastructure failure as the demo fixture signature at 0.91 confidence; Ariadne's deterministic evidence gate refused the action (evidence_signature_missing). The model was wrong and the system still did the right thing.
• With the action allowlist emptied, a genuine fixture failure produced human_required and HermesTriageHumanRequired reached FIRING in vmalert and active in Alertmanager; after an operator fixed the fixture and the job went green, the alert self-cleared through supersession.
• soteria returned unknown at 0.58 — below the 0.85 threshold and with no matching action, so two independent gates blocked it.
• Two defects were found by deliberately seeding a one-line break in a real repository (ariadne) rather than by reasoning about the code. First, the console evidence budget was filled entirely by successful tool output: build 404 retained six regions spanning lines 259-476 - Trivy setup, Sonar banners, and three lines whose only failure signal was the word coverage - and dropped the pytest failure at line 1487. Markers are now ranked by whether they assert a failure or merely name a tool that ran. Second, the defective file ariadne/utils/errors.py appeared zero times in the entire 174 KB console, because a pytest assertion that fails inside a test names only the test; candidate selection returned the wrong file. Ariadne now reads the failing test and follows its imports back to the module under test, widening what may be read without widening what may be written - the patch validator still gates writes on allowed_path_prefixes, so a failing test can be read for context and never patched, which also forecloses the classic bad fix of silencing a test instead of repairing the code.
• bstein-dev-home/446 was classified known_demo_fixture_failure at 0.99, even though build 445 of the same job had been classified correctly as sonarqube_node_runtime_missing. The evidence gate refused it again, so nothing ran, but the intermittency is the point: the same evidence produced a demo-only classification on one run and the right one on another. The triage prompt now interpolates the job under analysis and forbids the demo classification and repair_demo_fixture on any job other than hermes-triage-demo (Ariadne 0.1.0-403), removing the judgement call rather than asking the model to make it correctly every time.

Deviations from the plan

• Vault: the shared Hermes API key and the Gitea automation token are manually created Kubernetes Secrets, not Vault-sourced. robotuser returned 403 on userpass/ldap/oidc and no token was available. The manifests reference the Secrets by name, so the swap is config-only.

• Fixture storage: the demo fixture is a ConfigMap, not a PVC. Longhorn attach latency and per-node engine availability made a PVC unreliable for a fast repeatable demo.

• JUnit evidence: RESOLVED 2026-08-06. junit 1369.v15da_00283f06 and pipeline-stage-view 2.41 are installed and active, with scm-api bumped 724 -> 728 as their workflow-cps requires. This was never actually blocked on a core upgrade: only the latest junit (1418) needs core 2.533, while 1369 needs 2.504.3 and runs on 2.528.3. The earlier record was wrong because it read the plugin's latest release rather than the dynamic-stable-2.528.3 update-center channel.

  testReport and wfapi now return 200 where they returned 404 all night - ariadne build 414 reports 1183 passing tests through the API. No Ariadne change was needed: the evidence collector already fetched testReport/api/json and had simply always received a 404, so failed_tests now populates on its own.

  This removes the cause of both diagnosis defects fixed the same day. With no structured test data, console text was the only evidence, which is why the enforced failure could be crowded out of the byte budget and why the patcher could not locate a source file the console never named.

  Cost: installing them required restarting the controller, which rescheduled it onto a node without the image cached and produced a ~15 minute outage on the image pull alone, plus three killed builds. Pre-pull the controller image on the worker nodes before any future Jenkins restart; the agent image had already been pre-pulled for exactly this reason and the lesson was not carried across.

• Multibranch jobs are not detected: folder-style jobs expose a null lastBuild at the job root.

• OpenSearch namespace is still pinned to the demo namespace plus jenkins; real services get their Jenkins agent logs but not their own runtime logs.

Kill switches and rollback

ARIADNE_HERMES_AUTOREMEDIATION_ENABLED=false   # diagnose only
ARIADNE_HERMES_AUTOTRIAGE_ENABLED=false        # fully off
ARIADNE_HERMES_ISSUES_ENABLED=false            # stop filing issues
ARIADNE_HERMES_CODE_ENABLED=false              # stop proposing patches
ARIADNE_HERMES_ALLOWED_ACTIONS=""              # accept no action at all

All are Flux-managed in services/maintenance/apps/ariadne-deployment.yaml. The automatic loop can only ever create Jobs named hermes-demo-repair-* in hermes-triage-demo, rebuild allowlisted Jenkins jobs, file issues in explicitly mapped repositories, and push hermes-repair/* branches. One action per incident, ever.

Verified rollback and safety properties

• Emptying the action allowlist was exercised live and produced diagnosis without mutation.
• Restoring it was exercised live and the loop resumed.
• Hermes holds no Git credentials, no Kubernetes write access, and cannot read Secrets; its Flux-managed approval deny-list still blocks kubectl mutations, flux mutations and secret reads.