titan-iac/services/hermes/skills/master-hermes-on-atlas/references/incident-drills.md

Incident drills

Present only one Student packet before the student answers. Use the coach notes afterward to grade the failure classification and reasoning. These cases are based on retained Atlas rollout observations; treat them as training fixtures, not proof of current live state.

Case A — Metrics disappear after a Soteria run

Student packet

• The Soteria test stages ran.
• The post-stage log ends with: Syntax error: end of file unexpected (expecting "fi").
• The canonical Soteria Pushgateway series did not refresh.
• Other suites continued publishing.

Ask for: finding, confidence, evidence, likely cause, blast radius, read-only next checks, repo-side fix, and approval-required actions.

Coach notes

The primary class is pipeline glue/shell syntax, not a Pushgateway outage and not a product-test failure. Inspect the post block and the exact shell parsed by /bin/sh; confirm the last successful metric timestamp. The smallest likely fix is closing/correcting the conditional in the Jenkinsfile, then a normal SCM run and read-only metric verification. Editing, pushing, triggering, or manual metric backfill requires explicit approval.

Case B — UI tests fail before metrics publish

Student packet

• bstein-dev-home requires Playwright 1.59.1.
• The CI test container contains Playwright 1.51.0.
• The frontend test stage hard-fails.
• No fresh canonical quality metrics appear for the run.

Coach notes

Separate the direct test-environment mismatch from the telemetry-control-flow defect. The version mismatch explains the test failure; early pipeline abort explains missing metrics. A complete proposal aligns the image version and persists stage return codes so publishing still runs while the final gate still fails. Do not call this a Kubernetes capacity problem without node/pod evidence.

Case C — Promotion exits 127

Student packet

• Titan IaC validation and tests finish.
• The Promote stage uses python:3.12-slim.
• The stage returns exit code 127 at its first Git command.
• The repository and remote are reachable from other jobs.

Coach notes

Exit 127 means the invoked command is unavailable. Confirm the exact log line and image contents; the likely issue is that the slim image lacks git, not bad Git credentials. The minimal repo-side fix installs git and CA certificates in the job's dependency setup or uses a suitable pinned runner image.

Case D — Data Preppers cannot publish an image

Student packet

• Tests and local image build complete.
• Push to registry.bstein.dev/streaming/data-prepper:2.8.0 is denied.
• The configured Jenkins credential belongs to a different Harbor project.
• A Sonar evidence GET also returns 401 in the same run.

Coach notes

Do not collapse independent authentication failures into a generic network failure. Verify the Harbor repository/project and credential ID without reading the secret value; separately verify the Sonar token injection path and API scope. The likely image fix is selecting a streaming-scoped robot credential. Treat the Sonar 401 as a second issue unless evidence proves a shared cause.

Case E — Hermes local inference fails during game streaming

Student packet

• Wolf owns titan-24 according to the GPU-owner state.
• hermes-model-gate is ready but returns a deliberate unavailable response.
• Both Hermes dashboards remain healthy.
• One instance answers through Codex; the other reports missing provider auth.

Coach notes

This is expected resource arbitration plus an instance-specific fallback auth gap. Do not restart the agent or claim the GPU is underprovisioned. Confirm the gate response, fallback chain, and each instance's auth status without exposing tokens. Authorize the affected instance with its intended user's account. The separate credential stores are a security property, not configuration drift.