# Incident drills

Present only one `Student packet` before the student answers. Use the coach
notes afterward to grade the failure classification and reasoning. These cases
are based on retained Atlas rollout observations; treat them as training
fixtures, not proof of current live state.

## Case A — Metrics disappear after a Soteria run

### Student packet

- The Soteria test stages ran.
- The post-stage log ends with:
  `Syntax error: end of file unexpected (expecting "fi")`.
- The canonical Soteria Pushgateway series did not refresh.
- Other suites continued publishing.

Ask for: finding, confidence, evidence, likely cause, blast radius, read-only
next checks, repo-side fix, and approval-required actions.

### Coach notes

The primary class is pipeline glue/shell syntax, not a Pushgateway outage and
not a product-test failure. Inspect the post block and the exact shell parsed by
`/bin/sh`; confirm the last successful metric timestamp. The smallest likely
fix is closing/correcting the conditional in the Jenkinsfile, then a normal SCM
run and read-only metric verification. Editing, pushing, triggering, or manual
metric backfill requires explicit approval.

## Case B — UI tests fail before metrics publish

### Student packet

- `bstein-dev-home` requires Playwright `1.59.1`.
- The CI test container contains Playwright `1.51.0`.
- The frontend test stage hard-fails.
- No fresh canonical quality metrics appear for the run.

### Coach notes

Separate the direct test-environment mismatch from the telemetry-control-flow
defect. The version mismatch explains the test failure; early pipeline abort
explains missing metrics. A complete proposal aligns the image version and
persists stage return codes so publishing still runs while the final gate still
fails. Do not call this a Kubernetes capacity problem without node/pod evidence.

## Case C — Promotion exits 127

### Student packet

- Titan IaC validation and tests finish.
- The Promote stage uses `python:3.12-slim`.
- The stage returns exit code `127` at its first Git command.
- The repository and remote are reachable from other jobs.

### Coach notes

Exit 127 means the invoked command is unavailable. Confirm the exact log line
and image contents; the likely issue is that the slim image lacks `git`, not bad
Git credentials. The minimal repo-side fix installs `git` and CA certificates
in the job's dependency setup or uses a suitable pinned runner image.

## Case D — Data Preppers cannot publish an image

### Student packet

- Tests and local image build complete.
- Push to `registry.bstein.dev/streaming/data-prepper:2.8.0` is denied.
- The configured Jenkins credential belongs to a different Harbor project.
- A Sonar evidence GET also returns `401` in the same run.

### Coach notes

Do not collapse independent authentication failures into a generic network
failure. Verify the Harbor repository/project and credential ID without reading
the secret value; separately verify the Sonar token injection path and API
scope. The likely image fix is selecting a streaming-scoped robot credential.
Treat the Sonar 401 as a second issue unless evidence proves a shared cause.

## Case E — Hermes local inference fails during game streaming

### Student packet

- Wolf owns titan-24 according to the GPU-owner state.
- `hermes-model-gate` is ready but returns a deliberate unavailable response.
- Both Hermes dashboards remain healthy.
- One instance answers through Codex; the other reports missing provider auth.

### Coach notes

This is expected resource arbitration plus an instance-specific fallback auth
gap. Do not restart the agent or claim the GPU is underprovisioned. Confirm the
gate response, fallback chain, and each instance's auth status without exposing
tokens. Authorize the affected instance with its intended user's account. The
separate credential stores are a security property, not configuration drift.