96 lines
3.8 KiB
Markdown
96 lines
3.8 KiB
Markdown
|
# Incident drills
|
||
|
|
|
||
|
|
Present only one `Student packet` before the student answers. Use the coach
|
||
|
|
notes afterward to grade the failure classification and reasoning. These cases
|
||
|
|
are based on retained Atlas rollout observations; treat them as training
|
||
|
|
fixtures, not proof of current live state.
|
||
|
|
|
||
|
|
## Case A — Metrics disappear after a Soteria run
|
||
|
|
|
||
|
|
### Student packet
|
||
|
|
|
||
|
|
- The Soteria test stages ran.
|
||
|
|
- The post-stage log ends with:
|
||
|
|
`Syntax error: end of file unexpected (expecting "fi")`.
|
||
|
|
- The canonical Soteria Pushgateway series did not refresh.
|
||
|
|
- Other suites continued publishing.
|
||
|
|
|
||
|
|
Ask for: finding, confidence, evidence, likely cause, blast radius, read-only
|
||
|
|
next checks, repo-side fix, and approval-required actions.
|
||
|
|
|
||
|
|
### Coach notes
|
||
|
|
|
||
|
|
The primary class is pipeline glue/shell syntax, not a Pushgateway outage and
|
||
|
|
not a product-test failure. Inspect the post block and the exact shell parsed by
|
||
|
|
`/bin/sh`; confirm the last successful metric timestamp. The smallest likely
|
||
|
|
fix is closing/correcting the conditional in the Jenkinsfile, then a normal SCM
|
||
|
|
run and read-only metric verification. Editing, pushing, triggering, or manual
|
||
|
|
metric backfill requires explicit approval.
|
||
|
|
|
||
|
|
## Case B — UI tests fail before metrics publish
|
||
|
|
|
||
|
|
### Student packet
|
||
|
|
|
||
|
|
- `bstein-dev-home` requires Playwright `1.59.1`.
|
||
|
|
- The CI test container contains Playwright `1.51.0`.
|
||
|
|
- The frontend test stage hard-fails.
|
||
|
|
- No fresh canonical quality metrics appear for the run.
|
||
|
|
|
||
|
|
### Coach notes
|
||
|
|
|
||
|
|
Separate the direct test-environment mismatch from the telemetry-control-flow
|
||
|
|
defect. The version mismatch explains the test failure; early pipeline abort
|
||
|
|
explains missing metrics. A complete proposal aligns the image version and
|
||
|
|
persists stage return codes so publishing still runs while the final gate still
|
||
|
|
fails. Do not call this a Kubernetes capacity problem without node/pod evidence.
|
||
|
|
|
||
|
|
## Case C — Promotion exits 127
|
||
|
|
|
||
|
|
### Student packet
|
||
|
|
|
||
|
|
- Titan IaC validation and tests finish.
|
||
|
|
- The Promote stage uses `python:3.12-slim`.
|
||
|
|
- The stage returns exit code `127` at its first Git command.
|
||
|
|
- The repository and remote are reachable from other jobs.
|
||
|
|
|
||
|
|
### Coach notes
|
||
|
|
|
||
|
|
Exit 127 means the invoked command is unavailable. Confirm the exact log line
|
||
|
|
and image contents; the likely issue is that the slim image lacks `git`, not bad
|
||
|
|
Git credentials. The minimal repo-side fix installs `git` and CA certificates
|
||
|
|
in the job's dependency setup or uses a suitable pinned runner image.
|
||
|
|
|
||
|
|
## Case D — Data Preppers cannot publish an image
|
||
|
|
|
||
|
|
### Student packet
|
||
|
|
|
||
|
|
- Tests and local image build complete.
|
||
|
|
- Push to `registry.bstein.dev/streaming/data-prepper:2.8.0` is denied.
|
||
|
|
- The configured Jenkins credential belongs to a different Harbor project.
|
||
|
|
- A Sonar evidence GET also returns `401` in the same run.
|
||
|
|
|
||
|
|
### Coach notes
|
||
|
|
|
||
|
|
Do not collapse independent authentication failures into a generic network
|
||
|
|
failure. Verify the Harbor repository/project and credential ID without reading
|
||
|
|
the secret value; separately verify the Sonar token injection path and API
|
||
|
|
scope. The likely image fix is selecting a streaming-scoped robot credential.
|
||
|
|
Treat the Sonar 401 as a second issue unless evidence proves a shared cause.
|
||
|
|
|
||
|
|
## Case E — Hermes local inference fails during game streaming
|
||
|
|
|
||
|
|
### Student packet
|
||
|
|
|
||
|
|
- Wolf owns titan-24 according to the GPU-owner state.
|
||
|
|
- `hermes-model-gate` is ready but returns a deliberate unavailable response.
|
||
|
|
- Both Hermes dashboards remain healthy.
|
||
|
|
- One instance answers through Codex; the other reports missing provider auth.
|
||
|
|
|
||
|
|
### Coach notes
|
||
|
|
|
||
|
|
This is expected resource arbitration plus an instance-specific fallback auth
|
||
|
|
gap. Do not restart the agent or claim the GPU is underprovisioned. Confirm the
|
||
|
|
gate response, fallback chain, and each instance's auth status without exposing
|
||
|
|
tokens. Authorize the affected instance with its intended user's account. The
|
||
|
|
separate credential stores are a security property, not configuration drift.
|