Strategy Specification generation prompt consistently runs into blockers/warnings when prompt is used #26

New Issue

bstein · 2026-07-09T05:14:20Z

bstein commented

2026-07-09 05:14:20 +00:00

What happened

I used the generator prompt and got these errors:

blocker · Card Hover Notes

Card Hover Notes are not ready for versioned save/backfill: strong coverage: 108/110
Card Hover Notes
Use one structured note per required deck card with Deck role, Role type, Deck-specific job, Timing, Inputs / outputs, Mode map when the card has distinct modes/aspects, Partners / packages with exact in-deck card names, Role peers, Failure modes, and Matchup / visibility hooks.
warning · Reflection Template

Reflection Template is too shallow to produce useful tuning telemetry.
Ask for primary/secondary causes, win contributors, mana issues, sideboard or command-zone impact where legal, stranded cards, and over/underperformers.
warning · Reflection Template

Reflection Template is empty, so post-game reports will be less deck-specific.
Add win/loss cause questions and deck-specific tuning prompts.

What should have happened

The prompt should be complete (high enough quality) such that if chatgpt gets this prompt, it will produce a strategy specification that doesn't have these issues all the time.

Context

Page URL: https://veles.bstein.dev/lab
Route: lab
Path: /lab
Backend version: 0.5.28
Frontend version: 0.5.28
Sim version: 0.5.28

## What happened <img width="1440" alt="image.png" src="attachments/b1a3209c-62a4-482d-8efb-815ac8f3aaa9"> I used the generator prompt and got these errors: ``` blocker · Card Hover Notes Card Hover Notes are not ready for versioned save/backfill: strong coverage: 108/110 Card Hover Notes Use one structured note per required deck card with Deck role, Role type, Deck-specific job, Timing, Inputs / outputs, Mode map when the card has distinct modes/aspects, Partners / packages with exact in-deck card names, Role peers, Failure modes, and Matchup / visibility hooks. warning · Reflection Template Reflection Template is too shallow to produce useful tuning telemetry. Ask for primary/secondary causes, win contributors, mana issues, sideboard or command-zone impact where legal, stranded cards, and over/underperformers. warning · Reflection Template Reflection Template is empty, so post-game reports will be less deck-specific. Add win/loss cause questions and deck-specific tuning prompts. ``` ## What should have happened The prompt should be complete (high enough quality) such that if chatgpt gets this prompt, it will produce a strategy specification that doesn't have these issues all the time. ## Category - kind: metadata-content - severity: obvious ## Context - Page URL: https://veles.bstein.dev/lab - Route: lab - Path: /lab - Backend version: 0.5.28 - Frontend version: 0.5.28 - Sim version: 0.5.28

image.png

473 KiB

bstein added the

manual-review

kind/metadata

severity/obvious

labels 2026-07-09 05:14:20 +00:00

bstein commented

2026-07-09 06:33:45 +00:00

Verified in Cassandra 0.5.30.

Release proof: source bebd4c7 (Prepare Cassandra 0.5.30 feedback backlog closeout), Jenkins Veles #209 SUCCESS published version-only backend/frontend/sim-worker 0.5.30 images, Titan-IAC 1bf84977 deployed them, live /api/v1/app-config reports backend/frontend/sim_worker 0.5.30, /api/v1/ready is green, backend pod and both frontend pods are Running with 0 restarts, and VELES_SIM_IMAGE is registry.bstein.dev/veles/veles-sim-worker:0.5.30.

Local/source gate on the deployed source: bridge/runtime 42 tests OK; evaluator/audit/policy/API suite 504 tests OK; frontend maybeboard/compact-replay/prompt Playwright 22 tests passed; frontend build and git diff --check passed.

Issue-specific proof: live 0.5.30 backend pod proof showed generated reflection_requests present, primary-cause language present, commander-tax reporting present, and reflection_warning_ids=[] for the generated Strategy Specification path; prompt requirements now match the strict card-hover gate. Closing.

Verified in Cassandra 0.5.30. Release proof: source bebd4c7 (Prepare Cassandra 0.5.30 feedback backlog closeout), Jenkins Veles #209 SUCCESS published version-only backend/frontend/sim-worker 0.5.30 images, Titan-IAC 1bf84977 deployed them, live /api/v1/app-config reports backend/frontend/sim_worker 0.5.30, /api/v1/ready is green, backend pod and both frontend pods are Running with 0 restarts, and VELES_SIM_IMAGE is registry.bstein.dev/veles/veles-sim-worker:0.5.30. Local/source gate on the deployed source: bridge/runtime 42 tests OK; evaluator/audit/policy/API suite 504 tests OK; frontend maybeboard/compact-replay/prompt Playwright 22 tests passed; frontend build and git diff --check passed. Issue-specific proof: live 0.5.30 backend pod proof showed generated reflection_requests present, primary-cause language present, commander-tax reporting present, and reflection_warning_ids=[] for the generated Strategy Specification path; prompt requirements now match the strict card-hover gate. Closing.

bstein closed this issue

2026-07-09 06:33:46 +00:00

bstein commented

2026-07-09 07:02:45 +00:00

Follow-up shipped in Cassandra 0.5.31 for the generator/compiler diagnostics feedback.

What changed:

Strict Card Hover Notes blockers now name exact failing cards and reason buckets instead of only reporting strong coverage counts.
Missing structured fields are shown as human labels, e.g. Deck role, Role type, Deck-specific job, Inputs / outputs.
Reflection Template diagnostics no longer double-report empty templates as both empty and shallow.
Shallow Reflection Template warnings now name the missing telemetry prompts.

Proof:

The reported Lazav retry file now reduces to an actionable card-note diagnostic: strong coverage: 105/107; unresolved package labels: Damnation, Go for the Throat.
Focused compiler/card-note tests: 79 OK.
Frontend typecheck/build and git diff --check passed.
Jenkins Veles #210 SUCCESS published 0.5.31 backend/frontend/sim-worker images.
Titan-IAC 40e6c065 deployed 0.5.31; live app-config reports backend/frontend/sim_worker 0.5.31 and /ready is green.

Follow-up shipped in Cassandra 0.5.31 for the generator/compiler diagnostics feedback. What changed: - Strict Card Hover Notes blockers now name exact failing cards and reason buckets instead of only reporting strong coverage counts. - Missing structured fields are shown as human labels, e.g. Deck role, Role type, Deck-specific job, Inputs / outputs. - Reflection Template diagnostics no longer double-report empty templates as both empty and shallow. - Shallow Reflection Template warnings now name the missing telemetry prompts. Proof: - The reported Lazav retry file now reduces to an actionable card-note diagnostic: strong coverage: 105/107; unresolved package labels: Damnation, Go for the Throat. - Focused compiler/card-note tests: 79 OK. - Frontend typecheck/build and git diff --check passed. - Jenkins Veles #210 SUCCESS published 0.5.31 backend/frontend/sim-worker images. - Titan-IAC 40e6c065 deployed 0.5.31; live app-config reports backend/frontend/sim_worker 0.5.31 and /ready is green.

bstein commented

2026-07-09 08:25:33 +00:00

Follow-up shipped in Cassandra 0.5.32 for the remaining compiler diagnostic gap.

What changed:

Fixed the false positive where Damnation-style sweeper rules text (destruction of all creatures) was treated as an unresolved package label.
Real unresolved package labels now include exact card, field, matched phrase, and nearby context in the blocker message.
Embedded Reflection Template warnings now point to the exact heading line.
Generator/backfill prompts now warn against unresolved package labels and clarify that reflection questions belong in the separate Reflection Template field.

Proof:

The reported Lazav V2 file no longer fails on Damnation.
The remaining real blocker now reads: strong coverage: 109/110; unresolved package labels: Demonic Tutor (Deck-specific job uses "or artifact payoff" near "...tainted pact or artifact payoff").
Reflection warning points to line 2083 (### Reflection Template).
Focused tests: 101 OK.
Frontend typecheck/build and git diff --check passed.
Jenkins Veles #211 published 0.5.32 backend/frontend/sim-worker images.
Titan-IAC 1801e3e8 deployed 0.5.32; live app-config reports backend/frontend/sim_worker 0.5.32 and /ready is green.

Follow-up shipped in Cassandra 0.5.32 for the remaining compiler diagnostic gap. What changed: - Fixed the false positive where Damnation-style sweeper rules text (`destruction of all creatures`) was treated as an unresolved package label. - Real unresolved package labels now include exact card, field, matched phrase, and nearby context in the blocker message. - Embedded Reflection Template warnings now point to the exact heading line. - Generator/backfill prompts now warn against unresolved package labels and clarify that reflection questions belong in the separate Reflection Template field. Proof: - The reported Lazav V2 file no longer fails on Damnation. - The remaining real blocker now reads: strong coverage: 109/110; unresolved package labels: Demonic Tutor (Deck-specific job uses "or artifact payoff" near "...tainted pact or artifact payoff"). - Reflection warning points to line 2083 (`### Reflection Template`). - Focused tests: 101 OK. - Frontend typecheck/build and git diff --check passed. - Jenkins Veles #211 published 0.5.32 backend/frontend/sim-worker images. - Titan-IAC 1801e3e8 deployed 0.5.32; live app-config reports backend/frontend/sim_worker 0.5.32 and /ready is green.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: cassandra-alpha/feedback#26