# 05 — Prompt And Evaluation Guidelines

This file translates the research and repo audit into operating rules for prompts, structured contracts, and evaluation.

## Prompt Direction

The current prompt direction is right:

- `shot_staging` should emit structured cinematography/state fields.
- `scene_detail` should consume RenderPromptCard as primary contract.
- Prompt prose should explain consumption, not infer hidden semantics.
- Active prompts should avoid scenario-specific examples and fixed noun lists.

## Prompt Anti-Patterns

Avoid:

- semantic regex or phrase lists in code for open-world judgment,
- lists of body parts, object nouns, or scenario examples as logic,
- canonical names or arbitrary nouns as durable IDs,
- prose-only "remember this later" instructions,
- migration periods that accept both old and new fields silently,
- examples that look like real story content.

Accept:

- closed-world enum validation,
- closed-world short ID syntax validation,
- synthetic placeholders,
- LLM-produced structured fields,
- model-produced advisory observed-state proposals with confidence and source.

## Prompt Pack Checklist

Every new prompt pack that introduces a contract should include:

- schema version bump,
- prompt version bump,
- version_registry prompt dependency update,
- step_manifest schema update when checkpoint shape changes,
- latest-pack full copy if loader stem drift can occur,
- active prompt residue gate,
- test that loader picks the new pack,
- test that old field names are absent.

## Evaluation Direction

CANVAS uses targeted questions to choose candidates. TheRoad-I1 should adopt the contract-specific question design, but not the assumption that a VLM answer is a trustworthy final judgment. Current VLMs are too unreliable on character identity, small props, hands, spatial layout, and subtle background geometry to serve as closure gates.

### Current Gate Stack

```mermaid
flowchart LR
  Shape[Schema/helper tests]
  Version[Manifest/version tests]
  Prompt[Prompt residue gates]
  Contract[RenderPromptCard shape/hash]
  Runtime[verify_completion]
  Image[Advisory image review]

  Shape --> Version
  Version --> Prompt
  Prompt --> Contract
  Contract --> Runtime
  Runtime --> Image
```

The missing layer is the final image QA.

## Proposed ContinuityEval-Lite

Build a local evaluator inspired by CANVAS/ViStoryBench but tied to TheRoad-I1 contracts. Treat it as advisory unless a human or a deterministic/provenance check confirms the finding.

### Character continuity

Questions:

- Does each required visible character appear?
- Does appearance match the selected reference/outlook?
- If `subject_state` is non-alive, does the image reflect that state?
- Does the image avoid forbidden IDs/refs?

Inputs:

- generated image,
- attached refs,
- `visible_entities`,
- `id_policy`,
- `subject_state`,
- `asset_requirements`.

### Background continuity

Questions:

- If `background_ref_attached`, does the background layout match the reference?
- If `skipped_close_framing`, does the prompt/image avoid phantom reference claims?
- If `exact_background`, is same-room continuity preserved?
- If `atmosphere_reference`, is style carried without copying layout?

Inputs:

- previous/background ref image,
- `background_binding`,
- `previous_shot_refs`,
- `ref_usage`.

### Prop continuity

Questions:

- Is the prop present only when required by story state?
- If a prop is moved/removed/changed, does the image reflect current state?
- Does required prop ref attach when `render_contracts` requires it?

Inputs:

- `render_contracts`,
- `keep_elements`,
- prop refs,
- SceneContinuityPlan or StoryWorldMemory.

### Spatial contract

Questions:

- Does the first sentence of t2i_prompt reflect every frame spatial constraint?
- Do entities appear in expected zones/depth planes?
- Are close/medium/wide rules consistent with actual image?

Inputs:

- `render_strategy.frame_spatial_contract`,
- scene_detail variation echo,
- generated image.

## Evaluation Output Shape

Suggested structure:

```json
{
  "schema_version": 1,
  "scene_index": 1,
  "shot_index": 2,
  "overall_status": "pass|partial|fail",
  "scores": {
    "character": 0.0,
    "background": 0.0,
    "prop": 0.0,
    "spatial": 0.0,
    "prompt_alignment": 0.0
  },
  "findings": [
    {
      "dimension": "background",
      "severity": "fail",
      "expected": "exact_background",
      "observed": "layout changed",
      "source": "vlm_continuity_eval"
    }
  ],
  "memory_update_candidates": []
}
```

## How This Fits With Prompt Hygiene

The evaluator should not become a backdoor semantic regex layer or a fake source of truth. It should:

- read structured contracts,
- ask contract-specific questions only when image review is explicitly enabled,
- return structured observations,
- never infer by substring matching prompt prose,
- include null/uncertain when occluded.

## Minimal First Eval Gate

Start with deterministic/contract checks first:

1. Required refs were attached when the card required them.
2. Forbidden refs were not attached.
3. Background ref mode matches `background_binding`.
4. Prompt text does not claim a reference that is not attached.
5. State/memory records are written as intended state, not observed truth.

Advisory image review can come later for:

1. obviously missing visible character,
2. obviously wrong background ref,
3. obviously wrong state variant.

Those findings should route to manual review or regeneration hints, not automatic memory overwrite.

This is enough to catch high-impact drift without building a full benchmark immediately.

## Prompt Rewrite Standard

When a prompt rule becomes important enough to test, migrate it to one of:

- schema enum,
- RenderPromptCard field,
- StoryWorldMemory field,
- SceneContinuityPlan field,
- evaluator question,
- helper validation for closed-world IDs/enums.

Do not leave it as a standalone prose rule if downstream code or future prompts depend on it.