W21B Wave 4 FP Fidelity Implementation Brief
Floor-plan PNG self-fidelity, marker semantic validation, and layout-faithful BG anchor rendering.
0. Decision
Do not treat this as a W21B-wave-3 reuse regression. The reuse router, copy-less alias materialization, low-frequency character-state fix, and scene regeneration completed their own scope. The FP gallery exposed a separate existing fidelity gap in the floor-plan-to-background chain.
Recommended sequence:
- Phase A: FP PNG self-fidelity + marker semantic gate. A floor-plan image must match its own
t2i_promptandnumbered_elementsbefore it can be used as an authoritative visual reference. This phase adds a VLM semantic gate that checks whether each visible marker's image content matches the expected metadata label/object role, with evidence fields such assource_ref,observed_object_summary,confidence, andreasoning_basis. Mismatches trigger bounded retry or fail-closed state, not silent downstream use. - Phase B:
fp_seeded_anchorlayout-faithful BG anchor. Once the FP image is trustworthy, the BG anchor should preserve fixed structure, openings, persistent fixtures, persistent furniture, room scale, and spatial relationships from the validated FP. Style guidance can alter finish and realism, but must not replace FP layout with a different dwelling shape.
W21B-wave-3 commits should remain unpushed until the user confirms whether to close reuse as a separate deliverable or fold this wave into the same PR sequence.
1. Evidence
| Evidence | Observed fact | Implication |
|---|---|---|
background_render.data.groups.L05B06 |
floor_plan_used=true, ref_used=fp_only, attached_ref_labels=["fp:fp_l05_main"]. |
The FP was actually attached. This is not a missing-reference or gallery-mapping bug. |
floor_plan_prompt.data.floor_plans.fp_l05_main.numbered_elements |
17=main floor bedding mat, 18=main wardrobe closet, 24=wall-mounted family photo frame. |
Downstream prompt generation and shot-aware planning trust these number-to-label meanings. |
fp_l05_main.png visual review |
The rendered FP looks like a larger multi-room apartment with a proper bed; visible marker placement does not visually align with the metadata labels. | The FP image drifted from its own metadata and prompt. The problem starts before BG render. |
L05B06.png visual review |
Rendered as a modest room with floor bedding, wardrobe, window, and curtain partition. | The BG is largely faithful to metadata/prose, not to the visual FP PNG. |
floor_plan_geometry_readback |
VLM/readback records marker number, coarse cell, and base kind. It does not validate that marker 17 is actually drawn on bedding. |
Semantic marker-label mismatch can pass current gates. |
background_prompt/*/system.md and adapter guidance |
FP is described as a layout source, but prompts explicitly avoid preserving the top-down plan exactly; adapter style contract strongly pushes modest residential styling. | Even when FP is attached, content/layout can be overridden by metadata prose and style guidance. |
2. Root Cause
Important inversion: in the L05 case, the BG plate is closer to the floor-plan metadata than the FP PNG is. The floor-plan image is not a reliable visual source of truth.
- Floor-plan render self-fidelity gap.
render_one_floor_planrenders a PNG from the prompt and optional refs, but there is no post-generation check that the PNG matches the prompt's layout, furniture, scale, or numbered elements. - Marker semantic readback gap.
floor_plan_geometry_readbackandfloor_plan_vlm_providercheck marker number/kind/cell shape. They do not ask the VLM to confirm that each marker is placed on the expected object class or semantic label. - BG anchor layout-faithfulness gap.
fp_seeded_anchorattaches the FP, but renderer guidance treats base markers as spatial guidance and style/prose as content authority. This is too weak when the FP is supposed to be the visual layout anchor.
Upstream metadata risk: Phase A deliberately starts with the narrower contract "does the PNG match its own metadata?" It does not prove the metadata itself is the right scale or dwelling interpretation. If Phase A shows that the metadata is consistently wrong-scale or wrong-layout for the location, stop and move one layer upstream to floor-plan prompt/master-plan interpretation rather than forcing the PNG to faithfully render bad metadata.
3. Scope
In Scope
- Floor-plan prompt/render prompt changes that reduce PNG drift from
numbered_elementsandt2i_prompt. - A semantic readback/gate that confirms visible marker numbers correspond to expected object classes/labels at least for base structure, openings, persistent fixtures, and persistent furniture used downstream.
- Fail-closed or retry policy when an FP image violates self-fidelity, without deleting prior assets.
- Renderer guidance changes for
fp_seeded_anchorso BG anchors preserve FP fixed layout once the FP is validated. - Improved review artifact that shows scene, BG, FP, and marker set used by the BG, so whole-plan-vs-camera-view mismatch is easier to interpret.
Non-goals
- No scenario-specific code or prompt logic for rooftops, markets, or named locations.
- No destructive cleanup of existing PNGs or
ImageAssetrows. - No regex or literal keyword matching as semantic source of truth.
- No broad TDD around image quality. Tests stay on deterministic shape, wiring, and fail-closed behavior.
- No automatic rerun of full scene-image pipeline until FP/BG visual acceptance passes.
- No push of W21B-wave-3 local commits until the user agrees how this new fidelity wave is sequenced.
3.1 Specific Scenario Leakage Guard
Production logic must stay generic. The L05B06 / fp_l05_main case is evidence only. It may live in canary artifacts or review fixtures, but production prompts and code must not hard-code rooftops, bedding, wardrobes, family photos, scene numbers, Korean nouns, or any marker number as a semantic rule.
- Allowed generic contract terms:
floor_plan_id,marker_number,expected_label,base_layer_decision,observed_object_summary,semantic_match,source_ref,confidence. - Forbidden as production decision logic: substring/regex checks such as looking for words like bed, wardrobe, rooftop, market, or any location-specific noun.
- Semantic decisions must come from structured LLM/VLM extraction and visual review, not from lexical matching.
4. Proposed Implementation
| Step | Change | Expected effect |
|---|---|---|
| 1. FP prompt/render contract | Add a new floor-plan prompt/render pack or selector that makes numbered-element fidelity explicit: every numbered element must appear exactly once at the corresponding object/area; do not substitute beds for floor bedding, multi-room apartments for compact rooms, or extra furniture not in the inventory. | Reduces self-fidelity drift before any readback gate runs. |
| 2. Semantic readback sidecar | Extend or add a VLM readback sidecar with canonical field observed_marker_semantics[]. Each entry uses: number, expected_label, expected_layer, observed_object_summary, semantic_match, mismatch_reason, source_ref, confidence, reasoning_basis. The VLM should cite evidence from the image in generic visual terms, not scenario nouns embedded in code. |
Turns marker-label consistency into an explicit evidence-backed gate instead of assuming metadata and PNG agree. |
| 3. Retry/fail policy | If semantic readback finds structural/furniture marker mismatch, retry FP render within a small cap; after cap, mark the FP as needs_fix/failed for BG anchor consumption. Do not delete old assets. | Prevents untrustworthy FP images from becoming authoritative visual references. |
| 4. BG anchor guidance | Update fp_seeded_anchor guidance to preserve validated FP fixed layout: room count, openings, windows, persistent furniture positions, fixture positions, and scale relationships. Style contract may adjust material finish but must not relocate or replace validated objects. |
Makes BG anchor use FP as a real layout source rather than weak spatial inspiration. |
| 5. Review artifact | Generate active-only review rows with old/new scene, BG, FP, and used marker list. Highlight bg_id, fp_id, camera_unit, look_at_unit, and use_numbered_elements. |
Separates true mismatch from normal top-down-plan vs camera-view differences. |
5. Touchpoints
| Area | Likely files | Notes |
|---|---|---|
| Floor-plan prompt | prompts/_base/floor_plan_prompt/*, backend/app/modules/pipeline/floor_plan_prompt.py, backend/app/core/steps/floor_plan_prompt_step.py |
New pack or selector. Keep schema changes deterministic. Preserve scenario-general language. |
| Floor-plan render | backend/app/modules/pipeline/floor_plan_render.py, backend/app/core/steps/floor_plan_render_step.py |
Potential retry integration after semantic readback. Rendering itself remains image API call; no destructive cleanup. |
| Geometry/semantic readback | backend/app/modules/pipeline/floor_plan_vlm_provider.py, backend/app/modules/pipeline/floor_plan_geometry_readback.py, backend/app/core/steps/floor_plan_geometry_readback_step.py |
Existing readback observes number/cell/kind. New semantic sidecar should validate expected object semantics. |
| Base dossier | backend/app/modules/pipeline/base_location_dossier.py |
Should consume validated semantic readback state and surface warnings/blockers to downstream planners. |
| BG anchor render | backend/app/modules/pipeline/shot_aware_bg_render_adapter.py, prompts/_base/background_prompt/* |
Strengthen layout preservation only after FP has passed semantic fidelity gate. |
| Review tooling | /tmp operator runner or reusable read-only helper |
Show FP/BG/scene and marker usage together. Do not mutate DB or files. |
6. Canary Plan
Default mode is dry-run. Any runner or script for this wave must default to --generate=false or equivalent. The first pass designs and validates prompt/schema/contracts only. Real image or VLM calls require an explicit generate flag and a short scope/cap notice.
- Read-only preflight: run a small FP fidelity audit over W20E7 active FP set. Report marker semantic mismatch candidates, not as hard acceptance.
- FP-only canary: force floor-plan prompt/render/readback for W20E7 with a tight image cap. Do not rerun BG or scene yet.
- Visual acceptance: inspect FP PNGs first. Canary evidence may use the L05 case, where the FP should read as compact rooftop rental room with floor bedding and correct marker positions rather than a large furnished apartment. This example is review-only evidence, not production prompt or code logic.
- BG-anchor canary: after FP acceptance, force downstream dossier, shot-aware plan if needed, background prompt/render for affected BGs. Cap separately.
- Scene rerun: only after FP and BG visual acceptance, force scene-image pipeline if required. Preserve old images by demotion, not deletion.
7. Acceptance
Visual Acceptance
- FP PNG matches its own prompt and metadata: compact vs large-scale drift is gone.
- Numbered markers are visibly at their expected object or area. Example: marker
17should be on/near floor bedding if metadata says bedding. - BG anchor no longer contradicts validated FP layout in fixed room count, openings, windows, persistent fixtures, or anchor furniture.
- Review artifact makes it clear which part of the FP a camera-view BG is supposed to represent.
Deterministic Sanity Checks
- Schema/version selector loads new prompt/readback packs.
- Semantic readback output shape validates and fails closed on missing required fields.
- Retry/fail policy does not delete or overwrite unrelated assets.
- Review helper partitions active vs stale assets without mutation.
These checks do not decide image quality. Visual review, canary outputs, and focused experiments remain the acceptance gate for uncertain image outputs. Do not claim fidelity quality from pass counts alone.
8. Work Split
| Owner | Responsibility |
|---|---|
| Claude | Main production/prompt patching, operator runners, canary execution, and implementation notes. |
| Codex | Direction/brief, light deterministic test matrix, cross-review, visual/evidence evaluation, and closure guidance. |
9. Stop Conditions
- If VLM semantic readback cannot reliably judge marker-object matching, stop and use manual visual review as the gate rather than pretending deterministic confidence.
- If regenerated FP still drifts into a different dwelling type after two capped attempts, stop and inspect the floor-plan prompt rather than spending more image calls.
- If BG anchor remains inconsistent after FP is validated, treat it as a BG layout-faithfulness bug, not an FP bug.
- If user rejects the FP/BG target direction, pause implementation and revise the contract before more image calls.
9.1 Asset Safety
- Never delete floor-plan, background, or scene PNGs during this wave.
- Use UPSERT or status/demotion patterns only where the existing pipeline already uses them.
- Preserve stale artifacts for review unless the user explicitly authorizes cleanup.
- Any retry must write a new checkpoint/image result or update the intended row through the established asset path; it must not rewrite unrelated assets.
10. Retrospective Note
W21B-wave-3's active scene regeneration passed mechanical gates, but the FP comparison gallery exposed that mechanical success did not cover FP-to-BG semantic fidelity. Future visual reviews should include the specific visual source that downstream prompts claim to use, plus the metadata markers that bridge them.