# Open-World Semantic Heuristic Full Audit

**Status**: AUDIT_R1 (current checkout, no code changes)  
**Date**: 2026-05-12  
**HEAD**: `bdb4ca0` (`main == origin/main`)  
**Scope inspected**:
- Production code: `backend/app/**/*.py` — 237 files.
- Prompt assets: `prompts/_base/**/*.{md,json}` — 505 files.
- Active prompt packs resolved by latest prompt-loader ordering for the main relevant modules: `scene_detail/21.202605062217`, `shot_validator/5.202605081700`, `t2i_review/3.202605121200`, `shot_director/4.202605121200`, `shot_staging/9.202605121441`, `entity_extractor_v2/8.202605121200`, `prompt_sanitizer/2.202605112104`, `visual_world_rules/6.202605021400`.

This audit is not a simple forbidden-token grep. The workflow was:
1. Exhaustive candidate scan over all production code and prompt files for regex/list/keyword/noun/pattern surfaces.
2. Active prompt pack resolution and direct line reads.
3. Candidate-by-candidate classification into open-world semantic, closed-world mechanical, special migration, and archive/scenario residue.

---

## 1. Audit Rule

### 1.1 Hard Principle

Open-world semantic judgment must come from LLM-produced structured SOT. Code may consume only closed, explicit structures: enum, bool, ID, object fields.

This includes exceptional cases. If the system wants to treat a shot as special, the upstream LLM step must emit a structured signal for that special case. Downstream code must not rediscover it from free text with regex, substring, noun lists, or "known examples".

### 1.2 Allowed Closed-World Mechanisms

The following are allowed and are not findings:
- `C##`, `P##`, `L##`, `O##`, `C##O##`, `S##_Shot##` ID parsing.
- Prompt version, schema, status, hash, path, safe-id validation.
- Exact comparison against an enum already emitted by an LLM schema.
- Static CI/review gates that scan code identifiers to prevent new semantic heuristics. These are meta tooling, not runtime semantic judgment.

---

## 2. Executive Summary

### 2.1 What Is Fixed

Area A is structurally clean. `shot_staging` now asks the LLM to emit `directionality_class`; validator and consumer branch only on that enum:
- `shot_staging.py:22-34` checks `directionality_class in ORIENTATION_REQUIRED_CLASSES`, not element text.
- `detail_steps.py:847-873` maps `directionality_class` through `_DIRECTIVE_TEMPLATES`, not noun strings.
- `shot_staging/9.202605121441/schema.json:58-64` requires the enum.
- `shot_staging/9.202605121441/system.md:164-177` explicitly says to judge by meaning and not by object name.
- `tests/_gate/test_orientation_enum_alignment.py:16-25` locks producer/consumer enum drift.

### 2.2 What Is Still Not Fixed

There are still substantial legacy open-world semantic heuristics in production code and active prompts. The existing umbrella spec identified the main cluster correctly, but the live audit adds several current surfaces:
- active `shot_director/4` prompt still has pattern-based off-camera/gaze-target rules;
- `scene_consistency_step.py` has close/full classification from element-id and description keywords;
- active entity extraction prompts contain list-like inclusion/exclusion heuristics;
- scene segmentation has regex fallback / LLM-generated regex machinery outside the visual QA area;
- old prompt archive has scenario residue, not active but still hygiene debt.

### 2.3 Gate Reality

`backend/tests/_gate/test_semantic_regex_ban.py` blocks new changed-line identifiers with suffixes like `_NOUNS`, `_TOKENS`, `_KEYWORDS`, `_PATTERNS`, etc. It is a good forward gate, but it is explicitly changed-lines only. It does not prove legacy is gone.

---

## 3. Production Code Findings

Severity meanings:
- **BLOCKING migration target**: runtime open-world semantic judgment remains in code.
- **IMPORTANT migration target**: similar risk, but either narrower scope or already has a structured path that can replace it.
- **SPECIAL**: regex/list exists but belongs to safety, segmentation, or closed mechanical parsing; do not remove blindly.
- **ALLOWED**: closed-world system contract.

| ID | Severity | File / lines | Current shape | Why it violates the rule | Migration direction |
|---|---|---|---|---|---|
| C1 | BLOCKING | `backend/app/core/steps/render_prompt_card.py:326-357` | `_STORY_CRITICAL_CATEGORY_GROUPS` and noun tuples for photo/frame/document/map/key. | Story-critical prop status is open-world. New prop forms like snapshot, polaroid, memo, talisman, memorial tablet will silently miss. | Area B: LLM SOT field such as `reference_binding_required` / `story_criticality`, emitted upstream. |
| C2 | BLOCKING | `backend/app/core/steps/render_prompt_card.py:1706-1835` | `_noun_matches_text()` + prop id/name/category noun fallback. | Code decides prop relevance from free-text substring/word-boundary. This is exactly the noun-list failure mode the user rejected. | Area B/D: required prop refs should be emitted from structured SOT; resolver should attach required refs by SOT, not text similarity. |
| C3 | BLOCKING | `backend/app/core/steps/render_prompt_card.py:150-160`; `backend/app/core/visible_entities_validator.py:230-239` | reproduction surface list and downstream prompt substring exemption. | "photo/poster/painting/monitor/mirror/projection" is an open-world visual-surface category. | Area C: reuse Area A `directionality_class` or a dedicated LLM-emitted `depiction_surface` field. |
| C4 | BLOCKING | `backend/app/core/steps/render_prompt_card.py:376-478` | spatial/camera/framing keyword tuples for low/high/hip/frame-edge/shared-anchor/close/wide. | Spatial reasoning from keyword lists is brittle and language-bound. | Area E: move spatial classifications to LLM structured SOT, likely `shot_staging` or `scene_camera_flow` fields. |
| C5 | BLOCKING | `backend/app/modules/pipeline/shot_visibility.py:37-132`, `190-224`, `245-252` | Korean gaze stems, body-part nouns, directional tails, close-up markers, offscreen regex. | Off-camera / gaze-target visibility is open-world shot semantics. This is high risk because it changes visible entity lists. | Area G: extend `shot_director` / `shot_staging` with explicit `off_camera_entity_ids`, `gaze_subject_id`, `gaze_target_id`, `visibility_reason`. |
| C6 | IMPORTANT | `backend/app/core/ref_contract_validator.py:47-118`, `151-153` | token buckets classify `"from the reference"` as character/object/background. | Reference role is semantic when inferred from prompt text. | Area F: consume attached_meta / required_refs structured SOT only; if missing, fail-fast or legacy pass-through. |
| C7 | IMPORTANT | `backend/app/services/scene_reference_service.py:50-57`, `91-100` | live/person token regex in `keep_elements`. | Whether a free-text keep element contains a person is open-world. Current rule is conservative but still lexical. | Area D: require upstream `keep_elements` entries to carry `entity_type` / `is_environment_only` enum/bool. |
| C8 | BLOCKING | `backend/app/services/scene_reference_service.py:549-581` | prop attach by `P##`, prop name substring, or word-boundary. | Prop-reference binding still falls back to free-text name matching. | Area D after Area B: attach required prop refs by structured `required_refs`, not text matching. |
| C9 | IMPORTANT | `backend/app/core/steps/scene_consistency_step.py:65-80`, `103-120` | close/full framing classification from element id suffix and description keywords. | Fixed-element framing is visual semantics; element-id suffixes and description keywords are not robust SOT. | Add to Area E or create Area K: `fixed_element.framing_class` emitted by LLM schema. |
| C10 | IMPORTANT | `backend/app/core/steps/location_consistency_step.py:70-106` | location-scene mapping falls back to name substring if `scene_director.present_entity_ids` missing. | Location presence is open-world scene semantics. There is already an ID-based path, but fallback can silently mis-map. | Prefer fail-fast when `present_entity_ids` is unavailable for new checkpoints; legacy pass-through only. |
| C11 | SPECIAL | `backend/app/modules/llm/safety.py:26-73`, `160-175`, `230-235` | safety replacement tables and safety-message keywords. | This is policy rewrite / moderation routing, not normal visual semantics. Removing blindly can increase failures. | Area I: measure-based migration. Keep until a semantic safety contract replaces it. |
| C12 | SPECIAL | `backend/app/modules/pipeline/background_chain_render.py:44-48`, `175-177` | moderation error keyword detector. | This classifies exception text, not scene meaning. | Track separately. Not a visual semantic blocker. |
| C13 | SPECIAL | `backend/app/modules/pipeline/scene_extractor_v2.py:130-136`, `242-247`, `363-365`; `backend/app/core/steps/scene_steps.py:60-111` | scene segmentation regex fallback and LLM-generated regex. | This is script-structure parsing. It is semantically broad, but outside current visual QA migration. | Separate segmentation reliability review if user wants absolute no-regex beyond visual exceptions. |
| C14 | ALLOWED | `backend/app/core/visible_entities_validator.py:59-74`, `718-747`; `backend/app/services/export_service.py:49-51`; `backend/app/services/scene_checkpoint_loaders.py:90` | ID shape regex and P## presence checks. | Closed-world ID contract. | Keep. |
| C15 | ALLOWED | `backend/tests/_gate/test_semantic_regex_ban.py:35-59`, `131-156`, `196-218` | static gate scans identifiers. | Meta-test for preventing new heuristic lists, not runtime scene judgment. | Keep; consider adding a full legacy audit mode separately. |

---

## 4. Prompt Findings

### 4.1 Active Prompt Packs

Prompt loader resolves latest version by module pack and stem. Relevant latest packs on this checkout:

| Module | Active version / evidence | Classification |
|---|---|---|
| `shot_staging` | `9.202605121441`; schema has required `directionality_class` at `schema.json:58-64`. | Area A fixed, but residual example lists remain elsewhere in the prompt. |
| `scene_detail` | `21.202605062217`; prompt has close-framing keyword contract at `system.md:151-153`. | Area E. |
| `shot_validator` | `5.202605081700`; visible-human-action and motion/contact categories at `system.md:36-41`, `111-116`, `157-161`. | Area H / possible upstream SOT expansion. |
| `t2i_review` | `3.202605121200`; detection patterns at `scene_system.md:31-39`, `49-74`. | Area H / possible structured review issue types. |
| `shot_director` | `4.202605121200`; off-camera/gaze-target pattern prompt at `system.md:15-32`. | Missing from current umbrella; should be Area G/H. |
| `entity_extractor_v2` | `8.202605121200`; extraction/detail prompts have list-like inclusion/exclusion heuristics. | Add to Area B/D/H depending field. |
| `prompt_sanitizer` | `2.202605112104`; strategy prompt remains policy-level. | Area I / B-min already partially addressed via semantic override in code. |
| `visual_world_rules` | `6.202605021400`; mostly structured meta-rules and anti-scenario-summary examples. | Mostly OK; examples are non-exhaustive and scenario-abstract. |

### 4.2 Active Prompt Blockers / Risks

| ID | Severity | File / lines | Current shape | Why it matters | Migration direction |
|---|---|---|---|---|---|
| P-A | BLOCKING | `prompts/_base/shot_director/4.202605121200/system.md:15-32` | Gaze-target/off-camera/blocking/reaction-only "patterns". | It asks the LLM to reason, but as pattern rules. This directly affects `visible_entity_ids`. It is the prompt counterpart of `shot_visibility.py`. | Area G/H: replace pattern prose with explicit structured SOT fields and meaning-based instructions, not pattern templates. |
| P-B | BLOCKING | `prompts/_base/scene_detail/21.202605062217/system.md:151-153` | close/wide framing keyword lists and rules. | Close-framing drives ref-skipping and visibility behavior. Keyword-driven semantics keep leaking into downstream contracts. | Area E: `framing_scale` should be LLM structured SOT from staging/card, not prose keyword list. |
| P-C | IMPORTANT | `prompts/_base/t2i_review/3.202605121200/scene_system.md:31-39`, `49-74` | "검출 패턴" for close-framing ref leak, physical inconsistency, fg/bg shared anchor. | This is an LLM judge, so less dangerous than code regex, but the prompt still primes closed-list detection. | Area H: rewrite as non-exhaustive principles and require issue type reasoning from structured fields where possible. |
| P-D | IMPORTANT | `prompts/_base/shot_validator/5.202605081700/system.md:36-41`, `111-116`, `157-161` | movement/contact/body-part visible-human-action categories. | It is LLM-side, but hard lists can under-cover new languages/actions. | Area H or new `shot_validator` SOT: ask for explicit `motion_state`, `visible_human_action`, `entity_mapping_confidence` fields. |
| P-E | IMPORTANT | `prompts/_base/entity_extractor_v2/8.202605121200/system.md:20-30`; `turn2.md:7-12`; `turn4.md:5-6`; `turn1_7_detail_batch.md:13-18` | variation/prop/detail inclusion/exclusion examples and photo/painting/poster inner-person clause. | Entity extraction is an LLM step, so lists are not runtime code, but these lists define what becomes structured SOT. Under-coverage here propagates everywhere. | Add structured fields like `visual_reference_role`, `depicts_person`, `story_role`, `state_scope`. |
| P-F | MINOR | `prompts/_base/shot_staging/9.202605121441/system.md:22`, `82-92`, `145-150`, `188-196` | example lists remain, but several say "참고" or define broad style. | Area A directionality block is clean; remaining lists can still prime output but are not used as downstream classifier lists. | Prompt hygiene pass: mark all as non-exhaustive examples or convert to principles. |
| P-G | SPECIAL | `prompts/_base/prompt_sanitizer/2.202605112104/sanitize_system.md:10-20` | safety strategy list. | Policy rewrite domain. Patch B-min added semantic constraints in code, but system prompt still has generic rewrite tendencies. | Area I / B-next: make semantic override first-class in sanitizer prompt pack. |
| P-H | OK | `prompts/_base/visual_world_rules/6.202605021400/system.md:67-80` | good/bad examples for generic meta rules. | The prompt explicitly forbids scenario summary and uses abstract examples. | Keep; no immediate migration required. |

### 4.3 Archive / Non-Active Residue

These are not active under latest prompt pack resolution, but they remain in `prompts/_base` and can confuse future copy-forward work:
- `prompts/_base/shot_director/1`, `/2`, `/3` contain old scenario names and pattern examples.
- `prompts/_base/shot_essence_extraction/1`, `/2` contain scenario-specific examples with `민숙`, `수리영`.
- `prompts/_base/scene_consistency/4` contains scenario-specific example text.
- `prompts/_base/visual_world_rules/5` contains old bad examples with real project names; latest v6 has generic replacements.

Recommendation: do not treat archives as runtime blockers, but add an archive-copy-forward warning in prompt maintenance docs.

---

## 5. Area A Verification

Area A should remain the reference implementation pattern.

Verified good:
- LLM schema emits `directionality_class`: `prompts/_base/shot_staging/9.202605121441/schema.json:58-64`.
- Prompt asks for meaning-based classification and warns not to classify by object names: `prompts/_base/shot_staging/9.202605121441/system.md:164-177`.
- Producer validator checks only enum class and orientation emptiness: `backend/app/modules/pipeline/shot_staging.py:25-42`.
- Error raise is outside the broad `try/except`: `backend/app/modules/pipeline/shot_staging.py:208-245`.
- Consumer directive maps enum to template without element-string classification: `backend/app/core/steps/detail_steps.py:847-873`.
- Enum drift test exists: `backend/tests/_gate/test_orientation_enum_alignment.py:16-25`.

Residual caveat:
- `shot_staging` prompt still has general example lists for physical elements, body pose, framing, and scale at `system.md:22`, `82-92`, `145-150`, `188-196`. They are not the Area A directionality classifier anymore, but should be cleaned or marked more strongly as non-exhaustive principles in Area H.

---

## 6. Gap Against Current Umbrella Spec

The current umbrella spec (`docs/superpowers/specs/2026-05-12-llm-structured-sot-migration-design.md`) is correct but incomplete.

### 6.1 Already Covered

The umbrella covers:
- C1/C2 story-critical prop noun groups and gate.
- C3 reproduction surface.
- C4 spatial keyword rules.
- C5 shot_visibility.
- C7 ref_contract token bucket.
- C8/C9 scene_reference_service keep/person and prop attach.
- P1/P2/P3/P4 core prompt issues.
- C10 safety special migration.
- C6 scenario residue cleanup.

### 6.2 Missing Or Under-Specified

Add these to the umbrella inventory. The `U*` IDs below are audit-local addenda
to avoid colliding with the existing umbrella `C* / P*` IDs.

| Audit addendum | Location | Why add |
|---|---|---|
| U1 | `backend/app/core/steps/scene_consistency_step.py:65-120` | close/full fixed-element framing classification from element id and description keywords. |
| U2 | `backend/app/core/steps/location_consistency_step.py:70-106` | location-scene presence has an ID path but still uses name substring fallback. |
| U3 | `backend/app/modules/pipeline/scene_extractor_v2.py:130-136`, `242-247`, `363-365`; `backend/app/core/steps/scene_steps.py:60-111` | scene segmentation regex fallback / LLM-generated regex; outside visual QA but still "regex semantic" if the principle is expanded globally. |
| U4 | `prompts/_base/shot_director/4.202605121200/system.md:15-32` | active prompt contains off-camera/gaze-target pattern rules; pair with Area G. |
| U5 | `prompts/_base/entity_extractor_v2/8.202605121200/*` | entity extraction prompt examples define the SOT upstream, so list-like restrictions matter. |
| U6 | `prompts/_base/shot_staging/9.202605121441/system.md:22`, `82-92`, `145-150`, `188-196` | not Area A blocker, but active prompt still has example lists that can prime outputs. |

### 6.3 Gate Gap

Gate 1 is useful but limited:
- It scans only `backend/app/**/*.py` changed lines.
- It detects identifier names with suffixes like `_NOUNS`, `_TOKENS`, `_KEYWORDS`, `_PATTERNS`, `_STEMS`, `_TAILS`, `_MARKERS`.
- It does not detect inline local lists without those suffixes.
- It does not scan active prompts.
- It does not remove legacy.

Recommendation: add two follow-up gates:
1. **Active Prompt Closed-List Gate**: resolve active prompt packs, then scan changed prompt lines for headings like "검출 패턴", "판정 기준", "키워드", "다음 패턴", unless adjacent text explicitly says non-exhaustive and meaning-based.
2. **Full Legacy Audit Gate**: non-blocking report mode over all production code, producing the table in this document so the migration inventory cannot drift.

---

## 7. Recommended Migration Order

The previous order remains mostly right, but insert audit/active-prompt hygiene more explicitly.

| Order | Area | Scope | Rationale |
|---|---|---|---|
| 0 | Audit freeze | This document + umbrella inventory update. | Prevents more "we did not know this existed" loops. |
| 1 | Area C | reproduction surface using Area A SOT. | Small, directly depends on Area A; removes a known list. |
| 2 | Area B | story-critical prop SOT. | Big but central; removes C1/C2 and prepares resolver cleanup. |
| 3 | Area D | scene_reference_service prop attach / keep_elements. | Needs Area B structured required refs. |
| 4 | Area H-small | active prompt hygiene for shot_director, shot_validator, t2i_review, shot_staging examples. | Can run between larger code patches; reduces LLM priming. |
| 5 | Area E | spatial/close framing SOT. | Large and interconnected with scene_detail/render_prompt_card. |
| 6 | Area F | ref_contract token bucket. | Lower priority once required refs/attached_meta are stronger. |
| 7 | Area G | shot_visibility. | High operational risk; should be last or carefully staged with explicit SOT fields. |
| 8 | Area I | safety/prompt_sanitizer special migration. | Requires measurement; do not remove token safety blindly. |
| 9 | Area J | scenario residue cleanup. | Can be done anytime; mostly comments/archive hygiene. |
| Separate | Scene segmentation | regex fallback / LLM-generated regex. | Outside visual QA unless user wants global no-regex semantics. |

---

## 8. Immediate Decisions Needed

1. Update umbrella spec inventory with U1-U6 above.
2. Decide if scene segmentation regex is inside or outside the "무조건 LLM" principle. It is not visual QA, but it is still semantic segmentation.
3. Decide whether active prompt hygiene (Area H-small) should happen before Area C. It is cheaper and prevents more list priming.
4. Add active-prompt gate before next production patch.

---

## 9. Bottom Line

No, the repo is not yet free of regex/hardcoded semantic handling. Area A fixed one important class correctly, but legacy open-world semantic heuristics remain in `render_prompt_card`, `visible_entities_validator`, `shot_visibility`, `ref_contract_validator`, `scene_reference_service`, `scene_consistency_step`, active prompts, and entity extraction prompts.

The right path is not another noun-list patch. The right path is to migrate each legacy branch to explicit LLM-produced structured SOT, with fail-fast or legacy pass-through when the field is missing.