j UdZddlmZddlZddlZddlZddlmZmZ ddl Z ee ge fZ dddddd dd Z dd Zdd Zdd Zdd Ze Zded<ddZeZded<y#e $rdZ YLwxYw)z0Shared text preprocessing utilities for metrics.) annotationsN)CallableLiteralTFNFKC) lowercase strip_accentsremove_punctuation keep_emojinormalize_formctj||}|r|j}|s t|}|r t |}|r t |}t |}|jS)aNormalize text before metric processing. Args: text: Input string. lowercase: Whether to lowercase the text. strip_accents: Remove diacritical marks. remove_punctuation: Strip ASCII punctuation. keep_emoji: Preserve emoji characters; if False they are removed. normalize_form: Unicode normalization form to apply (default NFKC). ) unicodedata normalizelower _remove_emoji_strip_accents_remove_punctuation_collapse_whitespacestrip)textrrr r r normalizeds v/Users/manta/Documents/Projects/TheRoad-I1/backend/.venv/lib/python3.12/site-packages/opik/evaluation/preprocessing.pynormalize_textrsg(&&~tSkSoNr category.0chs r z _remove_emoji..9s& 2!5!5b!9!MBs&&)replace)emojijoin replace_emojirs rrr7s: }ww       tR 00rc^tjd|}djd|DS)NNFDrc3RK|]}tj|dk7s|!yw)MnNrr!s rr$z!_strip_accents..As#O "k.B.B2.F$.N2 s'')r rr')r decomposeds rrr?s)&&ud3J 77O O OOrctjddtj}|j |}t j dd|S)Nrz6[\u2010-\u2015\u2018-\u201f\u2020-\u2027\u2030-\u2043])str maketransstring punctuation translateresub)r translatorstrippeds rrrDs@r2v'9'9:J~~j)H 66A2x rc0tjdd|S)Nz\s+ )r5r6r)s rrrLs 66&#t $$r _NormalizerDEFAULT_NORMALIZERc t|dddS)NTF)rr r )rr)s rascii_normalizerr>Ss    rASCII_NORMALIZER)rr0rboolrr@r r@r r@r z%Literal['NFC', 'NFD', 'NFKC', 'NFKD']returnr0)rr0rAr0)__doc__ __future__rr5r2r typingrrr& ImportErrorr0r;rrrrrr<__annotations__r>r?rrrHs6" $ucz" $