j*ddlZddlmZddlmZmZmZmZmZm Z ddl m Z ddl m Z mZ ddlmZGdde j&Zed edfd ZGd d eZGd deZy#e$rdZYCwxYw)N)contextmanager)AnyIteratorListOptionalTupleUnion)MetricComputationError) base_metric score_result) bleu_scorec peZdZdZdededededeee deef fd Z dd Z d ed e e d ffd Z xZS)BaseBLEUa Base class containing shared BLEU logic, such as handling n-grams, smoothing, and weights initialization. This class is not intended to be used directly. References: - BLEU: Papineni et al., "BLEU: a Method for Automatic Evaluation of Machine Translation" (ACL 2002) https://aclanthology.org/P02-1040/ - NLTK BLEU documentation https://www.nltk.org/api/nltk.translate.bleu_score.html - Hugging Face Evaluate: BLEU metric overview https://huggingface.co/spaces/evaluate-metric/bleu Args: name: The name of the metric (e.g. "sentence_bleu_metric" or "corpus_bleu_metric"). track: Whether to track the metric (depends on your system). n_grams: Up to which n-gram order to use (1 through n_grams). smoothing_method: One of NLTK's SmoothingFunction methods (e.g. "method0", "method1", etc.). weights: Optional custom weights for n-gram orders. Must sum to 1.0. If None, defaults to uniform distribution across `n_grams`. project_name: Optional project name to track the metric in for the cases when there are no parent span/trace to inherit project name from. nametrackn_gramssmoothing_methodweights project_namecft||||t td||_||_|d|z g|z|_nUt||k7rtdt|d|dtt|dz dkDr td||_tj|_ y) N)rrrzV`nltk` library is required for BLEU score calculation. Install via `pip install nltk`.?zLength of weights (z) != n_grams (z).gư>zWeights must sum to 1.0) super__init__nltk_bleu_score ImportErrorrrrlen ValueErrorabssumSmoothingFunction _smootherselfrrrrrr __class__s /Users/manta/Documents/Projects/TheRoad-I1/backend/.venv/lib/python3.12/site-packages/opik/evaluation/metrics/heuristics/bleu.pyrzBaseBLEU.__init__&s d%lK  "2   0 ?'M?W4DL7|w& )#g,~gYbQ3w<#%&- !:;;"DL(::<returnclt|j|j|jjSN)getattrr!rmethod0)r#s r%_get_smoothing_funczBaseBLEU._get_smoothing_funcGs%t~~t'<'T>TUUr&max_len.ct|j|}|jd|}t|xsd}|Dcgc]}||z  }}t |Scc}w)Nr)minrrrtuple)r#r- used_order used_weightstotalw normalizeds r%_truncate_weightszBaseBLEU._truncate_weightsJsYw/ ||KZ0 L!(S)56Aa%i 6Z  7s A)r'z!nltk_bleu_score.SmoothingFunction)__name__ __module__ __qualname____doc__strboolintrrfloatrr,rr6 __classcell__r$s@r%rrsw.=== =  = $u+& =sm=BV!!ucz1B!r&rr'c#Ktj5tjddtdddddy#1swYyxYww)Nignorez5The hypothesis contains 0 counts of 2-gram overlaps\.znltk\.translate\.bleu_score)messagecategorymodule)warningscatch_warningsfilterwarnings UserWarningr&r%_suppress_bleu_warningsrKRs<  " L 2   # " "sA"A AA AceZdZdZ ddededededeee deef fd Z d ed e eeefd e d e jfd ZxZS) SentenceBLEUa Computes sentence-level BLEU for a single candidate string vs. one or more references. Example: >>> from opik.evaluation.metrics.heuristics.bleu import SentenceBLEU >>> metric = SentenceBLEU(n_grams=4, smoothing_method="method1") >>> result = metric.score("the cat is on the mat", "the cat is on the mat") >>> print(result.value) 1.0 rrrrrrc0t|||||||yN)rrrrrrrrr"s r%rzSentenceBLEU.__init__j) -%  r&output referenceignored_kwargsr'c |js tdt|tr;|js td|j j g}nQg}|D]J}|js td|j |j j L|j j }|jt|}|j} t5tj||||} dddtj |j d|j"d| d  S#1swY//1--c2B.CD 113 (**88$('5 +''4T5J5J4K3xX[n]   +*! H s* E, E %E, E)%E,, E:9E:)sentence_bleu_metricTmethod1NNr7r8r9r:r;r<r=rrr>rr rr rerkr?r@s@r%rMrM^s + ))-&*       $u+&  sm $@ @ d3i(@  @  ! ! @ r&rMceZdZdZ ddededededeee deef fd Z d eed ee eeefd e d e jfd ZxZS) CorpusBLEUa Computes corpus-level BLEU for multiple candidate strings vs. matching references. Each element in `output` corresponds to one candidate. The parallel `reference` element can be either a single string or a list of reference strings for that candidate. Example: >>> from opik.evaluation.metrics.heuristics.bleu import CorpusBLEU >>> metric = CorpusBLEU(n_grams=4, smoothing_method="method1") >>> outputs = ["the cat is on the mat", "there is a cat here"] >>> references = [ ... "the cat is on the mat", ... ["there is a cat here", "there is cat here"] ... ] >>> result = metric.score(outputs, references) >>> print(result.value) rrrrrrc0t|||||||yrOrPr"s r%rzCorpusBLEU.__init__rQr&rRrSrTr'c t|t|k7r tdg}g}t||D]\}}|js td|j j }t |tr;|js td|j j g} nQg} |D]J} | js td| j| j j L|j||j| td|D} |j| } |j} t5tj||| | }dddt!j"|j$d |j&d |d  S#1swYz#CorpusBLEU.score..sEndD nsrVNrXz Corpus-level BLEU (nltk, method=rYrZr[)rr zipr^r`rar_r;rbmaxr6r,rKr corpus_bleurdr rerr)r#rRrSrTall_candidatesall_references candidate_strref_itemrhrfr_linemax_candidate_lenr2rirjs r%rkzCorpusBLEU.scores2 v;#i. ((W +-02'*69'= #M8 &&(,-PQQ,224::< (C(~~'01TUU%^^-3356  &F!<<>4H$$V\\^%9%9%;< '  ! !"2 3  ! !) ,-(>0 EnEE--.?@ 113 (**66""('5 +''243H3H2IXVYN[   +*! H s*& G0G GG G GG)corpus_bleu_metricTrmrnNNror@s@r%rqrqs() ))-&*       $u+&  sm $N S N c49n-.N  N  ! ! N r&rq)rF contextlibrtypingrrrrrr opik.exceptionsr opik.evaluation.metricsr r nltk.translater rr BaseMetricrrKrMrqrJr&r%rs%>>2=< A!{%%A!H$^ 8^ Bs s kOsA**A43A4