jS+ddlmZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZeege efZGdde Zdd Zdd ZGd d ej0ZGd deZGddeZGddeZy)) annotationsN)Counter)AnyCallableDictIterableListOptionalProtocolSequence)MetricComputationError) base_metric score_resultc&eZdZ d ddZy) _JSDistanceFncyN)selfpqbases /Users/manta/Documents/Projects/TheRoad-I1/backend/.venv/lib/python3.12/site-packages/opik/evaluation/metrics/heuristics/distribution_metrics.py__call__z_JSDistanceFn.__call__sN).)rSequence[float]rrrzOptional[float]returnfloat)__name__ __module__ __qualname__rrrrrr s2      rrcN ddlm}|S#t$r}td|d}~wwxYw)Nr) jensenshannonzDInstall scipy via `pip install scipy` to use Jensen-Shannon metrics.)scipy.spatial.distancer# ImportError)r#errors r_load_jensen_shannon_distancer's78   R  s $ $c>|jjSr)lowersplit)texts r_default_tokenizerr,#s ::<   rcReZdZdZ d dfd ZddZddZxZS) _DistributionMetricBasea Internal helper for metrics that compare token distributions. Args: tokenizer: Optional tokenizer returning an iterable of tokens given text. name: Display name for the metric. track: Whether to automatically track metric results. project_name: Optional tracking project. normalize: When ``True`` the histogram is converted to probabilities. smoothing: Optional additive constant applied during KL-like computations. cxt|||||xst|_||_t d||_y)N)nametrack project_name)super__init__r, _tokenizer _normalizemax _smoothing)r tokenizerr0r1r2 normalize smoothing __class__s rr5z _DistributionMetricBase.__init__4s; d%lK#9'9#c9-rct|j|}t|dk(r tdt |}|j s-|j Dcic]\}}|t|c}}Stt|j}|j Dcic] \}}|||z  c}}Scc}}wcc}}w)Nrz4Tokenized text is empty (distribution-based metric).) listr6lenr rr7itemsrsumvalues)rr+tokenscountstokencounttotals r_build_distributionz+_DistributionMetricBase._build_distributionBsdood+, v;! (F  ``1.0``). normalize: Whether to normalise token counts to probabilities first. name: Display name for the metric result. track: Whether to automatically track metric results. project_name: Optional tracking project name. Note: Requires :mod:`scipy` to be installed. Example: >>> from opik.evaluation.metrics import JSDivergence >>> metric = JSDivergence() >>> result = metric.score( ... output="cat cat sat", ... reference="cat sat on mat", ... ) >>> round(result.value, 3) # doctest: +SKIP 0.812 cz|dkr tdt| |||||||_t |_y)N?zbase must be greater than 1.0)r:r0r1r2r;) ValueErrorr4r5_baser'_js_distance_fnrr:rr;r0r1r2r=s rr5zJSDivergence.__init__psM 3;<= = %   <>rc |js td|js td|j|}|j|}|j||}t dt dd|z }t j||jd|jdd|dd |dd |tj||jd  S) Nz/Candidate is empty (Jensen-Shannon divergence).z/Reference is empty (Jensen-Shannon divergence).r3rZz Jensen-Shannon similarity (base=g): .4fz (divergence=)) divergencedistancerrLr0reasonmetadata) stripr rI_js_divergencer8minr ScoreResultr0r\mathsqrt)routput referenceignored_kwargs output_distreference_distrdscores rrtzJSDivergence.scores ||~(A  (A ..v6 11)<((nE CS# "234''24::a.E#;O)#.a1) IIj1   rcttt|t|z}|sy|Dcgc]}|j|d}}|Dcgc]}|j|d}}|j|}|j|}t |j |||j } | dzScc}wcc}w)Nr3)r)sortedsetget_ensure_probability_vectorrr]r\) rp_distq_dist vocabularyrFp_vectorq_vectorp_probsq_probsres rrjzJSDivergence._js_divergences CK#f+56 8BC uFJJuc* C8BC uFJJuc* C11(;11(;--gwTZZ-PQ{DCs B0B5cht|}|dkr td|Dcgc]}||z  c}Scc}w)Nr3zADistribution is empty after tokenisation (Jensen-Shannon metric).)rBr )rrCrHrLs rrzz'JSDivergence._ensure_probability_vectors@F  C<(S ,226% 6222s /)N@Tjs_divergence_metricTNr:rNrrr;rPr0rOr1rPr2rQrrRrorOrprOrqrrzscore_result.ScoreResultr{rSr|rSrr)rCrrz List[float]) rr r!rTr5rtrjrzrUrVs@rrXrXVs6+/*&*?'?? ?  ?  ?$? ?*! ! !  ! " ! F !  $3rrXcheZdZdZ d dfd Z dfd ZxZS) JSDistanceu Return the raw Jensen–Shannon divergence instead of similarity. Args: tokenizer: Optional tokenizer function. base: Logarithm base used for the divergence calculation. normalize: Whether to normalise counts into probabilities. name: Display name for the metric result. track: Whether to automatically track metric results. project_name: Optional tracking project name. Example: >>> from opik.evaluation.metrics import JSDistance >>> metric = JSDistance() >>> result = metric.score("a a b", reference="a b b") >>> round(result.value, 3) # doctest: +SKIP 0.188 c0t|||||||y)N)r:rr;r0r1r2)r4r5r^s rr5zJSDistance.__init__s) %  rc |t|||}|jxsi}t|j dd}t|j dt j |}tj||jd|jdd|d||j d |jd  S) N)rorprdr3rez Jensen-Shannon divergence (base=r`rarbr)rerrf) r4rtrhrryrmrnrrlr0r\) rrorprq similarityrhrdrer=s rrtzJSDistance.scores W]&I]F &&,"8<< c:; j$))J2GHI''5djj^3zRUFVW$ VTZZ8   r)NrTjs_distance_metricTNrr)rr r!rTr5rtrUrVs@rrrs*+/(&* '        $   $     "   rrcreZdZdZ d dfd Z ddZddZxZS) KLDivergenceac Compute the (optionally symmetric) KL divergence between token distributions. Args: tokenizer: Optional tokenizer function. Defaults to whitespace split. direction: Direction to compute (``"pq"``, ``"qp"``, or ``"avg"`` for symmetric). normalize: Whether to normalise token counts to probabilities first. smoothing: Additive smoothing constant to avoid divide-by-zero. name: Display name for the metric result. track: Whether to automatically track metric results. project_name: Optional tracking project name. Example: >>> from opik.evaluation.metrics import KLDivergence >>> metric = KLDivergence(direction="avg") >>> result = metric.score("hello hello world", reference="hello world") >>> round(result.value, 4) # doctest: +SKIP 0.0583 c\|dvr tdt| ||||||||_y)N>pqqpavgz,direction must be one of {'pq', 'qp', 'avg'})r:r0r1r2r;r<)r[r4r5 _direction) rr: directionr;r<r0r1r2r=s rr5zKLDivergence.__init__sG / /KL L %  $rc |js td|js td|j|}|j|}|jdk(r|j ||}nJ|jdk(r|j ||}n(d|j |||j ||zz}t j ||jd|jd|d S) Nz*Candidate is empty (KL divergence metric).z*Reference is empty (KL divergence metric).rrg?zKL divergence (rarb)rLr0rg)rir rIr_klrrlr0)rrorprqr{r|rds rrtzKLDivergence.score*s ||~()UV V ()UV V))&1)))4 ??d "&&1J __ $&&1J 8488FF;S STJ''$T__$5SC8HI  rcd}|jD]U\}}|j|}|j|j|d}||tj||z zz }W|SrK)rArMryrmlog)rr{r|rdrFp_valq_vals rrzKLDivergence._klEsf "LLNLE5LL'ELLE3!78E %$((55="99 9J+r)NrTg-q=kl_divergence_metricTN)r:rNrrOr;rPr<rr0rOr1rPr2rQrrRrr)rr r!rTr5rtrrUrVs@rrrs.+/ *&*$'$$ $  $  $$$$ $,     "  6rr)rr)r+rOrz Iterable[str]) __future__rrm collectionsrtypingrrrrr r r r opik.exceptionsr opik.evaluation.metricsrrrO TokenizeFnrr'r, BaseMetricr.rXrrrrrrs" TTT2= seXc]* + H ,'k44,'^j3*j3Z8 8 vM*Mr