j-| ddlZddlZddlmZmZmZmZmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZddlmZddlmZdd lmZdd lmZdd l mZdd lmZd dlm Z m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'm(Z(d dl)m*Z*m+Z+m,Z,d dl-m.Z.m/Z/d dlm0Z0d dlm1Z1m2Z2m3Z3ddlm4Z4ddl5m6Z6e rdd l5mZ7ejpe9Z:dZ;dZd7ejd ejd&ejzd'e2d,ee@d.e?d/e?d8ee@deee@dee@de e"jeSffd?ZT dMd+e@d(ee'jd)eee.jd@e?d.e?d2ee3dAee@d"eee1de"jfdBZUdCe+jdDeee@efdeee@efgee@efffdEZW dNd&e ejzej|ejfdDeee@efdCee e@e+jfd(eee'jd)eee.jd*ee@d+ee@d,ee@d-eee@efd.e?dee?d/e?d0eejdeee@dee%jd3e?d"eee1d4eee@dee@de"jf(dFZX dLd9e@d&e ejzej|ejfd'e2d(eee'jd)eee.jd*ee@d+ee@d,ee@d-eee@efd.e?dee?d/e?d0eejd1eeejd2ee3deee@dee%jd3e?d"eee1d4eee@dee@de"jf,dGZY dOdHeee@efd'e2d(eee'jd)eee.jd,ee@d.e?d2ee3d@e?de"jfdIZ[d)eee.jd(eee'jd,ee@dee'jfdJZ\d+ee@d*ee@dee@fdKZ]y)PN) AnyCallableDictIteratorListOptionalTuple TYPE_CHECKINGUnioncast) base_prompt) opik_client)dataset experiment) dataset_item)helpers)execution_policy)chat_prompt_template)types)evaluation_suite)asyncio_supportengineevaluation_resultreportrest_operationssamplers) base_metric score_result)ModelCapabilities base_modelmodels_factory)scorer_functionscorer_wrapper_metric) test_result)ExperimentScoreFunctionLLMTaskScoringKeyMappingType) url_helpers)suite_result_constructorz>https://www.comet.com/docs/opik/evaluation/evaluate_multimodaldataset_ nb_samplesdataset_item_idsreturnc| t|S|$|jt||jS|S|jS)z;Calculate the total number of items that will be evaluated.)lendataset_items_countminr-r.r/s r/Users/manta/Documents/Projects/TheRoad-I1/backend/.venv/lib/python3.12/site-packages/opik/evaluation/evaluator.py_calculate_total_itemsr74sM ##$$  ' ' 3z8#?#?@ @  ' ''dataset_samplerdataset_filter_stringc|+|j||t|}t|||}||fStj dt |j||t|}|j |}t|t|fS)z Resolve dataset items for evaluation. Handles streaming vs sampling, and calculates total item count. Returns: Tuple of (items iterator, total item count or None). r.r/ batch_size filter_stringr5z)Dataset streaming disabled due to sampler) -__internal_api__stream_items_as_dataclasses__$EVALUATION_STREAM_DATASET_BATCH_SIZEr7LOGGERinfolistsampleiterr2)r-r.r/r9r: items_itertotal items_lists r6_resolve_dataset_itemsrIEsKK!-;/ L '!-  5   KK;<>>!-;/ ? J!'' 3J  S_ ,,r8rc |jj|jgy#t$r%tj d|jdYywxYw)N)idszKFailed to notify backend about the experiment completion. Experiment ID: %sTexc_info)experiments_rest_clientfinish_experimentsid ExceptionrAdebug)rs r6*_try_notifying_about_experiment_completionrSosQ **==:==/=R   Y MM   s'*+AAexperiment_scoring_functions test_resultsc|r|sgSg}|D]>} ||}t|tr|j|n|j|@|S#t$r"}t j d|dYd}~id}~wwxYw)z2Compute experiment-level scores from test results.z&Failed to compute experiment score: %sTrLN) isinstancerCextendappendrQrAwarning)rTrU all_scoresscore_functionscoreses r6_compute_experiment_scoresr_|s (| 13J6 #L1F&$'!!&)!!&)7   NN8    s;A A9A44A9rtaskscoring_metricsscoring_functionsexperiment_name_prefixexperiment_name project_nameexperiment_configverbose task_threadspromptpromptsscoring_key_mapping trial_countexperiment_tagsct|tjr |j}|gn|}t j | | }t j}t||}|j||j|||t|jdd}t|||}t|||||||| | | |||||S)u Performs task evaluation on a given dataset. You can use either `scoring_metrics` or `scorer_functions` to calculate evaluation metrics. The scorer functions doesn't require `scoring_key_mapping` and use reserved parameters to receive inputs and outputs from the task. Args: dataset: An Opik Dataset or DatasetVersion instance task: A callable object that takes dict with dataset item content as input and returns dict which will later be used for scoring. experiment_name_prefix: The prefix to be added to automatically generated experiment names to make them unique but grouped under the same prefix. For example, if you set `experiment_name_prefix="my-experiment"`, the first experiment created will be named `my-experiment-`. experiment_name: The name of the experiment associated with evaluation run. If None, a generated name will be used. project_name: The name of the project. If not provided, traces and spans will be logged to the `Default Project` experiment_config: The dictionary with parameters that describe experiment scoring_metrics: List of metrics to calculate during evaluation. Each metric has `score(...)` method, arguments for this method are taken from the `task` output, check the signature of the `score` method in metrics that you need to find out which keys are mandatory in `task`-returned dictionary. If no value provided, the experiment won't have any scoring metrics. scoring_functions: List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine: • dataset_item — a dictionary containing the dataset item content, • task_outputs — a dictionary containing the LLM task output. • task_span - the data collected during the LLM task execution [optional]. verbose: an integer value that controls evaluation output logs such as summary and tqdm progress bar. 0 - no outputs, 1 - outputs are enabled (default), 2 - outputs are enabled and detailed statistics are displayed. nb_samples: number of samples to evaluate. If no value is provided, all samples in the dataset will be evaluated. task_threads: number of thread workers to run tasks. If set to 1, no additional threads are created, all tasks executed in the current thread sequentially. are executed sequentially in the current thread. Use more than 1 worker if your task object is compatible with sharing across threads. prompt: Prompt object to link with experiment. Deprecated, use `prompts` argument instead. prompts: A list of Prompt objects to link with experiment. scoring_key_mapping: A dictionary that allows you to rename keys present in either the dataset item or the task output so that they match the keys expected by the scoring metrics. For example if you have a dataset item with the following content: {"user_question": "What is Opik ?"} and a scoring metric that expects a key "input", you can use scoring_key_mapping `{"input": "user_question"}` to map the "user_question" key to "input". dataset_item_ids: list of dataset item ids to evaluate. If not provided, all samples in the dataset will be evaluated. dataset_sampler: An instance of a dataset sampler that will be used to sample dataset items for evaluation. If not provided, all samples in the dataset will be evaluated. trial_count: number of times to run the task and evaluate the task output for every dataset item. experiment_scoring_functions: List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment. experiment_tags: Optional list of tags to associate with the experiment. dataset_filter_string: Optional OQL filter string to filter dataset items. Supports filtering by tags, data fields, metadata, etc. Supported columns include: - `id`, `source`, `trace_id`, `span_id`: String fields - `data`: Dictionary field (use dot notation, e.g., "data.category") - `tags`: List field (use "contains" operator) - `created_at`, `last_updated_at`: DateTime fields (ISO 8601 format) - `created_by`, `last_updated_by`: String fields Examples: - `tags contains "failed"` - Items with 'failed' tag - `data.category = "test"` - Items with specific data field value - `created_at >= "2024-01-01T00:00:00Z"` - Items created after date NrjrkrerdrPname dataset_namergrktagsdataset_version_idrcrbrfclientrrrarbrfrhr.rirlr/r9rmrTr:)rWrEvaluationSuiterexperiment_helpershandle_prompt_argsrget_client_cached_use_or_create_experiment_namecreate_experimentrsgetattrget_version_info_wrap_scoring_functions_evaluate_task)rrarbrcrdrerfrgrhr.rirjrkrlr/r9rmrTrnr:checked_promptsryrs r6evaluatersZ'+;;<//+28T!);;O  * * ,F4'5O )) \\+ "#;7#;#;#=tTJ *J.+'!O  '!!/)'%A3 r8)ryr/r:rdrerfrgrkrnrhrievaluator_modeloptimization_idexperiment_typeryrrrz!suite_types.EvaluationSuiteResultc <|tj}t||}t||j|| d| d}||xsd|d<||d<|j d i|}| dk\rUt j|j|j|jj }tj|t|||||| | | || \}}tj|}| dk\r.tj |j||| |j" |S) aI Run evaluation on a dataset configured as an evaluation suite. This function is designed for evaluation suites where evaluators and execution policies are stored in the dataset itself. Unlike the general `evaluate` function, this function: - Does not accept scoring_metrics (they come from the dataset) - Does not accept trial_count (it comes from the dataset's execution_policy) - Does not accept dataset_sampler or nb_samples (suites evaluate all items) Returns: EvaluationSuiteResult with pass/fail status for each item and the suite. Nrqr)rsrtrgrkevaluation_methodrurvtrialtyperr experiment_id dataset_id url_override) ryrrrarfrhrirr/r:)rhexperiment_url)rr}r~dictrsrr*get_experiment_url_by_idrPconfigrrdisplay_evaluation_in_progress_evaluate_suite_taskr+build_suite_resultdisplay_suite_resultsr)rraryr/r:rdrerfrgrkrnrhrirrrcreate_experiment_kwargs experiment_r eval_result total_time suite_results r6evaluate_suiter;s?@~..04'5O 04 \\+, 0"+:+Eg (6E !23*&**F-EFK!|$==%..zz33  --n=2 !!')3 K,>>{KL!|$$ LL  '66   r8c tj}tj5t||| | |\}}t j | | }t j||||}|j|||| d|||}dddtj|z }t| }|dk\r"tj|j|||tj|j|j|j j"}tj$||j't)||r|j*| t-j.|j|j|j||| | }|d k\r!tj0|j| |S#1swY=xYw) Nr-r.r/r9r: runs_per_itempass_thresholdryrfworkersrh dataset_itemsrarbrlrrdefault_execution_policy total_itemsrTrUrrr score_resultsrrrerUrrmexperiment_scoresr rtevaluation_results)timer)async_http_connections_expire_immediatelyrIdataset_execution_policyExecutionPolicyrEvaluationEngine run_and_scorer_rdisplay_experiment_resultsrsr*rrPrrdisplay_experiment_linkflushrSlog_experiment_scoresrEvaluationResult$display_evaluation_scores_statistics)ryrrrarbrfrhr.rirlr/r9rmrTr: start_timerFrGpolicyevaluation_enginerUrcomputed_experiment_scoresrevaluation_result_s r6rrs$J  B B D2!-+"7   E*99%&  #33%    )66$+ 3 "%+7  ' E<z)J"<%A!" !|)) LL*l4N !99 mm::]]//N  "".A LLN.z:"( ((7QR*;;:: mm"!%4!|33 1 W E Ds AGG )r/r:c tj} tj5|j|} |j} |j d|t | } |j}tj||||}|j| || d||| |d }dddtj| z }tj|j|j|jj}t!j"|j|j|j$|dg}|j't)|||fS#1swYxYw)Nr<rF) rrarbrlrrrrshow_scores_in_progress_barrrr)rrrget_evaluatorsget_execution_policyr?r@r3rrrr*rrPrrrrrsrrS)ryrrrarfrhrirr/r:rrbrrFrGrrUrrrs r6rrsXJ  B B D0'00A77779JWJJ-;/  ++"33%    )66$+ $+"%5(-7  # E:z)J 99 mm::]]//N +;;:: mm"!% LLN.z: z ))g E Ds A=EE'scoring_threadsrc |gn|}tj}tj} |r(tj d| j |} nt j| |} | j| j} t j| | |} | dj} t j| | }t||| }tj5t!j"| ||| }|j%| || }dddtj|z }t'| }|d k\r"t)j*| j,|||t/j0| j2| j2| j4j6}t)j8|t;| |r| j<|t?j@| j2| j2| j,||d |}|dk\r!t)jB| j,||S#1swY-xYw)uL Update the existing experiment with new evaluation metrics. You can use either `scoring_metrics` or `scorer_functions` to calculate evaluation metrics. The scorer functions doesn't require `scoring_key_mapping` and use reserved parameters to receive inputs and outputs from the task. Args: experiment_name: The name of the experiment to update. scoring_metrics: List of metrics to calculate during evaluation. Each metric has `score(...)` method, arguments for this method are taken from the `task` output, check the signature of the `score` method in metrics that you need to find out which keys are mandatory in `task`-returned dictionary. scoring_functions: List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine: • dataset_item — a dictionary containing the dataset item content, • task_outputs — a dictionary containing the LLM task output. • task_span - the data collected during the LLM task execution [optional]. scoring_threads: amount of thread workers to run scoring metrics. verbose: an integer value that controls evaluation output logs such as summary and tqdm progress bar. scoring_key_mapping: A dictionary that allows you to rename keys present in either the dataset item or the task output so that they match the keys expected by the scoring metrics. For example, if you have a dataset item with the following content: {"user_question": "What is Opik ?"} and a scoring metric that expects a key "input", you can use scoring_key_mapping `{"input": "user_question"}` to map the "user_question" key to "input". experiment_id: The ID of the experiment to evaluate. If not provided, the experiment will be evaluated based on the experiment name. experiment_scoring_functions: List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment. Nz5Getting experiment by id. Experiment name is ignored.)rP)ryre)rs)rr-rlr)rytrace_idrwr) test_casesrbrlrrrrrrr r)"rrr}rArBget_experiment_by_idrget_experiment_with_unique_name get_datasetrtget_experiment_test_casesrget_trace_project_namerrrrrscore_test_casesr_rrrsr*rrPrrrrSrrrr)rerbrcrrhrlrrTrryrr-rfirst_trace_idrfrrUrrrrs r6evaluate_experimentr<s5^+28T!J  * * ,F KL00M0B $DD? !!z'>'>!?H ::/J  ]++N"99L .+'!O  B B D"33%#   )99!+ 3:  Ez)J"<%A!" !|)) MM   &  !99 mm;;]]//N  "".A.z:"( ((7QR*;;;; mm"!%4!|33!1 w E Ds ,.IImodelmessagesc :ttjtjt ddtj t dddtj|dj}|Dchc]}j|ds|}}|rAdjt|}tjdt dd|tdt t"t$fd t t"t$fffd }|Scc}w) N model_name)visionvideoF)rvalidate_placeholdersz, zModel '%s' does not support %s content. Multimedia parts will be flattened to text placeholders. See %s for supported models and customization options.unknownprompt_variablesr0c|jd}j||}tj|5}||jdj j dcdddS#1swYyxYw)Nr) variablessupported_modalities template_type)model_providerrr)inputoutput)getformatr"get_provider_responsechoicesmessagecontent)rtemplate_type_overrideprocessed_messages llm_outputchat_prompt_template_rrs r6_prompt_evaluation_taskz>_build_prompt_evaluation_task.._prompt_evaluation_tasks~!1!5!5f!=299&!50:   - - += +$,,Q/77??   s &A..A7)r prompt_typesSupportedModalitiesr!supports_visionrsupports_videorChatPromptTemplaterequired_modalitiesrjoinsortedrArZMODALITY_SUPPORT_DOC_URLrstrr) rrrmodalityunsupported_modalitiesmodalities_listrrrs ` @@r6_build_prompt_evaluation_taskrs (('77|T2'55|T2   1CC0CCE,+H#''%8 + ))F+A$BC [ E< 3  $  $sCx.T#s(^ #"AsDc t|tjr |j}|gn|}t|trt j |}n't|tjs td|}|||jd}nd|vr||d<d|vr|j|d<tj}| r| gnd}t||}|j||j|||t!|j"dd }t%||| }t'j&}t)j*5t-|| | || \}}t/j0|| }t3j4||| | }|j7|t9|||dd|||}dddt'j&|z }t;|}| dk\r"t=j>|j|||tAjB|jD|jD|jFjH}t=jJ||jMtO||r|jP|tSjT|jD|jD|j||||} | dk\r!t=jV|j| | S#1swY=xYw)u3 Performs prompt evaluation on a given dataset. Args: dataset: An Opik Dataset or DatasetVersion instance messages: A list of prompt messages to evaluate. model: The name of the model to use for evaluation. Defaults to "gpt-3.5-turbo". scoring_metrics: List of metrics to calculate during evaluation. The LLM input and output will be passed as arguments to each metric `score(...)` method. scoring_functions: List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine: • dataset_item — a dictionary containing the dataset item content, • task_outputs — a dictionary containing the LLM task output. • task_span - the data collected during the LLM task execution [optional]. experiment_name_prefix: The prefix to be added to automatically generated experiment names to make them unique but grouped under the same prefix. For example, if you set `experiment_name_prefix="my-experiment"`, the first experiment created will be named `my-experiment-`. experiment_name: name of the experiment. project_name: The name of the project to log data experiment_config: configuration of the experiment. verbose: an integer value that controls evaluation output logs such as summary and tqdm progress bar. nb_samples: number of samples to evaluate. task_threads: amount of thread workers to run scoring metrics. prompt: Prompt object to link with experiment. dataset_item_ids: list of dataset item ids to evaluate. If not provided, all samples in the dataset will be evaluated. dataset_sampler: An instance of a dataset sampler that will be used to sample dataset items for evaluation. If not provided, all samples in the dataset will be evaluated. trial_count: number of times to execute the prompt and evaluate the LLM output for every dataset item. experiment_scoring_functions: List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment. experiment_tags: List of tags to be associated with the experiment. dataset_filter_string: Optional OQL filter string to filter dataset items. Supports filtering by tags, data fields, metadata, etc. Supported columns include: - `id`, `source`, `trace_id`, `span_id`: String fields - `data`: Dictionary field (use dot notation, e.g., "data.category") - `tags`: List field (use "contains" operator) - `created_at`, `last_updated_at`: DateTime fields (ISO 8601 format) - `created_by`, `last_updated_by`: String fields Examples: - `tags contains "failed"` - Items with 'failed' tag - `data.category = "test"` - Items with specific data field value - `created_at >= "2024-01-01T00:00:00Z"` - Items created after date N)rz<`model` must be either a string or an OpikBaseModel instance)prompt_templaterrrrqrPrrrwrrr)rrrrrrrr)rrrerUrrmrr r),rWrrzrrr#rr" OpikBaseModel ValueErrorrrr}r~rrsrrrrrrrIrrrrrrr_rrr*rrPrrrrrSrrrr)!rrrrbrcrdrerfrgrhr.rirjr/r9rmrTrnr: opik_modelryrkrrrFrGrrrUrrrrs! r6evaluate_promptrst'+;;<//+28T!%#''59 z77 8WXX  '**  $5 53; / 0 + +)3)>)> g &  * * ,F vhdG4'5O )) \\+ "#;7#;#;#=tTJ *J.+'!O J  B B D2!-+"7   E*99%&  #33%    )66$.Z(S+ $ "%+7  ' E<z)J"<%A!" !|)) LL*l4N !99 mm::]]//N  "".A LLN.z:"( ((7QR*;; mm::"!%4!|33 1 W E Ds A(K%%K/ct|tjr |j}|gn|}|g}t j | | }t |||}tj}t||}|j||j||d||t|jdd}t||||||| | | ||||||S) u' Performs task evaluation on a given dataset. Args: optimization_id: The ID of the optimization associated with the experiment. dataset: An Opik Dataset or DatasetVersion instance task: A callable object that takes dict with dataset item content as input and returns dict which will later be used for scoring. scoring_functions: List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine: • dataset_item — a dictionary containing the dataset item content, • task_outputs — a dictionary containing the LLM task output. • task_span - the data collected during the LLM task execution [optional]. experiment_name_prefix: The prefix to be added to automatically generated experiment names to make them unique but grouped under the same prefix. For example, if you set `experiment_name_prefix="my-experiment"`, the first experiment created will be named `my-experiment-`. experiment_name: The name of the experiment associated with evaluation run. If None, a generated name will be used. project_name: The name of the project. If not provided, traces and spans will be logged to the `Default Project` experiment_config: The dictionary with parameters that describe experiment scoring_metrics: List of metrics to calculate during evaluation. Each metric has `score(...)` method, arguments for this method are taken from the `task` output, check the signature of the `score` method in metrics that you need to find out which keys are mandatory in `task`-returned dictionary. If no value provided, the experiment won't have any scoring metrics. verbose: an integer value that controls evaluation output logs such as summary and tqdm progress bar. 0 - no outputs, 1 - outputs are enabled (default). nb_samples: number of samples to evaluate. If no value is provided, all samples in the dataset will be evaluated. task_threads: number of thread workers to run tasks. If set to 1, no additional threads are created, all tasks executed in the current thread sequentially. are executed sequentially in the current thread. Use more than 1 worker if your task object is compatible with sharing across threads. prompt: Prompt object to link with experiment. Deprecated, use `prompts` argument instead. prompts: A list of Prompt objects to link with experiment. scoring_key_mapping: A dictionary that allows you to rename keys present in either the dataset item or the task output so that they match the keys expected by the scoring metrics. For example if you have a dataset item with the following content: {"user_question": "What is Opik ?"} and a scoring metric that expects a key "input", you can use scoring_key_mapping `{"input": "user_question"}` to map the "user_question" key to "input". dataset_item_ids: list of dataset item ids to evaluate. If not provided, all samples in the dataset will be evaluated. dataset_sampler: An instance of a dataset sampler that will be used to sample dataset items for evaluation. If not provided, all samples in the dataset will be evaluated. trial_count: number of times to execute the prompt and evaluate the LLM output for every dataset item. experiment_scoring_functions: List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment. experiment_tags: A list of tags to associate with the experiment. dataset_filter_string: Optional OQL filter string to filter dataset items. Supports filtering by tags, data fields, metadata, etc. Supported columns include: - `id`, `source`, `trace_id`, `span_id`: String fields - `data`: Dictionary field (use dot notation, e.g., "data.category") - `tags`: List field (use "contains" operator) - `created_at`, `last_updated_at`: DateTime fields (ISO 8601 format) - `created_by`, `last_updated_by`: String fields Examples: - `tags contains "failed"` - Items with 'failed' tag - `data.category = "test"` - Items with specific data field value - `created_at >= "2024-01-01T00:00:00Z"` - Items created after date NrprwrqrrP)rsrtrgrkrrrurvrx)rWrrzrr{r|rrr}r~rrsrrr)rrrarbrcrdrerfrgrhr.rirjrkrlr/r9rmrTrnr:rryrs r6evaluate_optimization_trialrs Z'+;;<//+28T!(;;O .+'!O  * * ,F4'5O )) \\+ ' "#;7#;#;#=tTJ* J  '!!/)'%A3 r8itemsc Jt|||}|s+tjdtjgSt j }tj5t|D cgc]\} } tjd dd| i| !} } } tjdd} tj||||} | j!t#| |||d d | t%|  }d d d tjScc} } w#1swY%xYw) u Lightweight evaluation function that evaluates a task on dataset items (as dictionaries) without requiring a Dataset object or creating an experiment. This function is useful for optimization scenarios where you need to evaluate many candidate solutions quickly using Opik's metric infrastructure. It creates traces for tracking but doesn't require experiment setup or dataset management. Args: items: List of dataset item contents (dictionaries with the data to evaluate). task: A callable object that takes dict with dataset item content as input and returns dict which will later be used for scoring. scoring_metrics: List of metrics to calculate during evaluation. Each metric's `score(...)` method will be called with arguments taken from the dataset item and task output. scoring_functions: List of scorer functions to be executed during evaluation. Each scorer function accepts predefined arguments: • dataset_item — a dictionary containing the dataset item content, • task_outputs — a dictionary containing the LLM task output. project_name: The name of the project for logging traces. verbose: Controls evaluation output logs and progress bars. 0 - no outputs (default), 1 - enable outputs. scoring_key_mapping: A dictionary that allows you to rename keys present in either the dataset item or the task output to match the keys expected by scoring metrics. scoring_threads: Number of thread workers to run scoring metrics. Returns: EvaluationResultOnDictItems object containing test results and providing methods to aggregate scores, similar to the regular evaluation result. Example: ```python import opik from opik.evaluation.metrics import Equals items = [ {"input": "What is 2+2?", "expected_output": "4"}, {"input": "What is 3+3?", "expected_output": "6"}, ] def my_task(item): # Your LLM call here question = item["input"] # ... call model ... return {"output": model_output} result = opik.evaluate_on_dict_items( items=items, task=my_task, scoring_metrics=[Equals()], scoring_key_mapping={"reference": "expected_output"}, ) # Access individual test results for test_result in result.test_results: print(f"Score: {test_result.score_results[0].value}") # Get aggregated statistics aggregated = result.aggregate_evaluation_scores() print(f"Mean equals score: {aggregated['equals_metric'].mean}") ``` rwz0No scoring metrics provided for items evaluation)rUrP temp_item_rrrNrr)rrArZrEvaluationResultOnDictItemsrr}rr enumerater DatasetItemrrrrrrEr2)rrarbrcrfrhrlrryiitemrrrrUs r6evaluate_on_dict_itemsr s6`.+'!O IJ <<"MM  * * ,F  B B D%U+ +4  $ $ A*QC(8 AD A+  *99A #33%#   )66}-+ 3 %+M*7   E4  8 8! 3  E Ds$D3$DADDD"cl|r-tj||}|r|j|n|}|r|SgS)N)rf)r%wrap_scorer_functionsrX)rcrbrffunction_metricss r6rrsC 0FF L    " "#3 4.O-?525r8c:|r|S|rtj|Sy)N)r{generate_unique_experiment_namerqs r6r~r~s+!AA "  r8)NNNNNNrNr`NNNNNrNNN)Nr`rNNN)NNNNNNNrNr`NNNrNNN)NNNrNr`)^loggingrtypingrrrrrrr r r r api_objects.promptr api_objectsrrrapi_objects.datasetrapi_objects.experimentrr{rrapi_objects.prompt.chatrrrrrrrrrrmetricsrr modelsr!r"r#scorersr$r%r&r'r(r)r*$api_objects.dataset.evaluation_suiter+ suite_types getLogger__name__rArr@DatasetDatasetVersionintrr7BaseDatasetSamplerr rI ExperimentrS TestResult ScoreResultr_rz BaseMetricScorerFunction BasePromptrrOpikrrfloatrrrrrrrr rr~rr8r6r.s     -%-.BN:62/AA;JJKK   8 $D(+$(GOOW%;%;;<( (tCy)(c] ("'-GOOW%;%;;<'- '-tCy)'-h99: '- $C= '-  8L,, -x} <= '-T  %%     "&'>"?{--. , " "#@?CHL,0%)"&26 $/36:;?,0=ALP+/+/-a //1A1Q1QQa  a d;#9#9:; a  _%C%C DE a%SMac]a3-a S#X/aa aa [++ ,ad;112 3a ""78!a"tCy)#a$h99:%a&'a(#+40G+H"I)a*d3i(+a,$C=-a.''/aP*.,0+/,0%)"&266:+/%)%)%)#W __W W [%% & W tCy) W $C= W%SMWc]W3-W S#X/Wd;112 3Wd3i(WWWc]W c]!W"c]#W$)%Wt_   _%%_7??G$:$:: ; _  _ +001 _3-__ __""78_tCy)_h99:__#''>"?_ $C=!_"''#_X-1+/B*   B*%%B*__ B*  B* 3- B*B*B*c]B*tCy)B*$C=B*  - -u 45B*PIM;?#'LPKK+001K _%C%C DEK K  K ""78 KC=K#+40G+H"IK''K\4#  # #4#/3DcN/C4# tCH~S#X./4#x=A>BHL,0%)"&26 $/3,0=ALP+/+/+Z //1A1Q1QQZ4S>" Z E#z7778 9 Z d;#9#9:; Z _%C%C DEZ%SMZc]Z3-Z S#X/ZZ ZZ [++ ,Z tCy)!Z"h99:#Z$%Z&#+40G+H"I'Z(d3i()Z*$C=+Z,''-ZF?CHL,0%)"&26 $/36:;?,0=ALP+/+//ff //1A1Q1QQf  f d;#9#9:; f _%C%C DEf%SMfc]f3-f S#X/ff ff [++ ,f d;112 3!f"""78#f$tCy)%f&h99:'f()f*#+40G+H"I+f,d3i(-f.$C=/f0''1fX?CHL"&;?x S#X x xd;#9#9:;x _%C%C DE x 3- x  x""78xx22xv6_%C%C DE6d;#9#9:;63-6 + ! 6" c]