jP,ddlmZddlZddlZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZmZddlmZddlmZddlmZddlmZmZd d lmZmZmZd d lmZmZd d lm Z ddl!m"Z"e rddl#m$Z$ejJe&Z' d ddZ( ddZ) d ddZ*ddZ+ ddZ, d d dZ- d!dZ.y)") annotationsN)AnyDictIteratorListOptionalSet TYPE_CHECKING)OpikApi) dataset_itemdataset_version_public)streamer)retry_decorator)opik_query_languagerest_stream_parser)datasetr execution_policy) experiment constants)experiments_client)ApiError) llm_judgec #Ktjdd}d}|r t|nd} d|rFtjj |} | j } | rtj| |rtjd fd } | } t| dk(rd}n^| D]T}|j}|| || vr| j|d}|jrM|jDcgc]8}tj |j"|j$|j&:}}d}|j(r?tj*|j(j,|j(j.}tj0d |j|j2|j4|j6|j8||d|j:}||d z } |k\rd}n| Ct| dk(sSd}n|r| r&t| dkDrt<j?d | yyycc}ww) a Stream dataset items from the backend as a generator. Args: rest_client: The REST API client. dataset_name: Name of the dataset to stream items from. nb_samples: Maximum number of items to retrieve. If None, all items are streamed. batch_size: Maximum number of items to fetch per batch from the backend. dataset_item_ids: Optional list of specific item IDs to retrieve. filter_string: Optional OQL filter string to filter dataset items. dataset_version: Optional dataset version hash to filter items by a specific version. Yields: DatasetItem objects one at a time. NTrc tjjjtj S)N) dataset_namelast_retrieved_id steam_limitfiltersdataset_version)stream item_class nb_samples)rread_and_parse_streamdatasetsstream_dataset_itemsrest_dataset_item_read DatasetItem) batch_sizerr"r!rr% rest_clients/Users/manta/Documents/Projects/TheRoad-I1/backend/.venv/lib/python3.12/site-packages/opik/api_objects/dataset/rest_operations.py _fetch_batchz*stream_dataset_items.._fetch_batchFsM%;;"++@@!-&7 *#$3 A2==%  Fnametypeconfig runs_per_itempass_threshold)idtrace_idspan_idsource description evaluatorsrrz=The following dataset items were not found in the dataset: %s)returnz(List[rest_dataset_item_read.DatasetItem]) rDATASET_STREAM_BATCH_SIZEsetrOpikQueryLanguagefor_dataset_itemsget_filter_expressionsjsondumpsropik_rest_retrylenr7remover<r EvaluatorItemr1r2r3rExecutionPolicyItemr5r6r*r8r9r:r;dataLOGGERwarning)r,rr%r+dataset_item_ids filter_stringr"should_retrieve_more_items items_yieldeddataset_items_ids_leftoqlfilter_expressionsr. dataset_itemsitemitem_idr<erreconstructed_itemr!rs```` ` @@r-r(r(s^088 '+!%M!1 t"G!33EEmT 779 jj!34G $  ( (   ) % }  "). & !DggG ' %1"88*11':J"__  - !..VVVV xx -  $ $$#/#C#C"&"7"7"E"E#'#8#8#G#G$ ".!9!9 "77 {{ ,,%!1 ")) " % $ Q M%-:*E-2*%1c:P6QUV6V-2*e"- %T#&<"="A K " #BOs%C5I>=H=;B>I;I I.Ic |jj||S#t$r}|jdk(rYd}~yd}~wwxYw)a. Find a dataset version by version name. Args: rest_client: The REST API client. dataset_id: The dataset ID to search versions in. version_name: Version name to search for (e.g., 'v1', 'v2'). Returns: The DatasetVersionPublic if found, None otherwise. )r7 version_nameN)r'retrieve_dataset_versionr status_code)r, dataset_idr[rXs r-find_version_by_namer`sN ##<< =    ==C  s A==Acd}g}d}t||kr|jj||}t|jdk(r |S|jd|t|z D]\}t j |j |j||j}|r|j|j|^|dz }t||kr|S)Ndr)pagesizer)r1r;r,dataset_items_count) rGr' find_datasetscontentrDatasetr1r;re__internal_api__sync_hashes__append) r, max_results sync_items page_sizer'rc page_datasets dataset_ferndataset_s r- get_datasetsrqsI&(H D h-+ %#,,::; }$$ % * " O*112Q[3x=5PRL!&&(44'$0$D$D H668 OOH %S  - h-+ %0 Or/c |jj|j}|S#t$r/}|jdk(rt j d|d|d}~wwxYw)Nrr\zDataset with the name z not found.)r'get_dataset_by_identifierr7rr^ exceptionsDatasetNotFound)r,rr_rXs r-get_dataset_idrws{  ))CC%D "    ==C ,,(kB   s&* A"*AA"c d}g}d}t||kr|jj|||}t|jdk(r |S|jd|t|z D]U} |j t j | j| j| j|||| jW|dz }t||kr|S)Nrbr)rcrdr_r)r7r1rr,rrtags) rG experimentsfind_experimentsrgrjr Experimentr7r1rry) r,r_rkrrrmrzrcpage_experiments experiment_s r-get_dataset_experimentsrsI/1K D k [ (&22CC!D  '' (A - " ,334TkC DT6TUK   %%"~~$))!,!9!9 +%'9$)) V  / k [ (2 r/c|jj||d||jj|}|xstjj }i}|r?|D cgc]0} | j d| jjdd2c} |d<|jd d |jd d d |d <|jj|j|d|jScc} w)a Create a dataset of type 'evaluation_suite' and its initial version with evaluators and execution_policy persisted to the backend. Args: rest_client: The REST API client. dataset_name: The name of the dataset/suite. description: Optional description. evaluators: LLMJudge evaluators. exec_policy: Execution policy dict. Returns: The dataset ID. evaluation_suite)r1r;r2ryrsrTby_aliasr0r<r5rr6r4rr7requestoverride) r'create_datasetrtrDEFAULT_EXECUTION_POLICYcopyr1 to_config model_dumpgetapply_dataset_item_changesr7) r,rr;r< exec_policyryroresolved_policyrrXs r-create_evaluation_suite_datasetrs,'' {9KRV(''AA!BL"U%5%N%N%S%S%UO G !   #++-22D2A   !  ),,_a@)--.>B#G 33 ??Gd4 ??!! s&5C=c ||Dcgc]0}|jd|jjdd2c}|jdd|jdddd }|jj ||d y cc}w) a Update suite-level evaluators and execution_policy by creating a new dataset version based on the current latest version. Args: rest_client: The REST API client. dataset_id: The dataset ID. base_version_id: The current latest version UUID to base the update on. evaluators: Suite-level LLMJudge evaluators. exec_policy: Execution policy dict. rTrr0r5rr6r4) base_versionr<rFrN)r1rrrr'r)r,r_base_version_idr<rrXrs r-update_evaluation_suite_datasetr;s&(    #++-22D2A    )___a@)oo.>B G33 w4 s5B)NNNNN)r,r rstrr% Optional[int]r+rrNOptional[List[str]]rO Optional[str]r"rr=z"Iterator[dataset_item.DatasetItem])r,r r_rr[rr=z5Optional[dataset_version_public.DatasetVersionPublic])iT)r,r rkintrlboolr=zList[dataset.Dataset])r,r rrr=r) r,r r_rrkrrzstreamer.Streamerrz$experiments_client.ExperimentsClientr=zList[experiment.Experiment])N)r,r rrr;rr<z"Optional[List[llm_judge.LLMJudge]]rz*Optional[execution_policy.ExecutionPolicy]ryrr=r) r,r r_rrrr<zList[llm_judge.LLMJudge]rz execution_policy.ExecutionPolicyr=None)/ __future__rrDloggingtypingrrrrrr r opik.rest_apir opik.rest_api.typesr r)r opik.exceptionsruopik.message_processingropik.rest_client_configuratorropik.api_objectsrrrrrrrrest_api.core.api_errorr opik.evaluation.suite_evaluatorsr getLogger__name__rLr(r`rqrwrrrr>r/r-rs" JJJ!%,9D55$+/:   8 $ !% $,0#'%)w w w w  w * w ! w #w (w t; 6GK'*?CD $$$$ $ = $ ! $Z!% 11113 1 < 1  1 1h####) # 2 #  #r/