Ë ³«Žjãóî—ddlmZmZmZddlmZddlmZddlm Z ddl m Z dd lm Z m Z dd lmZ dd ed eed eedee j"deegefdeegefdededede j&fd„Zy)é)ÚOptionalÚListÚCallableé)Úasyncio_supporté)Ú opik_client)Úthreads_client)Úconversation_thread_metricé)Úevaluation_engineÚevaluation_result)ÚJsonListStringPublicÚ project_nameÚ filter_stringÚeval_project_nameÚmetricsÚtrace_input_transformÚtrace_output_transformÚverboseÚ num_workersÚmax_traces_per_threadÚreturnc ó—tj«} tj| «} t j «5t j| |||¬«} | j||||||¬«cddd«S#1swYyxYw)a%Evaluate conversation threads using specified metrics. This function evaluates conversation threads from a project using the provided metrics. It creates a ThreadsEvaluationEngine to fetch threads matching the filter string, converts them to conversation threads, applies the metrics, and logs feedback scores. Args: project_name: The name of the project containing the threads to evaluate. filter_string: Optional filter string to select specific threads for evaluation using Opik Query Language (OQL). The format is: " [AND ]*" Supported columns include: - `id`, `name`, `created_by`, `thread_id`, `type`, `model`, `provider`: String fields with full operator support - `status`: String field (=, contains, not_contains only) - `start_time`, `end_time`: DateTime fields (use ISO 8601 format, e.g., "2024-01-01T00:00:00Z") - `input`, `output`: String fields for content (=, contains, not_contains only) - `metadata`: Dictionary field (use dot notation, e.g., "metadata.model") - `feedback_scores`: Numeric field (use dot notation, e.g., "feedback_scores.accuracy") - `tags`: List field (use "contains" operator only) - `usage.total_tokens`, `usage.prompt_tokens`, `usage.completion_tokens`: Numeric usage fields - `duration`, `number_of_messages`, `total_estimated_cost`: Numeric fields Examples: 'status = "inactive"', 'id = "thread_123"', 'duration > 300' If None, all threads in the project will be evaluated. eval_project_name: Optional name for the evaluation project where evaluation traces will be stored. If None, the same project_name will be used. metrics: List of ConversationThreadMetric instances to apply to each thread. Must contain at least one metric. trace_input_transform: Function to transform trace input JSON to string representation. This function extracts the relevant user message from your trace's input structure. The function receives the raw trace input as a dictionary and should return a string. Example: If your trace input is {"content": {"user_question": "Hello"}}, use: lambda x: x["content"]["user_question"] This transformation is essential because trace inputs vary by framework, but metrics expect a standardized string format representing the user's message. trace_output_transform: Function to transform trace output JSON to string representation. This function extracts the relevant agent response from your trace's output structure. The function receives the raw trace output as a dictionary and should return a string. Example: If your trace output is {"response": {"text": "Hi there"}}, use: lambda x: x["response"]["text"] This transformation is essential because trace outputs vary by framework, but metrics expect a standardized string format representing the agent's response. verbose: Verbosity level for progress reporting (0=silent, 1=progress). Default is 1. num_workers: Number of concurrent workers for thread evaluation. Default is 8. max_traces_per_thread: Maximum number of traces to fetch per thread. Default is 1000. Returns: ThreadsEvaluationResult containing evaluation scores for each thread. Raises: ValueError: If no metrics are provided. MetricComputationError: If no threads are found or if evaluation fails. Example: >>> from opik.evaluation import evaluate_threads >>> from opik.evaluation.metrics import ConversationalCoherenceMetric, UserFrustrationMetric >>> >>> # Initialize the evaluation metrics >>> conversation_coherence_metric = ConversationalCoherenceMetric() >>> user_frustration_metric = UserFrustrationMetric() >>> >>> # Run the threads evaluation >>> results = evaluate_threads( >>> project_name="ai_team", >>> filter_string='thread_id = "0197ad2a-cf5c-75af-be8b-20e8a23304fe"', >>> eval_project_name="ai_team_evaluation", >>> metrics=[ >>> conversation_coherence_metric, >>> user_frustration_metric, >>> ], >>> trace_input_transform=lambda x: x["input"], >>> trace_output_transform=lambda x: x["output"], >>> ) )ÚclientrÚnumber_of_workersr)rrrrrrN) r Úget_client_cachedr Ú ThreadsClientrÚ)async_http_connections_expire_immediatelyr ÚThreadsEvaluationEngineÚevaluate_threads) rrrrrrrrrrÚthreads_client_Úengines úz/Users/manta/Documents/Projects/TheRoad-I1/backend/.venv/lib/python3.12/site-packages/opik/evaluation/threads/evaluator.pyr!r! s€ôx× *Ñ *Ó ,€FÜ$×2Ñ2°6Ó:€Oä × BÑ BÕ DÜ"×:Ñ:Ø"Ø%Ø)Øô  ˆð ×&Ñ&Ø'Ø/ØØ"7Ø#9Ø"7ð 'ó ÷ E× DÒ Dús ¾0A8Á8BN)r éiè)ÚtypingrrrÚrÚ api_objectsr Úapi_objects.threadsr Úmetrics.conversationr r rÚ opik.rest_apirÚstrÚConversationThreadMetricÚintÚThreadsEvaluationResultr!©ór$Úr2sÎðß+Ñ+åÝ&Ý1Ý=ß2Ý.ðØØ!%ñm Øðm à˜C‘=ðm ð  ‘}ðm ðÐ,×EÑEÑ Fð m ð $Ð%9Ð$:¸CÐ$?Ñ@ð m ð %Ð&:Ð%;¸SÐ%@ÑAð m ððm ððm ððm ð×.Ñ.ôm r1