i/dZddlmZddlZddlmZmZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZmZdd lmZmZdd lmZdd lmZdd lmZm Z erdd lm!Z!GddeZ"y)z Base RAG Ingestion class. Provides abstract methods for: - OCR - Chunking - Embedding - Vector Store operations Providers can inherit and override methods as needed. ) annotationsN)ABCabstractmethod) TYPE_CHECKINGAnyDictListOptionalTuplecast)verbose_logger)uuid4)DEFAULT_CHUNK_OVERLAPDEFAULT_CHUNK_SIZE)get_async_httpx_clienthttpxSpecialProvider)extract_text_from_pdf)RecursiveCharacterTextSplitter)RAGIngestOptionsRAGIngestResponse)RouterceZdZdZ d d dZddZeddZ d ddZ ddZ ddZ dd Z e dd Z d dd Zy)BaseRAGIngestiona Base class for RAG ingestion. Providers should inherit from this class and override methods as needed. For example, OpenAI handles embedding internally when attaching files to vector stores, so it overrides the embedding step to be a no-op. Nc||_||_dt|_|j d|_t tttf|j dxsddi|_ |j d|_ t tttf|j dxsi|_ |j d|_ |jy) Ningest_ocrchunking_strategytypeauto embedding vector_storename)ingest_optionsrouterr ingest_idget ocr_configr rstrrrembedding_configvector_store_config ingest_name_load_credentials_from_config)selfr#r$s u/Users/manta/Documents/Projects/TheRoad-I1/.venv/lib/python3.12/site-packages/litellm/rag/ingestion/base_ingestion.py__init__zBaseRAGIngestion.__init__,s - "57),),,U315 cN   2 3 G7G2 !/ 2 2; ?37 cNN..~>D"4  *--f5 **,cddlm}|jjd}|rYtj rH|j |}|jD]#\}}||jvs||j|<%yyy)z Load credentials from litellm_credential_name if provided in vector_store config. This allows users to specify a credential name in the vector_store config which will be resolved from litellm.credential_list. r)CredentialAccessorlitellm_credential_nameN).litellm.litellm_core_utils.credential_accessorr2r*r&litellmcredential_listget_credential_valuesitems)r-r2credential_namecredential_valueskeyvalues r.r,z.BaseRAGIngestion._load_credentials_from_configDs| V22667PQ w66 2 H H! 0557 Ud66649D,,S18 7?r0c:|jjddS)zGet the vector store provider.custom_llm_provideropenai)r*r&)r-s r.r>z$BaseRAGIngestion.custom_llm_providerWs''++,A8LLr0cfK|r |\}}}|||dfS|rttj}|j|d{}|j |j }|j ddxsd}|jjdd}|||dfS|rddd|fStd7mw) aG Upload / prepare file for ingestion. Args: file_data: Tuple of (filename, content_bytes, content_type) file_url: URL to fetch file from file_id: Existing file ID to use Returns: Tuple of (filename, file_content, content_type, existing_file_id) N) llm_provider/documentz content-typezapplication/octet-streamz,Must provide file_data, file_url, or file_id) rrRAGr&raise_for_statuscontentsplitheaders ValueError) r- file_datafile_urlfile_idfilename file_content content_type http_clientresponses r.uploadzBaseRAGIngestion.upload\s" 3< 0HlL\<= = 0>R>V>VWK(__X66H  % % '#++L~~c*2.<*H#++// :L\<= = tW, ,GHH7s?B1B/A.B1c K|jr|sy|jjdd}|r d|vrd\}}nd\}}tj|j d}d|d |}|j *|j j |d |||i d{}n#tj |d |||i d{}t|d r.|jr"d jd|jDSy7c7Aw)z Perform OCR on file content to extract text. Args: file_content: Raw file bytes content_type: MIME type of the file Returns: Extracted text or None if OCR not configured/needed Nmodelzmistral/mistral-ocr-latestimage) image_urlrW) document_urlrXutf-8zdata:z;base64,r)rUrDpagesz c3NK|]}t|ds|jyw)markdownN)hasattrr\).0pages r. z'BaseRAGIngestion.ocr..s"*<$j@Y * Hg&&|4;;GD <. > ;; "!%!1!1 (GX>"2"L ") (GX>"L < )l.@.@;;*6*<*< ! s%BDC?#D?DADDcd}|r|}n|r|s |jd}|sgS|j xsi}|jdt}|jdt}|jd d}||d } |r|| d <td i| } | j|S#t$rh|jdr;tjdt |}|s2tjdgcYStjdgcYSYwxYw) a Split text into chunks using RecursiveCharacterTextSplitter. Args: text: Text from OCR (if used) file_content: Raw file content bytes ocr_was_used: Whether OCR was performed Returns: List of text chunks NrYs%PDFz(PDF detected, attempting text extractionzkPDF text extraction failed. Install 'pypdf' or 'PyPDF2' for PDF support, or enable OCR with a vision model.z,Binary file detected, skipping text chunking chunk_size chunk_overlap separators)rmrn) rcUnicodeDecodeError startswithr debugrrr&rrr split_text) r-textrO ocr_was_used text_to_chunk splitter_argsrmrnrosplitter_kwargs text_splitters r.chunkzBaseRAGIngestion.chunks+$(,  M ,  , 3 3G < I..4" "&&|5GH %))/;PQ "&&|T: %*+  ,6OL )6II '' 66C& **73"(()ST$9,$GM(&,,A " "(()WXI) sBAD -D D cNK|jr|sy|jjdd}|j&|jj||d{}nt j||d{}|j Dcgc]}|d c}S7A7#cc}ww)z Generate embeddings for text chunks. Args: chunks: List of text chunks Returns: List of embeddings or None NrUztext-embedding-3-small)rUinputr )r)r&r$ aembeddingr5data)r-chunksembedding_modelrRitems r.embedzBaseRAGIngestion.embeds$$F//33G=UV ;; "![[33/QW3XXH$//oVTTH.6mm, .> ) #'88))$,$N ZZ#)!__D8 F $zzz88J59JJ)!)% 5?5/ +O^%>>" / 52&:*:  AB 9/   $ $'> "!f   srDB< DCB>9C5C6CC'C;D>CCC D :D DD DD)N)r#rr$zOptional['Router'])returnNone)rr()NNN)rK Optional[Tuple[str, bytes, str]]rL Optional[str]rMrrzCTuple[Optional[str], Optional[bytes], Optional[str], Optional[str]])rOOptional[bytes]rPrrr)rurrOrrvboolr List[str])rrrOptional[List[List[float]]]) rOrrNrrPrrrrrrz#Tuple[Optional[str], Optional[str]])rKrrLrrMrrr)__name__ __module__ __qualname____doc__r/r,propertyr>rSrr{rrrrrpr0r.rr#si&*-(-#-0:&MM 7;"&!% #I3#I #I #I M #IJ0%0$0  0d:7:7&:7 :7  :7x== %=2 %   $    0   -  67;"&!% D3D D D  Dr0r)#r __future__rraabcrrtypingrrrr r r r r5litellm._loggingr litellm._uuidrlitellm.constantsrr&litellm.llms.custom_httpx.http_handlerrr"litellm.rag.ingestion.file_parsersrlitellm.rag.text_splittersrlitellm.types.ragrrrrrpr0r.rsO # #HHH+GEEACsCr0