a SicE @sldZddlmZddlZddlZddlmZddlmZddl m Z m Z m Z m Z ddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZmZmZm Z m!Z!dddddZ"d3dddddddddZ#GdddZ$Gddde$Z%Gd d!d!e$Z&eed"d#d4d&d'dd(d)dd*d+d,d-d.Z'eed"d#d5d/dd*ddd&d0d1d2Z(dS)6z parquet compat ) annotationsN)Any)catch_warnings)FilePath ReadBufferStorageOptions WriteBuffer)import_optional_dependencyAbstractMethodError)doc) DataFrame MultiIndex get_option) _shared_docs)Version) IOHandles get_handle is_fsspec_urlis_urlstringify_pathstrBaseImpl)enginereturnc Cs|dkrtd}|dkr~ttg}d}|D]F}z |WStyl}z|dt|7}WYd}~q(d}~00q(td||dkrtS|dkrtStd dS) zreturn our implementationautozio.parquet.enginez - NzUnable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support. Trying to import the above resulted in these errors:pyarrow fastparquetz.engine must be one of 'pyarrow', 'fastparquet')r PyArrowImplFastParquetImpl ImportErrorr ValueError)rZengine_classes error_msgsZ engine_classerrr%M/var/www/html/django/DPS/env/lib/python3.9/site-packages/pandas/io/parquet.py get_engine$s( ( r'rbFz1FilePath | ReadBuffer[bytes] | WriteBuffer[bytes]rrboolzVtuple[FilePath | ReadBuffer[bytes] | WriteBuffer[bytes], IOHandles[bytes] | None, Any])pathfsstorage_optionsmodeis_dirrcCst|}t|r>|dur>td}|jj|fi|p2i\}}n|rZt|rR|dkrZtdd}|s|st|trt j |st ||d|d}d}|j }|||fS)zFile handling for PyArrow.Nfsspecr(z8storage_options passed with buffer, or non-supported URLFis_textr,)rrr core url_to_fsrr" isinstancerosr*isdirrhandle)r*r+r,r-r.path_or_handler/handlesr%r%r&_get_path_or_handleFs2    r:c@s>eZdZedddddZddddZd dd d d Zd S)rr None)dfrcCsxt|tstdt|jtr>tdd|jjDsRtdn|jjdvrRtdtdd|jj D}|sttddS) Nz+to_parquet only supports IO with DataFramescss|]}|jdvVqdS)>emptystringN) inferred_type).0xr%r%r& wsz.BaseImpl.validate_dataframe..z parquet must have string column names for all values in each level of the MultiIndex >r=r>z%parquet must have string column namescss |]}|durt|tVqdSN)r4r)r@namer%r%r&rBsz!Index level names must be strings) r4r r"columnsralllevelsr?indexnames)r<Z valid_namesr%r%r&validate_dataframeos    zBaseImpl.validate_dataframe)r<cKs t|dSrCr )selfr<r* compressionkwargsr%r%r&writeszBaseImpl.writeNrcKs t|dSrCr )rKr*rErMr%r%r&readsz BaseImpl.read)N)__name__ __module__ __qualname__ staticmethodrJrNrPr%r%r%r&rnsc @sHeZdZddddZdddd d d d dd ddZdd ddddZdS)rr;rOcCs&tdddddl}ddl}||_dS)Nrz(pyarrow is required for parquet support.extrar)r Zpyarrow.parquet(pandas.core.arrays.arrow.extension_typesapi)rKrpandasr%r%r&__init__s zPyArrowImpl.__init__snappyNr zFilePath | WriteBuffer[bytes] str | None bool | Nonerlist[str] | None)r<r*rLrHr,partition_colsrc Ks||d|ddi}|dur*||d<|jjj|fi|} t||dd|d|dud\} } |d<t| tjrt | drt| j t t fr| j } t| t r| } zX|dur|jjj| | f||d|n|jjj| | fd|i|W| dur| n| dur| 0dS) Nschemapreserve_index filesystemwb)r,r-r.rD)rLr_rL)rJpoprXTable from_pandasr:r4ioBufferedWriterhasattrrDrbytesdecodeparquetZwrite_to_dataset write_tableclose) rKr<r*rLrHr,r_rMZfrom_pandas_kwargstabler8r9r%r%r&rNsV      zPyArrowImpl.writeFr,rc Kszd|d<i}|rddl}|j||j||j||j| |j | |j | |j||j||j||j||j||j|i }|j|d<td} | dkrd|d<t||dd|d d \} } |d<zX|jjj| fd |i|j fi|} | dkrH| j!dd d } | W| dur^| "Sn| durt| "0dS)NTZuse_pandas_metadatarZ types_mapperzmode.data_managerarrayZ split_blocksrbr()r,r-rEF)copy)#rYrXint8 Int8Dtypeint16 Int16Dtypeint32 Int32Dtypeint64 Int64Dtypeuint8 UInt8Dtypeuint16 UInt16Dtypeuint32 UInt32Dtypeuint64 UInt64Dtypebool_ BooleanDtyper> StringDtypefloat32 Float32Dtypefloat64 Float64Dtypegetrr:rdrl read_table to_pandas _as_managerrn) rKr*rEuse_nullable_dtypesr,rMZto_pandas_kwargspdmappingmanagerr8r9resultr%r%r&rPs^      zPyArrowImpl.read)r[NNN)NFNrQrRrSrZrNrPr%r%r%r&rs8rc@s@eZdZddddZddddd d d Zdddd d dZdS)r r;rOcCstddd}||_dS)Nrz,fastparquet is required for parquet support.rU)r rX)rKrr%r%r&rZszFastParquetImpl.__init__r[Nr r)r<r,rc s||d|vr$|dur$tdnd|vr6|d}|durFd|d<t|}t|rrtdfdd|d<n r~td td d .|jj||f|||d |Wdn1s0YdS) N partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning dataZhiveZ file_schemer/csj|dfipiS)Nrcopenr*_r/r,r%r&/sz'FastParquetImpl.write.. open_withz?storage_options passed with file object or non-fsspec file pathT)record)rL write_indexr) rJr"rdrrr rrXrN)rKr<r*rLrHr_r,rMr%rr&rNs8   zFastParquetImpl.writerpc  s i}|dd}t|jjtdkr,d|d<|r8tdt|}d}t|rtdt|jjtdkrj|dfipzij |d <qȇfd d |d <n,t |t rt j |st|ddd }|j}z>|jj|fi|}|jfd|i|W|dur|Sn|dur|0dS)NrFz0.7.1Z pandas_nullszNThe 'use_nullable_dtypes' argument is not supported for the fastparquet enginer/z0.6.1r(r+csj|dfipiS)Nr(rrrr%r&rXsz&FastParquetImpl.read..rr0rE)rdrrX __version__r"rrr rr+r4rr5r*r6rr7Z ParquetFilerrn) rKr*rEr,rMZparquet_kwargsrr9Z parquet_filer%rr&rPAsD   zFastParquetImpl.read)r[NNN)NNrr%r%r%r&r s 3r r,)r,rr[r z$FilePath | WriteBuffer[bytes] | Noner\r]r^z bytes | None)r<r*rrLrHr,r_rc Ksrt|tr|g}t|}|dur(tn|} |j|| f||||d||durjt| tjsbJ| SdSdS)a} Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str, path object, file-like object, or None, default None String, path object (implementing ``os.PathLike[str]``), or file-like object implementing a binary ``write()`` function. If None, the result is returned as bytes. If a string, it will be used as Root Directory path when writing a partitioned dataset. The engine fastparquet does not accept file-like objects. .. versionchanged:: 1.2.0 engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. compression : {{'snappy', 'gzip', 'brotli', 'lz4', 'zstd', None}}, default 'snappy'. Name of the compression to use. Use ``None`` for no compression. The supported compression methods actually depend on which engine is used. For 'pyarrow', 'snappy', 'gzip', 'brotli', 'lz4', 'zstd' are all supported. For 'fastparquet', only 'gzip' and 'snappy' are supported. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. {storage_options} .. versionadded:: 1.2.0 kwargs Additional keyword arguments passed to the engine Returns ------- bytes if no path argument is provided else None N)rLrHr_r,)r4rr'rgBytesIOrNgetvalue) r<r*rrLrHr,r_rMimpl path_or_bufr%r%r& to_parquetls&<  rzFilePath | ReadBuffer[bytes])r*rrEr,rrcKs"t|}|j|f|||d|S)aY Load a parquet object from the file path, returning a DataFrame. Parameters ---------- path : str, path object or file-like object String, path object (implementing ``os.PathLike[str]``), or file-like object implementing a binary ``read()`` function. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir``. engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. columns : list, default=None If not None, only these columns will be read from the file. {storage_options} .. versionadded:: 1.3.0 use_nullable_dtypes : bool, default False If True, use dtypes that use ``pd.NA`` as missing value indicator for the resulting DataFrame. (only applicable for the ``pyarrow`` engine) As new dtypes are added that support ``pd.NA`` in the future, the output with this option will change to use those dtypes. Note: this is an experimental option, and behaviour (e.g. additional support dtypes) may change without notice. .. versionadded:: 1.2.0 **kwargs Any additional kwargs are passed to the engine. Returns ------- DataFrame )rEr,r)r'rP)r*rrEr,rrMrr%r%r& read_parquets6r)Nr(F)Nrr[NNN)rNNF))__doc__ __future__rrgr5typingrwarningsrpandas._typingrrrrpandas.compat._optionalr pandas.errorsr pandas.util._decoratorsr rYr rrpandas.core.shared_docsrpandas.util.versionrpandas.io.commonrrrrrr'r:rrr rrr%r%r%r&sF         %($tf  R