a Sic@sUdZddlmZddlmZddlZddlmZmZddl Z ddl Z ddl Z ddl m Z ddlmZmZmZmZmZmZmZmZmZmZddlZddlZddlmZmZdd lm Z m!Z"dd l#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-dd l.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEddlFmGZGddlHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQddlRmSZSddlTmUZUmVZVmWZWddlXmYmZZ[ddl\m]Z]m^Z^ddl_m`Z`ddlambZbddlcmdZdmeZeddlfmgZgddlhmiZimjZjerJddlkmlZlmmZmmnZnddlcmoZodZpdZqd d!Zrd"d#Zsd$d%Zte]Zud&d'd(d)Zvd*Zwd+exd,<d-Zyd+exd.<d/Zzd+exd0<d1d1d2d2d3Z{eIdgiZ|d4Z}d+exd5<d6Z~d+exd7<ed8@ejd9d:e}ejd;ejdd?ZddCdDdEdDdFdGdHdGdHdIdJdKdDdDdLdMdNdOZddCdDdDdQdFdFdRdHdFdS dTdUZdVdVdHdWdXdYZGdZd[d[ZGd\d]d]ZGd^d_d_ZGd`dadaeZGdbdcdceZGdddedeeZGdfdgdgeZGdhdidiZGdjdkdkeZGdldmdmeZGdndodoeZGdpdqdqeZGdrdsdseZGdtdudueZGdvdwdweZGdxdydyeZGdzd{d{eZGd|d}d}eZGd~ddeZGdddeZddd&dddddZdddddZeddddHddddZedddLdHddddZddddHddddZdDddDdDd_dddZdDdDdDddddZdDdddddZddDdDddddZddDdDddddZddDdDdDdddZdDdDdDdddZdDdHdddZdDddDdddZdDdDdddZddddZGdddZdS)zY High level interface to PyTables for reading and writing pandas data structures to disk ) annotations)suppressN)datetzinfo)dedent) TYPE_CHECKINGAnyCallableFinalHashableIteratorLiteralSequencecastoverload)config get_option)libwriters) timezones) AnyArrayLike ArrayLikeDtypeArgFilePathShapenpt)import_optional_dependency) patch_pickle)AttributeConflictWarningClosedFileErrorIncompatibilityWarningPerformanceWarningPossibleDataLossError)cache_readonly)find_stack_level) ensure_object is_bool_dtypeis_categorical_dtypeis_complex_dtypeis_datetime64_dtypeis_datetime64tz_dtypeis_extension_array_dtype is_list_likeis_string_dtypeis_timedelta64_dtypeneeds_i8_conversion)array_equivalent) DataFrame DatetimeIndexIndex MultiIndex PeriodIndexSeriesTimedeltaIndexconcatisna) Int64Index) Categorical DatetimeArray PeriodArray) PyTablesExprmaybe_expression) extract_array) ensure_index) ArrayManager BlockManager)stringify_path)adjoin pprint_thing)ColFileNode)Blockz0.15.2UTF-8cCst|tjr|d}|S)z(if we have bytes, decode them to unicoderK) isinstancenpbytes_decode)srQN/var/www/html/django/DPS/env/lib/python3.9/site-packages/pandas/io/pytables.py_ensure_decodeds  rScCs|dur t}|SN)_default_encodingencodingrQrQrR_ensure_encodingsrXcCst|trt|}|S)z Ensure that an index / column name is a str (python 3); otherwise they may be np.string dtype. Non-string dtypes are passed through unchanged. https://github.com/pandas-dev/pandas/issues/13492 )rLstrnamerQrQrR _ensure_strs r\int scope_levelcsV|dt|ttfr*fdd|D}nt|r>t|d}|dusNt|rR|SdS)z Ensure that the where is a Term or a list of Term. This makes sure that we are capturing the scope of variables that are passed create the terms here with a frame_level=2 (we are 2 levels down) cs0g|](}|durt|r(t|ddn|qS)Nr`r^)r?Term).0termlevelrQrR sz _ensure_term..r^N)rLlisttupler?ralen)wherer_rQrdrR _ensure_terms   rkz where criteria is being ignored as this version [%s] is too old (or not-defined), read the file in and write it out to a new file to upgrade (with the copy_to method) r incompatibility_doczu the [%s] attribute of the existing index is [%s] which conflicts with the new [%s], resetting the attribute to None attribute_conflict_docz your performance may suffer as PyTables will pickle object types that it cannot map directly to c-types [inferred_type->%s,key->%s] [items->%s] performance_docfixedtable)frotrpz; : boolean drop ALL nan rows when appending to a table dropna_docz~ : format default format writing format, if None, then put will default to 'fixed' and append will default to 'table' format_doczio.hdfZ dropna_tableF) validatordefault_format)rorpNcCsLtdurHddl}|att|jjdkaWdn1s>0YtS)Nrstrict) _table_modtablesrAttributeErrorfileZ_FILE_OPEN_POLICY!_table_file_open_policy_is_strict)ryrQrQrR_tabless  r}aTrwzFilePath | HDFStorerYDataFrame | Series int | None str | Noneboolint | dict[str, int] | None bool | None Literal[True] | list[str] | NoneNone) path_or_bufkeyvaluemode complevelcomplibappendformatindex min_itemsizedropna data_columnserrorsrWreturnc s|r$ f dd}n f dd}t|}t|trt||||d}||Wdq1s0Yn||dS)z+store this object, close it if we opened itc s|j d S)N)rrrnan_reprrrrW)rstore rrrWrrrrrrrrQrR szto_hdf..c s|j d S)N)rrrrrrrWrputrrrQrRrs)rrrN)rDrLrYHDFStore)rrrrrrrrrrrrrrrWrqrrQrrRto_hdfs   (rrzstr | list | Nonezlist[str] | None) rrrrjstartstopcolumnsiterator chunksizec  Ks|dvrtd|d|dur,t|dd}t|trN|jsDtd|} d} ntt|}t|tshtd zt j |} Wnt tfyd} Yn0| st d |d t|f||d | } d } zt|dur| }t|dkrtd|d}|ddD]}t||stdq|j}| j||||||| | dWStt tfyt|tstt| Wdn1s0YYn0dS)a) Read from the store, close it if we opened it. Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path_or_buf : str, path object, pandas.HDFStore Any valid string path is acceptable. Only supports the local file system, remote URLs and file-like objects are not supported. If you want to pass in a path object, pandas accepts any ``os.PathLike``. Alternatively, pandas accepts an open :class:`pandas.HDFStore` object. key : object, optional The group identifier in the store. Can be omitted if the HDF file contains a single pandas object. mode : {'r', 'r+', 'a'}, default 'r' Mode to use when opening the file. Ignored if path_or_buf is a :class:`pandas.HDFStore`. Default is 'r'. errors : str, default 'strict' Specifies how encoding and decoding errors are to be handled. See the errors argument for :func:`open` for a full list of options. where : list, optional A list of Term (or convertible) objects. start : int, optional Row number to start selection. stop : int, optional Row number to stop selection. columns : list, optional A list of columns names to return. iterator : bool, optional Return an iterator object. chunksize : int, optional Number of rows to include in an iteration when using an iterator. **kwargs Additional keyword arguments passed to HDFStore. Returns ------- item : object The selected object. Return type depends on the object stored. See Also -------- DataFrame.to_hdf : Write a HDF file from a DataFrame. HDFStore : Low-level access to HDF files. Examples -------- >>> df = pd.DataFrame([[1, 1.0, 'a']], columns=['x', 'y', 'z']) # doctest: +SKIP >>> df.to_hdf('./store.h5', 'data') # doctest: +SKIP >>> reread = pd.read_hdf('./store.h5') # doctest: +SKIP )rr+r~zmode zG is not allowed while performing a read. Allowed modes are r, r+ and a.Nr`r^z&The HDFStore must be open for reading.Fz5Support for generic buffers has not been implemented.zFile z does not exist)rrTrz]Dataset(s) incompatible with Pandas data types, not table, or no datasets found in HDF5 file.z?key must be provided when HDF5 file contains multiple datasets.)rjrrrrr auto_close) ValueErrorrkrLris_openOSErrorrDrYNotImplementedErrorospathexists TypeErrorFileNotFoundErrorgroupsri_is_metadata_of _v_pathnameselectKeyErrorrrzclose)rrrrrjrrrrrkwargsrrrrZcandidate_only_groupZgroup_to_checkrQrQrRread_hdf2sjO           (rrI)group parent_grouprcCsF|j|jkrdS|}|jdkrB|j}||kr:|jdkr:dS|j}qdS)zDCheck if a given group is a metadata group for a given parent_group.Fr`metaT)Z_v_depthZ _v_parent_v_name)rrcurrentparentrQrQrRrs  rc@s"eZdZUdZded<ded<ded<ded <ddd dddddZddddZeddZeddddZ ddddZ dddddZ ddddd Z dd!d"d#Z dddd$d%Zddd&d'Zddd(d)Zddd*d+Zddd,d-Zddd/d0d1d2Zd3dd4d5Zd6dd7d8Zd9d:Zdddd;dd?Zeddd@dAZddddBdCdDZdddEdFZddddGdHdIZddd d dJdKdLZdddd d dMdNdOZdddPdQdRZdddUd dVdWdddddX dYdZZ ddddd[d\Z!dddUd dVd]dWddd^d_d`Z"ddaddbdcddZ#ddd deddfdgdhZ$diddjdkZ%dddmdndodpZ&ddqddrdsZ'ddtddudvZ(ddd dddxdydzZ)ddd{d|Z*d}d~Z+dddddZ,dddddtdddZ-dddUd dVddddddZ.ddddZ/ddddddZ0dddddZ1d S)raa Dict-like IO interface for storing pandas objects in PyTables. Either Fixed or Table format. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path : str File path to HDF5 file. mode : {'a', 'w', 'r', 'r+'}, default 'a' ``'r'`` Read-only; no data can be modified. ``'w'`` Write; a new file is created (an existing file with the same name would be deleted). ``'a'`` Append; an existing file is opened for reading and writing, and if the file does not exist it is created. ``'r+'`` It is similar to ``'a'``, but the file must already exist. complevel : int, 0-9, default None Specifies a compression level for data. A value of 0 or None disables compression. complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib' Specifies the compression library to be used. As of v0.20.2 these additional compressors for Blosc are supported (default if no compressor specified: 'blosc:blosclz'): {'blosc:blosclz', 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy', 'blosc:zlib', 'blosc:zstd'}. Specifying a compression library which is not available issues a ValueError. fletcher32 : bool, default False If applying compression use the fletcher32 checksum. **kwargs These parameters will be passed to the PyTables open_file method. Examples -------- >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5') >>> store['foo'] = bar # write to HDF5 >>> bar = store['foo'] # retrieve >>> store.close() **Create or load HDF5 file in-memory** When passing the `driver` option to the PyTables open_file method through **kwargs, the HDF5 file is loaded or created in-memory and will only be written when closed: >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5', driver='H5FD_CORE') >>> store['foo'] = bar >>> store.close() # only now, data is written to disk z File | None_handlerY_moder] _complevelr _fletcher32r~NFrr)rr fletcher32rcKsd|vrtdtd}|dur@||jjvr@td|jjd|durX|durX|jj}t||_|durnd}||_d|_|r|nd|_ ||_ ||_ d|_ |j fd|i|dS) Nrz-format is not a defined argument for HDFStoreryzcomplib only supports z compression.r~rr)rrfiltersZ all_complibsZdefault_complibrD_pathrrr_complibr_filtersopen)selfrrrrrrryrQrQrR__init__"s&  zHDFStore.__init__rcCs|jSrTrrrQrQrR __fspath__DszHDFStore.__fspath__cCs||jdusJ|jjS)zreturn the root nodeN)_check_if_openrrootrrQrQrRrGsz HDFStore.rootcCs|jSrTrrrQrQrRfilenameNszHDFStore.filenamercCs ||SrT)getrrrQrQrR __getitem__RszHDFStore.__getitem__rrcCs|||dSrTr)rrrrQrQrR __setitem__UszHDFStore.__setitem__cCs ||SrT)removerrQrQrR __delitem__XszHDFStore.__delitem__rZc CsDz ||WSttfy"Yn0tdt|jd|ddS)z$allow attribute access to get stores'z' object has no attribute 'N)rrrrztype__name__)rr[rQrQrR __getattr__[s zHDFStore.__getattr__cCs8||}|dur4|j}||ks0|dd|kr4dSdS)zx check for existence of this key can match the exact pathname or the pathnm w/o the leading '/' Nr`TF)get_noder)rrnoder[rQrQrR __contains__es  zHDFStore.__contains__cCs t|SrT)rirrrQrQrR__len__qszHDFStore.__len__cCst|j}t|d|dS)N File path:  )rFrr)rpstrrQrQrR__repr__ts zHDFStore.__repr__cCs|SrTrQrrQrQrR __enter__xszHDFStore.__enter__cCs |dSrT)r)rexc_type exc_value tracebackrQrQrR__exit__{szHDFStore.__exit__pandas list[str])includercCs^|dkrdd|DS|dkrJ|jdus0Jdd|jjddd DStd |d dS) a# Return a list of keys corresponding to objects stored in HDFStore. Parameters ---------- include : str, default 'pandas' When kind equals 'pandas' return pandas objects. When kind equals 'native' return native HDF5 Table objects. .. versionadded:: 1.1.0 Returns ------- list List of ABSOLUTE path-names (e.g. have the leading '/'). Raises ------ raises ValueError if kind has an illegal value rcSsg|] }|jqSrQrrbnrQrQrRrfz!HDFStore.keys..nativeNcSsg|] }|jqSrQrrrQrQrRrfs/Table) classnamez8`include` should be either 'pandas' or 'native' but is 'r)rrZ walk_nodesr)rrrQrQrRkeys~s z HDFStore.keysz Iterator[str]cCs t|SrT)iterrrrQrQrR__iter__szHDFStore.__iter__zIterator[tuple[str, list]]ccs|D]}|j|fVqdS)' iterate on key->group N)rr)rgrQrQrRitemss zHDFStore.itemsccs$tjdttd|EdHdS)rzTiteritems is deprecated and will be removed in a future version. Use .items instead. stacklevelN)warningswarn FutureWarningr$rrrQrQrR iteritemss zHDFStore.iteritems)rrcKst}|j|krR|jdvr$|dvr$n(|dvrL|jrLtd|jd|jd||_|jr`||jr|jdkrtj|j|j|j d|_ t r|jrd }t ||j |j|jfi||_d S) a9 Open the file in the specified mode Parameters ---------- mode : {'a', 'w', 'r', 'r+'}, default 'a' See HDFStore docstring or tables.open_file for info about modes **kwargs These parameters will be passed to the PyTables open_file method. )r~w)rr)rzRe-opening the file [z ] with mode [z] will delete the current file!r)rzGCannot open HDF5 file, which is already opened, even in read-only mode.N)r}rrr"rrrFiltersrrrr|r open_filer)rrrrymsgrQrQrRrs*    z HDFStore.opencCs|jdur|jd|_dS)z0 Close the PyTables file handle N)rrrrQrQrRrs  zHDFStore.closecCs|jdurdSt|jjS)zF return a boolean indicating whether the file is open NF)rrZisopenrrQrQrRrs zHDFStore.is_open)fsyncrcCsT|jdurP|j|rPtt t|jWdn1sF0YdS)a Force all buffered modifications to be written to disk. Parameters ---------- fsync : bool (default False) call ``os.fsync()`` on the file handle to force writing to disk. Notes ----- Without ``fsync=True``, flushing may not guarantee that the OS writes to disk. With fsync, the operation will block until the OS claims the file has been written; however, other caching layers may still interfere. N)rflushrrrrfileno)rrrQrQrRrs    zHDFStore.flushcCsVt<||}|dur*td|d||WdS1sH0YdS)z Retrieve pandas object stored in file. Parameters ---------- key : str Returns ------- object Same type as object stored in file. NNo object named in the file)rrr _read_grouprrrrQrQrRr s  z HDFStore.get)rrc st||} | dur"td|dt|dd}|| fdd} t|| |j|||||d } | S) a Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str Object being retrieved from file. where : list or None List of Term (or convertible) objects, optional. start : int or None Row number to start selection. stop : int, default None Row number to stop selection. columns : list or None A list of columns that if not None, will limit the return columns. iterator : bool or False Returns an iterator. chunksize : int or None Number or rows to include in iteration, return an iterator. auto_close : bool or False Should automatically close the store when finished. Returns ------- object Retrieved object from file. Nrrr`r^csj|||dS)N)rrrjrread_start_stop_whererrPrQrRfuncWszHDFStore.select..funcrjnrowsrrrrr)rrrk_create_storer infer_axes TableIteratorr get_result) rrrjrrrrrrrritrQrrRrs(.    zHDFStore.selectrrrcCs8t|dd}||}t|ts(td|j|||dS)a return the selection as an Index .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection r`r^z&can only read_coordinates with a tablerjrr)rk get_storerrLrrread_coordinates)rrrjrrtblrQrQrRselect_as_coordinatesjs    zHDFStore.select_as_coordinates)rcolumnrrcCs,||}t|tstd|j|||dS)a~ return a single column from the table. This is generally only useful to select an indexable .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str column : str The column of interest. start : int or None, default None stop : int or None, default None Raises ------ raises KeyError if the column is not found (or key is not a valid store) raises ValueError if the column can not be extracted individually (it is part of a data block) z!can only read_column with a tabler rr)rrLrr read_column)rrr rrrrQrQrR select_columns#  zHDFStore.select_column)rc  szt|dd}t|ttfr.t|dkr.|d}t|trRj||||||| dSt|ttfshtdt|sxtd|dur|d}fdd |D |} d} t | |fgt |D]\\} } | durt d | d | jstd | jd | dur | j} q| j| krtdqdd D}tdd|Ddfdd}t| ||| ||||| d }|jddS)a Retrieve pandas objects from multiple tables. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- keys : a list of the tables selector : the table to apply the where criteria (defaults to keys[0] if not supplied) columns : the columns I want back start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection iterator : bool, return an iterator, default False chunksize : nrows to include in iteration, return an iterator auto_close : bool, default False Should automatically close the store when finished. Raises ------ raises KeyError if keys or selector is not found or keys is empty raises TypeError if keys is not a list or tuple raises ValueError if the tables are not ALL THE SAME DIMENSIONS r`r^r)rrjrrrrrrzkeys must be a list/tuplez keys must have a non-zero lengthNcsg|]}|qSrQ)rrbkrrQrRrfrz/HDFStore.select_as_multiple..zInvalid table []zobject [z>] is not a table, and cannot be used in all select as multiplez,all tables must have exactly the same nrows!cSsg|]}t|tr|qSrQ)rLrrbxrQrQrRrf rcSsh|]}|jddqSr)non_index_axesrbrrrQrQrR rz.HDFStore.select_as_multiple..cs*fddD}t|ddS)Ncsg|]}|jdqS)rjrrrr r+)rrrrrQrRrfsz=HDFStore.select_as_multiple..func..F)axisverify_integrity)r8 _consolidate)rrrobjs)r.rtblsr rRrsz)HDFStore.select_as_multiple..funcrT coordinates)rkrLrgrhrirYrrrr itertoolschainzipris_tablepathnamerrr)rrrjselectorrrrrrrrPrrrr%Z_tblsrrrQ)r.rrr2rRselect_as_multiplesd+           zHDFStore.select_as_multipleTrwrrr) rrrrrr track_timesrrcCsH|durtdpd}||}|j||||||||| | | | | |ddS)a Store object in HDFStore. Parameters ---------- key : str value : {Series, DataFrame} format : 'fixed(f)|table(t)', default is 'fixed' Format to use when storing object in HDFStore. Value can be one of: ``'fixed'`` Fixed format. Fast writing/reading. Not-appendable, nor searchable. ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. index : bool, default True Write DataFrame index as a column. append : bool, default False This will force Table format, append the input data to the existing. data_columns : list of columns or True, default None List of columns to create as data columns, or True to use all columns. See `here `__. encoding : str, default None Provide an encoding for strings. track_times : bool, default True Parameter is propagated to 'create_table' method of 'PyTables'. If set to False it enables to have the same h5 files (same hashes) independent on creation time. dropna : bool, default False, optional Remove missing values. .. versionadded:: 1.1.0 Nio.hdf.default_formatro) rrrrrrrrrWrr<r)r_validate_format_write_to_group)rrrrrrrrrrrrWrr<rrQrQrRr,s&4  z HDFStore.putc Cst|dd}z||}Wnty.Yntty@Ynbty}zJ|dur`td|||}|dur|jddWYd}~dSWYd}~n d}~00t |||r|j jddn|j std|j |||dSdS) a: Remove pandas object partially by specifying the where condition Parameters ---------- key : str Node to remove or delete rows from where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection Returns ------- number of rows removed (or None if not a Table) Raises ------ raises KeyError if key is not a valid store r`r^Nz5trying to remove a node with a non-None where clause!T recursivez7can only remove with where on objects written as tablesr) rkrrAssertionError ExceptionrrZ _f_removecomall_nonerr8delete)rrrjrrrPerrrrQrQrRrts2     $zHDFStore.remover)rrrrrrrrcCsl| durtd|dur td}|dur4tdp2d}||}|j||||||||| | | | ||||ddS)a Append to Table in file. Node must already exist and be Table format. Parameters ---------- key : str value : {Series, DataFrame} format : 'table' is the default Format to use when storing object in HDFStore. Value can be one of: ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. index : bool, default True Write DataFrame index as a column. append : bool, default True Append the input data to the existing. data_columns : list of columns, or True, default None List of columns to create as indexed data columns for on-disk queries, or True to use all columns. By default only the axes of the object are indexed. See `here `__. min_itemsize : dict of columns that specify minimum str sizes nan_rep : str to use as str nan representation chunksize : size to chunk the writing expectedrows : expected TOTAL row size of this table encoding : default None, provide an encoding for str dropna : bool, default False, optional Do not write an ALL nan row to the store settable by the option 'io.hdf.dropna_table'. Notes ----- Does *not* check if data being appended overlaps with existing data in the table, so be careful Nz>columns is not a supported keyword in append, try data_columnszio.hdf.dropna_tabler=rp)raxesrrrrrrr expectedrowsrrrWr)rrr>r?)rrrrrHrrrrrrrrrIrrrWrrQrQrRrs6;  zHDFStore.appenddict)drc s|durtdt|ts"td||vr2tdtttjttt d}d} g} | D]0\} dur| durtd| } qh| qh| durֈj |} | t| } t| | } | | || <|dur||}|r*fdd|D}t|}|D]}||}qj||d d}| D]h\} | |krT|nd}j|d }|durfd d | Dnd}|j| |f||d |q>dS)a Append to multiple tables Parameters ---------- d : a dict of table_name to table_columns, None is acceptable as the values of one node (this will get all the remaining columns) value : a pandas object selector : a string that designates the indexable table; all of its columns will be designed as data_columns, unless data_columns is passed, in which case these are used data_columns : list of columns to create as data columns, or True to use all columns dropna : if evaluates to True, drop rows from all tables if any single row in each table has all NaN. Default False. Notes ----- axes parameter is currently not accepted Nztaxes is currently not accepted as a parameter to append_to_multiple; you can create the tables independently insteadzQappend_to_multiple must have a dictionary specified as the way to split the valuez=append_to_multiple requires a selector that is in passed dictrzPrz.HDFStore.append_to_multiple..rr.csi|]\}}|vr||qSrQrQrbrr)vrQrR `rz/HDFStore.append_to_multiple..)rr)rrLrJrrgsetrangendim _AXES_MAPrrextendrH differencer3sorted get_indexertakevaluesnext intersectionlocpopreindexr)rrKrr:rrHrrr.Z remain_keyZ remain_valuesr%orderedZorddidxsZ valid_indexrrdcvalfilteredrQ)rRrrRappend_to_multiplesZ &    zHDFStore.append_to_multipler)roptlevelkindrcCsBt||}|durdSt|ts.td|j|||ddS)a Create a pytables index on the table. Parameters ---------- key : str columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError: raises if the node is not a table Nz1cannot create table index on a Fixed format store)rrirj)r}rrLrr create_index)rrrrirjrPrQrQrRcreate_table_indexfs  zHDFStore.create_table_indexrgcCs<t||jdusJtdus(Jdd|jDS)z Return a list of all the top-level nodes. Each node returned is not a pandas storage object. Returns ------- list List of objects. NcSsPg|]H}t|tjjst|jddsHt|ddsHt|tjjr|jdkr|qS) pandas_typeNrp) rLrxlinkLinkgetattr_v_attrsrprr)rbrrQrQrRrfs  z#HDFStore.groups..)r}rrrx walk_groupsrrQrQrRrs  zHDFStore.groupsrz*Iterator[tuple[str, list[str], list[str]]])rjrccst||jdusJtdus(J|j|D]}t|jdddurLq4g}g}|jD]B}t|jdd}|durt |tj j r| |j q^| |j q^|jd||fVq4dS)aS Walk the pytables group hierarchy for pandas objects. This generator will yield the group path, subgroups and pandas object names for each group. Any non-pandas PyTables objects that are not a group will be ignored. The `where` group itself is listed first (preorder), then each of its child groups (following an alphanumerical order) is also traversed, following the same procedure. Parameters ---------- where : str, default "/" Group where to start walking. Yields ------ path : str Full path to a group (without trailing '/'). groups : list Names (strings) of the groups contained in `path`. leaves : list Names (strings) of the pandas objects contained in `path`. Nrmr)r}rrrxrrrprqZ _v_childrenr]rLrGrouprrrrstrip)rrjrrleaveschildrmrQrQrRwalks  z HDFStore.walkz Node | NonecCs~||dsd|}|jdus(Jtdus4Jz|j|j|}Wntjjy`YdS0t|tj szJt ||S)z9return the node with the key or None if it does not existrN) r startswithrrxrr exceptionsZNoSuchNodeErrorrLrIr)rrrrQrQrRrs  zHDFStore.get_nodeGenericFixed | TablecCs8||}|dur"td|d||}||S)z| | vrj|rj| | || } t| trd} |rdd| j D} | j | | | t | dd| j dq>| j | | | j dq>| S) a; Copy the existing store to a new file, updating in place. Parameters ---------- propindexes : bool, default True Restore indexes in copied file. keys : list, optional List of keys to include in the copy (defaults to all). overwrite : bool, default True Whether to overwrite (remove and replace) existing nodes in the new store. mode, complib, complevel, fletcher32 same as in HDFStore.__init__ Returns ------- open file handle of the new store )rrrrNFcSsg|]}|jr|jqSrQ) is_indexedr[rbr~rQrQrRrf$rz!HDFStore.copy..r)rrrWrV)rrgrrLrhrrrrrHrrprWr)rr{rr{rrrr overwrite new_storer%rPdatarrQrQrRcopys6       z HDFStore.copyc Cs t|j}t|d|d}|jrt|}t|rg}g}|D]}z<||}|dur|t|j pj||t|p|dWqDt yYqDt y}z0||t|} |d| dWYd}~qDd}~00qD|t d||7}n|d7}n|d 7}|S) zg Print detailed information on the store. Returns ------- str rrNzinvalid_HDFStore nodez[invalid_HDFStore node: r& EmptyzFile is CLOSED) rFrrrrZrrirrr9rBrCrE) rroutputZlkeysrr]r%rPdetailZdstrrQrQrRinfo1s.     * z HDFStore.infocCs|jst|jddS)Nz file is not open!)rrrrrQrQrRr[szHDFStore._check_if_open)rrc CsLzt|}Wn6tyF}ztd|d|WYd}~n d}~00|S)zvalidate / deprecate formatsz#invalid HDFStore format specified [r&N) _FORMAT_MAPlowerrr)rrrGrQrQrRr>_s (zHDFStore._validate_formatrKzDataFrame | Series | None)rrWrrc s"durtttfstdfdd}ttjdd}ttjdd}|durƈdurttdustJtddsttj j rd}d }qtd n$ttrd }nd }dkr|d 7}d|vr(t t d} z | |} Wn0t y} z|d| WYd} ~ n d} ~ 00| |||dS|durˆdur|dkrtdd} | dur| jdkrnd}n| jdkrd}nB|dkrtdd} | dur| jdkrd}n| jdkrd}ttttttd} z | |} Wn0t y} z|d| WYd} ~ n d} ~ 00| |||dS)z"return a suitable class to operateNz(value must be None, Series, or DataFramec s$td|ddtdS)Nz(cannot properly create the storer for: [z ] [group->,value->z ,format->)rr)rrrrrrQrRerrorwsz&HDFStore._create_storer..errorrm table_typerp frame_table generic_tablezKcannot create a storer if the object is not existing nor a value are passedseriesframe_table)rr _STORER_MAPrWr series_tablerr`appendable_seriesappendable_multiseriesappendable_frameappendable_multiframe)rrrrrworm _TABLE_MAP)rLr6r1rrSrprqr}rxrpr SeriesFixed FrameFixedrnlevels GenericTableAppendableSeriesTableAppendableMultiSeriesTableAppendableFrameTableAppendableMultiFrameTable WORMTable)rrrrrWrrptttrclsrGrrrQrrRrisr                     zHDFStore._create_storer)rrrrrr<rcCst|ddr|dks|rdS|||}|j|||||d}|rr|jrZ|jrb|dkrb|jrbtd|jsz|n||js|rtd|j|||||| | | | | |||d t|t r|r|j |ddS) NemptyrprrozCan only append to Tablesz0Compression not supported on Fixed format stores) objrHrrrrrrrIrrrr<)r) rp_identify_grouprr8 is_existsrset_object_infowriterLrrk)rrrrrHrrrrrrrrIrrrrWrr<rrPrQrQrRr?s:   zHDFStore._write_to_grouprIrcCs||}||SrT)rrr )rrrPrQrQrRr s zHDFStore._read_group)rrrcCsN||}|jdusJ|dur8|s8|jj|ddd}|durJ||}|S)z@Identify HDF5 group based on key, delete/create group if needed.NTr@)rr remove_node_create_nodes_and_group)rrrrrQrQrRrs   zHDFStore._identify_groupcCsv|jdusJ|d}d}|D]P}t|s.q |}|dsD|d7}||7}||}|durl|j||}|}q |S)z,Create nodes from key and return group name.Nr)rsplitriendswithr create_group)rrpathsrpnew_pathrrQrQrRrs   z HDFStore._create_nodes_and_group)r~NNF)r)r~)F)NNNNFNF)NNN)NN)NNNNNFNF) NTFNNNNNNrwTF)NNN)NNTTNNNNNNNNNNrw)NNF)NNN)r)rTNNNFT)NNrKrw)NTFNNNNNNFNNNrwT)2r __module__ __qualname____doc____annotations__rrpropertyrrrrrrrrrrrrrrrrrrrrrrr#r;rrrrhrlrrwrrrrrr>rr?r rrrQrQrQrRrs A"   " -N$+~ H=]d(0 =* a>rc@sjeZdZUdZded<ded<ded<dddd dd d d d dZddZd dddZdd dddZdS)raa Define the iteration interface on a table Parameters ---------- store : HDFStore s : the referred storer func : the function to execute the query where : the where of the query nrows : the rows to iterate on start : the passed start value (default is None) stop : the passed stop value (default is None) iterator : bool, default False Whether to use the default iterator. chunksize : the passed chunking value (default is 100000) auto_close : bool, default False Whether to automatically close the store at the end of iteration. rrrrrzrPNFrr)rrPrrrrc Cs||_||_||_||_|jjrN|dur,d}|dur8d}|durD|}t||}||_||_||_d|_ |sr| dur| dur~d} t | |_ nd|_ | |_ dS)Nr順) rrPrrjr8minrrrr4r]rr) rrrPrrjrrrrrrrQrQrRrHs,    zTableIterator.__init__ccsv|j}|jdurtd||jkrjt||j|j}|dd|j||}|}|dust|sbq|Vq|dS)Nz*Cannot iterate until get_result is called.) rr4rrrrrrir)rrrrrQrQrRrrs  zTableIterator.__iter__rcCs|jr|jdSrT)rrrrrQrQrRrszTableIterator.closer3cCs|jdur4t|jtstd|jj|jd|_|S|rft|jtsLtd|jj|j|j|j d}n|j}| |j|j |}| |S)Nz0can only use an iterator or chunksize on a table)rjz$can only read_coordinates on a tabler) rrLrPrrrrjr4rrrr)rr4rjresultsrQrQrRrs    zTableIterator.get_result)NNFNF)F) rrrrrrrrrrQrQrQrRr0s  *rc @seZdZUdZdZded<dZded<gdZded<ded <dLdd d d ddZe ddddZ e ddddZ dd dddZ ddddZ dddddZdddd Ze ddd!d"Zd#ddd$d%d&d'Zd(d)Ze d*d+Ze d,d-Ze d.d/Ze d0d1Zd2d3ZdMd dd4d5Zd dd6d7Zd8dd d9d:d;ZdNdd?d@Zd ddAdBZd ddCdDZd ddEdFZd8d dGdHdIZ d8d dGdJdKZ!d S)OIndexCola an index column description class Parameters ---------- axis : axis which I reference values : the ndarray like converted values kind : a string description of this type typ : the pytables type pos : the position in the pytables Tris_an_indexableis_data_indexable)freqtz index_namerYr[cnameNrr)r[rrcCst|tstd||_||_||_||_|p0||_||_||_ ||_ | |_ | |_ | |_ | |_| |_||_|dur|||t|jtsJt|jtsJdS)Nz`name` must be a str.)rLrYrr]rjtypr[rr.posrrrrcrprmetadataset_pos)rr[r]rjrrr.rrrrrcrprrrQrQrRrs(   zIndexCol.__init__r]rcCs|jjSrT)ritemsizerrQrQrRrszIndexCol.itemsizecCs |jdS)N_kindrZrrQrQrR kind_attrszIndexCol.kind_attr)rrcCs$||_|dur |jdur ||j_dS)z,set the position of this column in the TableN)rrZ_v_pos)rrrQrQrRrszIndexCol.set_poscCs@ttt|j|j|j|j|jf}dddt gd|DS)N,cSsg|]\}}|d|qSz->rQrQrQrQrRrfsz%IndexCol.__repr__..)r[rr.rrj) rhmaprFr[rr.rrjjoinr7rtemprQrQrRrs zIndexCol.__repr__rotherrcstfdddDS)compare 2 col itemsc3s&|]}t|dt|dkVqdSrTrpr}rrrQrRrOsz"IndexCol.__eq__..)r[rr.rrLrrrQrrR__eq__szIndexCol.__eq__cCs || SrT)rrrQrQrR__ne__szIndexCol.__ne__cCs"t|jdsdSt|jj|jjS)z%return whether I am an indexed columnrNF)hasattrrprprNrr|rrQrQrRr|s zIndexCol.is_indexed np.ndarrayzCtuple[np.ndarray, np.ndarray] | tuple[DatetimeIndex, DatetimeIndex]r]rWrrc Cst|tjsJt||jjdur.||j}t|j}t ||||}i}t|j |d<|j durpt|j |d<t }t |jst|jrt}n|jdkrd|vrdd}z||fi|}Wn2tyd|vrd|d<||fi|}Yn0t||j} | | fS)zV Convert the data from this selection to the appropriate pandas type. Nr[ri8c[stfd|i|S)Nordinal)r5)r(kwdsrQrQrRr*sz"IndexCol.convert..)rLrMndarrayrdtypefieldsrrSrj_maybe_convertrrr3r)r*r2r_set_tzr) rr]rrWrval_kindrfactoryZ new_pd_indexZfinal_pd_indexrQrQrRconvert s,      zIndexCol.convertcCs|jS)zreturn the valuesr]rrQrQrR take_data:szIndexCol.take_datacCs|jjSrT)rprqrrQrQrRattrs>szIndexCol.attrscCs|jjSrTrp descriptionrrQrQrRrBszIndexCol.descriptioncCst|j|jdS)z!return my current col descriptionN)rprrrrQrQrRcolFsz IndexCol.colcCs|jSzreturn my cython valuesrrrQrQrRcvaluesKszIndexCol.cvaluescCs t|jSrT)rr]rrQrQrRrPszIndexCol.__iter__cCsPt|jdkrLt|tr$||j}|durL|jj|krLtj ||j d|_dS)z maybe set a string col itemsize: min_itemsize can be an integer or a dict with this columns name with an integer size stringN)rr) rSrjrLrJrr[rrr} StringColr)rrrQrQrRmaybe_set_sizeSs   zIndexCol.maybe_set_sizecCsdSrTrQrrQrQrRvalidate_names`szIndexCol.validate_namesAppendableTable)handlerrrcCs:|j|_||||||||dSrT)rp validate_col validate_attrvalidate_metadatawrite_metadataset_attr)rrrrQrQrRvalidate_and_setcs    zIndexCol.validate_and_setcCs^t|jdkrZ|j}|durZ|dur*|j}|j|krTtd|d|jd|jd|jSdS)z:validate this column: return the compared against itemsizerNz#Trying to store a string with len [z] in [z)] column but this column has a limit of [zC]! Consider using min_itemsize to preset the sizes on these columns)rSrjrrrr)rrcrQrQrRrks zIndexCol.validate_col)rrcCsB|r>t|j|jd}|dur>||jkr>td|d|jddS)Nzincompatible kind in col [ - r&)rprrrjr)rrZ existing_kindrQrQrRr~s zIndexCol.validate_attrc Cs|jD]}t||d}||ji}||}||vr|dur||kr|dvrt|||f}tj|tt dd||<t ||dqt d|jd|d|d|d q|dus|dur|||<qdS) z set/update the info for this indexable with the key/value if there is a conflict raise/warn as needed N)rrrzinvalid info for [z] for [z], existing_value [z] conflicts with new value [r&) _info_fieldsrp setdefaultr[rrmrrrr$setattrr)rrrridxexisting_valuewsrQrQrR update_infos*   zIndexCol.update_infocCs$||j}|dur |j|dS)z!set my state from the passed infoN)rr[__dict__update)rrrrQrQrRset_infos zIndexCol.set_infocCst|j|j|jdS)zset the kind for this columnN)rrrrjrrQrQrRrszIndexCol.set_attr)rrcCsB|jdkr>|j}||j}|dur>|dur>t||s>tddS)z:validate that kind=category does not change the categoriescategoryNzEcannot append a categorical with different categories to the existing)rr read_metadatarr0r)rrZ new_metadataZ cur_metadatarQrQrRrs  zIndexCol.validate_metadatacCs|jdur||j|jdS)zset the meta dataN)rrr)rrrQrQrRrs zIndexCol.write_metadata) NNNNNNNNNNNNN)N)N)"rrrrrrrrrrrrrrrrr|rrrrrrrrrrrrrrrrrrQrQrQrRrsf   , -       !rc@sDeZdZdZeddddZddddd d d Zd dd dZdS)GenericIndexColz:an index which is not represented in the data of the tablerrcCsdSNFrQrrQrQrRr|szGenericIndexCol.is_indexedrrYztuple[Int64Index, Int64Index]rcCs2t|tjsJt|ttt|}||fS)z Convert the data from this selection to the appropriate pandas type. Parameters ---------- values : np.ndarray nan_rep : str encoding : str errors : str )rLrMrrr:arangeri)rr]rrWrrrQrQrRrs zGenericIndexCol.convertrcCsdSrTrQrrQrQrRrszGenericIndexCol.set_attrN)rrrrrr|rrrQrQrQrRr s r c s<eZdZdZdZdZddgZd:dddd fd d Zedd d dZ edd ddZ dd ddZ dddddZ dddddZ ddZedddd d!Zed"d#Zedd$d%d&d'Zeddd%d(d)Zed*d+Zed,d-Zed.d/Zed0d1Zdd d2d3Zd4ddd5d6d7Zdd d8d9ZZS);DataCola3 a data holding column, by definition this is not indexable Parameters ---------- data : the actual data cname : the column name in the table to hold the data (typically values) meta : a string description of the metadata metadata : the actual metadata FrrcNrYzDtypeArg | Noner)r[rrc s2tj||||||||| | | d | |_| |_dS)N) r[r]rjrrrrrcrprr)superrrr)rr[r]rjrrrrrcrprrrr __class__rQrRrs zDataCol.__init__rcCs |jdS)N_dtyperZrrQrQrR dtype_attr szDataCol.dtype_attrcCs |jdS)N_metarZrrQrQrR meta_attr szDataCol.meta_attrcCs@ttt|j|j|j|j|jf}dddt gd|DS)NrcSsg|]\}}|d|qSrrQrQrQrQrRrf( sz$DataCol.__repr__..)r[rrrjshape) rhrrFr[rrrjrrr7rrQrQrRr! s zDataCol.__repr__rrrcstfdddDS)rc3s&|]}t|dt|dkVqdSrTrr}rrQrRrO0 sz!DataCol.__eq__..)r[rrrrrrQrrRr. szDataCol.__eq__r)rrcCs@|dus J|jdusJt|\}}||_||_t||_dSrT)r_get_data_and_dtype_namer_dtype_to_kindrj)rr dtype_namerQrQrRset_data5 s   zDataCol.set_datacCs|jS)zreturn the datarrrQrQrRr? szDataCol.take_datarG)r]rcCs|j}|j}|j}|jdkr&d|jf}t|trJ|j}|j||jj d}ntt |sZt |rf| |}nXt |rz||}nDt|rtj||dd}n&t|r|||}n|j||j d}|S)zW Get an appropriately typed and shaped pytables.Col object for values. r`rjrrr)rrrrVsizerLr;codes get_atom_datar[r)r*get_atom_datetime64r.get_atom_timedelta64r(r}Z ComplexColr-get_atom_string)rr]rrrratomrQrQrR _get_atomC s$     zDataCol._get_atomcCstj||ddS)Nrrr}rrrrrQrQrRr#c szDataCol.get_atom_stringz type[Col]rjrcCsR|dr$|dd}d|d}n"|dr4d}n|}|d}tt|S)z0return the PyTables column class for this columnuintNUIntrGperiodInt64Col)rx capitalizerpr})rrjZk4col_nameZkcaprQrQrRget_atom_coltypeg s    zDataCol.get_atom_coltypecCs|j|d|ddS)Nrrrr0rrrjrQrQrRr v szDataCol.get_atom_datacCstj|ddSNrr1r}r-rrrQrQrRr!z szDataCol.get_atom_datetime64cCstj|ddSr4r5r6rQrQrRr"~ szDataCol.get_atom_timedelta64cCst|jddS)Nr)rprrrQrQrRr sz DataCol.shapecCs|jSrrrrQrQrRr szDataCol.cvaluescCs`|r\t|j|jd}|dur2|t|jkr2tdt|j|jd}|dur\||jkr\tddS)zAvalidate that we have the same order as the existing & same dtypeNz4appended items do not match existing items in table!z@appended items dtype do not match existing items dtype in table!)rprrrgr]rrr)rrZexisting_fieldsZexisting_dtyperQrQrRr szDataCol.validate_attrr)r]rWrcCst|tjsJt||jjdur.||j}|jdus.cSsg|]}t|qSrQr fromtimestampr?rQrQrRrf rr ) categoriesrcFrOrrrWr)"rLrMrrrrrrrrrjrSrrrcrrasarrayobjectrravelr3float64r9anyastyper]cumsum_valuesr; from_codesr_unconvert_string_arrayr])rr]rrWr convertedrrjrrrcrrrCrmaskrQrQrRr sf              zDataCol.convertcCsHt|j|j|jt|j|j|j|jdus2Jt|j|j|jdS)zset the data for this columnN)rrrr]rrrrrrQrQrRr szDataCol.set_attr) NNNNNNNNNNNN)rrrrrrrrrrrrrrr classmethodr%r#r0r r!r"rrrrr __classcell__rQrQrrRrsX          erc@sZeZdZdZdZddddZeddZed d d d d ZeddZ eddZ dS)DataIndexableColz+represent a data column that can be indexedTrrcCst|jstddS)N-cannot have non-object label DataIndexableCol)r3r] is_objectrrrQrQrRr szDataIndexableCol.validate_namescCstj|dS)N)rr&r'rQrQrRr# sz DataIndexableCol.get_atom_stringrYrGr(cCs|j|dS)Nrr2r3rQrQrRr  szDataIndexableCol.get_atom_datacCs tSrTr5r6rQrQrRr! sz$DataIndexableCol.get_atom_datetime64cCs tSrTr5r6rQrQrRr" sz%DataIndexableCol.get_atom_timedelta64N) rrrrrrrSr#r r!r"rQrQrQrRrU s  rUc@seZdZdZdS)GenericDataIndexableColz(represent a generic pytables data columnN)rrrrrQrQrQrRrX srXc@seZdZUdZded<dZded<ded<ded <ded <d ed <d ed<ded<dZded<dPd d ddddddZeddddZ eddddZ edd Z ddd!d"Z ddd#d$Z ddd%d&Zed'd(Zed)d*Zed+d,Zed-d.Zeddd/d0Zeddd1d2Zed3d4Zddd5d6Zddd7d8Zed9d:Zeddd;d<Zed=d>Zd?dd@dAZdQdddCdDZdddEdFZdRdGdGdHdIdJZdKdLZdSdGdGddMdNdOZ dBS)TFixedz represent an object in my store facilitate read/write of various types of objects this is an abstract base class Parameters ---------- parent : HDFStore group : Node The group node where the table resides. rY pandas_kindro format_typetype[DataFrame | Series]obj_typer]rVrWrrrIrrFrr8rKrwr)rrrWrrcCsZt|tsJt|tdus"Jt|tjs:Jt|||_||_t||_||_ dSrT) rLrrrxrIrrrXrWr)rrrrWrrQrQrRr= s  zFixed.__init__rcCs*|jddko(|jddko(|jddkS)Nrr` )versionrrQrQrRis_old_versionL szFixed.is_old_versionztuple[int, int, int]cCs`tt|jjdd}z0tdd|dD}t|dkrB|d}WntyZd}Yn0|S) zcompute and set our versionpandas_versionNcss|]}t|VqdSrTr]r'rQrQrRrOU rz Fixed.version...r_r))rrr)rSrprrqrhrrirz)rr`rQrQrRr`P s    z Fixed.versioncCstt|jjddS)Nrm)rSrprrqrrQrQrRrm\ szFixed.pandas_typecCs^||j}|durXt|ttfrDddd|D}d|d}|jdd|d S|jS) (return a pretty representation of myselfNrcSsg|] }t|qSrQrFr'rQrQrRrff rz"Fixed.__repr__..[r&12.12z (shape->))rrrLrgrhrrm)rrPZjshaperQrQrRr` s zFixed.__repr__cCst|j|j_tt|j_dS)zset my pandas type & versionN)rYrZrrm_versionrbrrQrQrRrk szFixed.set_object_infocCst|}|SrTrD)rnew_selfrQrQrRrp s z Fixed.copycCs|jSrT)rrrQrQrRrt sz Fixed.shapecCs|jjSrTrrrrQrQrRr9x szFixed.pathnamecCs|jjSrT)rrrrQrQrRr| sz Fixed._handlecCs|jjSrT)rrrrQrQrRr szFixed._filterscCs|jjSrT)rrrrQrQrRr szFixed._complevelcCs|jjSrT)rrrrQrQrRr szFixed._fletcher32cCs|jjSrT)rrqrrQrQrRr sz Fixed.attrscCsdSzset our object attributesNrQrrQrQrR set_attrs szFixed.set_attrscCsdS)zget our object attributesNrQrrQrQrR get_attrs szFixed.get_attrscCs|jS)zreturn my storablerrrQrQrRstorable szFixed.storablecCsdSr rQrrQrQrRr szFixed.is_existscCst|jddS)Nr)rprprrQrQrRr sz Fixed.nrowszLiteral[True] | NonecCs|dur dSdS)z%validate against an existing storableNTrQrrQrQrRvalidate szFixed.validateNcCsdS)+are we trying to operate on an old version?NrQ)rrjrQrQrRvalidate_version szFixed.validate_versioncCs|j}|durdS|dS)zr infer the axes of my storer return a boolean indicating if we have a valid storer or not NFT)rpro)rrPrQrQrRr s zFixed.infer_axesrrrcCs tddS)Nz>cannot read on an abstract storer: subclasses should implementrrrjrrrrQrQrRr  sz Fixed.readcKs tddS)Nz?cannot write on an abstract storer: subclasses should implementrurrrQrQrRr sz Fixed.writerrrcCs0t|||r$|jj|jdddStddS)zs support fully deleting the node in its entirety (only) - where specification must be None Tr@Nz#cannot delete on an abstract storer)rDrErrrr)rrjrrrQrQrRrF sz Fixed.delete)rKrw)N)NNNN)NNN)!rrrrrr[r8rrrar`rmrrrrr9rrrrrrnrorprrrqrsrr rrFrQrQrQrRrY& sn                rYc@sFeZdZUdZedediZddeDZgZ de d<dd d d Z d d Z ddZ dd ddZedd ddZdd ddZdd ddZdd ddZd:dddddd Zd;dddd!d"d#d$Zdd!dd%d&d'Zdd(dd%d)d*Zddd5d6dd7d8d9ZdS)? GenericFixedza generified fixed versiondatetimer,cCsi|]\}}||qSrQrQ)rbr%rRrQrQrRrS rzGenericFixed.r attributesrYrcCs|j|dS)N)_index_type_mapr)rrrQrQrR_class_to_alias szGenericFixed._class_to_aliascCst|tr|S|j|tSrT)rLr_reverse_index_maprr3)raliasrQrQrR_alias_to_class s zGenericFixed._alias_to_classcCs|tt|dd}|tkr.d dd}|}n|tkrFd dd}|}n|}i}d|vrn|d|d<|turnt}d|vrt|dtr|d d|d<n |d|d<|tusJ||fS) N index_classr|cSs:tj|j|d}tj|dd}|dur6|d|}|S)NrrZUTC)r< _simple_newr]r2 tz_localize tz_convert)r]rrdtaresultrQrQrRrq s z*GenericFixed._get_index_factory..fcSstj||d}tj|ddS)NrrZ)r=rr5)r]rrparrrQrQrRrq srrzutf-8)NN)NN) rrSrpr2r5r3r7rLbytesrO)rrrrqrrrQrQrR_get_index_factory s*     zGenericFixed._get_index_factoryrcCs$|durtd|dur tddS)zE raise if any keywords are passed which are not-None Nzqcannot pass a column specification when reading a Fixed format store. this store must be selected in its entiretyzucannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety)r)rrrjrQrQrR validate_read szGenericFixed.validate_readrcCsdS)NTrQrrQrQrRr& szGenericFixed.is_existscCs|j|j_|j|j_dSrm)rWrrrrQrQrRrn* s zGenericFixed.set_attrsc CsRtt|jdd|_tt|jdd|_|jD]}t||tt|j|dq.dS)retrieve our attributesrWNrrw)rXrprrWrSrr{r)rrrQrQrRro/ s zGenericFixed.get_attrscKs |dSrT)rnrrrrQrQrRr7 szGenericFixed.writeNrrc Csddl}t|j|}|j}t|dd}t||jrD|d||}nztt|dd} t|dd} | durxtj| | d}n |||}| dkrt|d d} t || d d }n| d krtj |d d}|r|j S|SdS)z2read an array for the specified node (off of grouprN transposedF value_typerr<r7rTr8r:r;) ryrprrqrLZVLArrayrSrMrrrGT) rrrrryrrrretrrrrQrQrR read_array: s&      zGenericFixed.read_arrayr3)rrrrcCshtt|j|d}|dkr.|j|||dS|dkrVt|j|}|j|||d}|Std|dS)N_varietymultirtregularzunrecognized index variety: )rSrprread_multi_indexrread_index_noder)rrrrvarietyrrrQrQrR read_index\ s zGenericFixed.read_index)rrrcCst|tr,t|j|dd|||nt|j|ddtd||j|j}|||j t |j |}|j |j _ |j|j _t|ttfr|t||j _t|tttfr|j|j _t|tr|jdurt|j|j _dS)Nrrrr)rLr4rrwrite_multi_index_convert_indexrWr write_arrayr]rprrjrqr[r2r5r~rrr7rr_get_tz)rrrrQrrQrQrR write_indexj s     zGenericFixed.write_indexr4c Cst|j|d|jtt|j|j|jD]\}\}}}t|rJt d|d|}t |||j |j }| ||jt|j|} |j| j_|| j_t| j|d|||d|} | | |q,dS)N_nlevelsz=Saving a MultiIndex with an extension dtype is not supported._level_name_label)rrr enumerater7levelsrnamesr+rrrWrrr]rprrjrqr[) rrrilev level_codesr[ level_keyZ conv_levelr label_keyrQrQrRr s"  zGenericFixed.write_multi_indexcCst|j|d}g}g}g}t|D]l}|d|} t|j| } |j| ||d} || || j|d|} |j| ||d} || q&t|||ddS)NrrrtrT)rrrr/) rprrUrrrr[rr4)rrrrrrrrrrrrrrrQrQrRr s      zGenericFixed.read_multi_indexrI)rrrrc Cs|||}d|jvr>t|jjdkr>tj|jj|jjd}t|jj}d}d|jvrlt|jj }t|}|j}| |\}} |dkr|t |||j |j dfdti| } n |t |||j |j dfi| } || _ | S)Nrrr<r[rrr)rqrMprodrrrrSrjr\r[r_unconvert_indexrWrrH) rrrrrrjr[rrrrrQrQrRr s:      zGenericFixed.read_index_noder)rrrcCsJtd|j}|j|j||t|j|}t|j|j _ |j |j _ dS)zwrite a 0-len arrayr`N) rMrrVr create_arrayrrprYrrqrr)rrrarrrrQrQrRwrite_array_empty s  zGenericFixed.write_array_emptyrz Index | None)rrrrc CsJt|dd}||jvr&|j|j||jdk}d}t|jrFtd|s^t|dr^|j }d}d}|j durt t  t j|j}Wdn1s0Y|dur|s|jj|j|||j|j d}||dd<n |||nL|jjtjkr`tj|dd} |rn,| d krn t| ||f} tj| ttd |j|j|t } | |nt |jr|j!|j||"d d t#|j|j$_%nt&|jr|j!|j||j't#|j|} t(|j)| j$_)d | j$_%n\t*|jr|j!|j||"d d t#|j|j$_%n&|r$|||n|j!|j|||t#|j|j$_+dS)NT) extract_numpyrFz]Cannot store a category dtype in a HDF5 dataset that uses format="fixed". Use format="table".r)rskipnarrrr7r:),r@rrrrr'rrrrrrrr}AtomZ from_dtypeZ create_carrayrrrrMobject_r infer_dtypernrrr!r$Zcreate_vlarray ObjectAtomrr)rviewrprqrr*asi8rrr.r) rrrrrZ empty_arrayrr$ca inferred_typerZvlarrrrQrQrRr sf       .        zGenericFixed.write_array)NN)NN)NN)NN)N)rrrrr2r5r}rrr{rr~rrrrrrnrorrrrrrrrrrQrQrQrRry s2   .#& rycsVeZdZUdZdgZded<eddZddddd d d Zd d fdd Z Z S)rrr[r c Cs.zt|jjfWSttfy(YdS0dSrT)rirr]rrzrrQrQrRrA szSeriesFixed.shapeNrr6rxcCs<||||jd||d}|jd||d}t|||jdS)Nrrtr])rr[)rrrr6r[)rrjrrrrr]rQrQrRr H s zSeriesFixed.readrrc s<tj|fi||d|j|d||j|j_dS)Nrr])rrrrrr[rrrrQrRrU s zSeriesFixed.write)NNNN) rrrrZr{rrrr rrTrQrQrrRr; s   rcsZeZdZUddgZded<eddddZdd d d d d d Zddfdd ZZ S)BlockManagerFixedrVnblocksr]z Shape | NonercCsz|j}d}t|jD]8}t|jd|d}t|dd}|dur||d7}q|jj}t|dd}|durt|d|d}ng}|||WStyYdS0dS)Nrblock_itemsrr`) rVrUrrprZ block0_valuesrgrrz)rrVrrrrrQrQrRra s"    zBlockManagerFixed.shapeNrr1rxcCs||||d}g}t|jD]<}||kr<||fnd\}} |jd||| d} || q(|d} g} t|jD]Z}|d|d} |jd|d|| d}| | | }t |j ||dd }| |q|t | dkrt | dd }|j| d d }|St |d|dd S) Nr)NNr.rtrrrNr`rrrPF)rr)rr]_get_block_manager_axisrUrVrrrrr[r1rrir8rb)rrjrrrZ select_axisrHrrraxrdfs blk_itemsr]dfoutrQrQrRr | s(    zBlockManagerFixed.readrc stj|fi|t|jtr*|d}|j}|s@|}|j|j _t |j D]0\}}|dkrr|j srt d|d||qTt|j|j _t |jD]D\}}|j|j}|jd|d|j|d|d|d|qdS)Nrrz/Columns index has to be unique for fixed formatr.rN)rr)rrrL_mgrrB _as_manageris_consolidated consolidaterVrrrH is_uniquerrriblocksrrr\mgr_locsrr])rrrrrrblkrrrQrRr s    zBlockManagerFixed.write)NNNN) rrrr{rrrr rrTrQrQrrRr\ s &rc@seZdZdZeZdS)rrN)rrrrZr1r]rQrQrQrRr srcseZdZUdZdZdZded<ded<dZded <d Zd ed <d ed<ded<ded<ded<ded<ddddddfdd Z e dddd Z ddd!d"Z dd#d$d%Z ddd&d'Ze d(dd)d*Zd+d,d-d.d/Ze d0dd1d2Ze d(dd3d4Ze d5d6Ze d7d8Ze d9d:Ze d;d<Ze d=d>Ze d0dd?d@Ze d(ddAdBZe dCddDdEZdFddGdHZdIdJZdKddLdMZdddNdOdPZddQddRdSdTZddUdVdWZ dddXdYZ!dddZd[Z"dddd\d]Z#ddd^d_Z$e%d`daZ&ddbddcdddeZ'ddfdfdgdhdidjZ(e)d(dkdldmZ*dndoZ+ddpd(dqdrdsZ,e-dpd(dtdudvZ.ddwdpdxdydzZ/dfd(dfdFd{d|d}Z0ddfdfd~ddZ1dddfdfdddZ2Z3S)raa represent a table: facilitate read/write of various types of tables Attrs in Table Node ------------------- These are attributes that are store in the main table node, they are necessary to recreate these tables when read back in. index_axes : a list of tuples of the (original indexing axis and index column) non_index_axes: a list of tuples of the (original index axis and columns on a non-indexing axis) values_axes : a list of the columns which comprise the data of this table data_columns : a list of the columns that we are allowing indexing (these become single columns in values_axes) nan_rep : the string to use for nan representations for string objects levels : the names of levels metadata : the names of the metadata columns Z wide_tablerprYr[rr`zint | list[Hashable]rTzlist[IndexCol] index_axeszlist[tuple[int, Any]]r*z list[DataCol] values_axesrgrrrJrNrwrrIr)rrrrc sPtj||||d|pg|_|p$g|_|p.g|_|p8g|_| pBi|_| |_dS)Nr)rrrr*rrrr) rrrrWrrr*rrrrrrQrRr s      zTable.__init__rcCs|jddS)N_r)rrrrQrQrRtable_type_short szTable.table_type_shortc Cs|t|jrd|jnd}d|d}d}|jrZddd|jD}d|d}dd d|jD}|jd |d |jd |j d |j d|d|d S)rerr|z,dc->[r&rdcSsg|] }t|qSrQrYr'rQrQrRrf rz"Table.__repr__..rgcSsg|] }|jqSrQrZr}rQrQrRrf rrhz (typ->z,nrows->z,ncols->z ,indexers->[ri) rrirrrar`rrmrrncols)rZjdcreverZjverZ jindex_axesrQrQrRr s(  zTable.__repr__)rcCs"|jD]}||jkr|SqdS)zreturn the axis for cN)rHr[)rrr~rQrQrRr s   zTable.__getitem__c Cs|dur dS|j|jkr2td|jd|jddD]~}t||d}t||d}||kr6t|D]4\}}||}||krbtd|d|d|dqbtd|d|d|dq6dS) z"validate against an existing tableNz'incompatible table_type with existing [rr&)rr*rzinvalid combination of [z] on appending data [z] vs current table [)rrrprrrC)rrrsvovrsaxZoaxrQrQrRrq s:   zTable.validatercCs t|jtS)z@the levels attribute is 1 or a list in the case of a multi-index)rLrrgrrQrQrRis_multi_index: szTable.is_multi_indexrz tuple[DataFrame, list[Hashable]])rrc Cs`t|jj}z |}Wn.tyH}ztd|WYd}~n d}~00t|tsXJ||fS)ze validate that we can store the multi-index; reset and return the new object zBduplicate names/columns in the multi-index when storing as a tableN)rDfill_missing_namesrr reset_indexrrLr1)rrrZ reset_objrGrQrQrRvalidate_multiindex? s zTable.validate_multiindexr]cCstdd|jDS)z-based on our axes, compute the expected nrowscSsg|]}|jjdqSr))rrrbrrQrQrRrfS rz(Table.nrows_expected..)rMrrrrQrQrRnrows_expectedP szTable.nrows_expectedcCs d|jvS)zhas this table been createdrprrrQrQrRrU szTable.is_existscCst|jddSNrprprrrQrQrRrpZ szTable.storablecCs|jS)z,return the table group (this is my storable))rprrQrQrRrp^ sz Table.tablecCs|jjSrT)rprrrQrQrRrc sz Table.dtypecCs|jjSrTrrrQrQrRrg szTable.descriptioncCst|j|jSrT)r5r6rrrrQrQrRrHk sz Table.axescCstdd|jDS)z.the number of total columns in the values axescss|]}t|jVqdSrT)rir]r}rQrQrRrOr rzTable.ncols..)sumrrrQrQrRro sz Table.ncolscCsdSr rQrrQrQrR is_transposedt szTable.is_transposedztuple[int, ...]cCs(ttdd|jDdd|jDS)z@return a tuple of my permutated axes, non_indexable at the frontcSsg|]}t|dqSr)rcr}rQrQrRrf} rz*Table.data_orientation..cSsg|]}t|jqSrQ)r]r.r}rQrQrRrf~ r)rhr5r6r*rrrQrQrRdata_orientationx s zTable.data_orientationzdict[str, Any]csRdddddjD}fddjD}fddjD}t|||S)z.csg|]\}}|dfqSrTrQ)rbr.r]) axis_namesrQrRrf rcs&g|]}|jtjvr|j|fqSrQ)r[rTrrr?rrQrRrf s)rr*rrJ)rd1d2Zd3rQ)rrrR queryables s  zTable.queryablescCsdd|jDS)zreturn a list of my index colscSsg|]}|j|jfqSrQ)r.rrrQrQrRrf rz$Table.index_cols..rrrQrQrR index_cols szTable.index_colsrcCsdd|jDS)zreturn a list of my values colscSsg|] }|jqSrQrrrQrQrRrf rz%Table.values_cols..)rrrQrQrR values_cols szTable.values_colsrcCs|jj}|d|dS)z)return the metadata pathname for this keyz/meta/z/metarlr rQrQrR_get_metadata_path szTable._get_metadata_pathr)rr]rcCs,|jj||t|d|j|j|jddS)z Write out a metadata array to the key as a fixed-format Series. Parameters ---------- key : str values : ndarray rp)rrWrrN)rrrr6rWrr)rrr]rQrQrRr s zTable.write_metadatarcCs0tt|jdd|ddur,|j||SdS)z'return the meta data array for this keyrN)rprrrrrrQrQrRr  szTable.read_metadatacCspt|j|j_||j_||j_|j|j_|j|j_|j|j_|j|j_|j |j_ |j |j_ |j |j_ dS)zset our table type & indexablesN) rYrrrrr*rrrWrrrrrQrQrRrn s        zTable.set_attrscCst|jddpg|_t|jddp$g|_t|jddp8i|_t|jdd|_tt|jdd|_tt|jdd|_ t|jd dpg|_ d d |j D|_ d d |j D|_ dS) rr*NrrrrWrrwrcSsg|]}|jr|qSrQrr}rQrQrRrf rz#Table.get_attrs..cSsg|]}|js|qSrQrr}rQrQrRrf r)rprr*rrrrXrWrSrr indexablesrrrrQrQrRro szTable.get_attrscCs>|dur:|jr:tddd|jD}tj|ttddS)rrNrdcSsg|] }t|qSrQrr'rQrQrRrf rz*Table.validate_version..r)rarlrr`rrr r$)rrjrrQrQrRrs szTable.validate_versioncCsR|dur dSt|tsdS|}|D]&}|dkr4q&||vr&td|dq&dS)z validate the min_itemsize doesn't contain items that are not in the axes this needs data_columns to be defined Nr]zmin_itemsize has the key [z%] which is not an axis or data_column)rLrJrr)rrqr%rQrQrRvalidate_min_itemsize s  zTable.validate_min_itemsizec sg}jjjtjjD]j\}\}}t|}|}|durJdnd}|d}t|d} t|||| |j||d} || qt j t |fdd| fddtjj D|S) z/create/cache the indexables if they don't existNr r)r[r.rrjrrprrc st|tsJt}|vrt}t|}t|j}t|dd}t|dd}t|}|}t|dd} |||||||j | ||d } | S)Nrrr) r[rr]rjrrrprrr) rLrYrrUrp_maybe_adjust_namer`rr rp) rrklassr$adj_namer]rrjmdrr)base_posredescr table_attrsrQrRrqs0    zTable.indexables..fcsg|]\}}||qSrQrQ)rbrr)rqrQrRrf:rz$Table.indexables..)rrprrrrpr rrrTrrirXr) r _indexablesrr.r[r$rrrrj index_colrQ)rrerrqrrrRr s2      % zTable.indexablesrr(c CsV|s dS|durdS|dus(|dur8dd|jD}t|ttfsL|g}i}|dur`||d<|durp||d<|j}|D]}t|j|d}|dur"|jr|j }|j } |j } |dur| |kr| n| |d<|dur| |kr| n| |d<|jsP|j drtd |jfi|qz||jd d vrztd |d |d|dqzdS)aZ Create a pytables index on the specified columns. Parameters ---------- columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError if trying to create an index on a complex-type column. Notes ----- Cannot index Time64Col or ComplexCol. Pytables must be >= 3.0. NFTcSsg|]}|jr|jqSrQ)rrr}rQrQrRrfcrz&Table.create_index..rirjcomplexzColumns containing complex values can be stored but cannot be indexed when using table format. Either use fixed format, set index=False, or do not include the columns containing complex values to data_columns when initializing the table.rr`zcolumn z/ is not a data_column. In order to read column z: you must reload the dataframe into HDFStore and include z with the data_columns argument.)rrHrLrhrgrprprNr|rrirj remove_indexrrxrrkr*rz) rrrirjkwrprrRrZ cur_optlevelZcur_kindrQrQrRrk>sR   zTable.create_indexrz!list[tuple[ArrayLike, ArrayLike]]rxc CsZt||||d}|}g}|jD]2}||j|j||j|j|jd}| |q"|S)a Create the axes sniffed from the table. Parameters ---------- where : ??? start : int or None, default None stop : int or None, default None Returns ------- List[Tuple[index_values, column_values]] rrF) SelectionrrHrrrrrWrr) rrjrr selectionr]rr~resrQrQrR _read_axess   zTable._read_axesrcCs|S)zreturn the data for this objrQrrrrQrQrR get_objectszTable.get_objectcst|s gS|d\}|j|i}|ddkrL|rLtd|d||dur^t}n |durjg}t|trt|t|}|fdd | Dfd d |DS) zd take the input data_columns and min_itemize and create a data columns spec rrr4z"cannot use a multi-index on axis [z] with data_columns TNcs g|]}|dkr|vr|qSrrQr$)existing_data_columnsrQrRrfsz/Table.validate_data_columns..csg|]}|vr|qSrQrQ)rbr) axis_labelsrQrRrfr) rirrrrgrLrJrTrXr)rrrr*r.rrQ)rrrRvalidate_data_columnss.     zTable.validate_data_columnsr1)rrqc/stts,|jj}td|dtddur:dgfddD|rzd}d d|jDt|j }|j }nd }|j } |j d ksJt |j d krtd g} |durd}fdddDd} j| } t| } |r|j|j|j|| ||-| |d }.t?|drl|j@|._@|.A||r|r|.B||.S)a0 Create and return the axes. Parameters ---------- axes: list or None The names or numbers of the axes to create. obj : DataFrame The object to create axes on. validate: bool, default True Whether to validate the obj against an existing object already written. nan_rep : A value to use for string column nan_rep. data_columns : List[str], True, or None, default None Specify the columns that we want to create to allow indexing on. * True : Use all available columns. * None : Use no columns. * List[str] : Use the specified columns. min_itemsize: Dict[str, int] or None, default None The min itemsize for a column in bytes. z/cannot properly create the storer for: [group->rr&Nrcsg|]}|qSrQ)_get_axis_numberr})rrQrRrf rz&Table._create_axes..TcSsg|] }|jqSrQrPr}rQrQrRrfrFr_r`z.get_blk_items..)rrrQrrR get_blk_itemssz2Table._get_blocks_and_items..get_blk_itemsrrPcSs"i|]\}}t|||fqSrQ)rhtolist)rbbrrQrQrRrSsz/Table._get_blocks_and_items..rcSsg|] }t|qSrQrf)rbitemrQrQrRrfrz/Table._get_blocks_and_items..z+cannot match existing table structure for [z] on appending data)rLrrBrrrCrgrrir3rYrbrXr7rhr]rarrrrr)rrrrrr rrrr.r new_labelsrZby_items new_blocksZ new_blk_itemsearr"rrGZjitemsrQrQrRrsN          zTable._get_blocks_and_itemsr)rrc s|durt|}|durNjrNtjts.JjD]}||vr4|d|q4jD]\}}t|||qT|jdur|jD]$\}}fdd} | ||qS)zprocess axes filtersNrcsjD]}|}|}|dus*J||krfjrH|tj}||}j|d|S||vrtt |j }t|}t t rd|}||}j|d|Sqt d|ddS)NrPr`zcannot find the field [z] for filtering!) _AXIS_ORDERSr  _get_axisrunionr3rr`rArpr]rLr1r)fieldfiltr axis_numberZ axis_valuesZtakersr]roprrQrRprocess_filters"       z*Table.process_axes..process_filter) rgrrLrinsertr*rfilterr) rrrrrr.labelsr*r+r/rQr-rR process_axes s  ! zTable.process_axes)rrrIrcCs|durt|jd}d|d}dd|jD|d<|rj|durH|jpFd}tj|||pZ|jd }||d <n|jdur~|j|d <|S) z:create the description of the table from the axes & valuesNi'rp)r[rIcSsi|]}|j|jqSrQ)rrr}rQrQrRrSSrz,Table.create_description..r )rrrr)maxrrHrr}rrr)rrrrrIrKrrQrQrRcreate_descriptionDs       zTable.create_descriptionrtc Cs|||sdSt||||d}|}|jdur|jD]D\}}}|j|||dd} ||| j |||j }qBt |S)zf select coordinates (row numbers) from a table; return the coordinates object FrNr`rt) rsrr select_coordsr1rr"rr5ilocr]r3) rrjrrrcoordsr*r.r+rrQrQrRrcs   zTable.read_coordinatesr!cCs||sdS|dur$td|jD]z}||jkr*|jsNtd|dt|jj |}| |j |j ||||j |j|jd}tt|d|j|dSq*td|d dS) zj return a single column from the table, generally only indexables are interesting FNz4read_column does not currently accept a where clausezcolumn [z=] can not be extracted individually; it is not data indexablerFr`rZz] not found in the table)rsrrrHr[rrrprprNrrrrrWrr6rrr)rr rjrrr~rZ col_valuesrQrQrRr"}s*      zTable.read_column)NrwNNNNNN)N)NNN)NN)TNNN)N)NNN)NNN)4rrrrrZr[rrr8rrrrrrqrrrrrprprrrHrrrrrrrrr rnrorsrr#rrkrrSrr r staticmethodrr3r6rr"rTrQrQrrRr s   "          LW"*k>: rc@s4eZdZdZdZd dddddZdd d d ZdS) rz a write-once read-many table: this format DOES NOT ALLOW appending to a table. writing is a one-time operation the data are stored in a format that allows for searching the data on disk rNrrtcCs tddS)z[ read the indices and the indexing array, calculate offset rows and return z!WORMTable needs to implement readNrurvrQrQrRr s zWORMTable.readrrcKs tddS)z write in a format that we can search later on (but cannot append to): write out the indices and the values using _write_array (e.g. a CArray) create an indexing table so that we can search z"WORMTable needs to implement writeNrurwrQrQrRrszWORMTable.write)NNNN)rrrrrr rrQrQrQrRrs rc @sdeZdZdZdZdddddd d Zdd ddd d dZddddddddZdd d dddZdS)r(support the new appendable table formatsZ appendableNFTrr)rrrcCs|s|jr|j|jd|j||||| | d}|jD] }|q6|js|j|||| d}|| |d<|jj |jfi||j |j _ |jD]}| ||q|j || ddS)Nrp)rHrrqrrr)rrrrIr<)r)rrrrrrHrr6rnZ create_tablerrr write_data)rrrHrrrrrrrIrrrr<rpr~optionsrQrQrRrs4     zAppendableTable.writer)rrrcs|jj}|j}g}|rT|jD]6}t|jjdd}t|tj r| |j dddqt |r|d}|ddD] }||@}qp| }nd}dd |jD} t | } | dksJ| d d |jD} d d | D} g} t| D]6\} }|f|j|| | j}| | | |q|dur$d }tjt|||jd }||d}t|D]x} | |t| d||kr|q|j|fdd | D|dur|ndfdd | DdqNdS)z` we form the data into a 2-d including indexes,values,mask write chunk-by-chunk rrPu1FrDr`NcSsg|] }|jqSrQ)rr}rQrQrRrf$rz.AppendableTable.write_data..cSsg|] }|qSrQ)rr}rQrQrRrf*rc Ss,g|]$}|tt|j|jdqSr) transposerMrollr rVr?rQrQrRrf+rrr<csg|]}|qSrQrQr}end_istart_irQrRrf?rcsg|]}|qSrQrQr?rArQrRrfAr)indexesrRr])rrrrr9rrLrLrMrrrLrirIrrrreshaperrrUwrite_data_chunk)rrrrrmasksr~rRmrDnindexesr]bvaluesrrR new_shaperowschunksrQrArRr< sL        zAppendableTable.write_datarzlist[np.ndarray]znpt.NDArray[np.bool_] | None)rLrDrRr]rc Cs|D]}t|jsdSq|djd}|t|krFtj||jd}|jj}t|}t|D]\} } | ||| <q^t|D]\} }|||| |<q||dur|j t dd} | s|| }t|r|j ||j dS)z Parameters ---------- rows : an empty memory space where we are putting the chunk indexes : an array of the indexes mask : an array of the masks values : an array of the values Nrr<FrD)rMrrrirrrrrIrLrrLrprr) rrLrDrRr]rRrrrIrrrHrQrQrRrFDs&   z AppendableTable.write_data_chunkrtcCsb|dust|sf|dur:|dur:|j}|jj|jddn(|durH|j}|jj||d}|j|S|srdS|j}t ||||d}| }t | }t|} | r^| } t| | dkj} t| sdg} | d| kr| | | ddkr| dd| } t| D]@} |t| | }|j||jd||jddd| } q|j| S)NTr@rtr`rrB)rirrrrrpZ remove_rowsrrrr7r6 sort_valuesdiffrgrrr0rareversedr\rU)rrjrrrrprr]Z sorted_serieslnrOrpgrrLrQrQrRrFpsD       zAppendableTable.delete) NFNNNNNNFNNT)F)NNN) rrrrrrr<rFrFrQrQrQrRrs$<;,rc@s`eZdZUdZdZdZdZeZde d<e ddd d Z e dd d d Z ddddddZdS)rr;rrr_r\r]rrcCs|jdjdkS)Nrr`)rr.rrQrQrRrsz"AppendableFrameTable.is_transposedrcCs|r |j}|S)zthese are written transposed)rrrQrQrRrszAppendableFrameTable.get_objectNrrtcs0|sdSj|||d}tjrHjjddini}fddtjD}t|dkstJ|d}||d} g} tjD]N\} } | j vrq|| \} }|ddkrt | }n t | }|d}|dur|j |d d jr |}|}t | t| d dd }n|j}t | t| d dd }|}|jdkrlt|tjrl|d|jdf}t|tjrt|j||d }n.t|t rt|||d }ntj|g||d }|j|jksJ|j|jf| |qt| dkr| d}n t| dd}t|||d}j|||d}|S)Nrrcs"g|]\}}|jdur|qSr)r)rbrrrrQrRrfrz-AppendableFrameTable.read..r`rr4rTinplacer[rZrrP)rr) rsrrrir*rrrrHrr3r4 from_tuples set_namesrrprrVrLrMrrErr1 _from_arraysdtypesrrLrr8rr3)rrjrrrrrindsindrframesrr~Z index_valsrrNrr]Zindex_Zcols_rrrQrrRr sZ        "   zAppendableFrameTable.read)NNNN)rrrrrZrrVr1r]rrrrSrr rQrQrQrRrs  rcsneZdZdZdZdZdZeZe ddddZ e dd d d Z dfd d Z dddddfdd ZZS)rr;rrr_rrcCsdSr rQrrQrQrRrsz#AppendableSeriesTable.is_transposedrcCs|SrTrQrrQrQrRrsz AppendableSeriesTable.get_objectNc s<t|ts|jpd}||}tjf||jd|S)+we are going to write this as a frame tabler]rr)rLr1r[to_framerrrr!)rrrrr[rrQrRr!s   zAppendableSeriesTable.writerr6rxcs|j}|durB|rBt|jts"J|jD]}||vr(|d|q(tj||||d}|rj|j|jdd|jdddf}|j dkrd|_ |S)Nrr-TrSr]) rrLrrgr0rr  set_indexr8r[)rrjrrrrrrPrrQrRr (s   zAppendableSeriesTable.read)N)NNNN)rrrrrZrrVr6r]rrrSrrr rTrQrQrrRrs rcs(eZdZdZdZdZfddZZS)rr;rrc s^|jpd}||\}|_t|jts*Jt|j}||t||_tj fd|i|S)r\r]r) r[rrrLrgrr3rrr)rrrr[newobjrNrrQrRrHs    z AppendableMultiSeriesTable.write)rrrrrZrrrTrQrQrrRrBsrc@sjeZdZUdZdZdZdZeZde d<e ddd d Z e d d Z d dddZ eddZddZdS)rz:a table that read/writes the generic pytables table formatrrr_zlist[Hashable]rrYrcCs|jSrT)rZrrQrQrRrm\szGenericTable.pandas_typecCst|jddp|jSrrrrQrQrRrp`szGenericTable.storablercCsLg|_d|_g|_dd|jD|_dd|jD|_dd|jD|_dS)rNcSsg|]}|jr|qSrQrr}rQrQrRrfjrz*GenericTable.get_attrs..cSsg|]}|js|qSrQrr}rQrQrRrfkrcSsg|] }|jqSrQrZr}rQrQrRrflr)r*rrrrrrrrQrQrRrods zGenericTable.get_attrsc Cs|j}|d}|durdnd}tdd|j||d}|g}t|jD]^\}}t|tsZJt||}||}|durzdnd}t |||g||j||d} | | qD|S)z0create the indexables from the table descriptionrNr r)r[r.rprr)r[rr]rrprr) rr r rprZ_v_namesrLrYrprXr) rrKrrrrrrr$rerQrQrRrns.     zGenericTable.indexablescKs tddS)Nz cannot write on an generic tablerurwrQrQrRrszGenericTable.writeN)rrrrrZrrVr1r]rrrmrpror#rrrQrQrQrRrSs    "rcs`eZdZdZdZeZdZe dZ e ddddZ dfd d Z dd d d fdd ZZS)rza frame with a multi-indexrr_z ^level_\d+$rYrcCsdS)NZappendable_multirQrrQrQrRrsz*AppendableMultiFrameTable.table_type_shortNc sx|durg}n|dur |j}||\}|_t|jts@J|jD]}||vrF|d|qFtjf||d|S)NTrr]) rr!rrrLrgr0rr)rrrrrrrQrRrs  zAppendableMultiFrameTable.writerrtcsDtj||||d}|j}|jfdd|jjD|_|S)Nr-cs g|]}j|rdn|qSrT) _re_levelssearch)rbr[rrQrRrfrz2AppendableMultiFrameTable.read..)rr r_rrrVr)rrjrrrrrrrRr s  zAppendableMultiFrameTable.read)N)NNNN)rrrrrr1r]rVrecompilerarrrr rTrQrQrrRrs rr1r3)rr.r2rcCs||}t|}|dur"t|}|dus4||rB||rB|St|}|durlt|j|dd}||stddg|j}|||<|jt|}|S)NF)sort) r(rAequalsuniquer_slicerVr`rh)rr.r2rrslicerrQrQrRrs   rrz str | tzinfo)rrcCst|}|S)z+for a tz-aware type, return an encoded zone)r get_timezone)rzonerQrQrRrs rznp.ndarray | Indexr2)r]rr9rcCsdSrTrQr]rr9rQrQrRrsrrcCsdSrTrQrlrQrQrRrszstr | tzinfo | Noneznp.ndarray | DatetimeIndexcCst|tr"|jdus"|j|ks"J|durtt|trB|j}|j}n d}|}t|}t||d}|d|}n|rt j |dd}|S)a coerce the values to a DatetimeIndex if tz is set preserve the input shape if possible Parameters ---------- values : ndarray or Index tz : str or tzinfo coerce : if we do not have a passed timezone, coerce to M8[ns] ndarray NrZrM8[ns]r<) rLr2rr[rrIrSrrrMrG)r]rr9r[rQrQrRrs   )r[rrWrrc Cs~t|tsJ|j}t|\}}t|}t|}t|tsPt|j sPt |j rvt ||||t |ddt |dd|dSt|t rtdtj|dd} t|} | dkrtjdd | Dtjd }t ||dt|d S| d krt| ||}|j j} t ||d t| |d S| d vr.t |||||dSt|tjrH|j tksLJ|dks^J|t}t |||||d SdS)Nrr)r]rjrrrrzMultiIndex not supported here!FrrcSsg|] }|qSrQ) toordinalr?rQrQrRrf6rz"_convert_index..r<)rr)integerfloating)r]rjrrrH)rLrYr[rrrUr%r:r/rr&rrpr4rrrrMrGint32r}Z Time32Col_convert_string_arrayrrrrHr) r[rrWrrrQrrjr$rr]rrQrQrRrs^            r)rjrWrrcCs|dkrt|}n|dkr$t|}n|dkrvztjdd|Dtd}Wqtyrtjdd|Dtd}Yq0nT|dvrt|}n@|d vrt|d||d }n&|d krt|d }ntd ||S)Nr7r:rcSsg|]}t|qSrQr=r?rQrQrRrfZrz$_unconvert_index..r<cSsg|]}t|qSrQr@r?rQrQrRrf\r)rofloatrrrFrHrzunrecognized index type )r2r7rMrGrHrrP)rrjrWrrrQrQrRrQs&     rrr)r[rJrcCs|jtkr|Sttj|}|jj}tj|dd} | dkrBtdn&| dkrTtdn| dksh|dksh|St |} | } || | <tj| dd} | dkrt | j d D]V} | | } tj| dd} | dkrt || kr|| nd | }td |d | d qt| ||| j }|j}t|trBt||p>|dp>d }t|pLd |}|dur~||}|dur~||kr~|}|jd|dd}|S)NFrrz+[date] is not implemented as a table columnrzz>too many timezones in this block, create separate data columnsrrHrzNo.zCannot serialize the column [z2] because its data contents are not [string] but [z] object dtyper]z|SrD)rrHrrMrr[rrrr9rrUrrirrrErrLrJr]rr5rrL)r[rJr rrrWrrrrrRrrrZerror_column_labelrrZecirQrQrRrjsN       r)rrWrrcCs\t|r(t|j||j|j}t|}t dt |}t j |d|d}|S)a Take a string-like that is object dtype and coerce to a fixed size string type. Parameters ---------- data : np.ndarray[object] encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[fixed-length-string] r`Sr<)rir6rIrYencoderNrErr%r5 libwritersmax_len_string_arrayrMrG)rrWrZensuredrrQrQrRrrs rrcCs|j}tj|td}t|rvtt|}d|}t |dt r^t |j j ||dj}n|j|ddjtdd}|durd}t||||S) a* Inverse of _convert_string_array. Parameters ---------- data : np.ndarray[fixed-length-string] nan_rep : the storage repr of NaN encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[object] Decoded data. r<Ur)rFrDNr )rrMrGrIrHrirvrwr%rLrr6rYrOrNrL!string_array_replace_from_nan_reprE)rrrWrrrrrQrQrRrPs  rP)r]rrWrcCs6t|tsJt|t|r2t|||}||}|SrT)rLrYr _need_convert_get_converter)r]rrWrconvrQrQrRrs  rrjrWrcs8|dkrddS|dkr&fddStd|dS)Nr7cSstj|ddS)Nrmr<)rMrGr(rQrQrRrrz _get_converter..rcst|ddS)NrF)rPr~rrQrRrsz invalid kind )rr}rQrrRr{s r{r(cCs|dvr dSdS)N)r7rTFrQrrQrQrRrzsrzz Sequence[int])r[r`rcCslt|tst|dkrtd|ddkrh|ddkrh|ddkrhtd|}|rh|d}d|}|S) z Prior to 0.10.1, we named values blocks like: values_block_0 an the name values_0, adjust the given name if necessary. Parameters ---------- name : str version : Tuple[int, int, int] Returns ------- str z6Version is incorrect, expected sequence of 3 integers.rr`r^r_zvalues_block_(\d+)Zvalues_)rLrYrirrcrbr)r[r`rHgrprQrQrRrs$   r) dtype_strrcCst|}|ds|dr"d}n|dr2d}n|drBd}n|dsV|dr\d}nn|drld}n^|d r|d }nN|d rd }n>|d rd }n.|d rd}n|dkrd}ntd|d|S)zA Find the "kind" string describing the given dtype name. rrrsrr]r)ror7 timedeltar:rr r,rHzcannot interpret dtype of [r&)rSrxr)rrjrQrQrRr-s.       rrcCsbt|tr|j}|jjdd}|jjdvr@t| d}nt|t rP|j }t|}||fS)zJ Convert the passed data into a storable form and a dtype string. rgr)rHMr) rLr;rrr[rrjrMrGrr5r)rrrQrQrRrNs    rc@s>eZdZdZddddddddZd d Zd d Zd dZdS)rz Carries out a selection operation on a tables.Table object. Parameters ---------- table : a Table object where : list of Terms (or convertible to) start, stop: indices to start and/or stop selection Nrrr)rprrrcCsh||_||_||_||_d|_d|_d|_d|_t|r.t t t j |dd}|dksd|dkrt |}|jt jkr|j|j}}|durd}|dur|jj}t ||||_nVt|jjt jr|jdur||jks|jdur||jkrt d||_Wdn1s$0Y|jdurd|||_|jdurd|j\|_|_dS)NFrrobooleanrz3where must have index locations >= start and < stop)rprjrr conditionr1termsr4r,rrrrrMrGrbool_rr  issubclassrrorKgenerateevaluate)rrprjrrinferredrQrQrRrpsD     &   zSelection.__init__c Cs|dur dS|j}zt|||jjdWSty|}z:d|}td|d|d}t||WYd}~n d}~00dS)z'where can be a : dict,list,tuple,stringN)rrWrz- The passed where expression: a* contains an invalid variable reference all of the variable references must be a reference to an axis (e.g. 'index' or 'columns'), or a data_column The currently defined references are: z ) rprr>rW NameErrorrrrr)rrjrrGZqkeysrrQrQrRrs  zSelection.generatecCsX|jdur(|jjj|j|j|jdS|jdurB|jj|jS|jjj|j|jdS)( generate the selection Nrt) rrpZ read_whererrrr4rr rrQrQrRrs  zSelection.selectcCs|j|j}}|jj}|dur$d}n|dkr4||7}|durB|}n|dkrR||7}|jdurx|jjj|j||ddS|jdur|jSt ||S)rNrT)rrre) rrrprrZget_where_listrr4rMr )rrrrrQrQrRr7s"  zSelection.select_coords)NNN)rrrrrrrr7rQrQrQrRrds/ r) r~NNFNTNNNNrwrK) NrrwNNNNFN)N)F)F)F)r __future__r contextlibrrrzrrr5rrctextwraprtypingrrr r r r r rrrrnumpyrMpandas._configrr pandas._libsrrrvpandas._libs.tslibsrpandas._typingrrrrrrpandas.compat._optionalrZpandas.compat.pickle_compatr pandas.errorsrrr r!r"pandas.util._decoratorsr#pandas.util._exceptionsr$pandas.core.dtypes.commonr%r&r'r(r)r*r+r,r-r.r/pandas.core.dtypes.missingr0rr1r2r3r4r5r6r7r8r9pandas.core.apir:pandas.core.arraysr;r<r=pandas.core.commoncorecommonrDZ pandas.core.computation.pytablesr>r?pandas.core.constructionr@pandas.core.indexes.apirApandas.core.internalsrBrCpandas.io.commonrDpandas.io.formats.printingrErFryrGrHrIrJrjrUrSrXr\rarkrlrrmrnrrWrsrt config_prefixregister_optionis_boolis_one_of_factoryrxr|r}rrrrrrr rrUrXrYryrrrrrrrrrrrrrrrrrrrrPrr{rzrrrrrQrQrQrRs.   0       4 ,              &,: _p)!3e!`ogd1B+&AK' !