a +=icA@sdZddlmZddlmZddlmZddlZddlZddlZddl m Z ddl Z dd d Z dd dZ dddZddZGdddeZddZdS)z+Utilities for preprocessing sequence data. )absolute_import)division)print_functionN)rangeint32prec Cst|dstdt|}g}d}d} |D]^} z6|t| | rbt| rbt| jdd}d} Wq*tytdt| Yq*0q*|durt |}t |tj pt |tj } t |tjr|tkr| std |t|tj||f|||d } t|D]\} } t| sq|d kr4| | d}n$|d krL| d|}n td |tj||d }|jdd|krtd|jdd| |f|d kr|| | dt|f<n.|d kr|| | t| df<n td|q| S)aPads sequences to the same length. This function transforms a list of `num_samples` sequences (lists of integers) into a 2D Numpy array of shape `(num_samples, num_timesteps)`. `num_timesteps` is either the `maxlen` argument if provided, or the length of the longest sequence otherwise. Sequences that are shorter than `num_timesteps` are padded with `value` at the beginning or the end if padding='post. Sequences longer than `num_timesteps` are truncated so that they fit the desired length. The position where padding or truncation happens is determined by the arguments `padding` and `truncating`, respectively. Pre-padding is the default. # Arguments sequences: List of lists, where each element is a sequence. maxlen: Int, maximum length of all sequences. dtype: Type of the output sequences. To pad sequences with variable length strings, you can use `object`. padding: String, 'pre' or 'post': pad either before or after each sequence. truncating: String, 'pre' or 'post': remove values from sequences larger than `maxlen`, either at the beginning or at the end of the sequences. value: Float or String, padding value. # Returns x: Numpy array with shape `(len(sequences), maxlen)` # Raises ValueError: In case of invalid values for `truncating` or `padding`, or in case of invalid shape for a `sequences` entry. __len__z`sequences` must be iterable.TNFz=`sequences` must be a list of iterables. Found non-iterable: zo`dtype` {} is not compatible with `value`'s type: {} You should set `dtype=object` for variable length strings.)dtyperpostz#Truncating type "%s" not understoodzQShape of sample %s of sequence at position %s is different from expected shape %sz Padding type "%s" not understood)hasattr ValueErrorlenappendnpZasarrayshape TypeErrorstrmaxZ issubdtypeZstr_Zunicode_ isinstancesix string_typesobjectformattypefull enumerate) sequencesmaxlenr paddingZ truncatingvalueZ num_sampleslengthsZ sample_shapeflagxZ is_dtype_stridxstruncr r m/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/keras_preprocessing/sequence.py pad_sequencessZ(         r*h㈵>cCsVd}t|}d|d<|t||ddd|}||}td|t|S)a9Generates a word rank-based probabilistic sampling table. Used for generating the `sampling_table` argument for `skipgrams`. `sampling_table[i]` is the probability of sampling the word i-th most common word in a dataset (more common words should be sampled less frequently, for balance). The sampling probabilities are generated according to the sampling distribution used in word2vec: ``` p(word) = (min(1, sqrt(word_frequency / sampling_factor) / (word_frequency / sampling_factor))) ``` We assume that the word frequencies follow Zipf's law (s=1) to derive a numerical approximation of frequency(rank): `frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))` where `gamma` is the Euler-Mascheroni constant. # Arguments size: Int, number of possible words to sample. sampling_factor: The sampling factor in the word2vec formula. # Returns A 1D Numpy array of length `size` where the ith entry is the probability that a word of rank i should be sampled. gX9v?r rg??g(@)rarangelogminimumsqrt)sizeZsampling_factorgammaZrankZinv_fqfr r r)make_sampling_tableqs  "r4r,TFcsrg}g} t|D]\} } | sq|dur8|| tkr8qtd| |} tt|| |d} t| | D]F}|| krf||}|sqf|| |g|r| ddgqf| dqfq|dkr&tt| |}dd|Dt|fddt|D7}|r| ddgg|7} n| dg|7} |rj|durBt dd}t |t|t |t| || fS)a{Generates skipgram word pairs. This function transforms a sequence of word indexes (list of integers) into tuples of words of the form: - (word, word in the same window), with label 1 (positive samples). - (word, random word from the vocabulary), with label 0 (negative samples). Read more about Skipgram in this gnomic paper by Mikolov et al.: [Efficient Estimation of Word Representations in Vector Space](http://arxiv.org/pdf/1301.3781v3.pdf) # Arguments sequence: A word sequence (sentence), encoded as a list of word indices (integers). If using a `sampling_table`, word indices are expected to match the rank of the words in a reference dataset (e.g. 10 would encode the 10-th most frequently occurring token). Note that index 0 is expected to be a non-word and will be skipped. vocabulary_size: Int, maximum possible word index + 1 window_size: Int, size of sampling windows (technically half-window). The window of a word `w_i` will be `[i - window_size, i + window_size+1]`. negative_samples: Float >= 0. 0 for no negative (i.e. random) samples. 1 for same number as positive samples. shuffle: Whether to shuffle the word couples before returning them. categorical: bool. if False, labels will be integers (eg. `[0, 1, 1 .. ]`), if `True`, labels will be categorical, e.g. `[[1,0],[0,1],[0,1] .. ]`. sampling_table: 1D array of size `vocabulary_size` where the entry i encodes the probability to sample a word of rank i. seed: Random seed. # Returns couples, labels: where `couples` are int pairs and `labels` are either 0 or 1. # Note By convention, index 0 in the vocabulary is a non-word and will be skipped. Nrr cSsg|] }|dqS)rr ).0cr r r) zskipgrams..cs,g|]$}|ttddgqS)r )rrandomrandint)r6ivocabulary_sizewordsr r)r8sgcA) rr:rminrrrintshuffler;seed)sequencer>Z window_sizeZnegative_samplesrBZ categoricalZsampling_tablerCZcoupleslabelsr<ZwiZ window_startZ window_endjZwjZnum_negative_samplesr r=r) skipgramssJ-        rGcCsFgg}}t||D](\}}t||kr||||q||fS)aJRemoves sequences that exceed the maximum length. # Arguments maxlen: Int, maximum length of the output sequences. seq: List of lists, where each sublist is a sequence. label: List where each element is an integer. # Returns new_seq, new_label: shortened lists for `seq` and `label`. )ziprr)r seqlabelZnew_seqZ new_labelr%yr r r)_remove_long_seqs    rLc@s:eZdZdZdddZd d Zd d Zd dZddZdS)TimeseriesGeneratora+ Utility class for generating batches of temporal data. This class takes in a sequence of data-points gathered at equal intervals, along with time series parameters such as stride, length of history, etc., to produce batches for training/validation. # Arguments data: Indexable generator (such as list or Numpy array) containing consecutive data points (timesteps). The data should be at 2D, and axis 0 is expected to be the time dimension. targets: Targets corresponding to timesteps in `data`. It should have same length as `data`. length: Length of the output sequences (in number of timesteps). sampling_rate: Period between successive individual timesteps within sequences. For rate `r`, timesteps `data[i]`, `data[i-r]`, ... `data[i - length]` are used for create a sample sequence. stride: Period between successive output sequences. For stride `s`, consecutive output samples would be centered around `data[i]`, `data[i+s]`, `data[i+2*s]`, etc. start_index: Data points earlier than `start_index` will not be used in the output sequences. This is useful to reserve part of the data for test or validation. end_index: Data points later than `end_index` will not be used in the output sequences. This is useful to reserve part of the data for test or validation. shuffle: Whether to shuffle output samples, or instead draw them in chronological order. reverse: Boolean: if `true`, timesteps in each output sample will be in reverse chronological order. batch_size: Number of timeseries samples in each batch (except maybe the last one). # Returns A [Sequence](/utils/#sequence) instance. # Examples ```python from keras.preprocessing.sequence import TimeseriesGenerator import numpy as np data = np.array([[i] for i in range(50)]) targets = np.array([[i] for i in range(50)]) data_gen = TimeseriesGenerator(data, targets, length=10, sampling_rate=2, batch_size=2) assert len(data_gen) == 20 batch_0 = data_gen[0] x, y = batch_0 assert np.array_equal(x, np.array([[[0], [2], [4], [6], [8]], [[1], [3], [5], [7], [9]]])) assert np.array_equal(y, np.array([[10], [11]])) ``` r rNFc Cst|t|kr4tddt|dt|||_||_||_||_||_|||_|durpt|d}||_ ||_ | |_ | |_ |j|j krtd|j|j fdS)NzData and targets have to bez" of same length. Data length is {}z while target length is {}r zz`start_index+length=%i > end_index=%i` is disallowed, as no part of the sequence would be left to be used as current step.) rrrdatatargetslength sampling_ratestride start_index end_indexrBreverse batch_size) selfrOrPrQrRrSrTrUrBrVrWr r r)__init__Es0      zTimeseriesGenerator.__init__cCs$|j|j|j|j|j|jS)N)rUrTrWrSrXr r r)r gs    zTimeseriesGenerator.__len__csjr$tjjjjdjd}n>jjj|}t|t |jjjdj}t fdd|D}t fdd|D}j r|ddddddf|fS||fS)Nr )r1cs$g|]}j|j|jqSr )rOrQrRr6rowrZr r)r8tsz3TimeseriesGenerator.__getitem__..csg|]}j|qSr )rPr[rZr r)r8vr9.) rBrr:r;rTrUrWrSr-r@arrayrV)rXindexrowsr<ZsamplesrPr rZr) __getitem__ks&zTimeseriesGenerator.__getitem__c Cs|j}t|jjtjkr"|j}zt|}WntyLtd|Yn0|j }t|j jtjkrp|j }zt|}Wntytd|Yn0|||j |j |j |j |j|j|j|jd S)zReturns the TimeseriesGenerator configuration as Python dictionary. # Returns A Python dictionary with the TimeseriesGenerator configuration. zData not JSON Serializable:zTargets not JSON Serializable:) rOrPrQrRrSrTrUrBrVrW)rOr __module__r__name__tolistjsondumpsrrPrQrRrSrTrUrBrVrW)rXrOZ json_datarPZ json_targetsr r r) get_config|s2    zTimeseriesGenerator.get_configcKs(|}|jj|d}tj|fi|S)aReturns a JSON string containing the timeseries generator configuration. To load a generator from a JSON string, use `keras.preprocessing.sequence.timeseries_generator_from_json(json_string)`. # Arguments **kwargs: Additional keyword arguments to be passed to `json.dumps()`. # Returns A JSON string containing the tokenizer configuration. ) class_nameconfig)rg __class__rcrerf)rXkwargsriZtimeseries_generator_configr r r)to_jsons zTimeseriesGenerator.to_json)r r rNFFrN) rcrb __qualname____doc__rYr rargrlr r r r)rMs? "#rMcCsRt|}|d}t|d}||d<t|d}||d<tfi|S)aParses a JSON timeseries generator configuration file and returns a timeseries generator instance. # Arguments json_string: JSON string encoding a timeseries generator configuration. # Returns A Keras TimeseriesGenerator instance rirOrP)reloadsgetpoprM)Z json_stringZ full_configrirOrPr r r)timeseries_generator_from_jsons  rr)Nrrrr)r+)r5r,TFNN)rn __future__rrrnumpyrr:reZ six.movesrrr*r4rGrLrrMrrr r r r)s&     b ( [.