a +=icJ@sdZddlmZddlmZddlmZddlZddlZddlZddlm Z ddlm Z ddl m Z ddl Z ddlZdd lmZdd lmZejd krejZnejZdddZdddZdddZGdddeZddZdS)z(Utilities for text input preprocessing. )absolute_import)division)print_functionN) OrderedDict) defaultdict)md5)range)zip!!"#$%&()*+,-./:;<=>?@[\]^_`{|}~ T cs|r |}tjdkrt|tr>fdd|D}||}qtdkrht|t|}||}q|D]}||}qln$fdd|D}t|}||}| }dd|DS)aConverts a text to a sequence of words (or tokens). # Arguments text: Input text (string). filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to convert the input to lowercase. split: str. Separator for word splitting. # Returns A list of words (or tokens). r csi|]}t|tqS)ordunicode.0csplitri/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/keras_preprocessing/text.py /sz)text_to_word_sequence..csi|] }|qSrrrrrrr:cSsg|] }|r|qSrr)rirrr ?rz)text_to_word_sequence..) lowersys version_info isinstancer translatelen maketransreplacer)textfiltersrrZ translate_maprZtranslate_dictseqrrrtext_to_word_sequences$        r'cCst||t|||dS)aOne-hot encodes a text into a list of word indexes of size n. This is a wrapper to the `hashing_trick` function using `hash` as the hashing function; unicity of word to index mapping non-guaranteed. # Arguments text: Input text (string). n: int. Size of vocabulary. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. # Returns List of integers in [1, n]. Each integer encodes a word (unicity non-guaranteed). ) hash_functionr%rr) hashing_trickhash)r$nr%rrrrrone_hotBs r,csBdurtndkrddt||||d}fdd|DS)aConverts a text to a sequence of indexes in a fixed-size hashing space. # Arguments text: Input text (string). n: Dimension of the hashing space. hash_function: defaults to python `hash` function, can be 'md5' or any function that takes in input a string and returns a int. Note that 'hash' is not a stable hashing function, so it is not consistent across different runs, while 'md5' is a stable hashing function. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. # Returns A list of integer word indices (unicity non-guaranteed). `0` is a reserved index that won't be assigned to any word. Two or more words may be assigned to the same index, due to possible collisions by the hashing function. The [probability]( https://en.wikipedia.org/wiki/Birthday_problem#Probability_table) of a collision is in relation to the dimension of the hashing space and the number of distinct objects. NrcSstt|dS)N)intrencode hexdigest)wrrrr(sz$hashing_trick..hash_function)r%rrcs g|]}|ddqS)rr)rr1r(r+rrrrz!hashing_trick..)r*r')r$r+r(r%rrr&rr2rr)_s!r)c@sneZdZdZddd Zd d Zd d ZddZddZddZ ddZ d ddZ d!ddZ ddZ ddZdS)" TokenizeraOText tokenization utility class. This class allows to vectorize a text corpus, by turning each text into either a sequence of integers (each integer being the index of a token in a dictionary) or into a vector where the coefficient for each token could be binary, based on word count, based on tf-idf... # Arguments num_words: the maximum number of words to keep, based on word frequency. Only the most common `num_words-1` words will be kept. filters: a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the `'` character. lower: boolean. Whether to convert the texts to lowercase. split: str. Separator for word splitting. char_level: if True, every character will be treated as a token. oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls By default, all punctuation is removed, turning the texts into space-separated sequences of words (words maybe include the `'` character). These sequences are then split into lists of tokens. They will then be indexed or vectorized. `0` is a reserved index that won't be assigned to any word. Nr Tr Frc Ksd|vrtd|d}|r0tdt|t|_tt|_ ||_ ||_ ||_ ||_ ||_||_||_tt|_i|_i|_dS)NZnb_wordszDThe `nb_words` argument in `Tokenizer` has been renamed `num_words`.z Unrecognized keyword arguments: )warningswarnpop TypeErrorstrr word_countsrr. word_docsr%rr num_wordsdocument_count char_level oov_token index_docs word_index index_word) selfr;r%rrr=r>r<kwargsrrr__init__s"     zTokenizer.__init__cCsn|D]}|jd7_|js&t|trT|jrNt|trFdd|D}n|}|}nt||j|j|j}|D],}||jvr|j|d7<qld|j|<qlt |D]}|j |d7<qqt|j }|j dddd|j durg}n|j g}|d d |Dtt|ttdt|d|_d d |j D|_t|j D]\}}||j|j|<qNdS) aUpdates internal vocabulary based on a list of texts. In the case where texts contains lists, we assume each entry of the lists to be a token. Required before using `texts_to_sequences` or `texts_to_matrix`. # Arguments texts: can be a list of strings, a generator of strings (for memory-efficiency), or a list of list of strings. rcSsg|] }|qSrrrZ text_elemrrrrrz*Tokenizer.fit_on_texts..cSs|dS)Nrr)xrrrrz(Tokenizer.fit_on_texts..T)keyreverseNcss|]}|dVqdS)rNr)rwcrrr rz)Tokenizer.fit_on_texts..cSsi|]\}}||qSrr)rr1rrrrrrz*Tokenizer.fit_on_texts..)r<r=rlistrr'r%rr9setr:itemssortr>extenddictr rr!r@rAr?)rBtextsr$r&r1ZwcountsZ sorted_vocrrrr fit_on_textss>      zTokenizer.fit_on_textscCsD|jt|7_|D](}t|}|D]}|j|d7<q&qdS)a%Updates internal vocabulary based on a list of sequences. Required before using `sequences_to_matrix` (if `fit_on_texts` was never called). # Arguments sequences: A list of sequence. A "sequence" is a list of integer word indices. rN)r<r!rNr?)rB sequencesr&rrrrfit_on_sequencess zTokenizer.fit_on_sequencescCst||S)aNTransforms each text in texts to a sequence of integers. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments texts: A list of texts (strings). # Returns A list of sequences. )rMtexts_to_sequences_generator)rBrSrrrtexts_to_sequences s zTokenizer.texts_to_sequencesc cs|j}|j|j}|D]}|js,t|trZ|jrTt|trLdd|D}n|}|}nt||j |j|j }g}|D]X}|j|}|dur|r||kr|dur| |q| |qv|jdurv| |qv|VqdS)aTransforms each text in `texts` to a sequence of integers. Each item in texts can also be a list, in which case we assume each item of that list to be a token. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments texts: A list of texts (strings). # Yields Yields individual sequences. cSsg|] }|qSrrErFrrrr0rz:Tokenizer.texts_to_sequences_generator..N) r;r@getr>r=rrMrr'r%rappend) rBrSr;oov_token_indexr$r&vectr1rrrrrWs2       z&Tokenizer.texts_to_sequences_generatorcCst||S)aYTransforms each sequence into a list of text. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments sequences: A list of sequences (list of integers). # Returns A list of texts (strings) )rMsequences_to_texts_generator)rBrUrrrsequences_to_textsFs zTokenizer.sequences_to_textsccs|j}|j|j}|D]}g}|D]d}|j|}|durn|rb||krb|durl||j|q||q$|jdur$||j|q$d|}|VqdS)aTransforms each sequence in `sequences` to a list of texts(strings). Each sequence has to a list of integers. In other words, sequences should be a list of sequences Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments sequences: A list of sequences. # Yields Yields individual texts. Nr )r;r@rYr>rArZjoin)rBrUr;r[r&r\numwordrrrr]Ts     z&Tokenizer.sequences_to_texts_generatorbinarycCs||}|j||dS)zConvert a list of texts to a Numpy matrix. # Arguments texts: list of strings. mode: one of "binary", "count", "tfidf", "freq". # Returns A Numpy matrix. )mode)rXsequences_to_matrix)rBrSrcrUrrrtexts_to_matrixts zTokenizer.texts_to_matrixc CsR|js&|jrt|jd}q,tdn|j}|dkrB|jsBtdtt||f}t|D]\}}|sjq\tt }|D]}||krqv||d7<qvt | D]\}} |dkr| |||<q|dkr| t||||<q|dkrd|||<q|dkr@dt | } t d|jd|j |d} | | |||<qtd |qq\|S) aConverts a list of sequences into a Numpy matrix. # Arguments sequences: list of sequences (a sequence is a list of integer word indices). mode: one of "binary", "count", "tfidf", "freq" # Returns A Numpy matrix. # Raises ValueError: In case of invalid `mode` argument, or if the Tokenizer requires to be fit to sample data. rzKSpecify a dimension (`num_words` argument), or fit on some text data first.Ztfidfz7Fit the Tokenizer on some data before using tfidf mode.countfreqrbrzUnknown vectorization mode:)r;r@r! ValueErrorr<npZzeros enumeraterr.rMrOlogr?rY) rBrUrcr;rGrr&countsjrtfZidfrrrrds>   zTokenizer.sequences_to_matrixc Csht|j}t|j}t|j}t|j}t|j}|j|j|j |j |j |j |j |||||d S)a:Returns the tokenizer configuration as Python dictionary. The word count dictionaries used by the tokenizer get serialized into plain JSON, so that the configuration can be read by other projects. # Returns A Python dictionary with the tokenizer configuration. ) r;r%rrr=r>r<r9r:r?rAr@)jsondumpsr9r:r?r@rAr;r%rrr=r>r<)rBZjson_word_countsZjson_word_docsZjson_index_docsZjson_word_indexZjson_index_wordrrr get_configs$     zTokenizer.get_configcKs(|}|jj|d}tj|fi|S)aReturns a JSON string containing the tokenizer configuration. To load a tokenizer from a JSON string, use `keras.preprocessing.text.tokenizer_from_json(json_string)`. # Arguments **kwargs: Additional keyword arguments to be passed to `json.dumps()`. # Returns A JSON string containing the tokenizer configuration. ) class_nameconfig)rq __class____name__rorp)rBrCrstokenizer_configrrrto_jsons zTokenizer.to_json)Nr Tr FNr)rb)rb)ru __module__ __qualname____doc__rDrTrVrXrWr^r]rerdrqrwrrrrr3s& 6+ 7r3c Cst|}|d}t|d}t|d}t|d}dd|D}t|d}dd|D}t|d }tfi|}||_||_||_||_ ||_ |S) zParses a JSON tokenizer configuration file and returns a tokenizer instance. # Arguments json_string: JSON string encoding a tokenizer configuration. # Returns A Keras Tokenizer instance rsr9r:r?cSsi|]\}}t||qSrr.rkvrrrrrz'tokenizer_from_json..rAcSsi|]\}}t||qSrr{r|rrrrrr@) roloadsrYr6rOr3r9r:r?r@rA) Z json_stringrvrsr9r:r?rAr@Z tokenizerrrrtokenizer_from_jsons  r)r Tr )r Tr )Nr Tr )rz __future__rrrstringrr4 collectionsrrhashlibrronumpyriZ six.movesrr rr"r8r'r,r)objectr3rrrrrs@          )  ._