a SicX@sdZddlZddlZddlZddlZddlZddlmZedddd Z ed dd d Z ed dddZ edGddde Z edddZdS)aUtilities for text input preprocessing. Deprecated: `tf.keras.preprocessing.text` APIs are not recommended for new code. Prefer `tf.keras.utils.text_dataset_from_directory` and `tf.keras.layers.TextVectorization` which provide a more efficient approach for preprocessing text input. For an introduction to these APIs, see the [text loading tutorial] (https://www.tensorflow.org/tutorials/load_data/text) and [preprocessing layer guide] (https://www.tensorflow.org/guide/keras/preprocessing_layers). N) keras_exportz.keras.preprocessing.text.text_to_word_sequence!!"#$%&()*+,-./:;<=>?@[\]^_`{|}~ T csJ|r |}fdd|D}t|}||}|}dd|DS)aConverts a text to a sequence of words (or tokens). Deprecated: `tf.keras.preprocessing.text.text_to_word_sequence` does not operate on tensors and is not recommended for new code. Prefer `tf.strings.regex_replace` and `tf.strings.split` which provide equivalent functionality and accept `tf.Tensor` input. For an overview of text handling in Tensorflow, see the [text loading tutorial] (https://www.tensorflow.org/tutorials/load_data/text). This function transforms a string of text into a list of words while ignoring `filters` which include punctuations by default. >>> sample_text = 'This is a sample sentence.' >>> tf.keras.preprocessing.text.text_to_word_sequence(sample_text) ['this', 'is', 'a', 'sample', 'sentence'] Args: input_text: Input text (string). filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``'!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\\t\\n'``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to convert the input to lowercase. split: str. Separator for word splitting. Returns: A list of words (or tokens). csi|] }|qSr).0csplitrT/var/www/html/django/DPS/env/lib/python3.9/site-packages/keras/preprocessing/text.py Lz)text_to_word_sequence..cSsg|] }|r|qSrr)rirrr Qr z)text_to_word_sequence..)lowerstr maketrans translater ) input_textfiltersrr Ztranslate_dictZ translate_mapseqrrr text_to_word_sequence's"   rz keras.preprocessing.text.one_hotc Cst||t||||dS)a One-hot encodes a text into a list of word indexes of size `n`. Deprecated: `tf.keras.text.preprocessing.one_hot` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.Hashing` with `output_mode='one_hot'` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [preprocessing layer guide] (https://www.tensorflow.org/guide/keras/preprocessing_layers) for an overview of preprocessing layers. This function receives as input a string of text and returns a list of encoded integers each corresponding to a word (or token) in the given input string. Args: input_text: Input text (string). n: int. Size of vocabulary. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``` '!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n ```, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. analyzer: function. Custom analyzer to split the text Returns: List of integers in `[1, n]`. Each integer encodes a word (unicity non-guaranteed). ) hash_functionrrr analyzer) hashing_trickhash)rnrrr rrrr one_hotTs(rz&keras.preprocessing.text.hashing_trickcsTdurtndkrdd|dur8t||||d}n||}fdd|DS)aConverts a text to a sequence of indexes in a fixed-size hashing space. Deprecated: `tf.keras.text.preprocessing.hashing_trick` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.Hashing` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [preprocessing layer guide]( https://www.tensorflow.org/guide/keras/preprocessing_layers) for an overview of preprocessing layers. Args: text: Input text (string). n: Dimension of the hashing space. hash_function: defaults to python `hash` function, can be 'md5' or any function that takes in input a string and returns a int. Note that 'hash' is not a stable hashing function, so it is not consistent across different runs, while 'md5' is a stable hashing function. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\\t\\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. analyzer: function. Custom analyzer to split the text Returns: A list of integer word indices (unicity non-guaranteed). `0` is a reserved index that won't be assigned to any word. Two or more words may be assigned to the same index, due to possible collisions by the hashing function. The [probability]( https://en.wikipedia.org/wiki/Birthday_problem#Probability_table) of a collision is in relation to the dimension of the hashing space and the number of distinct objects. Nmd5cSstt|dS)N)inthashlibrencode hexdigest)wrrr r zhashing_trick..rrr cs g|]}|ddqS)r)rr#rrrr rr z!hashing_trick..)rr)textrrrrr rrrr'r rs,rz"keras.preprocessing.text.Tokenizerc@sneZdZdZdddZd d Zd d Zd dZddZddZ ddZ dddZ d ddZ ddZ ddZdS)! TokenizeragText tokenization utility class. Deprecated: `tf.keras.preprocessing.text.Tokenizer` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.TextVectorization` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [text loading tutorial](https://www.tensorflow.org/tutorials/load_data/text) for an overview of the layer and text handling in tensorflow. This class allows to vectorize a text corpus, by turning each text into either a sequence of integers (each integer being the index of a token in a dictionary) or into a vector where the coefficient for each token could be binary, based on word count, based on tf-idf... By default, all punctuation is removed, turning the texts into space-separated sequences of words (words maybe include the `'` character). These sequences are then split into lists of tokens. They will then be indexed or vectorized. `0` is a reserved index that won't be assigned to any word. Args: num_words: the maximum number of words to keep, based on word frequency. Only the most common `num_words-1` words will be kept. filters: a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the `'` character. lower: boolean. Whether to convert the texts to lowercase. split: str. Separator for word splitting. char_level: if True, every character will be treated as a token. oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls analyzer: function. Custom analyzer to split the text. The default analyzer is text_to_word_sequence NrTrFc Ksd|vrtd|d}|dd} |r.Nr%cSs|dS)Nr&r)xrrr r$7r z(Tokenizer.fit_on_texts..T)keyreversecss|]}|dVqdS)rNr)rZwcrrr =r z)Tokenizer.fit_on_texts..cSsi|]\}}||qSrr)rr#rrrr r Dr z*Tokenizer.fit_on_texts..)r+r6 isinstancelistrrrrr r2setr4itemssortr7extenddictziprangelenr9r:r8)r;textsr(rr#ZwcountsZ sorted_vocrrrr fit_on_texts sD        zTokenizer.fit_on_textscCsD|jt|7_|D](}t|}|D]}|j|d7<q&qdS)aUpdates internal vocabulary based on a list of sequences. Required before using `sequences_to_matrix` (if `fit_on_texts` was never called). Args: sequences: A list of sequence. A "sequence" is a list of integer word indices. r&N)r+rMrFr8)r; sequencesrr rrr fit_on_sequencesIs zTokenizer.fit_on_sequencescCst||S)aGTransforms each text in texts to a sequence of integers. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: texts: A list of texts (strings). Returns: A list of sequences. )rEtexts_to_sequences_generator)r;rNrrr texts_to_sequencesYs zTokenizer.texts_to_sequencesc cs|j}|j|j}|D]}|js,t|trZ|jrTt|trLdd|D}n|}|}n,|jdur|t ||j |j|j d}n ||}g}|D]X}|j|}|dur|r||kr|dur| |q| |q|jdur| |q|VqdS)aTransforms each text in `texts` to a sequence of integers. Each item in texts can also be a list, in which case we assume each item of that list to be a token. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: texts: A list of texts (strings). Yields: Yields individual sequences. cSsg|] }|qSrr>r?rrr r|r z:Tokenizer.texts_to_sequences_generator..Nr%) r5r9getr7r6rDrErrrrr append) r;rNr5oov_token_indexr(rvectr#r rrr rRgs8         z&Tokenizer.texts_to_sequences_generatorcCst||S)aRTransforms each sequence into a list of text. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: sequences: A list of sequences (list of integers). Returns: A list of texts (strings) )rEsequences_to_texts_generator)r;rPrrr sequences_to_textss zTokenizer.sequences_to_textsccs|j}|j|j}|D]}g}|D]d}|j|}|durn|rb||krb|durl||j|q||q$|jdur$||j|q$d|}|VqdS)aTransforms each sequence in `sequences` to a list of texts(strings). Each sequence has to a list of integers. In other words, sequences should be a list of sequences Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: sequences: A list of sequences. Yields: Yields individual texts. Nr)r5r9rTr7r:rUjoin)r;rPr5rVrrWnumwordrrr rXs     z&Tokenizer.sequences_to_texts_generatorbinarycCs||}|j||dS)zConvert a list of texts to a Numpy matrix. Args: texts: list of strings. mode: one of "binary", "count", "tfidf", "freq". Returns: A Numpy matrix. )mode)rSsequences_to_matrix)r;rNr^rPrrr texts_to_matrixs zTokenizer.texts_to_matrixc CsT|js&|jrt|jd}q,tdn|j}|dkrB|jsBtdtt||f}t|D]\}}|sjq\t t }|D]}||krqx||d7<qxt | D]\}} |dkr| |||<q|dkr| t||||<q|dkrd|||<q|dkrBdt | } t d|jd|j|d} | | |||<qtd |qq\|S) aConverts a list of sequences into a Numpy matrix. Args: sequences: list of sequences (a sequence is a list of integer word indices). mode: one of "binary", "count", "tfidf", "freq" Returns: A Numpy matrix. Raises: ValueError: In case of invalid `mode` argument, or if the Tokenizer requires to be fit to sample data. r&zKSpecify a dimension (`num_words` argument), or fit on some text data first.Ztfidfz7Fit the Tokenizer on some data before using tfidf mode.countfreqr]rzUnknown vectorization mode:)r5r9rM ValueErrorr+npzeros enumerater0r3rrErGlogr8rT) r;rPr^r5r@r rcountsjrtfZidfrrr r_sJ  zTokenizer.sequences_to_matrixc Csht|j}t|j}t|j}t|j}t|j}|j|j|j |j |j |j |j |||||d S)a:Returns the tokenizer configuration as Python dictionary. The word count dictionaries used by the tokenizer get serialized into plain JSON, so that the configuration can be read by other projects. Returns: A Python dictionary with the tokenizer configuration. ) r5rrr r6r7r+r2r4r8r:r9)jsondumpsr2r4r8r9r:r5rrr r6r7r+)r;Zjson_word_countsZjson_word_docsZjson_index_docsZjson_word_indexZjson_index_wordrrr get_configs$     zTokenizer.get_configcKs(|}|jj|d}tj|fi|S)aReturns a JSON string containing the tokenizer configuration. To load a tokenizer from a JSON string, use `keras.preprocessing.text.tokenizer_from_json(json_string)`. Args: **kwargs: Additional keyword arguments to be passed to `json.dumps()`. Returns: A JSON string containing the tokenizer configuration. ) class_nameconfig)rm __class____name__rkrl)r;r<rotokenizer_configrrr to_json-s zTokenizer.to_json)NrTrFNN)r])r])rq __module__ __qualname____doc__r=rOrQrSrRrYrXr`r_rmrsrrrr r)s&' $<0 <r)z,keras.preprocessing.text.tokenizer_from_jsonc Cst|}|d}t|d}t|d}t|d}dd|D}t|d}dd|D}t|d }tfi|}||_||_||_||_ ||_ |S) arParses a JSON tokenizer configuration and returns a tokenizer instance. Deprecated: `tf.keras.preprocessing.text.Tokenizer` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.TextVectorization` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [text loading tutorial](https://www.tensorflow.org/tutorials/load_data/text) for an overview of the layer and text handling in tensorflow. Args: json_string: JSON string encoding a tokenizer configuration. Returns: A Keras Tokenizer instance ror2r4r8cSsi|]\}}t||qSrrrkvrrr r Zr z'tokenizer_from_json..r:cSsi|]\}}t||qSrrwrxrrr r \r r9) rkloadsrTr.rGr)r2r4r8r9r:) json_stringrrror2r4r8r:r9 tokenizerrrr tokenizer_from_jsonBs   r~)rTr)rTrN)NrTrN)rvr0r rkr,numpyrd tensorflow.python.util.tf_exportrrrrobjectr)r~rrrr s>   , 2 :