a Sic@sdZddlZddlZddlmmZddlm Z ddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZejZejZejZejZejZd Zd ZGd ddejj Z!Gddde j"Z#Gddde j$Z%dS)z'Keras index lookup preprocessing layer.N)backend)base_layer_utils)base_preprocessing_layer)preprocessing_utils)layer_serialization) layer_utils)tf_utils) tf_loggingvocab idf_weightsc@s8eZdZdZddZeddZeddZdd Zd S) NullInitializerzEA placeholder initializer for restoring this layer from a SavedModel.cCs||_||_dS)zConstruct a table initializer object. Args: key_dtype: Type of the table keys. value_dtype: Type of the table values. N) _key_dtype _value_dtype)self key_dtype value_dtyperc/var/www/html/django/DPS/env/lib/python3.9/site-packages/keras/layers/preprocessing/index_lookup.py__init__/szNullInitializer.__init__cCs|jS)zThe expected table key dtype.)r rrrrr9szNullInitializer.key_dtypecCs|jS)zThe expected table value dtype.)rrrrrr>szNullInitializer.value_dtypecCsdS)z$Returns the table initialization op.Nr)rtablerrr initializeCszNullInitializer.initializeN) __name__ __module__ __qualname____doc__rpropertyrrrrrrrr ,s   r c@s4eZdZdZddZeddZddZdd Zd S) VocabWeightHandlerz;Adds the vocabulary as a layer weight during serialization.cCs||_|j|_tj|_dSN)_layervocabulary_dtype_dtypetf distribute get_strategy_distribute_strategy)rZ lookup_layerrrrrKszVocabWeightHandler.__init__cCsdS)Nrrrrr num_tensorsPszVocabWeightHandler.num_tensorscCs&t|d|j}|j||j_dSNr)r"convert_to_tensorr!r_lookup_table_from_tokens lookup_table)rweightstokensrrr set_weightsTszVocabWeightHandler.set_weightscCs"|jjdd}t||j}|gS)NF)include_special_tokens)rget_vocabularyr"r)r!)rr-rrr get_tensorsXszVocabWeightHandler.get_tensorsN) rrrrrrr'r.r1rrrrrHs  rcseZdZdZd:fdd ZddZd d Zd;d d ZddZddZ fddZ dr<)rur>as_listrIr rYr" TensorSpec)r input_spec output_shape output_dtyperrrcompute_output_signaturehsz$IndexLookup.compute_output_signatureTcsjdkrgg}}n<j\}}jr6||fn||f\}}||}}tfddt||fddt  D}j durj t krj |d<|s|d}|S)aqReturns the current vocabulary of the layer. Args: include_special_tokens: If True, the returned vocabulary will include mask and OOV tokens, and a term's index in the vocabulary will equal the term's index when calling the layer. If False, the returned vocabulary will not include any mask or OOV tokens. rcsjSr)rMrrrrz,IndexLookup.get_vocabulary..csg|] }|qSrr).0x)rgrr r}z.IndexLookup.get_vocabulary..N)r+sizeexportrI_tensor_vocab_to_numpynumpy collections defaultdictzipranger:rLr5rEra)rr/r indiceskeysvaluesr)rgrrr0os   zIndexLookup.get_vocabularycCs8tr"t|j|S|j|SdS)zGets the current size of the layer's vocabulary. Returns: The integer size of the vocabulary, including optional mask and oov indices. N)r"executing_eagerlyr3r+rrrarrrrr:s zIndexLookup.vocabulary_sizecCstd|S)Nz5vocab_size is deprecated, please use vocabulary_size.)loggingwarningr:rrrr vocab_sizes zIndexLookup.vocab_sizec sd|j|j|j|j|j|j|j|jt |j |j t |j d }t }tt|t|S)N) rIrJrKrMrLr5rNrOrmr r )rIrJrKrMrLr5rNrOutilslistify_tensorsrQr rRrW get_configdictlistitems)rconfig base_configrqrrrs   zIndexLookup.get_configcCsh|jtkr(|dur(td|jd|t|trptjj|sNtd ||jtkr`td| ||_ dSt st |st |rtd |jj|jt |r||}nt|ttfrt|}t |r|}nt|ttfrt|}|jdkrtd ||}|}|jg||jg|j}t||d|}|rh||d}n|}||}|rtd ||jdur|j|vrt ||jkd } td |d |jd | |jdur"|j!r"|j|vr"t ||jkd } td|d|jd | |t"|} |j#durX| |j#krXtd | |j#|$||_ |jtkrd|durtdt"|t"|krtd t"|t"||%|}|j&dkrtd t'||rd} d} n|} t(|} d}|j)r|j#dur|j#| t"|}nd}tj*|| |fd| |fd}tj+||j,d}|j-.||j-/|_0dS)aSets vocabulary (and optionally document frequency) data for this layer. This method sets the vocabulary and idf weights for this layer directly, instead of analyzing a dataset through `adapt`. It should be used whenever the vocab (and optionally document frequency) information is already known. If vocabulary data is already present in the layer, this method will replace it. Args: vocabulary: Either an array or a string path to a text file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D tensor containing the vocbulary terms. If passing a file path, the file should contain one line per term in the vocabulary. idf_weights: A tuple, list, 1D numpy array, or 1D tensor of inverse document frequency weights with equal length to vocabulary. Must be set if `output_mode` is `"tf_idf"`. Should not be set otherwise. Raises: ValueError: If there are too many inputs, the inputs do not match, or input data is missing. RuntimeError: If the vocabulary cannot be set when this function is called. This happens when `"multi_hot"`, `"count"`, and `"tf_idf"` modes, if `pad_to_max_tokens` is False and the layer itself has already been called. RuntimeError: If a tensor vocabulary is passed outside of eager execution. NzU`idf_weights` should only be set if output_mode is `'tf_idf'`. Received: output_mode=z and idf_weights=z"Vocabulary file {} does not exist.zGoutput_mode `'tf_idf'` does not support loading a vocabulary from file.zCannot set a tensor vocabulary on {} layer {} when not executing eagerly. Create this layer or call `set_vocabulary` outside of any `tf.function`s and with eager execution enabled.rz.Cannot set an empty vocabulary, you passed {}.zmThe passed vocabulary has at least one repeated term. Please uniquify your dataset. The repeated terms are {}r=aFound reserved mask token at unexpected location in `vocabulary`. Note that passed `vocabulary` does not need to include the OOV and mask tokens. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: z. Received: mask_token=z at vocabulary index aFound reserved OOV token at unexpected location in `vocabulary`. Note that passed `vocabulary` does not need to include the OOV and mask tokens. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: z. Received: oov_token=zpAttempted to set a vocabulary larger than the maximum vocab size. Passed vocab size is {}, max vocab size is {}.z2`idf_weights` must be set if output_mode is TF_IDFzb`idf_weights` must be the same length as vocabulary. len(idf_weights) is {}, len(vocabulary) is {}r&z4TF-IDF data must be a 1-index array, but received {}constantconstant_valuesr<)1r5rCrA isinstancestrr"iogfileexistsformat_lookup_table_from_filer+r is_tensor RuntimeErrorrHrnamerrtuplenparrayrrr]rarLrMrK array_equal_find_repeated_tokensargwhererIlenrJr*_convert_to_ndarrayndimtypeaveragerOpadr)rYr assignrbrc)rrmr Z oov_start token_startZspecial_tokensZfound_special_tokensr-Zrepeated_tokensZ mask_indexZ oov_indexnew_vocab_sizeZ front_paddingZfront_padding_valueZback_padding_valueZ back_paddingr,rrrrds                         zIndexLookup.set_vocabularycCs|jrtd|jjtj||jd}|jj dkr@t |d}|jj dkrXt |d}| |\}}|j |||j ||jtkrt dd|}| |\}}|j|||j|t|r|j|n|jt j|t jdddS)Nz^Cannot adapt {} layer after setting a static vocabulary via init argument or `set_vocabulary`.rrr&cSst|dSr()r"unique)rrrrr|r}z*IndexLookup.update_state..)out_type)rTrArrHrr ensure_tensorr r>rankr" expand_dims _num_tokensrjinsertrgr5rCmap_fnrkr is_raggedrl assign_addnrowsrU)rdatar-countsZdeduped_doc_dataZ doc_countsrrr update_statems4      zIndexLookup.update_statecCs|jst|jdr2|jtkr.|j|_ dS|j durV|j t |j g|j |jdurz|j t |jg|j |j\}}t||fddd}|}|jr|j|}|d|}t||}|||_|jtkr|j|}|||j}t||j}tj||dggt|d}|j rn|jdurntj|d|jt|ggdd}|j!||j|_ |"dS)Nrr=r)#rTr"equalrjrr5rCr rbrcrLremover)r rMrrlexsortrrarJgatherr*r+rkrg_inverse_document_frequencyrlcastrYr reduce_meanrOr reset_state)rr-rsorted_indicesrZmax_learned_tokensrkr rrrfinalize_statesP              zIndexLookup.finalize_statecCsP|jr dS|j|jd|jtkrL|j|jd|jddSr() rTrjrrr5rCrkrlrrrrrrs  zIndexLookup.reset_statecCs|tj||jd}|j}|jjdkr6||d}t|r\t |j | |j |j}n&t|rxt j|j |}n | |}|jtkr|jdkrt |d}|S|jr|jn|j}|jtkr|jnd}tj||j||j|j|dS)Nrrr=)r5rtr<rNr )_maybe_freeze_vocab_sizerrr r>r _expand_dimsr is_sparser" SparseTensorr _lookup_denser dense_shaperraggedmap_flat_valuesr5rEsqueezerOrJrPrCrcencode_categorical_inputsrYrN)rinputsoriginal_shapelookupsrtr rrrcalls@         zIndexLookup.callc Cs`tr$t|r$tj||jd}n |j|}|jdurXt ||j }t ||j |}|j rb|Sg}|jdkrt t |d}t||}tjd|f}tt t|d|g}||n`|jdkr&|jjrtj||j}ntjj||jd}||}t ||j} t | ||}t|t|WdS1sR0YdS)zALookup table values for a dense Tensor, handling masking and OOV.rNrr=zwWhen `num_oov_indices=0` all inputs should be in vocabulary, found OOV values {}, consider setting `num_oov_indices=1`.r&) num_buckets)r"rris_keras_tensor zeros_likerr+rgrLrr^wherer_rIrK gather_ndstringsrAssertrappendr rZmathfloormodto_hash_bucket_fastr]r[control_dependenciesidentity) rrrZmask_locationsZ lookup_checksZ oov_indicesZ oov_inputsmsg assertionZ oov_locationsrrrrs:        zIndexLookup._lookup_densecCsJt.t|j|j}tj||jWdS1s<0YdSr)r" init_scoper r rrgStaticHashTabler[)r initializerrrrre2s z'IndexLookup._uninitialized_lookup_tablec Cst|}|t|}|jr,|jn|j}tj|||d}|jrP||fn||f\}}tj |||j|j}tj ||j WdS1s0YdS)Nr) r"rrarrIr rrrgKeyValueTensorInitializerrr[) rr-rZ token_end indices_dtyperrrrrrrr*7s  z%IndexLookup._lookup_table_from_tokensc Cs|jrtjjj}tjjj}ntjjj}tjjj}t@tjj||j||j || d}tj ||j WdS1s~0YdS)N)filenamer key_indexr value_indexvalue_index_offset) rIr"rg TextFileIndex LINE_NUMBER WHOLE_LINErTextFileInitializerr rrarr[)rrrrrrrrrGs     z#IndexLookup._lookup_table_from_filecCst|ttfrt|S|Sr)rrrrr)rrrrrrYszIndexLookup._convert_to_ndarraycCs(t|rtj||St||SdSr)rrr"rNr)rraxisrrrr\s zIndexLookup._expand_dimscCs|jdur|jtkrdSdS)Nr&r)rLr5rErrrrr]bszIndexLookup._oov_start_indexcCs||jSr)r]rKrrrrragszIndexLookup._token_start_indexcCs|jtks|jrdSt0ts6td|j|}Wdn1sR0Y|| krztd|jn*|j dur||j krtd|j|j |||_ dS)Nz.N)setrrCounterr)rrmZvocabulary_setrrrrs  z!IndexLookup._find_repeated_tokenscCsPt|r|j}n t|r$|j}nt|dg}tj|tjd\}}}||fS)z?Count the number of tokens in a ragged, sparse or dense tensor.r=)out_idx) rrrr flat_valuesr"reshapeunique_with_countsrU)rrrr-_rrrrrs  zIndexLookup._num_tokenscCstjd|d|S)aComputes the inverse-document-frequency (IDF) component of "tf_idf". Uses the default weighting scheme described in https://en.wikipedia.org/wiki/Tf%E2%80%93idf. Args: token_document_counts: An array of the # of documents each token appears in. num_documents: An int representing the total number of documents Returns: An array of "inverse document frequency" weights. r&)r"rlog)rrkrlrrrrsz'IndexLookup._inverse_document_frequencycCs t|Sr)rVocabularySavedModelSaverrrrr_trackable_saved_model_saversz(IndexLookup._trackable_saved_model_savercCs|S)z3Converts a tensor vocabulary to a numpy vocabulary.)r)rrmrrrrsz"IndexLookup._tensor_vocab_to_numpy)NNFr3FF)T)N) rrrrrrur{r0r:rrrdrrrrrrer*rrrr]rarrrrrrr __classcell__rrrqrr2_sHJ<    ;$; +0   r2)&rrrrtensorflow.compat.v2compatv2r"kerasr keras.enginerrkeras.layers.preprocessingrrkeras.saving.saved_modelr keras.utilsrrtensorflow.python.platformr rrErBrFrGrCZ _VOCAB_NAMEZ_IDF_WEIGHTS_NAMErgrr TrackableWeightHandlerrPreprocessingLayerr2rrrrs*