a Sic>@s(dZddlZddlmmZddlmZddlmZddlm Z ddlm Z ddlm Z ddl m Z dd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZdZedgdGdddee jZ edgdGdddeee jZ!ddZ"ddZ#ddZ$dS)zGated Recurrent Unit layer.N) activations)backend) constraints) initializers) regularizers) base_layer) InputSpec)gru_lstm_utils) rnn_utils)RNN)DropoutRNNCellMixin)tf_utils) tf_logging) keras_exportzbRNN `implementation=2` is not supported when `recurrent_dropout` is set. Using `implementation=1`.zkeras.layers.GRUCell)v1csTeZdZdZdfd d Zejfd d ZdddZfddZ dddZ Z S)GRUCella! Cell class for the GRU layer. See [the Keras RNN API guide](https://www.tensorflow.org/guide/keras/rnn) for details about the usage of RNN API. This class processes one step within the whole time sequence input, whereas `tf.keras.layer.GRU` processes the whole sequence. For example: >>> inputs = tf.random.normal([32, 10, 8]) >>> rnn = tf.keras.layers.RNN(tf.keras.layers.GRUCell(4)) >>> output = rnn(inputs) >>> print(output.shape) (32, 4) >>> rnn = tf.keras.layers.RNN( ... tf.keras.layers.GRUCell(4), ... return_sequences=True, ... return_state=True) >>> whole_sequence_output, final_state = rnn(inputs) >>> print(whole_sequence_output.shape) (32, 10, 4) >>> print(final_state.shape) (32, 4) Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass None, no activation is applied (ie. "linear" activation: `a(x) = x`). recurrent_activation: Activation function to use for the recurrent step. Default: sigmoid (`sigmoid`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, (default `True`), whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. Default: `glorot_uniform`. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. Default: `orthogonal`. bias_initializer: Initializer for the bias vector. Default: `zeros`. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. Default: `None`. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_regularizer: Regularizer function applied to the bias vector. Default: `None`. kernel_constraint: Constraint function applied to the `kernel` weights matrix. Default: `None`. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_constraint: Constraint function applied to the bias vector. Default: `None`. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. Default: 0. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. Default: 0. reset_after: GRU convention (whether to apply reset gate after or before matrix multiplication). False = "before", True = "after" (default and cuDNN compatible). Call arguments: inputs: A 2D tensor, with shape of `[batch, feature]`. states: A 2D tensor with shape of `[batch, units]`, which is the state from the previous time step. For timestep 0, the initial state provided by user will be feed to cell. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. Only relevant when `dropout` or `recurrent_dropout` is used. tanhsigmoidTglorot_uniform orthogonalzerosNc s\|dkrtd|dtjjr4|dd|_n|dd|_tjfi|||_ t ||_ t ||_ ||_t ||_t ||_t ||_t ||_t | |_t | |_t | |_t | |_t | |_tdtd||_tdtd||_|d d }|jdkr<|d kr input_shape input_dimdefault_caching_device bias_shaper@rBrCrOsB     z GRUCell.buildcCshtj|r|dn|}|j||dd}|j||dd}|jrb|jsR|jd}}nt|j\}}|j dkrd|j krdkrnn&||d} ||d} ||d} n |} |} |} t | |j ddd|jf} t | |j dd|j|jdf} t | |j dd|jddf}|jrxt | |d|j} t | ||j|jd} t |||jdd}d|jkrdkrnn&||d}||d}||d}n |}|}|}t ||jddd|jf}t ||jdd|j|jdf}|jrP|jrPt ||d|j}t |||j|jd}|| |}|| |}|jrt ||jdd|jddf}|jrt |||jdd}||}n(t |||jdd|jddf}|||}n6d|j krdkr(nn ||d}t ||j }|jrJt ||}tj|ddd \} } }|jrt ||j}|jrt ||}n$t ||jdddd|jf}tj||j|jdgdd \}}}|| |}|| |}|jr||}n(t |||jddd|jdf}|||}||d||}tj|r\|gn|}||fS) NrrEcountrrrrrDaxis)rnest is_nestedget_dropout_mask_for_cell#get_recurrent_dropout_mask_for_cellr*r;rNunstackrr6rdotrFr&bias_addr7rMr)r(split)r>inputsstatestrainingh_tm1dp_mask rec_dp_mask input_biasrecurrent_biasinputs_zinputs_rinputs_hx_zx_rx_hh_tm1_zh_tm1_rh_tm1_h recurrent_z recurrent_rzr recurrent_hhhmatrix_x matrix_innerh new_staterBrBrCcalls    $         z GRUCell.callcs|jt|jt|j|jt|jt|jt|j t |j t |j t |j t|jt|jt|j|j|j|j|jd}|t|t}tt|t|S)N)r&r(r)r*r+r,r-r.r/r0r1r2r3r6r7rr;)r&r serializer(r)r*rr+r,r-rr.r/r0rr1r2r3r6r7rr;updater config_for_enable_caching_devicer$ get_configdictlistitemsr>config base_configr@rBrCr`sB      zGRUCell.get_configcCst||||SN)r #generate_zero_filled_state_for_cell)r>rb batch_sizedtyperBrBrCget_initial_stateszGRUCell.get_initial_state)rrTrrrNNNNNNrrT)N)NNN) __name__ __module__ __qualname____doc__r%r shape_type_conversionrOr}rr __classcell__rBrBr@rCr-s,JA* ~ $rzkeras.layers.GRUcseZdZdZd7fd d Zd8d dZeddZeddZeddZ eddZ eddZ eddZ eddZ eddZedd Zed!d"Zed#d$Zed%d&Zed'd(Zed)d*Zed+d,Zed-d.Zed/d0Zfd1d2Zed3d4Zd5d6ZZS)9GRUa<Gated Recurrent Unit - Cho et al. 2014. See [the Keras RNN API guide](https://www.tensorflow.org/guide/keras/rnn) for details about the usage of RNN API. Based on available runtime hardware and constraints, this layer will choose different implementations (cuDNN-based or pure-TensorFlow) to maximize the performance. If a GPU is available and all the arguments to the layer meet the requirement of the cuDNN kernel (see below for details), the layer will use a fast cuDNN implementation. The requirements to use the cuDNN implementation are: 1. `activation` == `tanh` 2. `recurrent_activation` == `sigmoid` 3. `recurrent_dropout` == 0 4. `unroll` is `False` 5. `use_bias` is `True` 6. `reset_after` is `True` 7. Inputs, if use masking, are strictly right-padded. 8. Eager execution is enabled in the outermost context. There are two variants of the GRU implementation. The default one is based on [v3](https://arxiv.org/abs/1406.1078v3) and has reset gate applied to hidden state before matrix multiplication. The other one is based on [original](https://arxiv.org/abs/1406.1078v1) and has the order reversed. The second variant is compatible with CuDNNGRU (GPU-only) and allows inference on CPU. Thus it has separate biases for `kernel` and `recurrent_kernel`. To use this variant, set `reset_after=True` and `recurrent_activation='sigmoid'`. For example: >>> inputs = tf.random.normal([32, 10, 8]) >>> gru = tf.keras.layers.GRU(4) >>> output = gru(inputs) >>> print(output.shape) (32, 4) >>> gru = tf.keras.layers.GRU(4, return_sequences=True, return_state=True) >>> whole_sequence_output, final_state = gru(inputs) >>> print(whole_sequence_output.shape) (32, 10, 4) >>> print(final_state.shape) (32, 4) Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). recurrent_activation: Activation function to use for the recurrent step. Default: sigmoid (`sigmoid`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, (default `True`), whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. Default: `glorot_uniform`. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. Default: `orthogonal`. bias_initializer: Initializer for the bias vector. Default: `zeros`. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. Default: `None`. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_regularizer: Regularizer function applied to the bias vector. Default: `None`. activity_regularizer: Regularizer function applied to the output of the layer (its "activation"). Default: `None`. kernel_constraint: Constraint function applied to the `kernel` weights matrix. Default: `None`. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_constraint: Constraint function applied to the bias vector. Default: `None`. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. Default: 0. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. Default: 0. return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence. Default: `False`. return_state: Boolean. Whether to return the last state in addition to the output. Default: `False`. go_backwards: Boolean (default `False`). If True, process the input sequence backwards and return the reversed sequence. stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. time_major: The shape format of the `inputs` and `outputs` tensors. If True, the inputs and outputs will be in shape `[timesteps, batch, feature]`, whereas in the False case, it will be `[batch, timesteps, feature]`. Using `time_major = True` is a bit more efficient because it avoids transposes at the beginning and end of the RNN calculation. However, most TensorFlow data is batch-major, so by default this function accepts input and emits output in batch-major form. reset_after: GRU convention (whether to apply reset gate after or before matrix multiplication). False = "before", True = "after" (default and cuDNN compatible). Call arguments: inputs: A 3D tensor, with shape `[batch, timesteps, feature]`. mask: Binary tensor of shape `[samples, timesteps]` indicating whether a given timestep should be masked (optional, defaults to `None`). An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. This argument is passed to the cell when calling it. This is only relevant if `dropout` or `recurrent_dropout` is used (optional, defaults to `None`). initial_state: List of initial state tensors to be passed to the first call of the cell (optional, defaults to `None` which causes creation of zero-filled initial state tensors). rrTrrrNrFc sv|dd|_|dd}|dkr,tdd|vrDd|di}ni}t|f|||||||| | | | ||||||d|d d d |}tj|f||||||d |t| |_ t d dg|_ |j t jtjfvo|jt jtjfvo|dko| o|o|otjj|_tjdrX|jrFttj|jnttj|jtrrt||d|_dS)Nreturn_runtimeFrrrzm`implementation=0` has been deprecated, and now defaults to `implementation=2`.Please update your layer call.rr trainableT)r(r)r*r+r,r-r.r/r0r1r2r3r6r7rr;rr)return_sequences return_state go_backwardsstatefulunroll time_majorrE)ndimGPUgru) r"_return_runtimer8warningrr'r$r%ractivity_regularizerr input_specr(rrrr)rr rr!_could_use_gpu_kernelrlist_logical_devicesr9r ZCUDNN_AVAILABLE_MSGrHZCUDNN_NOT_AVAILABLE_MSGuse_new_gru_lstm_implZ DefunWrapper_defun_wrapper)r>r&r(r)r*r+r,r-r.r/r0rr1r2r3r6r7rrrrrrr;r?r cell_kwargscellr@rBrCr% s        z GRU.__init__c st|\}}|du}||||d\}}}t|trH|d}t|}jr`|dn|d} jsd|i j fdd} tj | ||dj |j |dur|n| jjjd \} } } ttj}n|||||\} } }} jr2tjjjdt| djdjg}|jrPtj|| |j d}n| }jrj|gt| Sjrz||fS|SdS)Nrrrdcsj||fiSr)r) cell_inputs cell_statesr?r>rBrCstepszGRU.call..step) constantsrmaskr input_lengthrzero_output_for_maskreturn_all_outputs)r) rconvert_inputs_if_ragged_validate_args_if_ragged_process_inputs isinstancer int_shaperr_maybe_reset_cell_dropout_maskrrnnrrrrr runtimeZRUNTIME_UNKNOWN_defun_gru_callrrr rassignrccastr add_updatemaybe_convert_to_raggedrr)r>rbrrd initial_state row_lengthsis_ragged_input_rR timestepsr last_outputoutputsrcrupdatesoutputrBrrCr}rsh        zGRU.callcCs|jjSr)rr&r>rBrBrCr&sz GRU.unitscCs|jjSr)rr(rrBrBrCr(szGRU.activationcCs|jjSr)rr)rrBrBrCr)szGRU.recurrent_activationcCs|jjSr)rr*rrBrBrCr*sz GRU.use_biascCs|jjSr)rr+rrBrBrCr+szGRU.kernel_initializercCs|jjSr)rr,rrBrBrCr,szGRU.recurrent_initializercCs|jjSr)rr-rrBrBrCr-szGRU.bias_initializercCs|jjSr)rr.rrBrBrCr.szGRU.kernel_regularizercCs|jjSr)rr/rrBrBrCr/szGRU.recurrent_regularizercCs|jjSr)rr0rrBrBrCr0szGRU.bias_regularizercCs|jjSr)rr1rrBrBrCr1szGRU.kernel_constraintcCs|jjSr)rr2rrBrBrCr2szGRU.recurrent_constraintcCs|jjSr)rr3rrBrBrCr3szGRU.bias_constraintcCs|jjSr)rr6rrBrBrCr6sz GRU.dropoutcCs|jjSr)rr7rrBrBrCr7szGRU.recurrent_dropoutcCs|jjSr)rrrrBrBrCrszGRU.implementationcCs|jjSr)rr;rrBrBrCr;szGRU.reset_aftercs|jt|jt|j|jt|jt|jt|j t |j t |j t |j t |jt|jt|jt|j|j|j|j|jd}|t|jt}|d=tt|t|S)N)r&r(r)r*r+r,r-r.r/r0rr1r2r3r6r7rr;r) r&rr~r(r)r*rr+r,r-rr.r/r0rrr1r2r3r6r7rr;rr rrr$rrrrrr@rBrCrsJ     " zGRU.get_configcCs*d|vr|ddkrd|d<|fi|S)NrrrrB)clsrrBrBrC from_config&szGRU.from_configc Cs||j||dd}|dur,||d}tr|t|dt|jjt|jjt|jj||j |j ||j d }|j j fi|\}} } } n|t|dt|jjt|jjt|jj||j |j ||jd } | } | d|j itr|t}|tjks.|duoDtjdoD|dupDt||j }|rdtfi| \}} } } ntfi| \}} } } ntfi| \}} } } | g}|| | |fS)NrErVr) rbinit_hrFrMrNrrrsequence_lengthsr rbrrFrMrNrrrrrrr)reset_dropout_maskr\r rZread_variable_valuerrFrMrNrrrr defun_layerrcopyrrexecuting_eagerlyZget_context_device_typeGPU_DEVICE_NAMErris_cudnn_supported_inputsgpu_gru standard_grugru_with_backend_selection)r>rbrrdrr dropout_mask gru_kwargsrrnew_hrgpu_gru_kwargsnormal_gru_kwargs device_type can_use_gpurcrBrBrCr,s             zGRU._defun_gru_call)rrTrrrNNNNNNNrrFFFFFFT)NNN)rrrrr%r}propertyr&r(r)r*r+r,r-r.r/r0r1r2r3r6r7rr;r classmethodrrrrBrBr@rCrsg H                  ( rc  st|} |r| dn| d} t|\fdd} tj| ||gdd||||dur`|n| | | d \}}}|||dttjfS)a GRU with standard kernel implementation. This implementation can be run on all types of hardware. This implementation lifts out all the layer weights and make them function parameters. It has same number of tensor input params as the cuDNN counterpart. The RNN step logic has been simplified, eg dropout and mask is removed since cuDNN implementation does not support that. Args: inputs: Input tensor of GRU layer. init_h: Initial state tensor for the cell output. kernel: Weights for cell kernel. recurrent_kernel: Weights for cell recurrent kernel. bias: Weights for cell kernel bias and recurrent bias. The bias contains the combined input_bias and recurrent_bias. mask: Binary tensor of shape `(samples, timesteps)` indicating whether a given timestep should be masked. An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. time_major: Boolean, whether the inputs are in the format of [time, batch, feature] or [batch, time, feature]. go_backwards: Boolean (default False). If True, process the input sequence backwards and return the reversed sequence. sequence_lengths: The lengths of all sequences coming from a variable length input, such as ragged tensors. If the input has a fixed timestep size, this should be None. zero_output_for_mask: Boolean, whether to output zero for masked timestep. return_sequences: Boolean. If True, return the recurrent outputs for all timesteps in the sequence. If False, only return the output for the last timestep (which consumes less memory). Returns: last_output: output tensor for the last timestep, which has shape [batch, units]. outputs: - If `return_sequences=True`: output tensor for all timesteps, which has shape [batch, time, units]. - Else, a tensor equal to `last_output` with shape [batch, 1, units] state_0: the cell output, which has same shape as init_h. runtime: constant string tensor which indicate real runtime hardware. This value is for testing purpose and should be used by user. rrcs|d}t|}t|}tj|ddd\}}}t|}t|}tj|ddd\}} } t||} t|| } t|| | } | |d| | }||gfS)z5Step function that will be used by Keras RNN backend.rrErrX)rr_r`rrarr)rrreryrmrnrorzrsrtrwrurvrxr{rhrFrirMrBrCrs     zstandard_gru..stepNF)rrrrrrrr)rrrr^rr rZ RUNTIME_CPU)rbrrFrMrNrrrrrrrRrrrr new_statesrBrrCrs08   rc  CsD|durt||}|s8|dur8tj|dd}d\} } n|r@dnd\} } tj|| d}tj|ddd} | tj|ddd7} tt|d }tj d r| d| d | d <| d<| d | d| d<| d <|d|d |d <|d<|d |d|d<|d <tj | |t d gdd} |dur|r@tj ||| | d}tj j||d | dd||d\}}}}}|rtj ||| | d}tj|| gd}n6|rtj|d gd}tj j||d | ddd\}}}}|d }|s|dur| rtj|gdd}tj|| d}|dur|}| s0tj||r(d ndd}|||ttjfS)z>GRU with cuDNN implementation which is only available for GPU.N)rrr)perm)rr)rrrXrEr is_cuda_buildrrDT)weightsbiasesrGtranspose_weights)seq_axis batch_axisr)inputinput_hinput_cparams is_trainingrnn_moderr)rrrrrr)r calculate_sequence_by_maskr transpose expand_dimsrarflatten sysconfigget_build_infoZcanonical_to_paramsconstantreverse_sequenceraw_ops CudnnRNNV3reverseCudnnRNNsqueezerZ RUNTIME_GPU)rbrrFrMrNrrrrrrrrrrr{rrrBrBrCrs         rc  s|||||||||| | d ddtrjtjtjfddtjfddifdd\} } } }nfdtt }||d }t |tjt |}t |tj|}|fi\} } } }tj |fi| | | |fS) aCall the GRU with optimized backend kernel selection. Under the hood, this function will create two TF function, one with the most generic kernel and can run on all device condition, and the second one with cuDNN specific kernel, which can only run on GPU. The first function will be called with normal_lstm_params, while the second function is not called, but only registered in the graph. The Grappler will do the proper graph rewrite and swap the optimized TF function based on the device placement. Args: inputs: Input tensor of GRU layer. init_h: Initial state tensor for the cell output. kernel: Weights for cell kernel. recurrent_kernel: Weights for cell recurrent kernel. bias: Weights for cell kernel bias and recurrent bias. Only recurrent bias is used in this case. mask: Boolean tensor for mask out the steps within sequence. An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. time_major: Boolean, whether the inputs are in the format of [time, batch, feature] or [batch, time, feature]. go_backwards: Boolean (default False). If True, process the input sequence backwards and return the reversed sequence. sequence_lengths: The lengths of all sequences coming from a variable length input, such as ragged tensors. If the input has a fixed timestep size, this should be None. zero_output_for_mask: Boolean, whether to output zero for masked timestep. return_sequences: Boolean. If True, return the recurrent outputs for all timesteps in the sequence. If False, only return the output for the last timestep (which consumes less memory). Returns: List of output tensors, same as standard_gru. rbrrFrMrNrrrrrrc szdur$t d S f dd}  f dd} tjt | | dS)z.gpu_gru_with_fallback..cudnn_gru_fnc st  d S)NrrrB rNrrrbrFrrMrrrrrBrCstandard_gru_fnszRgru_with_backend_selection..gpu_gru_with_fallback..standard_gru_fn)true_fnfalse_fn)rrcondr r) rbrrFrMrNrrrrrrrrrBrrCgpu_gru_with_fallbacks(   z9gru_with_backend_selection..gpu_gru_with_fallbackcstfiSrrrBrrBrCz,gru_with_backend_selection..csfiSrrBrBrrrBrCrscstfiSrrrBrrBrCrr gru_)rr) r rr __internal__execute_fn_for_deviceZCPU_DEVICE_NAMErstruuiduuid4Zgenerate_defun_backendrZfunction_register)rbrrFrMrNrrrrrrrrrrapi_namesupportive_attributedefun_standard_gru defun_gpu_grurBr rCrtsZ3?  r)%rrtensorflow.compat.v2r v2rkerasrrrrr keras.enginerkeras.engine.input_specrZkeras.layers.rnnr r keras.layers.rnn.base_rnnr 'keras.layers.rnn.dropout_rnn_cell_mixinr keras.utilsr tensorflow.python.platformrr8 tensorflow.python.util.tf_exportrr:BaseRandomLayerrrrrrrBrBrBrCs>               ^ m{