a 8Sic@s(ddlZddlZddlZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZd d lmZejejd Zd e e ee d ddZGddde ZGdddeZGdddeZGdddeZGddde ZGdddeZGdddeZGdddeZdS)!N)ListTupleOptionaloverload)Tensor)Module) Parameter)PackedSequence)init)_VF)RNN_TANHRNN_RELU)tensor permutationdimreturncCs |||SN) index_select)rrrrP/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/nn/modules/rnn.pyapply_permutationsrc seZdZUgdZdgZeed<eed<eed<eed<eed<eed<e ed <eed <eed <d5eeeeeee eedd fdd Z fddZ ddddZ fddZ ddddZeeedddd Zeeeeeeefdd!d"Zd6eeeeefedd$d%d&Zeeeed'd(d)Zeeed*d+d,Zedd-d.Zfd/d0Zeeeedd1d2Zfd3d4ZZS)7RNNBase) mode input_size hidden_size num_layersbias batch_firstdropout bidirectional proj_size all_weightsrrrrrr r!r"r#rTFrN) rrrrrr r!r"r#rc s| | d} tt|_|_|_|_|_|_t |_ |_ | _ |rZdnd} t |tjrd|kr~dkrnn t |trtd|dkr|dkrtd||| dkrtd| |krtd|d krd |}n>|d krd |}n,|d kr |}n|dkr|}n td|g_g_t|D]t| D]}| dkr\| n|}dkrn|n|| }ttj||ffi| }ttj||ffi| }ttj|fi| }ttj|fi| }d}j dkr|r||||f}n||f}n:ttj| |ffi| }|r4|||||f}n |||f}|dkrLdndddg}|rj|ddg7}j dkr|dg7}fdd|D}t||D]\}}t||qj|j|qHq:fddjD_ !dS)Ndevicedtyper rrzbdropout should be a number in range [0, 1] representing the probability of an element being zeroedzdropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout={} and num_layers={}zEproj_size should be a positive integer or zero to disable projectionsz,proj_size has to be smaller than hidden_sizeLSTMGRUr rrzUnrecognized RNN mode: r_reverseweight_ih_l{}{}weight_hh_l{}{} bias_ih_l{}{} bias_hh_l{}{}weight_hr_l{}{}csg|]}|qSrformat.0xlayersuffixrr rz$RNNBase.__init__..csg|]}fdd|qS)cst|rt|SdSrhasattrgetattrwnselfrryr<z-RNNBase.__init__...rr6rArBrrr;yr<)"superr__init__rrrrrr floatr!r"r# isinstancenumbersNumberbool ValueErrorwarningswarnr4_flat_weights_names _all_weightsranger torchemptyzipsetattrextendappend _flat_weightsflatten_parametersreset_parameters)rCrrrrrr r!r"r#r'r(factory_kwargsnum_directionsZ gate_size directionreal_hidden_sizeZlayer_input_sizew_ihw_hhb_ihb_hhZ layer_paramsZw_hrZ param_namesnameparam __class__r9rCr:rrG's   $            zRNNBase.__init__cs@t|dr*||jvr*|j|}||j|<tt|||dS)NrP)r>rPindexrYrFr __setattr__)rCattrvalueidxrfrrrj~s  zRNNBase.__setattr__rcCst|jt|jkrdS|jD]}t|tsdSq|jd}|j}|jD]:}t|jtr~|jj|kr~|jjr~tj j |jsJdSqJt dd|jD}t|t|jkrdStj |ddlm m m}tptr@|jrdnd}|jdkr |d7}t|j||j||j|j|j|j|jt|j Wdn1sV0YWdn1sv0YdS)zResets parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it's a no-op. Nrcss|]}|VqdSr)data_ptr)r6prrr r<z-RNNBase.flatten_parameters..r*r r)lenrYrPrIrr(datais_cudarSbackendscudnnZ is_acceptablesetcuda device_ofZtorch.backends.cudnn.rnnrnnno_gradZ_use_cudnn_rnn_flatten_weightrr#Z_cudnn_rnn_flatten_weightrZget_cudnn_moderrrr rLr")rCwZfirst_fwr(fwZunique_data_ptrsrzZ num_weightsrrrrZs<         zRNNBase.flatten_parameterscs2tt|}fddjD_|S)Ncsg|]}fdd|qS)cst|rt|SdSrr=r@rBrrrDr<z+RNNBase._apply...rrErBrrr;r<z"RNNBase._apply..)rFr_applyrPrYrZ)rCfnretrfrBrr~szRNNBase._applycCs@|jdkrdt|jnd}|D]}t|| |q&dSNrg?rmathsqrt parametersr uniform_rCZstdvweightrrrr[s zRNNBase.reset_parametersinput batch_sizesrcCs\|dur dnd}||kr0td|||j|dkrXtd|j|ddS)Nr r z%input must have {} dimensions, got {}z?input.size(-1) must be equal to input_size. Expected {}, got {})r RuntimeErrorr4rsize)rCrrZexpected_input_dimrrr check_inputs  zRNNBase.check_inputcCsr|durt|d}n|jr&|dn|d}|jr:dnd}|jdkr\|j|||jf}n|j|||jf}|SNrrr )intr rr"r#rrrCrrZ mini_batchr]expected_hidden_sizerrrget_expected_hidden_sizes z RNNBase.get_expected_hidden_sizeExpected hidden size {}, got {})hxrmsgrcCs(||kr$t||t|dSr)rrr4list)rCrrrrrrcheck_hidden_sizes zRNNBase.check_hidden_sizerhiddenrcCs(||||||}|||dSr)rrr)rCrrrrrrrcheck_forward_argss  zRNNBase.check_forward_args)rrcCs|dur |St||SrrrCrrrrrpermute_hiddenszRNNBase.permute_hiddencCsd}|jdkr|d7}|jdkr(|d7}|jdur:|d7}|jdurL|d 7}|jdkr^|d 7}|jdurp|d 7}|jfi|jS) N{input_size}, {hidden_size}rz, proj_size={proj_size}rz, num_layers={num_layers}T , bias={bias}Fz, batch_first={batch_first}z, dropout={dropout}z, bidirectional={bidirectional})r#rrr r!r"r4__dict__rCsrrr extra_reprs      zRNNBase.extra_reprcstt|d|vr"|d_d|vr0d_tjddtrHdSj}jrXdnd}g_ g_t |D] t |D]}|dkrdndgd}fd d |D}j rjdkrj|g7_j |n,j|dd g7_j |dd q~jdkrdj|ddg|d dg7_j |dd|d dgq~j|ddg7_j |ddq~qpfd d j D_ dS)Nr$r#rr rr,r-)r.r/r0r1r2csg|]}|qSrr3r5r8rrr;r<z(RNNBase.__setstate__..r*rcsg|]}fdd|qS)cst|rt|SdSrr=r@rBrrrD$r<z1RNNBase.__setstate__...rrErBrrr;$r<)rFr __setstate__rQr#rIstrrr"rPrRrrWrY)rCdrr]r^weightsrfrhrrs8   &$zRNNBase.__setstate__csfddjDS)Ncsg|]}fdd|DqS)csg|]}t|qSr)r?)r6rrBrrr;(r<z2RNNBase.all_weights...r)r6rrBrrr;(r<z'RNNBase.all_weights..)rQrBrrBrr$&szRNNBase.all_weightscs2tt|}|jdd|_|jdd|_|Sr)rFr_replicate_for_data_parallelrYrP)rCreplicarfrrr*sz$RNNBase._replicate_for_data_parallel)rTFr%FrNN)r)__name__ __module__ __qualname__ __constants__Z__jit_unused_properties__r__annotations__rrLrHrGrjrZr~r[rrrrrrrrrrpropertyrr r$r __classcell__rrrfrrsH W 0    &rcseZdZdZfddZeejjd e e e e e e fdddZ eejjd e e e e e e fdddZ d d dZ ZS) RNNaApplies a multi-layer Elman RNN with :math:`\tanh` or :math:`\text{ReLU}` non-linearity to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: h_t = \tanh(x_t W_{ih}^T + b_{ih} + h_{t-1}W_{hh}^T + b_{hh}) where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input at time `t`, and :math:`h_{(t-1)}` is the hidden state of the previous layer at time `t-1` or the initial hidden state at time `0`. If :attr:`nonlinearity` is ``'relu'``, then :math:`\text{ReLU}` is used instead of :math:`\tanh`. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two RNNs together to form a `stacked RNN`, with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1 nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'`` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each RNN layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional RNN. Default: ``False`` Inputs: input, h_0 * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for the input sequence batch. Defaults to zeros if not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned} Outputs: output, h_n * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the RNN, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for each element in the batch. Attributes: weight_ih_l[k]: the learnable input-hidden weights of the k-th layer, of shape `(hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(hidden_size, num_directions * hidden_size)` weight_hh_l[k]: the learnable hidden-hidden weights of the k-th layer, of shape `(hidden_size, hidden_size)` bias_ih_l[k]: the learnable input-hidden bias of the k-th layer, of shape `(hidden_size)` bias_hh_l[k]: the learnable hidden-hidden bias of the k-th layer, of shape `(hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional RNNs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_rnn_determinism.rst .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.RNN(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> output, hn = rnn(input, h0) cspd|vrtd|dd|_|jdkr.d}n |jdkr>d}ntd|jtt|j|g|Ri|dS) Nr#=proj_size argument is only supported for LSTM, not RNN or GRU nonlinearitytanhrrelurzUnknown nonlinearity '{}')rMpoprr4rFrrG)rCargskwargsrrfrrrGs  z RNN.__init__NrrrcCsdSrrrCrrrrrforwardsz RNN.forwardcCsdSrrrrrrrsc Csx|}t|tr(|\}}}}t|d}nd}|dk}|jrBdnd} |s|| }|dur|dkr|td|d|d}n(|dur|dkrtd|d|jr|dn|d}d}d}|dur |jrdnd} t j |j | ||j |j |jd}n |||}|dus&J|||||jd ksP|jd ksPJ|dur|jd krt|||j|j|j |j|j|j|j } n(t|||j|j|j |j|j|j|j } nZ|jd krt||||j|j|j |j|j|j } n&t||||j|j|j |j|j|j } | d} | d} t|trNt| |||}||| |fS|sh| | } | d} | || |fS) Nrr rr 7For unbatched 2-D input, hx should also be 2-D but got -D tensor5For batched 3-D input, hx should also be 3-D but got r(r'rr)rIr rrr unsqueezerrr"rSzerosrrr(r'rrrrrnn_tanhrYrr!trainingrnn_relusqueezerCrr orig_inputrsorted_indicesunsorted_indicesmax_batch_size is_batched batch_dimr]resultoutputr output_packedrrrrsx                   )N)N)Nrrr__doc__rGrrS _jit_internal_overload_methodrrrrr rrrrfrr3sf $$rc seZdZdZfddZeeeeeeefdddZ eeeefeeddd Z eeefeeeeefd d d Z e e jjdeeeeefeeeeeffdddZe e jjdeeeeefeeeeeffdddZdddZZS)r)aW$Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: \begin{array}{ll} \\ i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{t-1} + b_{hi}) \\ f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{t-1} + b_{hf}) \\ g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{t-1} + b_{hg}) \\ o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{t-1} + b_{ho}) \\ c_t = f_t \odot c_{t-1} + i_t \odot g_t \\ h_t = o_t \odot \tanh(c_t) \\ \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{t-1}` is the hidden state of the layer at time `t-1` or the initial hidden state at time `0`, and :math:`i_t`, :math:`f_t`, :math:`g_t`, :math:`o_t` are the input, forget, cell, and output gates, respectively. :math:`\sigma` is the sigmoid function, and :math:`\odot` is the Hadamard product. In a multilayer LSTM, the input :math:`x^{(l)}_t` of the :math:`l` -th layer (:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random variable which is :math:`0` with probability :attr:`dropout`. If ``proj_size > 0`` is specified, LSTM with projections will be used. This changes the LSTM cell in the following way. First, the dimension of :math:`h_t` will be changed from ``hidden_size`` to ``proj_size`` (dimensions of :math:`W_{hi}` will be changed accordingly). Second, the output hidden state of each layer will be multiplied by a learnable projection matrix: :math:`h_t = W_{hr}h_t`. Note that as a consequence of this, the output of LSTM network will be of different shape as well. See Inputs/Outputs sections below for exact dimensions of all variables. You can find more details in https://arxiv.org/abs/1402.1128. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two LSTMs together to form a `stacked LSTM`, with the second LSTM taking in outputs of the first LSTM and computing the final results. Default: 1 bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each LSTM layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional LSTM. Default: ``False`` proj_size: If ``> 0``, will use LSTM with projections of corresponding size. Default: 0 Inputs: input, (h_0, c_0) * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. * **c_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{cell})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{cell})` containing the initial cell state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{cell} ={} & \text{hidden\_size} \\ H_{out} ={} & \text{proj\_size if } \text{proj\_size}>0 \text{ otherwise hidden\_size} \\ \end{aligned} Outputs: output, (h_n, c_n) * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the LSTM, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. When ``bidirectional=True``, `output` will contain a concatenation of the forward and reverse hidden states at each time step in the sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for each element in the sequence. When ``bidirectional=True``, `h_n` will contain a concatenation of the final forward and reverse hidden states, respectively. * **c_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{cell})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{cell})` containing the final cell state for each element in the sequence. When ``bidirectional=True``, `c_n` will contain a concatenation of the final forward and reverse cell states, respectively. Attributes: weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer `(W_ii|W_if|W_ig|W_io)`, of shape `(4*hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(4*hidden_size, num_directions * hidden_size)`. If ``proj_size > 0`` was specified, the shape will be `(4*hidden_size, num_directions * proj_size)` for `k > 0` weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer `(W_hi|W_hf|W_hg|W_ho)`, of shape `(4*hidden_size, hidden_size)`. If ``proj_size > 0`` was specified, the shape will be `(4*hidden_size, proj_size)`. bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer `(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer `(b_hi|b_hf|b_hg|b_ho)`, of shape `(4*hidden_size)` weight_hr_l[k] : the learnable projection weights of the :math:`\text{k}^{th}` layer of shape `(proj_size, hidden_size)`. Only present when ``proj_size > 0`` was specified. weight_ih_l[k]_reverse: Analogous to `weight_ih_l[k]` for the reverse direction. Only present when ``bidirectional=True``. weight_hh_l[k]_reverse: Analogous to `weight_hh_l[k]` for the reverse direction. Only present when ``bidirectional=True``. bias_ih_l[k]_reverse: Analogous to `bias_ih_l[k]` for the reverse direction. Only present when ``bidirectional=True``. bias_hh_l[k]_reverse: Analogous to `bias_hh_l[k]` for the reverse direction. Only present when ``bidirectional=True``. weight_hr_l[k]_reverse: Analogous to `weight_hr_l[k]` for the reverse direction. Only present when ``bidirectional=True`` and ``proj_size > 0`` was specified. .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional LSTMs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: For bidirectional LSTMs, `h_n` is not equivalent to the last element of `output`; the former contains the final forward and reverse hidden states, while the latter contains the final forward hidden state and the initial reverse hidden state. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_rnn_determinism.rst .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.LSTM(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> c0 = torch.randn(2, 3, 20) >>> output, (hn, cn) = rnn(input, (h0, c0)) cs"tt|jdg|Ri|dS)Nr))rFr)rGrCrrrfrrrGsz LSTM.__init__rcCsT|durt|d}n|jr&|dn|d}|jr:dnd}|j|||jf}|Sr)rr rr"rrrrrrget_expected_cell_sizeszLSTM.get_expected_cell_sizercCsD|||||d|||d||d|||ddS)Nrz"Expected hidden[0] size {}, got {}rz"Expected hidden[1] size {}, got {})rrrr)rCrrrrrrrs zLSTM.check_forward_args)rrrcCs(|dur |St|d|t|d|fS)NrrrrrrrrszLSTM.permute_hiddenNrcCsdSrrrrrrrsz LSTM.forwardcCsdSrrrrrrrsc Cs|}d}t|tr0|\}}}}|d}t|}nNd}|dk}|jrJdnd} |s\|| }|jrl|dn|d}d}d}|dur|jrdnd} |jdkr|jn|j } t j |j | || |j |jd} t j |j | ||j |j |jd} | | f}n|dur|rR|ddks&|ddkrd|dd|dd}t|nj|ddksv|ddkrd |dd|dd}t||dd|ddf}|||}|||||dur t|||j|j|j |j|j|j|j }n&t||||j|j|j |j|j|j }|d}|dd}t|trnt||||}||||fS|s|| }|dd|ddf}||||fSdS) Nrr rr rz=For batched 3-D input, hx and cx should also be 3-D but got (z-D, z -D) tensorsz?For unbatched 2-D input, hx and cx should also be 2-D but got ()rIr rrr rrr"r#rrSrrr(r'rrrrlstmrYrr!rr)rCrrrrrrrrrr]r_Zh_zerosZc_zerosrrrrrrrrrsz         $   $       )N)N)N)rrrrrGrrrrrrrrrSrrrr rrrrfrr)s.     r)cseZdZdZfddZeejjd e e e e e e fdddZ eejjd e e e e e e fdddZ d d dZ ZS) r+aApplies a multi-layer gated recurrent unit (GRU) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: \begin{array}{ll} r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ h_t = (1 - z_t) * n_t + z_t * h_{(t-1)} \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{(t-1)}` is the hidden state of the layer at time `t-1` or the initial hidden state at time `0`, and :math:`r_t`, :math:`z_t`, :math:`n_t` are the reset, update, and new gates, respectively. :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. In a multilayer GRU, the input :math:`x^{(l)}_t` of the :math:`l` -th layer (:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random variable which is :math:`0` with probability :attr:`dropout`. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two GRUs together to form a `stacked GRU`, with the second GRU taking in outputs of the first GRU and computing the final results. Default: 1 bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each GRU layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional GRU. Default: ``False`` Inputs: input, h_0 * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for the input sequence. Defaults to zeros if not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned} Outputs: output, h_n * **output**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the GRU, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for the input sequence. Attributes: weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer (W_ir|W_iz|W_in), of shape `(3*hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(3*hidden_size, num_directions * hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer (W_hr|W_hz|W_hn), of shape `(3*hidden_size, hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer (b_ir|b_iz|b_in), of shape `(3*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer (b_hr|b_hz|b_hn), of shape `(3*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional GRUs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.GRU(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> output, hn = rnn(input, h0) cs2d|vrtdtt|jdg|Ri|dS)Nr#rr+)rMrFr+rGrrfrrrGsz GRU.__init__NrcCsdSrrrrrrrsz GRU.forwardcCsdSrrrrrrrsc Cs|}t|tr,|\}}}}|d}t|}nd}|dk}|jrFdnd} |s|| }|dur|dkrtd|d|d}n(|dur|dkrtd|d|jr|dn|d}d}d}|dur|jrdnd} t j |j | ||j |j |jd}n |||}|||||dur^t|||j|j|j |j|j|j|j } n&t||||j|j|j |j|j|j } | d} | d} t|trt| |||}||| |fS|s| | } | d} | || |fSdS) Nrr rr rrrr)rIr rrr rrrr"rSrrrr(r'rrrgrurYrr!rrrrrrrs^              )N)N)Nrrrrfrr+sm $$r+cszeZdZUgdZeed<eed<eed<eed<eed<deeeeddfd d Ze d d d Z dd ddZ Z S) RNNCellBase)rrrrrr weight_ih weight_hhN)rrr num_chunksrcs||d}tt|||_||_||_ttj|||ffi||_ ttj|||ffi||_ |rttj||fi||_ ttj||fi||_ n| dd| dd|dS)Nr&bias_ihbias_hh)rFrrGrrrr rSrTrrrrregister_parameterr[)rCrrrrr'r(r\rfrrrGs     zRNNCellBase.__init__rncCsNd}d|jvr |jdur |d7}d|jvr<|jdkr<|d7}|jfi|jS)NrrTrrrz, nonlinearity={nonlinearity})rrrr4rrrrrs zRNNCellBase.extra_reprcCs@|jdkrdt|jnd}|D]}t|| |q&dSrrrrrrr[s zRNNCellBase.reset_parameters)NN) rrrrrrrLrrGrrr[rrrrfrrs  rcsZeZdZUdZgdZeed<d eeeeddfdd Z de e e e d d d Z Z S)RNNCellarAn Elman RNN cell with tanh or ReLU non-linearity. .. math:: h' = \tanh(W_{ih} x + b_{ih} + W_{hh} h + b_{hh}) If :attr:`nonlinearity` is `'relu'`, then ReLU is used in place of tanh. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'`` Inputs: input, hidden - **input**: tensor containing input features - **hidden**: tensor containing the initial hidden state Defaults to zero if not provided. Outputs: h' - **h'** of shape `(batch, hidden_size)`: tensor containing the next hidden state for each element in the batch Shape: - input: :math:`(N, H_{in})` or :math:`(H_{in})` tensor containing input features where :math:`H_{in}` = `input_size`. - hidden: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the initial hidden state where :math:`H_{out}` = `hidden_size`. Defaults to zero if not provided. - output: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the next hidden state. Attributes: weight_ih: the learnable input-hidden weights, of shape `(hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` Examples:: >>> rnn = nn.RNNCell(10, 20) >>> input = torch.randn(6, 3, 10) >>> hx = torch.randn(3, 20) >>> output = [] >>> for i in range(6): hx = rnn(input[i], hx) output.append(hx) )rrrrrTrN)rrrrrcs4||d}tt|j|||fddi|||_dS)Nr&rr)rFrrGr)rCrrrrr'r(r\rfrrrG.s  zRNNCell.__init__rcCs|dvs Jd|d|dk}|s:|d}|durbtj|d|j|j|jd}n|sp|dn|}|jdkrt |||j |j |j |j}n<|jdkrt |||j |j |j |j}n|}td |j|s|d}|S) Nrr z6RNNCell: Expected input to be 1-D or 2-D but received rr rrrrzUnknown nonlinearity: {})rrrSrrrr(r'rr rnn_tanh_cellrrrr rnn_relu_cellrr4rrCrrrrrrrr4s8       zRNNCell.forward)TrNN)N)rrrrrrrrrLrGrrrrrrrfrrs 5 rcsVeZdZdZd eeeddfdd Zd eee eefe eefddd Z Z S) LSTMCella A long short-term memory (LSTM) cell. .. math:: \begin{array}{ll} i = \sigma(W_{ii} x + b_{ii} + W_{hi} h + b_{hi}) \\ f = \sigma(W_{if} x + b_{if} + W_{hf} h + b_{hf}) \\ g = \tanh(W_{ig} x + b_{ig} + W_{hg} h + b_{hg}) \\ o = \sigma(W_{io} x + b_{io} + W_{ho} h + b_{ho}) \\ c' = f * c + i * g \\ h' = o * \tanh(c') \\ \end{array} where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` Inputs: input, (h_0, c_0) - **input** of shape `(batch, input_size)` or `(input_size)`: tensor containing input features - **h_0** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the initial hidden state - **c_0** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the initial cell state If `(h_0, c_0)` is not provided, both **h_0** and **c_0** default to zero. Outputs: (h_1, c_1) - **h_1** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the next hidden state - **c_1** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the next cell state Attributes: weight_ih: the learnable input-hidden weights, of shape `(4*hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(4*hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(4*hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(4*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision` for backward. Examples:: >>> rnn = nn.LSTMCell(10, 20) # (input_size, hidden_size) >>> input = torch.randn(2, 3, 10) # (time_steps, batch, input_size) >>> hx = torch.randn(3, 20) # (batch, hidden_size) >>> cx = torch.randn(3, 20) >>> output = [] >>> for i in range(input.size()[0]): hx, cx = rnn(input[i], (hx, cx)) output.append(hx) >>> output = torch.stack(output, dim=0) TNrrrrcs.||d}tt|j|||fddi|dS)Nr&rr*)rFrrGrCrrrr'r(r\rfrrrGs zLSTMCell.__init__rcCs|dvs Jd|d|dk}|s:|d}|durjtj|d|j|j|jd}||f}n$|s|dd|ddfn|}t |||j |j |j |j }|s|dd|ddf}|S)Nrz7LSTMCell: Expected input to be 1-D or 2-D but received rr rrr)rrrSrrrr(r'r lstm_cellrrrrr)rCrrrrrrrrrs$   $zLSTMCell.forward)TNN)N) rrrrrrLrGrrrrrrrrfrrWs;rcsFeZdZdZd eeeddfdd Zd eeeeddd Z Z S) GRUCellai A gated recurrent unit (GRU) cell .. math:: \begin{array}{ll} r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\ z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\ n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\ h' = (1 - z) * n + z * h \end{array} where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` Inputs: input, hidden - **input** : tensor containing input features - **hidden** : tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. Outputs: h' - **h'** : tensor containing the next hidden state for each element in the batch Shape: - input: :math:`(N, H_{in})` or :math:`(H_{in})` tensor containing input features where :math:`H_{in}` = `input_size`. - hidden: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the initial hidden state where :math:`H_{out}` = `hidden_size`. Defaults to zero if not provided. - output: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the next hidden state. Attributes: weight_ih: the learnable input-hidden weights, of shape `(3*hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(3*hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(3*hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(3*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision` for backward. Examples:: >>> rnn = nn.GRUCell(10, 20) >>> input = torch.randn(6, 3, 10) >>> hx = torch.randn(3, 20) >>> output = [] >>> for i in range(6): hx = rnn(input[i], hx) output.append(hx) TNrcs.||d}tt|j|||fddi|dS)Nr&rr )rFrrGrrfrrrGs zGRUCell.__init__rcCs|dvs Jd|d|dk}|s:|d}|durbtj|d|j|j|jd}n|sp|dn|}t |||j |j |j |j }|s|d}|S)Nrz6GRUCell: Expected input to be 1-D or 2-D but received rr rr)rrrSrrrr(r'rgru_cellrrrrrrrrrrs"    zGRUCell.forward)TNN)N) rrrrrrLrGrrrrrrrfrrs=r)r) rrNrJtypingrrrrrSrmoduler parameterr Z utils.rnnr r-r rrrZ _rnn_implsrrrrr)r+rrrrrrrrs8      O8+bY