a 8SicW@sddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd d Zd d ZGd ddeZGdddeZdddeee eeedddZGdddeZd eeeee eedddZdS)!)EnumautoN)Tensor) parametrize)Module) functional)OptionalcCsV|d|d}}tj||j|jd}d|t|jj}tj|j|||dS)Ndtypedeviceg$@)atol) sizetorcheyer rfinfoepsallclosemH)QrnkIdr[/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/nn/utils/parametrizations.py_is_orthogonal srcCs<t|\}}tj||}||jdddd9}|S)z Assume that A is a tall matrix. Compute the Q factor s.t. A = QR (A may be complex) and diag(R) is real and non-negative r r dim1dim2)rgeqrflinalghouseholder_productdiagonalsgn unsqueeze)AXtaurrrr_make_orthogonalsr*c@seZdZeZeZeZdS) _OrthMapsN)__name__ __module__ __qualname__r matrix_expcayley householderrrrrr+sr+csfeZdZUeed<ddeddfddZejejdd d Zej ejejd d d Z Z S) _OrthogonalbaseTuse_trivializationN)orthogonal_mapreturncsFt|r$|tjkr$td|j|_||_|rB|dddS)NzAThe householder parametrization does not support complex tensors.r3) super__init__ is_complexr+r1 ValueErrorshaper6register_buffer)selfweightr6r5 __class__rrr9(s z_Orthogonal.__init__)r(r7c Cs|d|d}}||k}|r2|j}||}}|jtjksL|jtjkr|}||krtj|| |||j g|j ddddRgdd}||j }|jtjkrt|}nH|jtjkrtj ||j|jd}tjtj||ddtj||dd}||krl|dd|f}nN|jdd }d d ||jdd}tj||}||jddd d}t|d r|j|}|r|j}|S)Nr r dimr g)alphag?.)r$g@g?rr3)rmTr6r+r/r0trilrcat new_zerosexpandr<rrr rr"solveaddsumr#r$intr&hasattrr3) r>r(rr transposedr'rrr)rrrforwardCs4 <   &    z_Orthogonal.forward)rr7c Csj|j|jkr&td|jd|jd|}|d|d}}||k}|r\|j}||}}t|ds|jtjks~|jtjkrt dt |\}}|j ddd |j ddd|d kd9<|r|jS|S||krt|st|}n|}nHt j|dd|||f|j|jd }t j||gdd }t|}||_t |} | j dddd | SdS) Nz0Expected a matrix or batch of matrices of shape z. Got a tensor of shape .r r r3ztIt is not possible to assign to the matrix exponential or the Cayley parametrizations when use_trivialization=False.rgr rBg)r<r;rrErNr6r+r0r/NotImplementedErrorrr!r$sign_rr*clonerandnr rrGr3 zeros_likefill_) r>rZQ_initrr transposer'r)NZneg_Idrrr right_inverseks8       . z_Orthogonal.right_inverse) r,r-r.r__annotations__r+r9rrPautogradno_gradrZ __classcell__rrr@rr2%s (r2r?Tr4)modulenamer6r5r7cCst||d}t|ts&td|||jdkrBtd|jd|durn|d|dksf|rjdnd }tt|d}|durtd |t |||d }t j |||d d |S)aApplies an orthogonal or unitary parametrization to a matrix or a batch of matrices. Letting :math:`\mathbb{K}` be :math:`\mathbb{R}` or :math:`\mathbb{C}`, the parametrized matrix :math:`Q \in \mathbb{K}^{m \times n}` is **orthogonal** as .. math:: \begin{align*} Q^{\text{H}}Q &= \mathrm{I}_n \mathrlap{\qquad \text{if }m \geq n}\\ QQ^{\text{H}} &= \mathrm{I}_m \mathrlap{\qquad \text{if }m < n} \end{align*} where :math:`Q^{\text{H}}` is the conjugate transpose when :math:`Q` is complex and the transpose when :math:`Q` is real-valued, and :math:`\mathrm{I}_n` is the `n`-dimensional identity matrix. In plain words, :math:`Q` will have orthonormal columns whenever :math:`m \geq n` and orthonormal rows otherwise. If the tensor has more than two dimensions, we consider it as a batch of matrices of shape `(..., m, n)`. The matrix :math:`Q` may be parametrized via three different ``orthogonal_map`` in terms of the original tensor: - ``"matrix_exp"``/``"cayley"``: the :func:`~torch.matrix_exp` :math:`Q = \exp(A)` and the `Cayley map`_ :math:`Q = (\mathrm{I}_n + A/2)(\mathrm{I}_n - A/2)^{-1}` are applied to a skew-symmetric :math:`A` to give an orthogonal matrix. - ``"householder"``: computes a product of Householder reflectors (:func:`~torch.linalg.householder_product`). ``"matrix_exp"``/``"cayley"`` often make the parametrized weight converge faster than ``"householder"``, but they are slower to compute for very thin or very wide matrices. If ``use_trivialization=True`` (default), the parametrization implements the "Dynamic Trivialization Framework", where an extra matrix :math:`B \in \mathbb{K}^{n \times n}` is stored under ``module.parametrizations.weight[0].base``. This helps the convergence of the parametrized layer at the expense of some extra memory use. See `Trivializations for Gradient-Based Optimization on Manifolds`_ . Initial value of :math:`Q`: If the original tensor is not parametrized and ``use_trivialization=True`` (default), the initial value of :math:`Q` is that of the original tensor if it is orthogonal (or unitary in the complex case) and it is orthogonalized via the QR decomposition otherwise (see :func:`torch.linalg.qr`). Same happens when it is not parametrized and ``orthogonal_map="householder"`` even when ``use_trivialization=False``. Otherwise, the initial value is the result of the composition of all the registered parametrizations applied to the original tensor. .. note:: This function is implemented using the parametrization functionality in :func:`~torch.nn.utils.parametrize.register_parametrization`. .. _`Cayley map`: https://en.wikipedia.org/wiki/Cayley_transform#Matrix_map .. _`Trivializations for Gradient-Based Optimization on Manifolds`: https://arxiv.org/abs/1909.09501 Args: module (nn.Module): module on which to register the parametrization. name (str, optional): name of the tensor to make orthogonal. Default: ``"weight"``. orthogonal_map (str, optional): One of the following: ``"matrix_exp"``, ``"cayley"``, ``"householder"``. Default: ``"matrix_exp"`` if the matrix is square or complex, ``"householder"`` otherwise. use_trivialization (bool, optional): whether to use the dynamic trivialization framework. Default: ``True``. Returns: The original module with an orthogonal parametrization registered to the specified weight Example:: >>> orth_linear = orthogonal(nn.Linear(20, 40)) >>> orth_linear ParametrizedLinear( in_features=20, out_features=40, bias=True (parametrizations): ModuleDict( (weight): ParametrizationList( (0): _Orthogonal() ) ) ) >>> Q = orth_linear.weight >>> torch.dist(Q.T @ Q, torch.eye(20)) tensor(4.9332e-07) Nz5Module '{}' has no parameter ot buffer with name '{}'rz8Expected a matrix or batch of matrices. Got a tensor of z dimensions.r r r/r1zLorthogonal_map has to be one of "matrix_exp", "cayley", "householder". Got: r4T)unsafe) getattr isinstancerr;formatndimrr:r+r2rregister_parametrization)r_r`r6r5r?Z orth_enumZorthrrr orthogonals,W     $ rgcseZdZdejeeeddfdd Zejejdd d Zej ejedd d d Z ejejdddZ ejejdddZ ZS) _SpectralNormr-q=N)r?n_power_iterationsrCrr7c st|j}||ks"|| krBtd|d|dd|d|dkrXtd||dkrd|n|||_||_|dkr||_| |}| \}}| | dd} | | dd} | dtj| d|jd | d tj| d|jd ||d dS) Nz5Dimension out of range (expected to be in range of [-z, riz ] but got )rzIExpected n_power_iterations to be positive, but got n_power_iterations={}_urCr_v)r8r9re IndexErrorr;rdrCrrk_reshape_weight_to_matrixr new_emptynormal_r=F normalize _power_method) r>r?rkrCrre weight_mathwuvr@rrr9s2    z_SpectralNorm.__init__)r?r7csL|jdksJjdkrB|jjgfddt|DR}|dS)Nrirc3s|]}|jkr|VqdSNrB).0dr>rr Dz:_SpectralNorm._reshape_weight_to_matrix..)rerCpermuterangeflatten)r>r?rrrrr>s *z'_SpectralNorm._reshape_weight_to_matrix)rxrkr7cCsh|jdksJt|D]L}tjt||jd|j|jd|_tjt| |jd|j|jd|_qdS)Nrir)rCrout) rerrurvrmvrorrmt)r>rxrk_rrrrwHs"   z_SpectralNorm._power_methodcCsz|jdkrtj|d|jdS||}|jr:|||j|jj t j d}|j j t j d}t |t ||}||SdS)Nrirrn) memory_format)rerurvrrrtrainingrwrkrmrTrcontiguous_formatrodotr)r>r?rxr{r|sigmarrrrPus  z_SpectralNorm.forward)valuer7cCs|Sr}r)r>rrrrrZsz_SpectralNorm.right_inverse)rirrj)r,r-r.rrrMfloatr9rrr\r]rwrPrZr^rrr@rrhs! ,rhrirj)r_r`rkrrCr7c Cspt||d}t|ts&td|||durTt|tjjtjjtjj frPd}nd}t ||t |||||S)a Applies spectral normalization to a parameter in the given module. .. math:: \mathbf{W}_{SN} = \dfrac{\mathbf{W}}{\sigma(\mathbf{W})}, \sigma(\mathbf{W}) = \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2} When applied on a vector, it simplifies to .. math:: \mathbf{x}_{SN} = \dfrac{\mathbf{x}}{\|\mathbf{x}\|_2} Spectral normalization stabilizes the training of discriminators (critics) in Generative Adversarial Networks (GANs) by reducing the Lipschitz constant of the model. :math:`\sigma` is approximated performing one iteration of the `power method`_ every time the weight is accessed. If the dimension of the weight tensor is greater than 2, it is reshaped to 2D in power iteration method to get spectral norm. See `Spectral Normalization for Generative Adversarial Networks`_ . .. _`power method`: https://en.wikipedia.org/wiki/Power_iteration .. _`Spectral Normalization for Generative Adversarial Networks`: https://arxiv.org/abs/1802.05957 .. note:: This function is implemented using the parametrization functionality in :func:`~torch.nn.utils.parametrize.register_parametrization`. It is a reimplementation of :func:`torch.nn.utils.spectral_norm`. .. note:: When this constraint is registered, the singular vectors associated to the largest singular value are estimated rather than sampled at random. These are then updated performing :attr:`n_power_iterations` of the `power method`_ whenever the tensor is accessed with the module on `training` mode. .. note:: If the `_SpectralNorm` module, i.e., `module.parametrization.weight[idx]`, is in training mode on removal, it will perform another power iteration. If you'd like to avoid this iteration, set the module to eval mode before its removal. Args: module (nn.Module): containing module name (str, optional): name of weight parameter. Default: ``"weight"``. n_power_iterations (int, optional): number of power iterations to calculate spectral norm. Default: ``1``. eps (float, optional): epsilon for numerical stability in calculating norms. Default: ``1e-12``. dim (int, optional): dimension corresponding to number of outputs. Default: ``0``, except for modules that are instances of ConvTranspose{1,2,3}d, when it is ``1`` Returns: The original module with a new parametrization registered to the specified weight Example:: >>> snm = spectral_norm(nn.Linear(20, 40)) >>> snm ParametrizedLinear( in_features=20, out_features=40, bias=True (parametrizations): ModuleDict( (weight): ParametrizationList( (0): _SpectralNorm() ) ) ) >>> torch.linalg.matrix_norm(snm.weight, 2) tensor(1.0000, grad_fn=) Nz5Module '{}' has no parameter or buffer with name '{}'rir) rbrcrr;rdrnnConvTranspose1dConvTranspose2dConvTranspose3drrfrh)r_r`rkrrCr?rrr spectral_normsL    r)N)r?N)r?rirjN)enumrrrrutilsrmodulesrrrutypingr rr*r+r2strboolrgrhrMrrrrrrs>        qq