a 8Sicс@sHUddlZddlmZmZmZddlmZddlmZddlZddl m Z ddl m Z m Z mZmZmZdaiaeeeefe efed<e dd Zd d ZGd d d eZeddddZeeddddZddeeeeedddZd$ee eedddZd%eeeedddZeeddd Z d&eee eed!d"d#Z!dS)'N) ModuleList ModuleDictModule) Parameter)Tensor)contextmanager)UnionOptionalDictTupleSequence_cacheccs:td7azdVWtd8ats6iantd8ats4ia0dS)a]Context manager that enables the caching system within parametrizations registered with :func:`register_parametrization`. The value of the parametrized objects is computed and cached the first time they are required when this context manager is active. The cached values are discarded when leaving the context manager. This is useful when using a parametrized parameter more than once in the forward pass. An example of this is when parametrizing the recurrent kernel of an RNN or when sharing weights. The simplest way to activate the cache is by wrapping the forward pass of the neural network .. code-block:: python import torch.nn.utils.parametrize as P ... with P.cached(): output = model(inputs) in training and evaluation. One may also wrap the parts of the modules that use several times the parametrized tensors. For example, the loop of an RNN with a parametrized recurrent kernel: .. code-block:: python with P.cached(): for x in xs: out_rnn = self.rnn_cell(x, out_rnn) N)_cache_enabledr rrV/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/nn/utils/parametrize.pycacheds"rcCs(t|tr|||n |||dSN) isinstancerregister_parameterregister_buffer)modulenameXrrr_register_parameter_or_buffer9s rcsjeZdZUdZeed<eed<deee ee feddfdd Z edd d d Z ed d dZ ZS)ParametrizationLista\A sequential container that holds and manages the ``original`` or ``original0``, ``original1``, ... parameters or buffers of a parametrized :class:`torch.nn.Module`. It is the type of ``module.parametrizations[tensor_name]`` when ``module[tensor_name]`` has been parametrized with :func:`register_parametrization`. If the first registered parmetrization has a ``right_inverse`` that returns one tensor or does not have a ``right_inverse`` (in which case we assume that ``right_inverse`` is the identity), it will hold the tensor under the name ``original``. If it has a ``right_inverse`` that returns more than one tensor, these will be registered as ``original0``, ``original1``, ... .. warning:: This class is used internally by :func:`register_parametrization`. It is documented here for completeness. It shall not be instantiated by the user. Args: modules (sequence): sequence of modules representing the parametrizations original (Parameter or Tensor): parameter or buffer that is parametrized unsafe (bool): a boolean flag that denotes whether the parametrization may change the dtype and shape of the tensor. Default: `False` Warning: the parametrization is not checked for consistency upon registration. Enable this flag at your own risk. originalunsafeFN)modulesrrreturnc s:t|dkrtdt|||_|j}|j}tN|}t |D]0}t |drHz| |}WqHt yvYqH0qHWdn1s0Yt |tst |tjjstdt|jt |t|_|jrdnt||_|jrT|j|jkrtd|jd|jt||Wdn1s<0Yt|d|nnt|D]d\}} t | tstd |d t| jd t |trt| } | |jt|d|| q\|js6|} t | tstd t| jd | j|krtd |d| j| j|kr6td|d| jdS)Nrz1ParametrizationList requires one or more modules. right_inversezT'right_inverse' must return a Tensor or a Sequence of tensors (list, tuple...). Got rzVWhen `right_inverse` outputs one tensor, it may not change the dtype. original.dtype: z right_inverse(original).dtype: rz\'right_inverse' must return a Tensor or a Sequence of tensors (list, tuple...). Got element z of the sequence with type .,A parametrization must return a tensor. Got z}Registering a parametrization may not change the dtype of the tensor, unless `unsafe` flag is enabled. unparametrized dtype: z parametrized dtype: z}Registering a parametrization may not change the shape of the tensor, unless `unsafe` flag is enabled. unparametrized shape: z parametrized shape: )len ValueErrorsuper__init__rshapedtypetorchno_gradreversedhasattrr NotImplementedErrorrr collectionsabcr type__name__ is_tensorntensorsset_r enumeraterrequires_grad_ requires_grad) selfrrrZoriginal_shapeZoriginal_dtypenewriZ originaliZ __class__rrr&\s      &  *       zParametrizationList.__init__valuerc Cstvt|D]0}t|dr.||}qtdt|jdq|jrt |t sjt dt|j|j |j j krt d|j d|j j |j |nt |tjjst dt|jdt||jkrt d |jd t|dt|D]z\}}t|d |}t |t s4t d |d t|j|j |j krft d|d|j d|d|j ||qWdn1s0YdS)aCalls the methods ``right_inverse`` (see :func:`register_parametrization`) of the parametrizations in the inverse order they were registered in. Then, it stores the result in ``self.original`` if ``right_inverse`` outputs one tensor or in ``self.original0``, ``self.original1``, ... if it outputs several. Args: value (Tensor): Value to which initialize the module r zparametrization z" does not implement right_inverse.z,`right_inverse` should return a tensor. Got z1The tensor returned by `right_inverse` has dtype z while `original` has dtype z7'right_inverse' must return a sequence of tensors. Got r!z<'right_inverse' must return a sequence of tensors of length z. Got a sequence of lenght rz?`right_inverse` must return a sequence of tensors. Got element z of type zTensor z' returned by `right_inverse` has dtype z while `originalz ` has dtype N)r)r*r+r,r RuntimeErrorr0r1r2rrr$r(rr4r.r/r r#r3r5getattr)r8r?rr:tensorZ original_irrrr sd       z!ParametrizationList.right_inversercsfjrdj}n$fddtjD}d|}d}tt|rb||}|d7}q>|S)Nrc3s|]}td|VqdS)rN)rA).0r:r8rr z.ParametrizationList.forward..r)r2rranger3r,str)r8xZ originalsZcurr_idxrrErforwards   zParametrizationList.forward)F)r1 __module__ __qualname____doc__r__annotations__boolr rrrr&r rK __classcell__rrr<rr@s g=r)rrcCs2|j}dd}td|j|fd|i}||_dS)zSets up a module to be parametrized. This works by substituting the class of the module by a class that extends it to be able to inject a property Args: module (nn.Module): module into which to inject the property cSs tddS)NzSerialization of parametrized modules is only supported through state_dict(). See: https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training)r@rErrrgetstatesz#_inject_new_class..getstateZ Parametrized __getstate__N)r=r0r1)rclsrRZ param_clsrrr_inject_new_classs  rU)r tensor_namercsjtrJtjjtdfdd tdfdd }tddfdd }tjt||dS) aInjects a property into module[tensor_name]. It assumes that the class in the module has already been modified from its original one using _inject_new_class and that the tensor under :attr:`tensor_name` has already been moved out Args: module (nn.Module): module into which to inject the property tensor_name (str): name of the name of the property to create rCcs0tf}t|}|dur,|}|t|<|Sr)idr get)parametrizationkeyrB)rrVrrget_cached_parametrization=s   z4_inject_property..get_cached_parametrizationcsN|j}trDtjr"tdqJtjdur:tdqJ|Sn|SdS)NzTCaching is not implemented for scripting. Either disable caching or avoid scripting.z4Cannot trace a model while caching parametrizations.)parametrizationsrr)jit is_scriptingr@_C_get_tracing_state)r8rY)r[rVrrget_parametrizedGs     z*_inject_property..get_parametrizedNr>cs|j|dSr)r\r )r8r?)rVrr set_originalWsz&_inject_property..set_original)r,r)r]unusedrsetattrr=property)rrVrarbr)r[rrVr_inject_property.s  rfFr)rrVrYrrc Cs0||jt||r|s`t||}||}t|tsPtdt|jd|j |j krtd|d|j d|d|j |j |j krtd|d|j d|d |j t |d r`z| |}Wnt yYn0t|tstd t|j|j |j kr.td |d |d|j d|j |j |j kr`td|d |d|j d|j t|jtsrJ|j|||j|j|O_n||jvs||jvrt||}t|g||d}t||t|st|t|_t||t|jts J||j|<ntd|d|d|S)aiAdds a parametrization to a tensor in a module. Assume that ``tensor_name="weight"`` for simplicity. When accessing ``module.weight``, the module will return the parametrized version ``parametrization(module.weight)``. If the original tensor requires a gradient, the backward pass will differentiate through :attr:`parametrization`, and the optimizer will update the tensor accordingly. The first time that a module registers a parametrization, this function will add an attribute ``parametrizations`` to the module of type :class:`~ParametrizationList`. The list of parametrizations on the tensor ``weight`` will be accessible under ``module.parametrizations.weight``. The original tensor will be accessible under ``module.parametrizations.weight.original``. Parametrizations may be concatenated by registering several parametrizations on the same attribute. The training mode of a registered parametrization is updated on registration to match the training mode of the host module Parametrized parameters and buffers have an inbuilt caching system that can be activated using the context manager :func:`cached`. A :attr:`parametrization` may optionally implement a method with signature .. code-block:: python def right_inverse(self, X: Tensor) -> Union[Tensor, Sequence[Tensor]] This method is called on the unparametrized tensor when the first parametrization is registered to compute the initial value of the original tensor. If this method is not implemented, the original tensor will be just the unparametrized tensor. If all the parametrizations registered on a tensor implement `right_inverse` it is possible to initialize a parametrized tensor by assigning to it, as shown in the example below. It is possible for the first parametrization to depend on several inputs. This may be implemented returning a tuple of tensors from ``right_inverse`` (see the example implementation of a ``RankOne`` parametrization below). In this case, the unconstrained tensors are also located under ``module.parametrizations.weight`` with names ``original0``, ``original1``,... .. note:: If unsafe=False (default) both the forward and right_inverse methods will be called once to perform a number of consistency checks. If unsafe=True, then right_inverse will be called if the tensor is not parametrized, and nothing will be called otherwise. .. note:: In most situations, ``right_inverse`` will be a function such that ``forward(right_inverse(X)) == X`` (see `right inverse `_). Sometimes, when the parametrization is not surjective, it may be reasonable to relax this. .. warning:: If a parametrization depends on several inputs, :func:`~register_parametrization` will register a number of new parameters. If such parametrization is registered after the optimizer is created, these new parameters will need to be added manually to the optimizer. See :meth:`torch.Optimizer.add_param_group`. Args: module (nn.Module): module on which to register the parametrization tensor_name (str): name of the parameter or buffer on which to register the parametrization parametrization (nn.Module): the parametrization to register Keyword args: unsafe (bool): a boolean flag that denotes whether the parametrization may change the dtype and shape of the tensor. Default: `False` Warning: the parametrization is not checked for consistency upon registration. Enable this flag at your own risk. Raises: ValueError: if the module does not have a parameter or a buffer named :attr:`tensor_name` Examples: >>> import torch >>> import torch.nn as nn >>> import torch.nn.utils.parametrize as P >>> >>> class Symmetric(nn.Module): >>> def forward(self, X): >>> return X.triu() + X.triu(1).T # Return a symmetric matrix >>> >>> def right_inverse(self, A): >>> return A.triu() >>> >>> m = nn.Linear(5, 5) >>> P.register_parametrization(m, "weight", Symmetric()) >>> print(torch.allclose(m.weight, m.weight.T)) # m.weight is now symmetric True >>> A = torch.rand(5, 5) >>> A = A + A.T # A is now symmetric >>> m.weight = A # Initialize the weight to be the symmetric matrix A >>> print(torch.allclose(m.weight, A)) True >>> class RankOne(nn.Module): >>> def forward(self, x, y): >>> # Form a rank 1 matrix multiplying two vectors >>> return x.unsqueeze(-1) @ y.unsqueeze(-2) >>> >>> def right_inverse(self, Z): >>> # Project Z onto the rank 1 matrices >>> U, S, Vh = torch.linalg.svd(Z, full_matrices=False) >>> # Return rescaled singular vectors >>> s0_sqrt = S[0].sqrt().unsqueeze(-1) >>> return U[..., :, 0] * s0_sqrt, Vh[..., 0, :] * s0_sqrt >>> >>> linear_rank_one = P.register_parametrization(nn.Linear(4, 4), "weight", RankOne()) >>> print(torch.linalg.matrix_rank(linear_rank_one.weight).item()) 1 r"r!zrRegistering a parametrization may not change the dtype of the tensor, unless the `unsafe` flag is enabled. module.z.dtype: z parametrization(module.z ).dtype: zrRegistering a parametrization may not change the shape of the tensor, unless the `unsafe` flag is enabled. module.z.shape: z ).shape: r z9parametrization.right_inverse must return a tensor. Got: zXThe tensor returned by parametrization.right_inverse must have the same dtype as module.z., unless the `unsafe` flag is enabled. module.z returned dtype: zXThe tensor returned by parametrization.right_inverse must have the same shape as module.z returned shape: rgzModule 'zL' does not have a parameter, a buffer, or a parametrized element with name '')traintrainingis_parametrizedrArrr$r0r1r(r'r,r r-r\rappendr_buffers _parametersrdelattrrUrf) rrVrYrYrr;rr\rrrregister_parametrization\s{               rqcCsBt|dd}|dust|ts"dS|dur6t|dkS||vSdS)aJReturns ``True`` if module has an active parametrization. If the argument :attr:`tensor_name` is specified, returns ``True`` if ``module[tensor_name]`` is parametrized. Args: module (nn.Module): module to query name (str, optional): attribute in the module to query Default: ``None`` r\NFr)rArrr#)rrVr\rrrrk+s  rkT)rrVleave_parametrizedrc Csrt||std|d|t|jts.J|j|}|jr|j}|rtt ||}Wdn1sp0Ytft |tj ur| |n      rvcCs t|r|jjdSt|SdS)zReturns the module type before parametrizations were applied and if not, then it returns the module type. Args: module (nn.Module): module to get type of rN)rkr=rsr0)rrrrtype_before_parametrizationss rw) from_module to_modulerVrcCst|r t|jtsJ|dur(|jn|g}t|dssB      *Q/ P S