a 8Sic6@sjddlmZmZddlZddlmZddlmZddl Z ddl Z Gddde Z e Z Gddde ZdS) ) defaultdictabcN)deepcopy)chainc@seZdZdZddZdS)_RequiredParameterzCSingleton class representing a required parameter for an Optimizer.cCsdS)NzselfrrQ/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/optim/optimizer.py__repr__ sz_RequiredParameter.__repr__N)__name__ __module__ __qualname____doc__r rrrr r src@speZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ de dddZ ddZddZdS) OptimizeraXBase class for all optimizers. .. warning:: Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don't satisfy those properties are sets and iterators over values of dictionaries. Args: params (iterable): an iterable of :class:`torch.Tensor` s or :class:`dict` s. Specifies what Tensors should be optimized. defaults: (dict): a dict containing default values of optimization options (used when a parameter group doesn't specify them). cCstjd||_|t|tjr8tdt|t t |_ g|_ t |}t|dkrdtdt|dt s|d|ig}|D]}||qd|_dS)Nzpython.optimizerzZparams argument given to the optimizer should be an iterable of Tensors or dicts, but got rz%optimizer got an empty parameter listparamsT)torch_C_log_api_usage_oncedefaults_hook_for_profile isinstanceTensor TypeErrortypenamerdictstate param_groupslistlen ValueErroradd_param_group$_warned_capturable_if_run_uncaptured)r rrr param_grouprrr __init__!s"      zOptimizer.__init__cCs|j|j|jdS)Nrrrr%rrrr __getstate__=szOptimizer.__getstate__cCs|j||dSN)__dict__updater)r rrrr __setstate__Ds zOptimizer.__setstate__cCsp|jjd}t|jD]L\}}|d7}|d|7}t|D] }|dkr@|d|||7}q@q|d7}|S)Nz ( zParameter Group {0} rz {0}: {1} )) __class__r enumeraterformatsortedkeys)r format_stringigroupkeyrrr r Hs zOptimizer.__repr__cCshtjrdtjrdtj}|r<|jds.profile_hook_step..wrapper) functoolswraps)rFrGrrEr profile_hook_stepjsz6Optimizer._hook_for_profile..profile_hook_stephookedT)r/r-r _zero_grad_profile_namer;steprK)r rJrKrrr rgs  zOptimizer._hook_for_profilecsLidfddfdd|jD}fdd|jD}||dS) aKReturns the state of the optimizer as a :class:`dict`. It contains two entries: * state - a dict holding current optimization state. Its content differs between optimizer classes. * param_groups - a list containing all parameter groups where each parameter group is a dict rcsbdd|D}fddt|dDfdd|dD|d<t|d7|S)NcSsi|]\}}|dkr||qS)rr.0kvrrr z.pack_group..cs&i|]\}}t|vrt||qSrid)rOr3pparam_mappingsrr rRs  rcsg|]}t|qSrrT)rOrVrWrr rSz.pack_group..)itemsr)r.r)r4packed)rX start_indexrr pack_groups "z(Optimizer.state_dict..pack_groupcsg|] }|qSrrrOg)r]rr rYrSz(Optimizer.state_dict..cs.i|]&\}}t|tjr$t|n||qSr)rrrrUrNrWrr rRsz(Optimizer.state_dict..rr)rrrZ)r rZ packed_stater)r]rXr\r state_dictys  zOptimizer.state_dictc s&t|}|j}|d}t|t|kr.tddd|D}dd|D}tddt||Drjtddd ttd d|Dtd d|DD}dfd d tt }|d D]0\}} ||vr||} | | || <q| ||<qddfddt||D} | || dd S)zLoads the optimizer state. Args: state_dict (dict): optimizer state. Should be an object returned from a call to :meth:`state_dict`. rzrSz,Optimizer.load_state_dict..css|]}t|dVqdSrbrcr^rrr rdrScss|]\}}||kVqdSr'r)rOZp_lens_lenrrr rdrSz]loaded state dict contains a parameter group that doesn't match the size of optimizer's groupcSsi|]\}}||qSrr)rOZold_idrVrrr rRrSz-Optimizer.load_state_dict..css|]}|dVqdSrbrr^rrr rdrScss|]}|dVqdSrbrr^rrr rdrSNcst|tjr8|dkr4r(|j}|j}|St|trZfdd|DSt|t j rt |fdd|DS|SdS)zBMake a deep copy of value, casting all tensors to device of param.rMcs i|]\}}|||dqS))r5rrNcastparamrr rRrSz;Optimizer.load_state_dict..cast..c3s|]}|VqdSr'r)rOrQrfrr rdrSz:Optimizer.load_state_dict..cast..N) rrris_floating_pointtodtypedevicerrZcontainer_abcsIterabletype)rhvaluer5)rg)rhr rgs     z'Optimizer.load_state_dict..castrcSs|d|d<|S)Nrr)r4 new_grouprrr update_groups z/Optimizer.load_state_dict..update_groupcsg|]\}}||qSrr)rOr_Zng)rrrr rYsz-Optimizer.load_state_dict..r`)N) rrrr anyzipr from_iterablerrrZr*) r ragroupsZ saved_groupsZ param_lensZ saved_lensid_maprrPrQrhrr)rgrrr load_state_dicts4  zOptimizer.load_state_dictF) set_to_nonec Cs|jdd}t|ds ||r0tdd}tjj|j |j D]}|dD]r}|j durT|rnd|_ qT|j j dur|j n |j d|r|j jr|j qT||j j|j j|j qTqH|r|D] \}}|D]}t|qqWdn1s0YdS)aSets the gradients of all optimized :class:`torch.Tensor` s to zero. Args: set_to_none (bool): instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s are guaranteed to be None for params that did not receive a gradient. 3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether). foreachFrLcSsttSr')rrrrrr rSz%Optimizer.zero_grad..rN)rgethasattrrrrr>r?r@rLrgradgrad_fndetach_requires_grad_ is_sparsezero_rlrkappendrZvaluesZ_foreach_zero_) r ryrzper_device_and_dtype_gradsr4rVrDper_dtype_gradsgradsrrr zero_grads*            zOptimizer.zero_gradcCstdS)afPerforms a single optimization step (parameter update). Args: closure (callable): A closure that reevaluates the model and returns the loss. Optional for most optimizers. .. note:: Unless otherwise specified, this function should not modify the ``.grad`` field of the parameters. N)NotImplementedError)r closurerrr rMs zOptimizer.stepcCsFt|tsJd|d}t|tjr2|g|d<n t|trFtdn t||d<|dD]0}t|tjs|tdt||jsZt dqZ|j D]2\}}|t ur||vrt d|q| ||q|d}t|tt|krtjddd t}|jD]}|t|dq|t|ds6t d |j|d S) aAdd a param group to the :class:`Optimizer` s `param_groups`. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the :class:`Optimizer` as training progresses. Args: param_group (dict): Specifies what Tensors should be optimized along with group specific optimization options. zparam group must be a dictrzoptimizer parameters need to be organized in ordered collections, but the ordering of tensors in sets will change between runs. Please use a list instead.z>optimizer can only optimize Tensors, but one of the params is z can't optimize a non-leaf TensorzJparameter group didn't specify a value of required optimization parameter zoptimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information) stacklevelz7some parameters appear in more than one parameter groupN)rrrrsetrrris_leafr rrZrequired setdefaultrwarningswarnrr) isdisjointr)r r#rrhnamedefault param_setr4rrr r!s>          zOptimizer.add_param_groupN)F)r r rrr$r&r*r r=rrarxboolrrMr!rrrr rs @) r) collectionsrrrmrcopyr itertoolsrrrHobjectrrrrrrr s