a 8Sic.3@sddlZddlZddlmZddlmZddlmZmZGdddeZdeeeeeeeeeeeee e e e e e e d d d Z eeeeeeeeeeeee e e e e e d d d Z eeeeeeeeeeeee e e e e e d ddZ dS)N)Tensor) Optimizer)ListOptionalcsJeZdZdZdeedfd d Zfd d Ze dd dZ Z S)NAdama Implements NAdam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma_t \text{ (lr)}, \: \beta_1,\beta_2 \text{ (betas)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \: \lambda \text{ (weight decay)}, \:\psi \text{ (momentum decay)} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0 \leftarrow 0 \text{ ( second moment)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm} \mu_t \leftarrow \beta_1 \big(1 - \frac{1}{2} 0.96^{t \psi} \big) \\ &\hspace{5mm} \mu_{t+1} \leftarrow \beta_1 \big(1 - \frac{1}{2} 0.96^{(t+1)\psi}\big)\\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow \mu_{t+1} m_t/(1-\prod_{i=1}^{t+1}\mu_i)\\[-1.ex] & \hspace{11mm} + (1-\mu_t) g_t /(1-\prod_{i=1}^{t} \mu_{i}) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Incorporating Nesterov Momentum into Adam`_. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 2e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) momentum_decay (float, optional): momentum momentum_decay (default: 4e-3) foreach (bool, optional): whether foreach implementation of optimizer is used (default: None) .. _Incorporating Nesterov Momentum into Adam: https://openreview.net/forum?id=OM0jvwB8jIp57ZJjtNEZ Mb`?g?g+?:0yE>rMbp?N)foreachc sd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd |d|kstd |t||||||d }tt|||dS) NzInvalid learning rate: {}zInvalid epsilon value: {}r?z%Invalid beta parameter at index 0: {}rz%Invalid beta parameter at index 1: {}zInvalid weight_decay value: {}z Invalid momentum_decay value: {})lrbetaseps weight_decaymomentum_decayr ) ValueErrorformatdictsuperr__init__) selfparamsrrrrrr defaults __class__M/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/optim/nadam.pyr9s"zNAdam.__init__cst||jD]}|ddqt|j}t|dkoNt |dd}|st|D]}t t |d|d<qXt|dkot |dd}|s|D]}t |d|d<qdS)Nr rstep mu_product) r __setstate__ param_groups setdefaultliststatevalueslentorch is_tensortensorfloat)rr&group state_valuesstep_is_tensorsZmu_product_is_tensorrrrr"Ls  zNAdam.__setstate__cCsxd}|dur:t|}Wdn1s00Y|jD]0}g}g}g}g}g}g} |d\} } |dD]} | jdurr|| | jjrtd|| j|j| } t| dkrt d| d<t d| d <tj | tj d | d <tj | tj d | d <|| d || d || d | | dqrt |||||| | | |d |d|d|d|dd q@|S)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrrz'NAdam does not support sparse gradientsrr r rr!) memory_formatexp_avg exp_avg_sqrrrrr )beta1beta2rrrrr ) r) enable_gradr#gradappend is_sparse RuntimeErrorr&r(r+ zeros_likepreserve_formatnadam)rclosurelossr-params_with_gradgradsexp_avgs exp_avg_sqs mu_products state_stepsr4r5pr&rrrr ZsV $       z NAdam.step)rr r rr N)N) __name__ __module__ __qualname____doc__rboolrr"r)no_gradr __classcell__rrrrrs0 r) rrArBrCrDrEr r4r5rrrrcCstdd|Dstdtdd|Ds4td|dur@d}|rVtjrVtd|rjtjsjt} nt} | ||||||||| | | | d dS) zpFunctional API that performs NAdam algorithm computation. See :class:`~torch.optim.NAdam` for details. cSsg|]}t|tjqSr isinstancer)r.0trrr znadam..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorscSsg|]}t|tjqSrrNrPrrrrSrTzPAPI has changed, `mu_products` argument must contain a list of singleton tensorsNFz6torch.jit.script not supported with foreach optimizers)r4r5rrrr)allr:r)jit is_scripting_multi_tensor_nadam_single_tensor_nadam)rrArBrCrDrEr r4r5rrrrfuncrrrr=s0r=) rrArBrCrDrEr4r5rrrrcCsHt|D]8\} } || }|| }|| }|| }|| }|d7}|}d||}| dkrl|j| | d}|ddd|| }|ddd|d| }||9}|||}||j|d|d||j||d|d||| }| j||| d|d|d| j||| |d|dqdS)Nrralphar?Q?)value) enumerateitemaddmul_add_addcmul_divsqrtaddcdiv_)rrArBrCrDrEr4r5rrrriparamr7r2r3r!step_tr bias_correction2mumu_nextZmu_product_nextdenomrrrrYs(  &rYcsPt|dkrdSt|dfdd|D} fdd|D} fdd|D}fdd|D}t||| dkrtj||| dt|tj||ddt|t|||dt|}d d| D}t||t|| }fd dt||D}fd dt||D}t ||||t ||||dS) Nrrcsg|]}d|qSrrarQr )r4rrrSrTz'_multi_tensor_nadam..csg|]}d|qSrprqrr)r5rrrSrTcs(g|] }ddd|qS)rr]r^rqrrr4rrrrSrTcs,g|]$}ddd|dqS)rr]r^rrqrrrsrrrSsr[cSsg|]}t|qSr)mathrg)rQbcrrrrS"rTcs,g|]$\}}d|d|dqSrrq)rQr!rmrrrrS&scs,g|]$\}}|d||dqSrvrq)rQr!rnrxrrrS(s) r(r) _foreach_add_ _foreach_mul__foreach_addcmul_ _foreach_sqrt _foreach_div_ _foreach_addzip_foreach_addcdiv_)rrArBrCrDrEr4r5rrrrbias_correction1rlmusZmu_nextsexp_avg_sq_sqrtZbias_correction_sqrtroZstep_size_gradsZstep_size_expavgr)r4r5rrrrXs8           rX)N)rtr)r optimizerrtypingrrrrKr,r=rYrXrrrrs^   5 .