a 8Sic& @sddlZddlmZddlmZddlmZmZGdddeZdeeeeeeeeee e e e ed d d Z eeeeeeeee e e e ed d d Z eeeeeeeee e e e ed ddZ dS)N)Tensor) Optimizer)ListOptionalcsReZdZdZdddeeed fd d Zfd d Ze dddZ Z S)Adadeltaa Implements Adadelta algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)}, \: \rho \text{ (decay)}, \: \lambda \text{ (weight decay)} \\ &\textbf{initialize} : v_0 \leftarrow 0 \: \text{ (square avg)}, \: u_0 \leftarrow 0 \: \text{ (accumulate variables)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm} v_t \leftarrow v_{t-1} \rho + g^2_t (1 - \rho) \\ &\hspace{5mm}\Delta x_t \leftarrow \frac{\sqrt{u_{t-1} + \epsilon }}{ \sqrt{v_t + \epsilon} }g_t \hspace{21mm} \\ &\hspace{5mm} u_t \leftarrow u_{t-1} \rho + \Delta x^2_t (1 - \rho) \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \Delta x_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `ADADELTA: An Adaptive Learning Rate Method`_. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups rho (float, optional): coefficient used for computing a running average of squared gradients (default: 0.9) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-6) lr (float, optional): coefficient that scale delta before it is applied to the parameters (default: 1.0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) foreach (bool, optional): whether foreach implementation of optimizer is used (default: None) maximize (bool, optional): maximize the params based on the objective, instead of minimizing (default: False) .. _ADADELTA\: An Adaptive Learning Rate Method: https://arxiv.org/abs/1212.5701 ??ư>rNF)maximize)foreachr c sd|kstd|d|kr*dks:ntd|d|ksPtd|d|ksftd|t||||||d}tt|||dS)NgzInvalid learning rate: {}rzInvalid rho value: {}zInvalid epsilon value: {}zInvalid weight_decay value: {})lrrhoeps weight_decayr r ) ValueErrorformatdictsuperr__init__) selfparamsr rrrr r defaults __class__P/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/optim/adadelta.pyr7s zAdadelta.__init__cs4t||jD]}|dd|ddqdS)Nr r F)r __setstate__ param_groups setdefault)rstategrouprrrrFs   zAdadelta.__setstate__c Cs`d}|dur:t|}Wdn1s00Y|jD]}g}g}g}g}|d|d|d|d|d|df\}} } } } } |dD]}|jdurq|||jjrtd ||j|j|}t|d krd |d <tj |tj d |d <tj |tj d |d<||d ||d|d d7<qt |||||| | | | | d q@|S)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr rrrr r rz*Adadelta does not support sparse gradientsrstep) memory_format square_avg acc_deltar)r rrrr r ) torch enable_gradrgradappend is_sparse RuntimeErrorr len zeros_likepreserve_formatadadelta)rclosurelossr!Zparams_with_gradgrads square_avgs acc_deltasr rrrr r pr rrrr"LsV $       z Adadelta.step)rr r rN)N) __name__ __module__ __qualname____doc__rboolrrr&no_gradr" __classcell__rrrrrs. r) rr2r3r4r r rrrr c CsX|dur d}|r"tjr"td|r6tjs6t} nt} | ||||||||| d dS)zvFunctional API that performs Adadelta algorithm computation. See :class:`~torch.optim.Adadelta` for details. NFz6torch.jit.script not supported with foreach optimizers)r rrrr )r&jit is_scriptingr+_multi_tensor_adadelta_single_tensor_adadelta) rr2r3r4r r rrrr funcrrrr/s"r/) rr2r3r4r rrrr cCst||||D]\} } } } |s"| n| } |dkr>| j| |d} t| rft| } t| } t| } | |j| | d|d| |} | || | }| |j||d|dt| rt |}| j || dqdSNr)alphar)value) zipaddr& is_complex view_as_realmul_addcmul_sqrt_div_view_as_complexadd_)rr2r3r4r rrrr paramr(r$r%stddeltarrrr@s       r@) rr2r3r4r rrrr c Cst|dkrdS|rt|}|dkr6tj|||dt||tj|||d|dt||} t| t||} t| t| | t| |tj|| | dt||tj|| | d|ddSrB) r,r&Z _foreach_negZ _foreach_add_Z _foreach_mul_Z_foreach_addcmul_Z _foreach_addZ_foreach_sqrt_Z _foreach_div_) rr2r3r4r rrrr rPZdeltasrrrr?s"          r?)N) r&r optimizerrtypingrrrr:floatr/r@r?rrrrsJ   )