a Sic)T@sjdZddlmmZddlmZddlmZddl m Z e dGdddej Z Gd d d ej Z dS) zAdam optimizer implementation.N)backend_config) optimizer_v2) keras_exportzkeras.optimizers.AdamcsfeZdZdZdZdfdd Zd d Zfd d ZfddZdddZ dddZ fddZ Z S)AdamaOptimizer that implements the Adam algorithm. Adam optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. According to [Kingma et al., 2014](http://arxiv.org/abs/1412.6980), the method is "*computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters*". Args: learning_rate: A `Tensor`, floating point value, or a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use, The learning rate. Defaults to 0.001. beta_1: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use. The exponential decay rate for the 1st moment estimates. Defaults to 0.9. beta_2: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use, The exponential decay rate for the 2nd moment estimates. Defaults to 0.999. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to 1e-7. amsgrad: Boolean. Whether to apply AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and beyond". Defaults to `False`. name: Optional name for the operations created when applying gradients. Defaults to `"Adam"`. **kwargs: keyword arguments. Allowed arguments are `clipvalue`, `clipnorm`, `global_clipnorm`. If `clipvalue` (float) is set, the gradient of each weight is clipped to be no higher than this value. If `clipnorm` (float) is set, the gradient of each weight is individually clipped so that its norm is no higher than this value. If `global_clipnorm` (float) is set the gradient of all weights is clipped so that their global norm is no higher than this value. Usage: >>> opt = tf.keras.optimizers.Adam(learning_rate=0.1) >>> var1 = tf.Variable(10.0) >>> loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1 >>> step_count = opt.minimize(loss, [var1]).numpy() >>> # The first step is `-learning_rate*sign(grad)` >>> var1.numpy() 9.9 Reference: - [Kingma et al., 2014](http://arxiv.org/abs/1412.6980) - [Reddi et al., 2018]( https://openreview.net/pdf?id=ryQu7f-RZ) for `amsgrad`. Notes: The default value of 1e-7 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since Adam uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). TMbP??+?Hz>Fc sftj|fi||d|d||d|j|d||d||pXt|_||_dS)N learning_ratelrdecaybeta_1beta_2super__init__ _set_hyperget_initial_decayrepsilonamsgradselfr r rrrnamekwargs __class__^/var/www/html/django/DPS/env/lib/python3.9/site-packages/keras/optimizers/optimizer_v2/adam.pyrhs   z Adam.__init__cCsL|D]}||dq|D]}||dq|jrH|D]}||dq6dSNmvvhatadd_slotrrvar_listvarrrr _create_slotszszAdam._create_slotsc st|||t|jd|}t|d|}t|d|}t||}t||}|||fdtd|d|} |||f t | t |j |||d|||d|ddSNr rlr_t)r rbeta_1_t beta_1_powerone_minus_beta_1_tbeta_2_t beta_2_powerone_minus_beta_2_t r_prepare_localtfcast iterationsidentity _get_hyperpowsqrtupdatedictconvert_to_tensorr r var_device var_dtype apply_state local_stepr,r/r-r0r rrrr3s*    zAdam._prepare_localcsN|j}tt|dd}t|d|dkr>|dt|}t|dSNr*weightsintlenr set_weightsrrGparamsnum_varsrrrrJs zAdam.set_weightsNc Cs|j|jj}}|pi||fp,|||}||d}||d}|jstjj |j |j |j |d|d|d|d|d|d||j d S||d } tjj |j |j |j | j |d|d|d|d|d|d||j d SdS) Nr r!r-r0r+r,r/r) r'r r! beta1_power beta2_powerr beta1beta2rgrad use_lockingr") r'r r!r"rNrOr rPrQrrRrS) devicedtype base_dtyper_fallback_apply_stateget_slotrr4raw_opsResourceApplyAdamhandle _use_lockingResourceApplyAdamWithAmsgrad) rrRr'rAr?r@ coefficientsr r!r"rrr_resource_apply_densesJ    zAdam._resource_apply_densecCs|j|jj}}|pi||fp,|||}||d}||d} tjjj |||d|j d} t | g| ||| } Wdn1s0Y||d} |||d} tjjj | | |d|j d} t | g| | || } Wdn1s 0Y|j s`t| }tjjj||d| ||d |j d}tj|| | gS||d }t|| }t |g&tjjj |||j d}Wdn1s0Yt|}tjjj||d| ||d |j d}tj|| | |gSdS) Nr r.r,)rSr!r1r/r rr")rTrUrVrrWrXr4compatv1assignr\control_dependencies_resource_scatter_addrr: assign_subgroupmaximum)rrRr'indicesrAr?r@r^r m_scaled_g_valuesm_tr!v_scaled_g_valuesv_tv_sqrt var_updatev_hatv_hat_t v_hat_sqrtrrr_resource_apply_sparsesZ   , .   &  zAdam._resource_apply_sparsec s>t}||d|j|d|d|j|jd|SNr r r)r r r rrrr get_configr;_serialize_hyperparameterrrrrconfigrrrrus  zAdam.get_config)rrrr Fr)N)N) __name__ __module__ __qualname____doc___HAS_AGGREGATE_GRADrr(r3rJr_rrru __classcell__rrrrrsJ  ( 1rcs~eZdZdZdZdfd d Zd d Zfd dZfddZe j dddddZ e j dddddZ fddZ ZS) NonFusedAdama Optimizer that implements the Adam algorithm without fused kernels. Adam optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. According to the paper [Adam: A Method for Stochastic Optimization. Kingma et al., 2014](http://arxiv.org/abs/1412.6980), the method is "*computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters*". For AMSGrad see [On The Convergence Of Adam And Beyond. Reddi et al., 5-8](https://openreview.net/pdf?id=ryQu7f-RZ). **If amsgrad = False**: initialize $m_0$ as 1st moment vector initialize $v_0$ as 2nd moment vector The update rule for $\theta$ with gradient $g$ uses an optimization described at the end of section 2 of the paper: $$lr_t = \mathrm{learning\_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t = \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * g^2$$ $$\theta_t = \theta_{t-1} - lr_t * m_t / (\sqrt{v_t} + \epsilon)$$ **If amsgrad = True**: initialize $m_0$ as 1st moment vector initialize $v_0$ as 2nd moment vector initialize $\hat{v}_0$ as 2nd moment vector The update rule for $\theta$ with gradient $g$ uses an optimization described at the end of section 2 of the paper: $$lr_t = \mathrm{learning\_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t = \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * g^2$$ $$\hat{v}_t = \max(\hat{v}_{t-1}, v_t)$$ $$\theta_t = \theta_{t-1} - lr_t * m_t / (\sqrt{\hat{v}_t} + \epsilon)$$ The default value of 1e-7 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since Adam uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). Usage: >>> opt = tf.keras.optimizers.Adam(learning_rate=0.1) >>> var1 = tf.Variable(10.0) >>> loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1 >>> step_count = opt.minimize(loss, [var1]).numpy() >>> # The first step is `-learning_rate*sign(grad)` >>> var1.numpy() 9.9 Trrrr Frc sftj|fi||d|d||d|j|d||d||pXt|_||_dS)a Construct a new Adam optimizer. Args: learning_rate: A `Tensor`, floating point value, or a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use, The learning rate. Defaults to 0.001. beta_1: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use. The exponential decay rate for the 1st moment estimates. Defaults to 0.9. beta_2: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use, The exponential decay rate for the 2nd moment estimates. Defaults to 0.999. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to 1e-7. amsgrad: Boolean. Whether to apply AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and beyond". Defaults to `False`. name: Optional name for the operations created when applying gradients. Defaults to "Adam". **kwargs: keyword arguments. Allowed to be {`clipnorm`, `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use `learning_rate` instead. r r r r rNrrrrrr\s*  zNonFusedAdam.__init__cCsL|D]}||dq|D]}||dq|jrH|D]}||dq6dSrr#r%rrrr(szNonFusedAdam._create_slotsc st|||t|jd|}t|d|}t|d|}t||}t||}|||fdtd|d|} |||f t | t |j |||d|||d|ddSr)r2r>rrrr3s*    zNonFusedAdam._prepare_localcsN|j}tt|dd}t|d|dkr>|dt|}t|dSrCrFrKrrrrJs zNonFusedAdam.set_weights) jit_compileNc Cs|j|jj}}|pi||fp,|||}||d}||d}|dtd|dd|d} |||d|d|t ||d|d|j r||d } | t | || }| || t||d dS) Nr r!r+r*r0r-r,r/r"r)rTrUrVrrWrXr4r: assign_addsquarerrbrgre) rrRr'rAr?r@r^r r!alphar"rrrr_s*      z"NonFusedAdam._resource_apply_densec Cs|j|jj}}|pi||fp,|||}||d}||d} |||d|t | |||d} |||d} | | |d| t | ||j s| |d|t | |dnB||d } | t | | | |d|t | |ddS) Nr r.r,r!r1r/r rr")rTrUrVrrWrXrb scatter_addr4 IndexedSlicesrrer:rg) rrRr'rhrAr?r@r^r rir!rkrorrrrrs6     z#NonFusedAdam._resource_apply_sparsec s>t}||d|j|d|d|j|jd|Srsrtrwrrrrus  zNonFusedAdam.get_config)rrrr Fr)N)N)ryrzr{r|r}rr(r3rJr4functionr_rrrur~rrrrrs"H2     r)r|tensorflow.compat.v2r`v2r4kerasrkeras.optimizers.optimizer_v2r tensorflow.python.util.tf_exportr OptimizerV2rrrrrrs   w