a .Sicf1@sdZddlmZddlmZddlmZddlmZddlmZddlm Z ddl m Z dd l m Z dd l mZed gd Gd dde jZdS)zAdam for TensorFlow.)context)ops)control_flow_ops)math_ops)resource_variable_ops) state_ops) optimizer) training_ops) tf_exportztrain.AdamOptimizer)v1csreZdZdZdfdd Zd d Zd d ZddZddZddZ ddZ ddZ ddZ ddZ ddZZS) AdamOptimizera Optimizer that implements the Adam algorithm. References: Adam - A Method for Stochastic Optimization: [Kingma et al., 2015](https://arxiv.org/abs/1412.6980) ([pdf](https://arxiv.org/pdf/1412.6980.pdf)) @compatibility(TF2) tf.compat.v1.train.AdamOptimizer is compatible with eager mode and `tf.function`. When eager execution is enabled, `learning_rate`, `beta1`, `beta2`, and `epsilon` can each be a callable that takes no arguments and returns the actual value to use. This can be useful for changing these values across different invocations of optimizer functions. To switch to native TF2 style, use [`tf.keras.optimizers.Adam`] (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam) instead. Please notice that due to the implementation differences, `tf.keras.optimizers.Adam` and `tf.compat.v1.train.AdamOptimizer` may have slight differences in floating point numerics even though the formula used for the variable updates still matches. #### Structural Mapping to Native TF2 Before: ```python optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=0.001) ``` After: ```python optimizer = tf.keras.optimizers.Adam(learning_rate=0.001) ``` #### How to Map Arguments |TF1 Arg Name |TF2 Arg Name |Note | |----------------------|-------------|----------------------| |learning_rate |learning_rate|Be careful of setting learning_rate as a : : : tensor value computed from the global : : : step. In TF1 this was usually meant to : : : imply a dynamic learning rate and would : : : recompute in each step. In TF2 (eager + : : : function) it will treat it as a scalar : : : value that only gets computed once : : : instead of a symbolic placeholder to be : : : computed each time. : |beta1 |beta_1 | | |beta2 |beta_2 | | |epsilon |epsilon | Default value is 1e-08 in TF1, but : : : 1e-07 in TF2. : |use_locking |N/A |Not applicable in TF2. | #### Before & After Usage Example Before: ```python x = tf.Variable([1,2,3], dtype=tf.float32) grad = tf.constant([0.1, 0.2, 0.3]) optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=0.001) optimizer.apply_gradients(zip([grad], [x])) ``` After: ```python x = tf.Variable([1,2,3], dtype=tf.float32) grad = tf.constant([0.1, 0.2, 0.3]) optimizer = tf.keras.optimizers.Adam(learning_rate=0.001) optimizer.apply_gradients(zip([grad], [x])) ``` @end_compatibility MbP??+?:0yE>FAdamcsFtt|||||_||_||_||_d|_d|_d|_ d|_ dS)aX Construct a new Adam optimizer. Initialization: $$m_0 := 0 \text{(Initialize initial 1st moment vector)}$$ $$v_0 := 0 \text{(Initialize initial 2nd moment vector)}$$ $$t := 0 \text{(Initialize timestep)}$$ The update rule for `variable` with gradient `g` uses an optimization described at the end of section 2 of the paper: $$t := t + 1$$ $$\text{lr}_t := \mathrm{learning_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t := \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t := \beta_2 * v_{t-1} + (1 - \beta_2) * g * g$$ $$\text{variable} := \text{variable} - \text{lr}_t * m_t / (\sqrt{v_t} + \epsilon)$$ The default value of 1e-8 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since AdamOptimizer uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). Args: learning_rate: A Tensor or a floating point value. The learning rate. beta1: A float value or a constant float tensor. The exponential decay rate for the 1st moment estimates. beta2: A float value or a constant float tensor. The exponential decay rate for the 2nd moment estimates. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. use_locking: If True use locks for update operations. name: Optional name for the operations created when applying gradients. Defaults to "Adam". N) superr __init___lr_beta1_beta2_epsilon_lr_t_beta1_t_beta2_t _epsilon_t)self learning_ratebeta1beta2epsilon use_lockingname __class__[/var/www/html/django/DPS/env/lib/python3.9/site-packages/tensorflow/python/training/adam.pyrjs;zAdamOptimizer.__init__cCs^tBtrd}nt}|jd|d|jd|dfWdS1sP0YdS)N beta1_power)graph beta2_power)r init_scoperexecuting_eagerlyget_default_graph_get_non_slot_variable)rr(r%r%r&_get_beta_accumulatorss   z$AdamOptimizer._get_beta_accumulatorscCsbt|ddd}|j|jd|d|j|jd|d|D]$}||d|j||d|jq8dS) NcSs|jSNr")xr%r%r&z-AdamOptimizer._create_slots..)keyr') initial_valuer" colocate_withr)mv)min_create_non_slot_variablerr _zeros_slot_name)rvar_list first_varr8r%r%r& _create_slotsszAdamOptimizer._create_slotscCst||j}||j}||j}||j}tj|dd|_tj|dd|_tj|dd|_ tj|dd|_ dS)Nrr0rrr ) _call_if_callablerrrrrconvert_to_tensorrrrr)rlrrrr r%r%r&_prepares    zAdamOptimizer._preparec Cs||d}||d}|\}}tj|||t||jjt||jjt|j|jjt|j |jjt|j |jjt|j |jj||j d j SNr7r8r!)get_slotr.r apply_adamrcastdtype base_dtyperrrr _use_lockingoprgradvarr7r8r'r)r%r%r& _apply_denses    zAdamOptimizer._apply_densec Cs||d}||d}|\}}tj|j|j|jt||jjt||jjt|j |jjt|j |jjt|j |jjt|j |jj||j d SrD)rFr.r resource_apply_adamhandlerrHrIrJrrrrrKrMr%r%r&_resource_apply_denses    z#AdamOptimizer._resource_apply_densecCs|\}}t||jj}t||jj}t|j|jj}t|j|jj}t|j|jj} t|j|jj} |t d|d|} | |d} |d|} t j | | ||j d}t|g|| || }Wdn1s0Y| |d}||d| }t j ||| |j d}t|g||||}Wdn1sV0Yt |}t j|| ||| |j d}tj|||gS)Nr7rEr8)r.rrHrIrJrrrrsqrtrFrassignrKrcontrol_dependencies assign_subrgroup)rrNrOindices scatter_addr'r)lr_tZbeta1_tZbeta2_tZ epsilon_trBr7m_scaled_g_valuesm_tr8v_scaled_g_valuesv_tv_sqrt var_updater%r%r&_apply_sparse_shareds.   * , z"AdamOptimizer._apply_sparse_sharedcs|j||jfddS)Ncstj|||jdS)NrE)rr[rK)r1ir8rr%r&r2s z-AdamOptimizer._apply_sparse..)rcvaluesrZ)rrNrOr%rer& _apply_sparses  zAdamOptimizer._apply_sparsecCsDtt|j||g|WdS1s60YdSr/)rrWrresource_scatter_addrRvalue)rr1rdr8r%r%r&_resource_scatter_addsz#AdamOptimizer._resource_scatter_addcCs|||||jSr/)rcrj)rrNrOrZr%r%r&_resource_apply_sparse!s z$AdamOptimizer._resource_apply_sparsec Cst|r|\}}t|<|j||j|jd}|j||j|jd}Wdn1sd0YWdn1s0Ytj |||gd|iS)NrEr") rrWr.r6rVrrKrrrY)r update_ops name_scoper'r)Z update_beta1Z update_beta2r%r%r&_finish%s     B zAdamOptimizer._finish)r rrrFr)__name__ __module__ __qualname____doc__rr.r?rCrPrSrcrgrjrkrn __classcell__r%r%r#r&r s$NG   r N)rrtensorflow.python.eagerrtensorflow.python.frameworkrtensorflow.python.opsrrrrtensorflow.python.trainingrr tensorflow.python.util.tf_exportr Optimizerr r%r%r%r&s