a .SicZ@sbdZddlmZddlmZddlmZddlmZddlm Z e dgdGd d d ej Z d S) zMomentum for TensorFlow.)ops)math_ops) optimizer) training_ops) tf_exportztrain.MomentumOptimizer)v1csReZdZdZdfdd ZddZdd Zd d Zd d ZddZ ddZ Z S)MomentumOptimizera Optimizer that implements the Momentum algorithm. Computes (if `use_nesterov = False`): ``` accumulation = momentum * accumulation + gradient variable -= learning_rate * accumulation ``` Note that in the dense version of this algorithm, `accumulation` is updated and applied regardless of a gradient's value, whereas the sparse version (when the gradient is an `IndexedSlices`, typically because of `tf.gather` or an embedding) only updates variable slices and corresponding `accumulation` terms when that part of the variable was used in the forward pass. @compatibility(TF2) tf.compat.v1.train.MomentumOptimizer is compatible with eager mode and `tf.function`. When eager execution is enabled, `learning_rate`,`momentum`, can each be a callable that takes no arguments and returns the actual value to use. This can be useful for changing these values across different invocations of optimizer functions. To switch to native TF2 style, please directly use [`tf.keras.optimizers.SGD`] (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD) with the `momentum` argument. #### Structural mapping to native TF2 Before: ```python optimizer = tf.compat.v1.train.MomentumOptimizer( learning_rate=learning_rate, momentum=momentum, use_nesterov=use_nesterov) ``` After: ```python optimizer = tf.keras.optimizers.SGD( learning_rate=learning_rate, momentum=momentum, nesterov=use_nesterov) ``` #### How to map arguments | TF1 Arg Name | TF2 Arg Name | Note | | ------------------ | ------------- | ------------------------------- | | `learning_rate` | `learning_rate`| Be careful of setting | : : : learning_rate tensor value computed from the global step. : : : : In TF1 this was usually meant to imply a dynamic learning rate and : : : : would recompute in each step. In TF2 (eager + function) it will : : : : treat it as a scalar value that only gets computed once instead of : : : : a symbolic placeholder to be computed each time. : | `momentum` | `momentum` | - | | `use_locking` | - | Not applicable in TF2. | | `use_nesterov` | `nesterov` | - | #### Before & after usage example Before: ```python x = tf.Variable([1,2,3], dtype=tf.float32) grad = tf.constant([0.1, 0.2, 0.3]) optimizer = tf.compat.v1.train.MomentumOptimizer( learning_rate=0.001, momentum=0.9, use_nesterov=False) optimizer.apply_gradients(zip([grad], [x])) ``` After: ```python x = tf.Variable([1,2,3], dtype=tf.float32) grad = tf.constant([0.1, 0.2, 0.3]) optimizer = tf.keras.optimizers.SGD( learning_rate=0.001, momentum=0.9, nesterov=False) optimizer.apply_gradients(zip([grad], [x])) ``` @end_compatibility FMomentumcs(tt|||||_||_||_dS)a Construct a new Momentum optimizer. Args: learning_rate: A `Tensor` or a floating point value. The learning rate. momentum: A `Tensor` or a floating point value. The momentum. use_locking: If `True` use locks for update operations. name: Optional name prefix for the operations created when applying gradients. Defaults to "Momentum". use_nesterov: If `True` use Nesterov Momentum. See (Sutskever et al., 2013). This implementation always computes gradients at the value of the variable(s) passed to the optimizer. Using Nesterov Momentum makes the variable(s) track the values called `theta_t + mu*v_t` in the paper. This implementation is an approximation of the original formula, valid for high values of momentum. It will compute the "adjusted gradient" in NAG by assuming that the new gradient will be estimated by the current average gradient plus the product of momentum and the change in the average gradient. References: On the importance of initialization and momentum in deep learning: [Sutskever et al., 2013] (http://proceedings.mlr.press/v28/sutskever13.html) ([pdf](http://proceedings.mlr.press/v28/sutskever13.pdf)) N)superr__init___learning_rate _momentum _use_nesterov)self learning_ratemomentum use_lockingname use_nesterov __class___/var/www/html/django/DPS/env/lib/python3.9/site-packages/tensorflow/python/training/momentum.pyr tszMomentumOptimizer.__init__cCs|D]}||d|jqdS)Nr) _zeros_slot_name)rvar_listvrrr _create_slotsszMomentumOptimizer._create_slotscCsL|j}t|r|}tj|dd|_|j}t|r8|}tj|dd|_dS)Nr)rr)r callablerconvert_to_tensor_learning_rate_tensorr _momentum_tensor)rrrrrr_prepareszMomentumOptimizer._preparec CsF||d}tj||t|j|jj|t|j|jj|j |j dj SNr)rr) get_slotrapply_momentumrcastr dtype base_dtyper! _use_lockingroprgradvarmomrrr _apply_denses zMomentumOptimizer._apply_densec CsH||d}tj|j|jt|j|jj|t|j |jj|j |j dSr#) r$rresource_apply_momentumhandlerr&r r'r(r!r)rr+rrr_resource_apply_denses z'MomentumOptimizer._resource_apply_densec CsL||d}tj||t|j|jj|j|j t|j |jj|j |j dj Sr#)r$rsparse_apply_momentumrr&r r'r(valuesindicesr!r)rr*r+rrr _apply_sparses zMomentumOptimizer._apply_sparsec CsF||d}tj|j|jt|j|j||t|j|j|j |j dSr#) r$rresource_sparse_apply_momentumr1rr&r r'r!r)r)rr,r-r5r.rrr_resource_apply_sparses z(MomentumOptimizer._resource_apply_sparse)Fr F) __name__ __module__ __qualname____doc__r rr"r/r2r6r8 __classcell__rrrrrs["    rN) r<tensorflow.python.frameworkrtensorflow.python.opsrtensorflow.python.trainingrr tensorflow.python.util.tf_exportr Optimizerrrrrrs