a Sic @sdZddlZddlmmZddlmZddlm Z ddlm Z ddl m Z ddlmZddlmZdd lmZGd d d ejjjZd Zed gdGdddeZGdddeZejjjjdddejjjjddddddgdejdee_dS)zfBase class of optimizer. This is under development, and subject to interface/implementation changes. N)logging)backend) initializers)utils)learning_rate_schedule) keras_export) doc_controlsc@s`eZdZdZdEddZdd Zd d Zd d ZddZe j ddZ e j ddddZddZdFddZddZeddZejddZeddZejd dZeejd!d"Zejd#d"Zd$d%Ze j d&d'Zd(d)ZdGd+d,ZdHd-d.ZdId/d0Zd1d2Zd3d4Z d5d6Z!d7d8Z"d9d:Z#d;d<Z$d=d>Z%d?d@Z&e'dAdBZ(ejdCdDZ)dS)J_BaseOptimizerzBOptimizer base class, which only supports non-distribute use case.NFGz?Tc Ks||_||_||_||_||_||_tjds6d|_|r||dksJ|dkrXt d||r|t |t rn|dkr|t d|||_ ||_ |jdur|jdurt d|jd|jd ||| g|_dS) NGPUFrzC`ema_momentum` must be in the range [0, 1]. Received: ema_momentum=z\`ema_overwrite_frequency` must be an integer > 1 or None. Received: ema_overwrite_frequency=zOAt most one of `clipnorm` and `global_clipnorm` can be set. Received: clipnorm=z, global_clipnorm=.)nameclipnormglobal_clipnorm clipvalueuse_ema jit_compiletfconfiglist_physical_devices ValueError isinstanceint ema_momentumema_overwrite_frequency_create_iteration_variable_process_kwargs _variables selfrrrrrrrrkwargsr"m/var/www/html/django/DPS/env/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py__init__&sL   z_BaseOptimizer.__init__cCsBt&tjddtjdd|_Wdn1s40YdS)z'Create the iterations counter variable.r iterationFrdtype trainableN)r init_scopeVariableint64 _iterationsr r"r"r#rZs  z)_BaseOptimizer._create_iteration_variablecCsF|ddhd}|D](}||vr2td|qt|dqdS)Nis_legacy_optimizer>gradient_aggregatordecaygradient_transformerslrzg%s is deprecated in `optimizer_experimental.Optimizer`, please check the docstring for valid arguments.zY is not a valid argument, kwargs should be empty for `optimizer_experimental.Optimizer`.)poprwarning TypeError)r r!Z legacy_kwargskr"r"r#rcs z_BaseOptimizer._process_kwargscCs|jS).Get a unique identifier of the given variable.) _unique_idr variabler"r"r#_var_keyxsz_BaseOptimizer._var_keyc Cstg}|D]f}t|tjrd|j}|j}t|\}}tj||t|d}| t|||j q| |q|S)aBDeduplicate sparse gradient. For sparse gradients, i.e., gradient is of type `tf.IndexedSlices`, it is possible that `gradient.indices` has duplicated indices. This function adds up values for the duplicated indices, and returns a `tf.IndexedSlices` with indices of unique values. r) rr IndexedSlicesvaluesindicesuniquemathunsorted_segment_sumshapeappend dense_shape) r gradsZprocessed_gradsgradr=r>unique_indicesnew_index_positions summed_valuesr"r"r#_deduplicate_sparse_grads   z'_BaseOptimizer._deduplicate_sparse_gradcCstdS)aiFunction to update variable value based on given gradients. This method must be implemented in customized optimizers. Args: gradient: backpropagated gradient of the given variable. variable: variable whose value needs to be updated. Returns: An `Operation` that applies the specified gradients. NNotImplementedErrorr gradientr:r"r"r# update_stepsz_BaseOptimizer.update_step)rcCs |||S)aA wrapper of `update_step` to enable XLA acceleration. Due to `tf.function` tracing mechanism, for (gradient, variable) pairs of the same shape and dtype, the execution graph always invoke the first pair it has seen. Thus, we need a `key` argument to make each (gradient, variable) pair unique. In additions, XLA cannot understand string input, so the key is an integer. Args: gradient: backpropagated gradient of the given variable. variable: variable whose value needs to be updated. key (int): a unique key that identifies the variable. Returns: An `Operation` that applies the specified gradients. ) _update_step)r rNr:keyr"r"r#_update_step_xlasz_BaseOptimizer._update_step_xlacCsFt|dddurdS|||jvr6td|jd|||dS)Nr8z(The optimizer cannot recognize variable z. This usually means that you're reusing an optimizer previously created for a different model. Try creating a new optimizer instance.)getattrr; _index_dictKeyErrorrrOrMr"r"r#rPs z_BaseOptimizer._update_stepcCst|s&|dur&td|d|d|dur6t}t|rr| |||}Wdn1sh0Y|||}tt||S)aECompute gradients of loss on trainable variables. Args: loss: `Tensor` or callable. If a callable, `loss` should take no arguments and return the value to minimize. var_list: list or tuple of `Variable` objects to update to minimize `loss`. tape: (Optional) `tf.GradientTape`. If `loss` is provided as a `Tensor`, the tape that computed the `loss` must be provided. Returns: A list of (gradient, variable) pairs. Variable is always present, but gradient can be `None`. NzB`tape` is required when a `Tensor` loss is passed. Received: loss=z, tape=r )callablerr GradientTapewatchrNlistzip)r lossvar_listtaperEr"r"r#compute_gradientss  $ z _BaseOptimizer.compute_gradientscCsg}|jrJ|jdkrJ|D],}|dur0||q|t||jq|S|jrl|jdkrlt||jdS|jr|jdkr|D]4}|dur||q|tj||j |jdq|S|S)Nr)clip_value_minclip_value_max)rrCr clip_by_normrclip_by_global_normr clip_by_value)r rE clipped_gradsgr"r"r#_clip_gradientss,  z_BaseOptimizer._clip_gradientscCs|jS)zThe number of training steps this `optimizer` has run. By default, iterations would be incremented by one every time `apply_gradients()` is called. )r,r-r"r"r# iterationssz_BaseOptimizer.iterationscCs&t|ddrtd|d||_dS)N_builtFzCannot set `iterations` to a new Variable after the Optimizer weights have been created. Here it is attempting to set `iterations` to z.Usually this means you are trying to set `iterations` after calling `apply_gradients()`. Please set `iterations` before calling `apply_gradients()`.)rS RuntimeErrorr,r9r"r"r#rgs cCs8t|dr|jdurtd|j}t|tjr4|jS|S)N_learning_ratezPMissing learning rate, please set self.learning_rate at optimizer creation time.)hasattrrjrrrLearningRateSchedule_current_learning_rate)r r2r"r"r# learning_rates z_BaseOptimizer.learning_ratecCs:t|tjr||_n"t|jtjr*td|j|dS)NaThis optimizer was created with a `LearningRateSchedule` object as its `learning_rate` constructor argument, hence its learning rate is not settable. If you need the learning rate to be settable, you should instantiate the optimizer with a float `learning_rate` argument.)rrrlrjr5assignr rnr"r"r#rn+scCs|jS)zAlias of `learning_rate()`. `lr()` is heavily called in workflows using `optimizer_v2.OptimizerV2`, so we keep it for backward compabitliy. rnr-r"r"r#r2>sz_BaseOptimizer.lrcCs ||_dSNrqrpr"r"r#r2HscCsBt|tjr,tj||jdtjdd|_|Stj|dt ddS)NrnFr&) rrrlrr*rgfloat32rmrfloatxrpr"r"r#_build_learning_rateLs z#_BaseOptimizer._build_learning_ratecCsLt|ddrdS|||jrHg|_|D]}|j|j|d|dq*dS)aInitialize the optimizer's variables, such as momemtum variables. This function has to be implemented by subclass optimizers, and subclass optimizers need to call `super().build(var_list)`. Args: var_list: List of model variables to build optimizers on. For example, SGD optimizer with momentum will store one momentum variable corresponding to each model variable. rhFNaverage) initial_value)rS_build_index_dictr_model_variables_moving_averagerCadd_variable_from_reference)r r\varr"r"r#build_s  z_BaseOptimizer.buildcCs0i|_t|D]\}}||}||j|<qdS)zBuild variable to index dictionary. Build a dictionary that maps variable to the index of it in the given var_list. Args: var_list: List of variables to build index dict on. Returns: None N)rT enumerater;)r r\ir{var_keyr"r"r#rxys  z _BaseOptimizer._build_index_dictzeroscCsVt|trt|}|dur$t}|dur0g}tj||||dd}|j ||S)aCreate an optimizer variable. Args: shape: A list of integers, a tuple of integers, or a 1-D Tensor of type int32. Defaults to scalar if unspecified. dtype: The DType of the optimizer variable to be created. Defaults to `tf.keras.backend.floatx` if unspecified. initializer: string or callable. Initializer instance. name: The name of the optimizer variable to be created. Returns: An optimizer variable, in the format of tf.Variable. NF)rwrr() rstrrgetrrtrr*rrC)r rBr' initializerrr:r"r"r# add_variables    z_BaseOptimizer.add_variablecCsd|dur4|dur$tj|j|jd}ntj||jd}tj||d|j|jdd}|j||S)akCreate an optimizer variable from model variable. Create an optimizer variable based on the information of model variable. For example, in SGD optimizer momemtum, for each model variable, a corresponding momemtum variable is created of the same shape and dtype. Args: model_variable: tf.Variable. The corresponding model variable to the optimizer variable to be created. variable_name: String. The name prefix of the optimizer variable to be created. The create variables name will follow the pattern `{variable_name}/{model_variable.name}`, e.g., `momemtum/dense_1`. shape: List or Tuple, defaults to None. The shape of the optimizer variable to be created. If None, the created variable will have the same shape as `model_variable`. initial_value: A Tensor, or Python object convertible to a Tensor, defaults to None. The initial value of the optimizer variable, if None, the initial value will be default to 0. Returns: An optimizer variable. N)rBr')r'/F)rwrr'r()rrrBr'r* _shared_namerrC)r model_variable variable_namerBrwr:r"r"r#rzs z*_BaseOptimizer.add_variable_from_referencecCs||||}||dS)aMinimize `loss` by updating `var_list`. This method simply computes gradient using `tf.GradientTape` and calls `apply_gradients()`. If you want to process the gradient before applying then call `tf.GradientTape` and `apply_gradients()` explicitly instead of using this function. Args: loss: `Tensor` or callable. If a callable, `loss` should take no arguments and return the value to minimize. var_list: list or tuple of `Variable` objects to update to minimize `loss`. tape: (Optional) `tf.GradientTape`. Returns: None N)r^apply_gradients)r r[r\r]grads_and_varsr"r"r#minimizesz_BaseOptimizer.minimizec Cs t|jtjrLt|dr.|j||jntj ||jdtj dd|_t |}t t|dkrjdSt|\}}|jp~d}t|Bt||Wdn1s0YWdn1s0Y||}||}tt||}||dS)zApply gradients to variables. Args: grads_and_vars: List of (gradient, variable) pairs. Returns: None Raises: TypeError: If `grads_and_vars` is malformed. rmrnFr&rN optimizer)rrjrrlrkrmrorgrr*rsoptimizer_utilsfilter_empty_gradientslenrYrZr name_scoper)r|rfrJ_internal_apply_gradients)r rrEtrainable_variables scope_namer"r"r#rs2         F  z_BaseOptimizer.apply_gradientsc CsX|jr.|D] \}}|||t||q n|D]\}}|||q2|jddS)zHelper function of apply gradients. This is required for separating out distributed training logic. Args: grads_and_vars: List of (gradient, variable) pairs. r N)rrRidr;rPrg assign_add)r rrFr{r"r"r#rs   z(_BaseOptimizer._internal_apply_gradientscCs>|jr:t||jD]&\}}||j|d|j|qdS)8Update the stored moving average using the latest value.r N)rrZryror)r r\r{rvr"r"r#&_update_model_variables_moving_average!s z5_BaseOptimizer._update_model_variables_moving_averagecCs@t|t|jkr2tdt|dt|jd||dS)z2Overwrite model variables with its moving average.zThe length of model variables (zT) to override does not match the length of model variables stored in the optimizer (z:). Please check if the optimizer was called on your model.N)rryr4_overwrite_model_variables_with_average_value_helperr r\r"r"r#-_overwrite_model_variables_with_average_value+s z<_BaseOptimizer._overwrite_model_variables_with_average_valuecCs$t||jD]\}}||q dS)z0Helper function that overwrites model variables.N)rZryro)r r\r{ average_varr"r"r#r7s zC_BaseOptimizer._overwrite_model_variables_with_average_value_helpercCs|jr||dS)aSet the final value of model's trainable variables. Sometimes there are some extra steps before ending the variable updates, such as overriding the model variables with its average value. Args: var_list: list of model variables. N)rrrr"r"r#finalize_variable_values>s z'_BaseOptimizer.finalize_variable_valuescCs<t|tjrt|St|tjr*|St|r8|S|S)z=Serialize a hyperparameter that can be a numeric or callable.)rrrl serializerr*numpyrV)r Zhyperparameterr"r"r#_serialize_hyperparameterMs  z(_BaseOptimizer._serialize_hyperparameterc Cs(|j|j|j|j|j|j|jdd}|S)aReturns the config of the optimizer. An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration. Subclass optimizer should override this method to include other hyperparameters. Returns: Python dictionary. F)rrrrrrrr.)rrrrrrr)r rr"r"r# get_configYs z_BaseOptimizer.get_configcCs6d|vr(t|dtr(t|d|d<|fi|S)aDCreates an optimizer from its config. This method is the reverse of `get_config`, capable of instantiating the same optimizer from the config dictionary. Args: config: A Python dictionary, typically the output of get_config. Returns: An optimizer instance. rn)rdictr deserialize)clsrr"r"r# from_configss z_BaseOptimizer.from_configcCs|jS)zReturns variables of this Optimizer. We override the `variable` property method of `tf.Module` for the sake of backward compatibility with `optimizer_v2.Optimizer`'s `variable()` method. )rr-r"r"r# variablessz_BaseOptimizer.variables)NNNFr NT)N)NrN)NN)N)*__name__ __module__ __qualname____doc__r$rrr;rJabcabstractmethodrOrfunctionrRrPr^rfpropertyrgsetterrnrdo_not_generate_docsr2rur|rxrrzrrrrrrrrr classmethodrrr"r"r"r#r #sj 4              ) -    r aname: String. The name to use for momentum accumulator weights created by the optimizer. clipnorm: Float. If set, the gradient of each weight is individually clipped so that its norm is no higher than this value. clipvalue: Float. If set, the gradient of each weight is clipped to be no higher than this value. global_clipnorm: Float. If set, the gradient of all weights is clipped so that their global norm is no higher than this value. use_ema: Boolean, defaults to False. If True, exponential moving average (EMA) is applied. EMA consists of computing an exponential moving average of the weights of the model (as the weight values change after each training batch), and periodically overwriting the weights with their moving average. ema_momentum: Float, defaults to 0.99. Only used if `use_ema=True`. This is # noqa: E501 the momentum to use when computing the EMA of the model's weights: `new_average = ema_momentum * old_average + (1 - ema_momentum) * current_variable_value`. ema_overwrite_frequency: Int or None, defaults to None. Only used if `use_ema=True`. Every `ema_overwrite_frequency` steps of iterations, we overwrite the model variable by its moving average. If None, the optimizer # noqa: E501 does not overwrite model variables in the middle of training, and you need to explicitly overwrite the variables at the end of training by calling `optimizer.finalize_variable_values()` (which updates the model # noqa: E501 variables in-place). When using the built-in `fit()` training loop, this happens automatically after the last epoch, and you don't need to do anything. jit_compile: Boolean, defaults to True. If True, the optimizer will use XLA # noqa: E501 compilation. If no GPU device is found, this flag will be ignored. **kwargs: keyword arguments only used for backward compatibility.z'keras.optimizers.experimental.Optimizer)v1csreZdZdZdfdd Zdfdd Zfd d Zd d Zdfdd ZddZ ddZ ddZ ddZ Z S) OptimizeraAbstract optimizer base class. This class supports distributed training. If you want to implement your own optimizer, please subclass this class instead of _BaseOptimizer. Args: {{base_optimizer_keyword_args}} ### Usage ```python # Create an optimizer with the desired parameters. opt = tf.keras.optimizers.experimental.SGD(learning_rate=0.1) var1, var2 = tf.Variable(1.0), tf.Variable(2.0) # `loss` is a callable that takes no argument and returns the value # to minimize. loss = lambda: 3 * var1 * var1 + 2 * var2 * var2 # Call minimize to update the list of variables. opt.minimize(loss, var_list=[var1, var2]) ``` ### Processing gradients before applying them Calling `minimize()` takes care of both computing the gradients and applying them to the variables. If you want to process the gradients before applying them you can instead use the optimizer in three steps: 1. Compute the gradients with `tf.GradientTape`. 2. Process the gradients as you wish. 3. Apply the processed gradients with `apply_gradients()`. Example: ```python # Create an optimizer. opt = tf.keras.optimizers.experimental.SGD(learning_rate=0.1) var1, var2 = tf.Variable(1.0), tf.Variable(2.0) # Compute the gradients for a list of variables. with tf.GradientTape() as tape: loss = 3 * var1 * var1 + 2 * var2 * var2 grads = tape.gradient(loss, [var1, var2]) # Process the gradients. grads[0] = grads[0] + 1 # Ask the optimizer to apply the gradients on variables. opt.apply_gradients(zip(grads, [var1, var2])) ``` ### Dynamic learning rate Dynamic learning rate can be achieved by setting learning rate as a built-in or customized `tf.keras.optimizers.schedules.LearningRateSchedule`. Example: >>> var = tf.Variable(np.random.random(size=(1,))) >>> learning_rate = tf.keras.optimizers.schedules.ExponentialDecay( ... initial_learning_rate=.01, decay_steps=20, decay_rate=.1) >>> opt = tf.keras.optimizers.experimental.SGD(learning_rate=learning_rate) >>> loss = lambda: 3 * var >>> opt.minimize(loss, var_list=[var]) ### Gradients clipping Users can clip the gradients before applying to variables by setting `clipnorm`, `clipvalue` and `global_clipnorm`. Notice that `clipnorm` and `global_clipnorm` can only have one being set. Example: >>> opt = tf.keras.optimizers.experimental.SGD(learning_rate=1, clipvalue=1) >>> var1, var2 = tf.Variable(2.0), tf.Variable(2.0) >>> with tf.GradientTape() as tape: ... loss = 2 * var1 + 2 * var2 >>> grads = tape.gradient(loss, [var1, var2]) >>> print([grads[0].numpy(), grads[1].numpy()]) [2.0, 2.0] >>> opt.apply_gradients(zip(grads, [var1, var2])) >>> # Without clipping, we should get [0, 0], but as gradients are clipped >>> # to >>> # have max value 1, we get [1.0, 1.0]. >>> print([var1.numpy(), var2.numpy()]) [1.0, 1.0] ### Using exponential moving average. Empirically it has been found that using the exponential moving average (EMA) of the trained parameters of a deep network achieves a better performance than using its trained parameters directly. Keras optimizers allows users to compute this moving average and overwrite the model variables at desired time. Example: ```python # Create an SGD optimizer with EMA on. `ema_momentum` controls the decay # rate of the moving average. `ema_momentum=1` means no decay and the stored # moving average is always model variable's initial value before training. # Reversely, `ema_momentum=0` is equivalent to not using EMA. # `ema_overwrite_frequency=3` means every 3 iterations, we overwrite the # trainable variables with their moving average values. opt = tf.keras.optimizers.experimental.SGD( learning_rate=1, use_ema=True, ema_momentum=0.5, ema_overwrite_frequency=3) var1, var2 = tf.Variable(2.0), tf.Variable(2.0) with tf.GradientTape() as tape: loss = var1 + var2 grads = tape.gradient(loss, [var1, var2]) # First iteration: [var1, var2] = [1.0, 1.0] opt.apply_gradients(zip(grads, [var1, var2])) print([var1, var2]) # Second iteration: [var1, var2] = [0.0, 0.0] opt.apply_gradients(zip(grads, [var1, var2])) print([var1, var2]) # Third iteration, without EMA, we should see [var1, var2] = [-1.0, -1.0], # but overwriting results in [var1, var2] = [-0.125, -0.125]. The full # calculation for the moving average of var1 is: # var1=2*0.5**3+1*(1-0.5)*0.5**2+0*(1-0.5)*0.5**1+(-1)*(1-0.5)=-0.125. opt.apply_gradients(zip(grads, [var1, var2])) print([var1, var2]) ``` When optimizer is constructed with `use_ema=True`, in custom training loop, users can explicitly call `finalize_variable_values()` to overwrite trainable variables with their EMA values. `finalize_variable_values()` is by default called at the end of `model.fit()`. ### Use with `tf.distribute.Strategy` This optimizer class is `tf.distribute.Strategy` aware, which means it automatically sums gradients across all replicas. To aggregate gradients yourself, call `apply_gradients` with `skip_aggregate_gradients` set to True. This is useful if you need to process aggregated gradients. ```python # This example is not runnable, it consists of dummy code for simple # tutorial. strategy = tf.distribute.experimental.TPUStrategy() with strategy.scope(): opt = tf.keras.optimizers.experimental.SGD() model = magic_function_that_returns_model() gradients = magic_function_that_returns_gradients() # Custom logic to aggregate gradients. gradients = strategy.reduce("SUM", gradients, axis=None) opt.apply_gradients(zip(gradients, model.trainable_variables), skip_aggregate_gradients=True) ``` ### Creating a custom optimizer If you intend to create your own optimization algorithm, please inherit from this class and override the following methods: - `build`: Create your optimizer-related variables, such as `momentums` in SGD optimizer. - `update_step`: Implement your optimizer's updating logic. - `get_config`: serialization of the optimizer, include all hyper parameters. Your optimizer would automatically be compatible with tensorflow distributed training if you subclass `optimizer_experimental.Optimizer`. NFr Tc s2tj||||||||fi| tj|_dS)zCreate a new Optimizer.N)superr$r distribute get_strategy_distribution_strategyr __class__r"r#r$_s  zOptimizer.__init__csLtj}|j|"t||||WdS1s>0YdSrr)rrrextendedcolocate_vars_withrrz)r rrrBrwstrategyrr"r#rzzs  z%Optimizer.add_variable_from_referencecst|dr|}t|S)r7_distributed_container)rkrrr;r9rr"r#r;s zOptimizer._var_keycCs t|S)aXAggregate gradients on all devices. By default we will perform reduce_sum of gradients across devices. Users can implement their own aggregation logic by overriding this method. Args: grads_and_vars: List of (gradient, variable) pairs. Returns: List of (gradient, variable) pairs. )rall_reduce_sum_gradientsr rr"r"r#aggregate_gradientss zOptimizer.aggregate_gradientscs|s||}t|dS)aApply gradients to variables. Args: grads_and_vars: List of (gradient, variable) pairs. skip_gradients_aggregation: If true, gradients aggregation will not be performed inside optimizer. Usually this arg is set to True when you write custom code aggregating gradients outside the optimizer. Returns: None Raises: TypeError: If `grads_and_vars` is malformed. RuntimeError: If called in a cross-replica context. N)rrr)r rZskip_gradients_aggregationrr"r#rs zOptimizer.apply_gradientscCstjjj|j|j|dSrr)r __internal__rinterimmaybe_merge_call_distributed_apply_gradients_fnrrr"r"r#rs  z#Optimizer._internal_apply_gradientscCs8|j}t||jD] \}}|jj|dd|fdqdS)zHelper function to _overwrite_model_variables_with_average_value. This function overwrites variables on each device. Args: var_list: list of model variables. cSs ||Srr)ro)abr"r"r#zPOptimizer._overwrite_model_variables_with_average_value_helper..)argsN)rrZryrupdate)r r\rr{rr"r"r#rs  z>Optimizer._overwrite_model_variables_with_average_value_helpercsDjr@fdd}t|jD] \}}jjj|||fddqdS)rcs"|j|dj|dS)Nr )ror)rvr{r-r"r#update_averageszHOptimizer._update_model_variables_moving_average..update_averageFrgroupN)rrZryrrr)r r\rr{rvr"r-r#rs   z0Optimizer._update_model_variables_moving_averagec  sfdd}|D]\}}|jj|||fddqjdjrt|\}jrjjdk}tj t |tj fddd dd d S) z1`apply_gradients` using a `DistributionStrategy`.cs.jr||t|S||SdSrr)rrRrr;rP)r{rFr-r"r#apply_grad_to_update_varszKOptimizer._distributed_apply_gradients_fn..apply_grad_to_update_varFrr rcs Srr)rr"rr"r#rsz;Optimizer._distributed_apply_gradients_fn..cSsdSrrr"r"r"r"r#rr)true_fnfalse_fnN) rrrgrrrZrrrcondcastbool) r distributionrr!rrFr{_Zshould_overwrite_model_varsr"rr#rs"        z)Optimizer._distributed_apply_gradients_fn)NNNFr NT)NN)F)rrrrr$rzr;rrrrrr __classcell__r"r"rr#rs&/ rcs$eZdZfddZddZZS)RestoredOptimizercstddS)Nr)rr$r-rr"r#r$szRestoredOptimizer.__init__cCs tddS)NzRestoring functional Optimizers from SavedModels is not currently supported. Please file a feature request if this limitation bothers you.rKr-r"r"r#rszRestoredOptimizer.get_config)rrrr$rrr"r"rr#rs rZexperimentalOptimizercCs t|tSrr)rr)objr"r"r#rrrcCstSrr)r)protor"r"r#r rr )object_factoryversionmin_producer_versionmin_consumer_version)versionsz{{base_optimizer_keyword_args}}) rrtensorflow.compat.v2compatv2rabslrkerasrrkeras.optimizers.optimizer_v2rrkeras.optimizers.schedulesr tensorflow.python.util.tf_exportrtensorflow.tools.docsrrtracking AutoTrackabler base_optimizer_keyword_argsrr saved_modelloadregister_revived_typeVersionedTypeRegistrationreplacer"r"r"r#sD       s G