a Sicډ @sdZddlZddlZddlmmZddlmmZ ddl m Z ddl m ZddlmZddlmZddlmZddlmZeZdJd d Zde jdddd ddde jje jjddf d d ZddZddZddZ ddZ!dZ"ddZ#ddZ$dKddZ%ddZ&dd Z'd!d"Z(d#d$Z)d%d&Z*d'd(Z+e j,-e+Gd)d*d*Z.Gd+d,d,Z/d-d.Z0d/d0Z1d1d2Z2dLd4d5Z3d6d7Z4da5ed8gd9d:d;Z6edZ7d?d@Z8GdAdBdBZ9dCdDZ:dEdFZ;dGdHZt|r6|}| rNtj|||| d}ntj|||d}|j}t|}|durzd}| durtj ||||||||| | | |r|ndd St j |||||||| | | |r|ndd SdS)aTemporary util to create a variable (relies on `variable_scope.variable`). Some reuse-related technicalities prevent us from using `variable_scope.get_variable()` directly, so we use a subcomponent that has fewer constraints (`variable_scope.variable()`). In the longer term, it seems like a similar "default variable creator" method should exist in `Trackable` instead. When this happens, we can get rid of this temporary solution. TODO(fchollet): remove this method when no longer needed. Args: name: Variable name. shape: Variable shape. dtype: The type of the variable. Defaults to `self.dtype` or `float32`. initializer: Initializer instance (callable). trainable: Whether the variable should be part of the layer's "trainable_variables" (e.g. variables, biases) or "non_trainable_variables" (e.g. BatchNorm mean, stddev). Note, if the current variable scope is marked as non-trainable then this parameter is ignored and any added variables are also marked as non-trainable. `trainable` defaults to `True` unless `synchronization` is set to `ON_READ`. caching_device: Passed to `tf.Variable`. validate_shape: Passed to `tf.Variable`. constraint: Constraint instance (callable). use_resource: Whether to use a `ResourceVariable`. collections: List of graph collections keys. The new variable is added to these collections. Defaults to `[GraphKeys.GLOBAL_VARIABLES]`. synchronization: Indicates when a distributed a variable will be aggregated. Accepted values are constants defined in the class `tf.VariableSynchronization`. By default the synchronization is set to `AUTO` and the current `DistributionStrategy` chooses when to synchronize. If `synchronization` is set to `ON_READ`, `trainable` must not be set to `True`. aggregation: Indicates how a distributed variable will be aggregated. Accepted values are constants defined in the class `tf.VariableAggregation`. partitioner: Not handled at this time. layout: the optional DTensor layout, used for creating DVariable. Returns: Variable instance. FNT)r layout)r ) initial_valuer trainablecaching_devicer validate_shape constraint use_resource collectionssynchronization aggregationshape) rr rrr rrrrrr) callablerisclass functoolspartial base_dtypetf TensorShapetf1Variabledtensor DVariable)r rr initializerrrrrrrrr partitionerrinitializing_from_valueinit_valvariable_dtypevariable_shaperrr make_variable,sZ=    r/cCsdd}tj||S)zRetrieves the output mask(s) of the previous node. Args: input_tensors: An arbitrary structure of Tensors. Returns: A mask tensor or list of mask tensors. cSs t|ddS)N _keras_maskgetattr)xrrr_collect_previous_masksz5collect_previous_mask.._collect_previous_maskr#nest map_structure) input_tensorsr4rrrcollect_previous_masks r9cCstddtj|DS)Ncss|]}t|dVqdS_keras_historyNhasattr.0r3rrr z*have_all_keras_metadata..)allr#r6flatten)tensorsrrrhave_all_keras_metadatasrEcCstj|tdS)N)rr )r% placeholderrfloatx)rrrr generate_placeholders_from_shapesrHcCst|tg\}}|S)a!Wraps TensorFlow Operations for compatibility with the Functional API. This method checks to see if a Tensor in `tensors` is missing Keras metadata and has its origin in a Keras `Input` Layer. If so, this method will replace the raw TensorFlow Operations that created this tensor with `TensorFlowOpLayer` instances that create identical operations. Any Tensors not originating from a Keras `Input` Layer will be treated as constants when constructing `TensorFlowOpLayer` instances. Args: tensors: A structure of Tensors, some of which come from raw TensorFlow operations and need to have Keras metadata assigned to them. Returns: created_layers: List. The `TensorFlowOpLayer` instances created to wrap the raw Tensorflow operations. )_create_keras_history_helperset)rD_created_layersrrrcreate_keras_historysrMFc Cstrtdddlm}tj|}g}g}|D]l}t|dddurLq4t |tj tj frl| |j q4t|r| |q4|j }||vr4t|j} i} g} t| D]\} } t| r| | qtjot }tt}|s|str| | | <qt$tg| g| | <Wdq1s40Yqt| } t| ||\}}|j}|j !}|j"|| |d}| ||j#| fi|j$d|%|gq4|s|rd}td j&|t'|t'|d ||fS) a$Helper method for `create_keras_history`. Args: tensors: A structure of Tensors for which to create Keras metadata. processed_ops: Set. TensorFlow operations that have already been wrapped in `TensorFlowOpLayer` instances. created_layers: List. The `TensorFlowOpLayer` instances created. Returns: Tuple. First element is the updated set of TensorFlow Operations that have been wrapped in `TensorFlowOpLayer` instances. Second element is a list of the `TensorFlowOpLayer` instances created. zB`create_keras_history` should only be called if eager is disabled!r) base_layerr;N) constantsr )argskwargsoutputsz weights_mult = lambda x: tf.sparse.sparse_dense_matmul(x, weights) output = tf.keras.layers.Lambda(weights_mult)(input) aTensorflow ops that generate ragged or sparse tensor outputs are currently not supported by Keras automatic op wrapping. Please wrap these ops in a Lambda layer: ``` {example} ``` Sparse ops encountered: {sparse_ops} Ragged tensors encountered: {ragged_tensors} )example sparse_opsragged_tensors)(r%#executing_eagerly_outside_functions ValueError keras.enginerNr#r6rCr2 isinstance SparseTensorSparseTensorValueappendopr is_raggedlistinputs enumerateuses_keras_history distributein_cross_replica_contextrGraphOrParentsInXlaContextget_default_graph_UNSAFE_GRAPH_OP_LAYER_CREATION init_scoperfunctionunnest_if_single_tensorrIr node_defSerializeToStringTensorFlowOpLayer_set_connectivity_metadatarRupdateformatstr)rD processed_opsrLrN tensor_listrTrUtensorr] op_inputsrO layer_inputsiop_inputds_with_session using_xlar rkop_layerlambda_examplerrrrIs           6     rIcCs.tj|}t|ts*t|dkr*|d}|S)Nr)r#r6rCrYdictlen)r8flat_input_tensorsrrrrjFs rjcCs:tj|}tjr|sdStdd|Dr2dSt|S)aCheck if any Tensors need to be wrapped in TensorFlowOpLayers. This will never return True inside a sublayer, because sublayers do not need to create Keras History. Otherwise, this returns True if one or more of `tensors` originates from a `keras.Input` and does not have `_keras_history` set. Args: tensors: An arbitrary nested structure of Tensors. ignore_call_context: Whether to ignore the check of if currently outside of a `call` context. This is `True` when creating KerasHistory inside `Node`, where we always know that Tensors are being used with the Functional API. Returns: Bool, whether at least one Tensor needs to be wrapped. Fcss|]}t|ddduVqdSr:r1)r?rtrrrr@fsz&needs_keras_history..)r#r6rC call_contextin_callrBrb)rDignore_call_contextr8rrrneeds_keras_historyQs  rcCstjS)z7Returns if currently executing inside of a Keras graph.)rin_keras_graphrrrris_in_keras_graphosrcCstp tS)z4Returns if in eager mode or inside of a tf.function.)r#executing_eagerlyis_in_tf_functionrrrris_in_eager_or_tf_functiontsrcCsJts dStsdStr"dSt}t|ddrF|jdrFdSdS)z#Returns if inside of a tf.function.Fr wrapped_functionT) r%rVr#inside_functionrrfr2r startswith)graphrrrrysrc Cst}tj|}|rg}|D]n}t||vr0q|t|t|dddurPqt|dddurfdSz||jj Wqt yYq0q|}qt |dS)a*Check if at least one Tensor originates from a `keras.Input`. This is `True` if at least one Tensor has its origin in a `keras.Input`. Any Tensor that originates from a `keras.Input` will have a dependency Tensor with a `_keras_history` attribute attached. Tensors that have already been checked to not originate from a `keras.Input` are marked as `_keras_history_checked`. Args: tensors: An arbitrary nested structure of Tensors. Returns: Bool, whether at least one Tensor originates from a `keras.Input`. _keras_history_checkedNr;TF) rJr#r6rCidaddr2extendr]r`AttributeError mark_checked)rDchecked_tensorstensors_to_checknew_tensors_to_checkrtrrrrbs&   rbcCsdd}tj||dS)zMarks that these Tensors should not be tracked. This prevents Layers from attempting to create TensorFlowOpLayers for these Tensors. Args: tensors: An arbitrary structure of Tensors. cSs d|_dS)NT)rrtrrr _mark_checkedsz#mark_checked.._mark_checkedNr5)rDrrrrrs rcCs$ttdd}|dur t}|t_|S)z'Returns currently active `CallContext`.rN)r2 _call_context CallContextr)call_ctxrrrrs  rc@sveZdZdZddZdddZeddZed d Zed d Z ed dZ eddZ eddZ eddZ dS)rasKeeps track of properties currently inside a Layer/Model's `call`. Attributes: in_call: Whether currently inside the `call` of a Layer. layer: The `Layer` whose `call` is currently active. inputs: The inputs to the currently active `Layer`. build_graph: Whether currently inside a Graph or FuncGraph. training: Whether currently executing in training or inference mode. saving: Whether currently saving to SavedModel. frozen: Whether currently executing inside a `Layer` with `trainable` set to `False`. in_keras_graph: Whether executing inside the Keras Graph. cCs"d|_dddddd|_d|_dS)NFlayerr` build_graphtrainingsaving)r_state_in_keras_graphselfrrr__init__s zCallContext.__init__NcCs|||||d}t||S)aPush a Layer and its inputs and state onto the current call context. Args: layer: The `Layer` whose `call` is currently active. inputs: The inputs to the currently active `Layer`. build_graph: Whether currently inside a Graph or FuncGraph. training: Whether currently executing in training or inference mode. saving: Whether currently saving to SavedModel. Returns: Context manager. r)CallContextManager)rrr`rrrstaterrrenterszCallContext.entercCs |jdS)Nrrrrrrr szCallContext.layercCs |jdS)Nr`rrrrrr`szCallContext.inputscCs |jdSNrrrrrrrszCallContext.build_graphcCs |jdS)NrrrrrrrszCallContext.trainingcCs |jdS)NrrrrrrrszCallContext.savingcCs|jd}|sdS|j S)NrF)rr)rrrrrfrozens zCallContext.frozencCs&tr dS|jp$ttdddkS)NFr keras_graph)r#rrr2r get_graphrrrrr&s zCallContext.in_keras_graph)N)__name__ __module__ __qualname____doc__rrpropertyrr`rrrrrrrrrrs"       rc@s(eZdZdZddZddZddZdS) rz"Context manager for `CallContext`.cCs||_||_|d|_dSr) _call_ctxr _build_graph)rrrrrrr5szCallContextManager.__init__cCsR|j}|j|_|j|_d|_|j|_|jrN|j|_|jpJtt dddk|_dS)NTr r) rr _prev_in_callr _prev_staterr_prev_in_keras_graphr2rr)rrrrr __enter__:szCallContextManager.__enter__cGs(|j}|j|_|j|_|jr$|j|_dSN)rrrrrrrr)rexc_inforrrr__exit__Ks zCallContextManager.__exit__N)rrrrrrrrrrrr2srcCs6tt|jdd|}||d|vo4|dduS)zDReturns whether a user passed the `training` argument in `__call__`.Nr)r~ziprPro)argspecrPrQ full_argsrrrtraining_arg_passed_to_callTs rcCs |jddko|jddkS)zEReturns True if the object is a subclassed layer or subclassed model.z keras.enginez keras.layersrfindrrrr is_subclassed\srcCs|jddkS)z6Returns whether the layer is loaded from a SavedModel.zkeras.saving.saved_modelrrrrrrfrom_saved_modeldsradd_losscCs|strt|dr|jjr|dkrDd}d}td|d|d|dkrVd }d }n|d krhd }d }nd}d}tdj|||ddS)agChecks that tensors passed to `add_*` method match the Keras graph. When one of the `add_*` method is called inside a V2 conditional branch, the underlying tensor gets created in a FuncGraph managed by control_flow_v2. We need to raise clear error messages in such cases. Args: tensor: Tensor to check, or `False` if it is known that an error should be raised. method: Caller method, one of {'add_metric', 'add_loss', 'add_update'}. force_raise: If an error should be raised regardless of `tensor`. Raises: RuntimeError: In case of an out-of-graph tensor. ractivity_regularizerag class TestModel(tf.keras.Model): def __init__(self): super(TestModel, self).__init__(name='test_model') self.dense = tf.keras.layers.Dense(2, activity_regularizer='l2') def call(self, x, training=None): if training: return self.dense(x) else: return self.dense(x) a class TestModel(tf.keras.Model): def __init__(self): super(TestModel, self).__init__(name='test_model') self.dense = tf.keras.layers.Dense(2, activity_regularizer='l2') def call(self, x, training=None): return self.dense(x) zRYou are using a layer with `activity_regularizer` in a control flow branch, e.g.: z This is currently not supported. Please move your call to the layer with `activity_regularizer` out of the control flow branch, e.g.: aC You can also resolve this by marking your outer model/layer dynamic (eager-only) by passing `dynamic=True` to the layer constructor. Any kind of control flow is supported with dynamic layers. Note that using `dynamic=True` requires you to implement static shape inference in the `compute_output_shape(input_shape)` method. add_metricz def call(self, inputs, training=None): if training: metric = compute_metric(inputs) self.add_metric(metric, name='my_metric', aggregation='mean') return inputs z def call(self, inputs, training=None): if training: metric = compute_metric(inputs) else: metric = 0. self.add_metric(metric, name='my_metric', aggregation='mean') return inputs rz def call(self, inputs, training=None): if training: loss = compute_loss(inputs) self.add_loss(loss) return inputs z def call(self, inputs, training=None): if training: loss = compute_loss(inputs) else: loss = 0. self.add_loss(loss) return inputs z def call(self, inputs, training=None): if training: self.add_update(self.w.assign_add(1)) return inputs z def call(self, inputs, training=None): if training: increment = 1 else: increment = 0 self.add_update(self.w.assign_add(increment)) return inputs aYou are using the method `{method}` in a control flow branch in your layer, e.g.: {bad_example} This is not currently supported. Please move your call to {method} out of the control flow branch, e.g.: {correct_example} You can also resolve this by marking your layer as dynamic (eager-only) by passing `dynamic=True` to the layer constructor. Any kind of control flow is supported with dynamic layers. Note that using `dynamic=True` requires you to implement static shape inference in the `compute_output_shape(input_shape)` method.)method bad_examplecorrect_exampleN)r%rVr=ris_control_flow_graph RuntimeErrorrp)rtr force_raiserrrrrcheck_graph_consistencyis@      rcsfdd}tj||S)z@Marks `outputs` as the return values for automatic control deps.csZt|s|S|}t|dddur8|j|_nd|_t|dddurV|j|_|S)z>Marks `tensor` as the return value for automatic control deps.r0N_tfp_distribution)r# is_tensormark_as_returnr2r0r)rt return_tensoracdrr_mark_as_returns  z'mark_as_return.._mark_as_returnr5)rRrrrrrrs rz%keras.layers.enable_v2_dtype_behavior)v1cCsdadS)aEnable the V2 dtype behavior for Keras layers. By default, the V2 dtype behavior is enabled in TensorFlow 2, so this function is only useful if `tf.compat.v1.disable_v2_behavior` has been called. Since mixed precision requires V2 dtype behavior to be enabled, this function allows you to use mixed precision in Keras layers if `disable_v2_behavior` has been called. When enabled, the dtype of Keras layers defaults to floatx (which is typically float32) instead of None. In addition, layers will automatically cast floating-point inputs to the layer's dtype. >>> x = tf.ones((4, 4, 4, 4), dtype='float64') >>> layer = tf.keras.layers.Conv2D(filters=4, kernel_size=2) >>> print(layer.dtype) # float32 since V2 dtype behavior is enabled float32 >>> y = layer(x) # Layer casts inputs since V2 dtype behavior is enabled >>> print(y.dtype.name) float32 A layer author can opt-out their layer from the automatic input casting by passing `autocast=False` to the base Layer's constructor. This disables the autocasting part of the V2 behavior for that layer, but not the defaulting to floatx part of the V2 behavior. When a global `tf.keras.mixed_precision.Policy` is set, a Keras layer's dtype will default to the global policy instead of floatx. Layers will automatically cast inputs to the policy's compute_dtype. TNV2_DTYPE_BEHAVIORrrrrenable_v2_dtype_behaviors rz&keras.layers.disable_v2_dtype_behaviorcCsdadS)ztDisables the V2 dtype behavior for Keras layers. See `tf.compat.v1.keras.layers.enable_v2_dtype_behavior`. FNrrrrrdisable_v2_dtype_behavior(srcCstdurtjjStS)z1Returns True if the V2 dtype behavior is enabled.N)rr# __internal__tf2enabledrrrrv2_dtype_behavior_enabled2s rc@s<eZdZdZddZeddZddZdd Zd d Z d S) TrackableWeightHandleraKeras wrapper for handling tracking.Trackable object saving and restoring. This class handles Trackables in both V1 and V2 modes, ensuring that they can be saved and restored with the correct data and without adding additional ops on every save. Attributes: trackable: The trackable to wrap. num_tensors: The number of tensors that this trackable requires for saving. csHt|tjjjst|d|_tj_ tjj | }|sbd_ dd_ dd_nt|dkr*t|d}tr|_tj_ fdd_ fdd_npg_|_tjj_ jjD]"}|j}jt|j|jq܈jjd_j_ fd d_ntd |d t|d dS) Nz is not a Trackable object.rcSsdSrrweightsrrrSrAz1TrackableWeightHandler.__init__..cSsgSrrrrrrrTrAr}cs|dSr) _saveablerestorerrrrrascsddjDS)NcSsg|] }|jqSrrr?specrrr dsETrackableWeightHandler.__init__....rspecsrrrrrdscsddjjDS)NcSsg|] }|jqSrrrrrrrxsrrrrrrrxsz?Only Trackables with one Saveable are supported. The Trackable z has z Saveables.)rYr#rtracking TrackablerW _trackablerc get_strategy_distribute_strategysaveable_objects_from_trackablevalues _num_tensors_setter_getterrr_r%rVrr_placeholder_tensorsrtr\rFr rr _assign_op_set_weights_v1)r trackable saveablessaveablerrtrrrrFsL     zTrackableWeightHandler.__init__cCs|jSr)rrrrr num_tensorssz"TrackableWeightHandler.num_tensorscCs@t||jkr2td|jd|jdt|d||dS)NzWeight handler for trackable z3 received an incorrect number of weights: expected z weights, got z weights.)rrrWrr)rrrrr set_weightss z"TrackableWeightHandler.set_weightscCs|Sr)rrrrr get_tensorssz"TrackableWeightHandler.get_tensorscCs:i}t|D]\}}|||j|<q t|j|dSr)rarr get_sessionrunr)rr feed_dictidxrtrrrrsz&TrackableWeightHandler._set_weights_v1N) rrrrrrrrrrrrrrr9s ;  rcCs8tj|}tdd|Dr4td|d|ddS)Ncss|]}t|tjVqdSr)rYr# RaggedTensorr>rrrr@rAz$no_ragged_support..zLayer z; does not support RaggedTensors as input. Inputs received: z@. You can try converting your input to a dense (uniform) tensor.)r#r6rCanyrW)r` layer_name input_listrrrno_ragged_supports rcCst|dpt|dS)z@Returns True if `v` is a PartionedVariable or a ShardedVariable._variable_list _variablesr<)vrrris_split_variablesr cCs(t|}t|do&t|do&t|t S)Ntrainable_weightsnon_trainable_weights)typer=rY)objobj_typerrr has_weightss   rz8This layer's losses have been added to the parent layer.)N)F)NrF)>rr threadingtensorflow.compat.v1compatrr%tensorflow.compat.v2v2r#r r keras.dtensorrr' keras.utilsrrr tensorflow.python.util.tf_exportrlocalrrfloat32VariableSynchronizationAUTOVariableAggregationNONEr/r9rErHrMrgrIrjrrrrrbrrrregister_call_context_functionrrrrrrrrrrrrrr rREVIVED_LOSS_PLACEHOLDERrrrrsz        x!f - ["   "  `