a SicY@s*dZddlZddlZddlZddlZddlmmZddl m Z ddl m Z ddl m Z ddlmZddlmZddlmZgd Zead d Zed gd GdddejjZejjej_edgd eddejddZddZddZ ddZ!ddZ"ddZ#ddZ$d d!Z%d"d#Z&dS)$z5Library for map layout and corresponding tf.Variable.N) dtensor_api) lazy_variable)utils) base_layer) deprecated) keras_export)_self_tracked_trackables_trainable_weights_non_trainable_weights_captured_weight_regularizercCs ttddS)N layout_map)getattr _LAYOUT_MAPrrT/var/www/html/django/DPS/env/lib/python3.9/site-packages/keras/dtensor/layout_map.pyget_current_layout_map1srz$keras.dtensor.experimental.LayoutMap)v1c@sReZdZdZdddZddZddZd d Zd d Zd dZ ddZ ddZ dS) LayoutMapaA dict-like object that maps string to `Layout` instances. `LayoutMap` uses a string as key and a `Layout` as value. There is a behavior difference between a normal Python dict and this class. The string key will be treated as a regex when retrieving the value. See the docstring of `get` for more details. See below for a usage example. You can define the naming schema of the `Layout`, and then retrieve the corresponding `Layout` instance. To use the `LayoutMap` with a `Model`, please see the docstring of `tf.keras.dtensor.experimental.layout_map_scope`. ```python map = LayoutMap(mesh=None) map['.*dense.*kernel'] = layout_2d map['.*dense.*bias'] = layout_1d map['.*conv2d.*kernel'] = layout_4d map['.*conv2d.*bias'] = layout_1d layout_1 = map['dense_1.kernel'] # layout_1 == layout_2d layout_2 = map['dense_1.bias'] # layout_2 == layout_1d layout_3 = map['dense_2.kernel'] # layout_3 == layout_2d layout_4 = map['dense_2.bias'] # layout_4 == layout_1d layout_5 = map['my_model/conv2d_123/kernel'] # layout_5 == layout_4d layout_6 = map['my_model/conv2d_123/bias'] # layout_6 == layout_1d ``` Args: mesh: An optional `Mesh` that can be used to create all replicated layout as default when there isn't a layout found based on the input string query. NcCst|_||_dSN) collections OrderedDict _layout_map _default_mesh)selfmeshrrr__init__Ys zLayoutMap.__init__cCs>||jvr|j|S|jD]}t||r|j|SqdS)aRetrieve the corresponding layout by the string key. When there isn't an exact match, all the existing keys in the layout map will be treated as a regex and map against the input key again. The first match will be returned, based on the key insertion order. Return None if there isn't any match found. Args: key: the string key as the query for the layout. Returns: Corresponding layout based on the query. N)rrematch)rkeykrrr __getitem__]s     zLayoutMap.__getitem__cCsT||jvr$t|d|j|dt|tjsFt|dt|||j|<dS)Nz+ already exist in the LayoutMap with value z.. Please make sure to not use duplicated keys.z& should be a dtensor.Layout type, got )r ValueError isinstancedtensorLayouttype)rrlayoutrrr __setitem__ss  zLayoutMap.__setitem__cCs |j|Sr)rpop)rrrrr __delitem__szLayoutMap.__delitem__cCs t|jSr)lenrrrrr__len__szLayoutMap.__len__cCs t|jSr)iterrr+rrr__iter__szLayoutMap.__iter__cCs|jS)zReturn the default `Mesh` set at instance creation. The `Mesh` can be used to create default replicated `Layout` when there isn't a match of the input string query. )rr+rrrget_default_meshszLayoutMap.get_default_meshcCst|S)a Apply layout to all `tf.Variable` instances created under the scope. All `tf.Variable` instances created under this scope will be lazily initialized first. Once they are attached as the model or layer attributes, and there is a stable layout mapping for it, the variables will be reinitialized into a `tf.experimental.dtensor.DVariable` with corresponding layout. Note that the layout mapping will use object/attribute names as the keys to map the variable to the layout. For subclassed models, the full object/attribute name is used as the key. For Functional/Sequential models, we use `layer.name` as the key for the layer, followed by the attribute name. Keras ensures name uniqueness among the layers within a Functional/Sequential model. See the following examples that show variable object names for different Keras model types: ```python layout_map = layout_map_lib.LayoutMap(mesh=self.mesh) layout_map['d1.kernel'] = layout_1 layout_map['d1.bias'] = layout_2 layout_map['d2.kernel'] = layout_3 layout_map['d2.bias'] = layout_4 ## Subclassed model class SubclassModel(tf.keras.Model): def __init__(self, name=None): super().__init__(name=name) self.d1 = tf.keras.layers.Dense(1000) self.d2 = tf.keras.layers.Dense(1000) def call(self, inputs): x = self.d1(inputs) return self.d2(x) with layout_map.scope(): model = SubclassModel() inputs = tf.zeros((10, 10)) results = model(inputs) model.d1.kernel.layout == layout_1 model.d1.bias.layout == layout_2 model.d2.kernel.layout == layout_3 model.d2.bias.layout == layout_4 ## Functional model with layout_map.scope(): inputs = tf.keras.Input((10,), batch_size=10) x = tf.keras.layers.Dense(20, name='d1')(inputs) output = tf.keras.layers.Dense(30, name='d2')(x) model = tf.keras.Model(inputs, output) d1 = model.layers[1] d2 = model.layers[2] d1.kernel.layout == layout_1 d1.bias.layout == layout_2 d1.kernel.layout == layout_3 d1.bias.layout == layout_4 ## Sequential model with layout_map.scope(): model = tf.keras.Sequential([ tf.keras.layers.Dense(20, name='d1', input_shape=(10,)), tf.keras.layers.Dense(30, name='d2') ]) d1 = model.layers[0] d2 = model.layers[1] d1.kernel.layout == layout_1 d1.bias.layout == layout_2 d1.kernel.layout == layout_3 d1.bias.layout == layout_4 ``` Returns: A context that will lazily initialize all `tf.Variable` objects within the model, with their attributed layouts. )layout_map_scoper+rrrscopesUzLayoutMap.scope)N) __name__ __module__ __qualname____doc__rr r'r)r,r.r/r1rrrrr5s" rz+keras.dtensor.experimental.layout_map_scopez Layout. When a layout is not found for the variable, a default all replicated layout will be created for the variable. Yields: A context that will lazily initialize all `tf.Variable` objects within the model, with their attributed layouts. N)rrr rZlazy_init_scope)r Zprevious_layout_maprrrr0s _ r0csi}|jtddD]T\}fddtDr.qdddD}t|||}t||||t|<q|jddd D]}t||qx|jtddD] \}|t|}t||qt||t |||S) z0Map/Replace LazyInitVariable for subclass model.T predicate with_pathcsg|]}|vr|qSrr.0apathrr ez0_map_subclass_model_variable...cSsg|] }t|qSrstrr:itemrrrr>hr?cSs t|tjSr)r"rLayerorrror?z._map_subclass_model_variable..r7) _flatten_is_lazy_init_variable_KERAS_ATTRIBUTES_TO_SKIPjoin_create_dvariable_set_object_by_pathid _config_dvariable_regularization_init_state_variable_for_rng_update_trackable_reference)modelr %lazy_init_variable_to_tf_variable_mapvariable object_path new_variablelayer tf_variablerr<r_map_subclass_model_variableYs4      r[c si}|jD]}|j}|jtddD]`\}fddtDr>q"dddD}|d|}t|||}t||||t|<q"t |||jtddD] \}|t|}t||qq t ||t |||S)z=Map/Replace LazyInitVariable for functional/sequential model.Tr6csg|]}|vr|qSrrr9r<rrr>r?z2_map_functional_model_variable..r@cSsg|] }t|qSrrArCrrrr>r?) layersnamerJrKrLrMrNrOrPrQrRrS) rTr rUrY layer_namerVrWrXrZrr<r_map_functional_model_variables4       r_c Cs|jdddD]}|j}|jr2|jdur2tdt|dr^t|jjr^t|d|jj|j_qt | | Wdq1s0YqdS)aIInit the state variable in tf.ranodm.Generator. Since the BaseRandomLayer in keras explicitly untrack the tf.random.Generator, the variable in it will stay as LazyInitVariable, which cause runtime error if we don't replace them with proper DVariable. Since user usually are not aware the existence of those variable, we will just give them replicated layout since they are tiny. Args: model: the model whose layers will be checked to find the BaseRandomLayers. layout_map: used to get the default mesh information to create DVariable. cSs t|tjSr)r"rZBaseRandomLayerrFrrrrHr?z._init_state_variable_for_rng..rINzKeras is expected to use tf.random.Generator when using DTensor API. Please call `tf.keras.backend.experimental.enable_tf_random_generator` at the beginning of your program. _generator) rJ_random_generator_builtr`r!hasattrrK _state_varrNr#run_onr/ _maybe_init)rTr lZkeras_generatorrrrrRs    rRcCsL|jD]:\}}}t|s&td||t|}||||qg|_dS)a`Update the weights regularizer for newly created `DVariable`. The weight regularization usually happens when `layer.add_weight()` is called, at which point the library will first create a `LazyInitVariable`, and then replace it with a `DVariable`. We will defer the creation of those losses, until the DVariable is created. See `layer._captured_weight_regularizer` for more details. Args: layer: the layer instance for DVariable regularization config. lazy_init_variable_to_tf_variable_map: the dict between LazyInitVariable ID and newly created DVariable. zFExpect the regularization loss are created from LazyInitVariable, got N)r rKr!rP_handle_weight_regularization)rYrUr]rVZ regualarizer d_variablerrrrQs rQcCs||}|dur,|jj}tjj||d}|j}t|rpt t ||}Wdq|1sd0Yn t ||}|j }|dr|dd}tj||j|d}|S)aCreate a new variable instead of using the LazyInitVariable. We choose to do this since even the LazyInitVariable might behavior like a normal tf.Variable/DVariable, it is not future proof for any new changes to variable class. It will also fail the instance type check in python, which could affect user's code when they do any filtering based on type to find any variables. Args: layout_map: a LayoutMap which contains the variable_object_path (string) -> Layout. object_path: string, the object attribute path for the variable. variable: LazyInitVariable which will be replaced by the newly created tf.Variable. Returns: A new tf.Variable with correct layout information. N)rrankz:0) trainabler])shaperkr#r$ replicatedr/_initial_valuecallablerZdisable_init_variable_creatorrcall_with_layout copy_to_meshr]endswith DVariablerm)r rWrVr&Z variable_rankinit_val variable_namerXrrrrNs$ ,   rNcCsft|D]X\}}|t|dkrBt|tr4|||<q`t|||qt|trV||}qt||}qdS)a5Set the attribute of instance to the object. Args: object_to_set: the instance whose attribute should be set. path: the tuple/list of string and ints, representing the attribute names. Int means that the attribute to set is a item a list. value: the value of the attribute. N) enumerater*r"intsetattrr )Z object_to_setr=valuei attr_namerrrrO$s     rOcCs^tjj|}|\}}|D]:}|D](\}}t|r.|j|t ||ddq.qdS)aUpdate the trackable object references for the model. Note that this method is only needed because of a corner case for model checkpoint, where it could accidently catch a LazyInitVariable in checkpoint dependency and not visible to the model attribute graph itself. Args: model: the keras model instance whose checkpoint dependency will be examed. lazy_init_variable_to_tf_variable_map: the dict between LazyInitVariable ID and newly created DVariable. T) overwriteN) tf __internal__trackingObjectGraphViewbreadth_first_traversal_trackable_childrenitemsrK_track_trackablerP)rTrUZ object_graph trackables_ trackableZref_namerefrrrrS?s  rScCs t|tjSr)r"rZLazyInitVariable)objrrrrKZsrK)'r5r contextlibr threadingtensorflow.compat.v2compatv2r keras.dtensorrr#rr keras.enginer"tensorflow.python.util.deprecationr tensorflow.python.util.tf_exportrrLlocalrrabcMutableMappingrr getcontextmanagerr0r[r_rRrQrNrOrSrKrrrrs@       7  e).)-