a ==ici5 @sddlZddlmZmZmZmZmZmZmZm Z m Z ddl m Z ddl mZedddZe jeeee e jeddd Ze jeed d d Ze jeeed ddZedddfe jeeeeee e jeee e jedddZe je jhe_e jhe_ejeeeddddZe jee jdddZe jee jdddZd$e jeeee jee j eeee jefdd d!Z!Gd"d#d#Z"dS)%N) AnyCallableDict GeneratorOptionalSetTupleTypecast) _BatchNormreturncOsdS)z A simple wrapper policy that always returns ``True``, i.e. when passed as the `auto_wrap_policy` into FSDP, this will result in all submodules being wrapped as distinct FSDP instances. T)argskwargsrrl/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.pyalways_wrap_policysr)modulerecurseunwrapped_paramstransformer_layer_clsr cCs|rdSt|t|SdS)ae A convenient auto wrap policy for transformer models. If the submodule is an instance of transformer_layer_cls, the submodule will be wrapped as a FSDP unit. Otherwise, all the other remainder submodules are wrapped by the outermost FSDP unit. Right now, FSDP requires submodules that share weights to be wrapped in the same FSDP unit, this auto wrap policy can conviniently wrap the shared embeddings into the same FSDP unit for transformer models. In the near future, FSDP will support submodules that share weights to be wrapped in the separated FSDP units. Return if a module should be wrapped during FSDP auto wrapping. The first three parameters are required by :func:`_recursive_wrap`. Args: module (nn.Module): The module to be considered in this decision. recurse (bool): Indicate if this is called to make a decision on whether we should recurse down a subgraph of the module structure. If False, it means this function is called to make a decision on whether we should wrap the said module. unwrapped_params (int): The number of parameters yet to be wrapped in this module. transformer_layer_cls (int): Submodules with one of the `transformer_layer_cls` names will be wrapped as seperated FSDP units TN) isinstancetuple)rrrrrrrtransformer_auto_wrap_policy s$r)rrr cOs|rdSt|tSdS)zM A policy that wraps ``BatchNorm`` instances in their own FSDP unit. TN)rr )rrrrrrr_wrap_batchnorm_individuallyKs r)rrrr cstfdd|DS)zv A policy that wraps ``module`` if any policy in the passed in iterable of ``policies`` returns ``True``. c3s|]}|VqdSNr).0policyrrrrr fsz_or_policy..)any)rrrZpoliciesrrr _or_policy\s r!gחA)rrrmin_num_paramsforce_leaf_modulesexclude_wrap_modulesr cCs\|durtjn|}|dur tjn|}||k}|rD|oBt|t| S|oVt|t| SdS)aA size based auto_wrap_policy function for FSDP API. Return if a module should be wrapped during FSDP auto wrapping. The first three parameters are used by :func:`_recursive_wrap`. If you write a custom version of this policy function, your version needs to at least accept the first three parameters and free to do whatever you want in the function. Args: module (nn.Module): The module to be considered in this decision. recurse (bool): Indicate if this is called to make a decision on whether we should recurse down a subgraph of the module structure. If False, it means this function is called to make a decision on whether we should wrap the said module. unwrapped_params (int): The number of parameters yet to be wrapped in this module. min_num_params (int): Customizable policy input. It controls the size threshold on how big should a module be to be considered wrapped. force_leaf_modules (Set[Type[nn.Module]]): set of module types to keep as leaves, i.e., their children will never be wrapped. exclude_wrap_modules (Set[Type[nn.Module]]): Customizable set of module types to be excluded in wrapping. N)size_based_auto_wrap_policyFORCE_LEAF_MODULESEXCLUDE_WRAP_MODULESrr)rrrr"r#r$Zis_largerrrr%ks'r%)NNN) wrapper_clswrapper_kwargsr cksHid|i|}tfi|dVWdn1s:0YdS)a Context manager to wrap modules using a wrapper. Useful for when you'd like to apply the same configuration arguments to all child modules that you wrap. A particularly important use case is wrapping large layers so that they get sharded (in-place) during initialization, to avoid running out of system memory. Large layers can indicate that they should be sharded via the ``wrap`` annotation and this context manager can provide the exact configuration for these nested instances. Usage:: with enable_wrap(wrapper_cls, **params): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: wrapper_cls: Class that `wrap` annotation will `wrap` modules with, such as `FullyShardedDataParallel`. **wrapper_kwargs: Configuration settings that will be passed to all ``wrap`` instances inside the context r(N)_ConfigAutoWrap)r(r)rrrr enable_wrapsr+)rwrap_overridesr cKs:tjr6tjdusJitj|}t|tjfi|S|S)a Annotate that a module should be wrapped. Annotated modules will only be wrapped if inside of an :func:`enable_wrap` context manager. This allows a module to be initialized both with and without a wrapper without code change. The class that this function wraps the passed in ``nn.Module`` with is the passed in ``wrapper_cls`` argument into ``enable_wrap``. Both ``enable_wrap`` and ``wrap`` can take in kwargs specifying how to construct the ``wrapper_cls`` instance. In the case of duplicate kwargs in ``enable_wrap`` and ``wrap``, the argument passed into ``wrap`` will be respected. Usage:: with enable_wrap(wrapper_cls=FSDP, **fsdp_config): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: module (nn.Module): module to wrap (if in :func:`enable_wrap` context) **wrap_overrides: configuration overrides that will take priority over the values provided by the :func:`enable_wrap` context N)r*in_autowrap_contextr(r_wrap)rr,rrrwrapsr/)rr(r cKsD|dus Jt|dr4i||j}||fi|S||fi|S)N_wrap_overrides)hasattrr0)rr(rZ overridesrrrr.s   r.F)rauto_wrap_policyr(ignored_modulesignored_paramsonly_wrap_childrenrr c s|dusJd|dus Jd|D]&\}}||vr:q(t|tt|r(Jq(tfdd|D} |dusvJ||d| drd} |D]D\} }||vrqtf||||d |\} } t|| | | | 7} q| | }|s ||d |dr t ||fi|| fS|| fS|dfS) a Automatically wrap child modules of *module* that meet the given criteria with :func:`auto_wrap`. Does not rely on _ConfigAutoWrap. Args: module (nn.Module): module to recursively wrap auto_wrap_policy (Callable): A callable specifying a policy to recursively wrap layers with FSDP. ignored_modules (Set[torch.nn.Module]): Modules to ignore when wrapping. ignored_params (Set[torch.nn.Parameter]): Parameters to ignore when wrapping; these should be the parameters contained in the modules in ``ignored_modules``. Returns: (nn.Module, int): Wrapped module and the number parameters wrapped recursively. NzMust specify auto_wrap_policy.zMust specify wrapper_clsc3s|]}|vr|VqdSr)Znumel)rpr4rrr"sz"_recursive_wrap..Trr)rr2r(r3r4F) Z named_modulesrr typesum parametersZnamed_children_recursive_wrapsetattrr.)rr2r(r3r4r5r_childZ num_paramsZtotal_wrapped_paramsnameZ wrapped_childZnum_wrapped_params remainderrr7rr;sB     r;c@seZdZUdZdZeed<dZee ed<iZ e e e fed<e e e fddd Zee dd d d Zedd ddZdd ddZe e e ddddZdS)r*z Helper class to wrap modules based on default config args via a context manager. See :func:`enable_wrap` for more information. Fr-Nr(rrcKs ||_dSrrA)selfrrrr__init__Osz_ConfigAutoWrap.__init__)rr cCsHtjrtddt_d|vs(Jdtt|dt_|d=|t_dS)Nz]You are already within an autowrap context and we currently do not supported nested autowrap.Tr(z9Expected to pass in wrapper_cls arg into _ConfigAutoWrap.)r*r-NotImplementedErrorkeysr rr(rrArrrenable_autowrap_contextRs z'_ConfigAutoWrap.enable_autowrap_contextr cCsdt_dt_it_dS)NF)r*r-r(rrrrrdisable_autowrap_contextbsz(_ConfigAutoWrap.disable_autowrap_contextcCs||jdSr)rFr)rBrrr __enter__hsz_ConfigAutoWrap.__enter__)exc_typeexc_valexc_tbr cCs |dSr)rG)rBrIrJrKrrr__exit__ksz_ConfigAutoWrap.__exit__)__name__ __module__ __qualname____doc__r-bool__annotations__r(rrrrstrrrC staticmethodrFrGrHrLrrrrr*Es  r*)F)# contextlibtypingrrrrrrrr r Ztorch.nnnnZtorch.nn.modules.batchnormr rQrModuleintrrr!r%Z ModuleListZ ModuleDictr'ZMultiheadAttentionr&contextmanagerr+r/r. Parameterr;r*rrrrsf,     ,   : #% F