a NSicz@sddlZddlZddlmZddlmZddlmZ ddl m Z m Z m Z gZddZddZdd d Zd d Zeje ejeed ddZe ee feeddddZdS)N) _get_stream) is_namedtuple)DictAnyListcs,fddz|}Wdnd0|S)z4 Recursively moves input to the target_gpu. c sttjrˆjtdkr$fSs4fSt}tj|}Wdn1sh0Ytj.tj}| || |Wdn1s0Y|fSt rfddt t DSttrtdkrtt t Sttr$gSttjjrtdkrzfddt t DWStyddt t DYS0ttjjrtdkrz"fddt t DWStyddt t DYS0gS) Ncudacsg|]}t|qStype).0argsobjrS/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/distributed/utils.py (z1_recursive_to..to_map..rcsg|]}t|qSrr r ir rrr1rcSsg|] }t|qSr)listrrrrr4rcsg|]}t|qSrr rr rrr7rcSsg|] }t|qSr)dictrrrrr:r) isinstancetorchTensordevicetorrstreamcurrent_stream wait_stream record_stream_is_namedtuplezipmaptuplelenrstr collectionsabcSequence TypeErrorMappingitems)rroutputr target_gputo_map!use_side_stream_for_tensor_copiesr rr.s<  (  ( ""z_recursive_to..to_mapNr)inputsr-r/resrr,r _recursive_to s ,  r2cCs|rt|||ng}|r$t|||ng}t|t|kr^|ddtt|t|Dn4t|t|kr|ddtt|t|Dt|}t|}||fS)NcSsg|]}dqS)rrr _rrrrQrz_to_kwargs..cSsg|]}iqSrrr3rrrrSr)r2r#extendranger")r0kwargs device_idr/rrr _to_kwargsEs&$r9cCst|||S)N)dist_verify_params_across_processes) process_grouptensorsloggerrrr$_verify_param_shape_across_processesXsr?c Csfg}|D]\}}||vr ||q |D]\}}||vr4||q4t||||dS)a2 Syncs ``module``'s parameters and buffers state so that all ranks contain the same module state across all ranks. Note that this API assumes that all parameter shapes are consistent before running the synchronization. This can be checked with ``_verify_param_shape_across_processes``. N)named_parametersappenddetach named_buffers_sync_params_and_buffers) moduler<broadcast_bucket_sizesrcparams_and_buffers_to_ignore module_statesnameparambufferrrr_sync_module_states[s rMr<rIrFrGcCs t|dkrt||||dS)zu Synchronizes ``module_states`` (list of tensors) across all processes by broadcasting them from rank 0. rN)r#r:_broadcast_coalescedrNrrrrDxs rD) state_dict old_prefix new_prefixreturncCsX||krtdt|D]6}||s,q||t|d}||||<||=qdS)a Replace all keys that match a given old_prefix with a new_prefix (in-place). Usage:: state_dict = {"layer.xyz": torch.tensor(1)} replace_by_prefix_(state_dict, "layer.", "module.layer.") assert state_dict == {"module.layer.xyz": torch.tensor(1)} z*old_prefix and new_prefix must be distinctN) ValueErrorrkeys startswithr#)rPrQrRkeyZnew_keyrrr_replace_by_prefixs  rX)N)r%rtorch.distributed distributedr:Ztorch.nn.parallel._functionsrZ torch.nn.parallel.scatter_gatherrrtypingrrr__all__r2r9r?rM ProcessGrouprintrDr$rXrrrrs(   8