a ==ic50@sVddlZddlmZddlmZddlmZmZejfddZ dejfddZ dejfdd Z ej ejfd d Z ej ejfd d ZejfddZejfddZddejfddZej ejfddZGdddeZGdddeZGdddeZGdddeZGdddeZGd d!d!eZGd"d#d#eZGd$d%d%eZGd&d'd'eZdS)(N)Function)groupReduceOpcCst|||S)a Broadcasts the tensor to the whole group. ``tensor`` must have the same number of elements in all processes participating in the collective. Arguments: tensor (Tensor): Data to be sent if ``src`` is the rank of current process. src (int): Source rank. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Received tensor from the broadcast op. ) _Broadcastapply)tensorsrcrr p/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/torch/distributed/nn/functional.py broadcast sr cCst|||S)aT Gathers a list of tensors in a single process. Arguments: tensor (Tensor): Input tensor. dst (int, optional): Destination rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: tuple[Tensor]: List of appropriately-sized tensors with the gathered data. )_Gatherr)rdstrr r r gathers rcCstj||g|RS)a Scatters a list of tensors to all processes in a group. Each process will receive exactly one tensor and store its data in the ``tensor`` argument. Arguments: tensors (list[Tensor]): List of tensors to scatter on the source rank. Receivers must pass ``None`. src (int, optional): Source rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output tensor from the scatter operation. )_Scatterr)tensorsrrr r r scatter,srcCst||||S)a Reduces the tensor data across all machines. Only the process with rank ``dst`` is going to receive the final result. Arguments: tensor (Tensor): Input of the collective. dst (int): Destination rank. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. )_Reducer)rr oprr r r reduce@srcCstj|||g|RS)a Reduces, then scatters a list of tensors to all processes in a group. Arguments: output (Tensor): Output tensor. input_list (list[Tensor]): List of tensors to reduce and scatter. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. )_Reduce_Scatterr)outputZ input_listrrr r r reduce_scatterUsrcCs t||S)a Gathers tensors from the whole group in a list. Arguments: tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. ) _AllGatherr)rrr r r all_gatherhs rcCstj||g|RS)a Each process scatters list of input tensors to all processes in a group and return gathered list of tensors in output list. Arguments: out_tensor_list (list[Tensor]): list of tensors to gather one per rank. input_tensor_list (list[Tensor]): List of tensors to scatter one per rank. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. ) _AlltoAllr)Zoutput_tensor_listinput_tensor_listrr r r all_to_allwsrcCst|||||S)a Each process splits input tensor and then scatters the split list to all processes in a group. Then concatenate the received tensors from all the processes in the group and return single output tensor. Arguments: output (Tensor): Gathered cancatenated output tensor. input (Tensor): Input tensor to scatter. output_split_sizes: (list[Int], optional): Output split sizes for dim 0 if specified None or empty, dim 0 of ``output`` tensor must divide equally by ``world_size``. input_split_sizes: (list[Int], optional): Input split sizes for dim 0 if specified None or empty, dim 0 of ``input`` tensor must divide equally by ``world_size``. Returns: Tensor: Output of the collective. )_AlltoAllSingler)rinputoutput_split_sizesinput_split_sizesrr r r all_to_all_singles r!cCst|||S)a* Reduces the tensor data across all machines in such a way that all get the final result. After the call the returned tensor is going to be bitwise identical in all processes. Arguments: tensor (Tensor): Input of the collective. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective ) _AllReducer)rrrr r r all_reducesr#c@s$eZdZeddZeddZdS)rcCs2||_||_t|_|}tj|||d|S)Nr)rrdistget_rankrankcloner )ctxrrrr r r forwards  z_Broadcast.forwardcCs4t|jtj|j|}|j|jkr*|dd|fSN)rrrrSUMrr'Zzero_)r) grad_outputgxr r r backwards z_Broadcast.backwardN__name__ __module__ __qualname__ staticmethodr*r/r r r r rs rc@s$eZdZeddZeddZdS)r csp||_||_fddttj|dD}tj|d|krVtj|||dntjd||dt|S)Ncsg|]}tqSr )torch zeros_like.0irr r sz#_Gather.forward..r$) r rranger%get_world_size contiguousr&rtuple)r)r rr tensor_listr r:r r*s z_Gather.forwardcGsdtj|j|jg|RfSNNN)rrr r)r) grad_outputsr r r r/sz_Gather.backwardNr0r r r r r s r c@s$eZdZeddZeddZdS)rcsr||_||_tfddDs&Jtd}tj|d|kr\tj|t||dntj|d||d|S)Nc3s"|]}|dkVqdS)rNsizer8trr r z#_Scatter.forward..rr$) rrallr5r6r%r&rlist)r)rrrrr rHr r*sz_Scatter.forwardcCsdt|j|j|SrA)r rrrr)r-r r r r/sz_Scatter.backwardNr0r r r r rs rc@s$eZdZeddZeddZdS)rcCs*||_||_|}tj||||d|SNrr)rrr(r%r)r)rrrrr r r r*s z_Reduce.forwardcCsdt|j|j|fSN)NNN)rrrrrMr r r r/sz_Reduce.backwardNr0r r r r rs rc@s$eZdZeddZeddZdS)rcGs2||_tdd|D}tj|t|||d|S)Ncss|]}|VqdSr+r>rFr r r rIrJz*_Reduce_Scatter.forward..rO)rr?r%rrL)r)rrrrr r r r* sz_Reduce_Scatter.forwardcCsdt|j|SrP)rrrrMr r r r/sz_Reduce_Scatter.backwardNr0r r r r r s rc@s$eZdZeddZeddZdS)rcs@||_fddttj|dD}tj||dt|S)Ncsg|]}tqSr r5 empty_like)r8_r:r r r;sz&_AllGather.forward..r$)rr<r%r=rr>r?)r)rrout_tensor_listr r:r r*s  z_AllGather.forwardcGstj|jdtjjurHt}t||}tj t j |j|g|Rn8dd|D}t j |j|g|R}tj t|dd}d|fS)Nr$cSsg|]}t|qSr rR)r8rr r r r;+rJz'_AllGather.backward..r)dim)r% get_backendrBackendZNCCLr&r5rSrrrr,rsumstack)r)rCr'r.r@Zgxsr r r r/"sz_AllGather.backwardNr0r r r r rs rc@s$eZdZeddZeddZdS)rcs||_fddttj|dD|_tj|d}tddDtj|dtjj urttj|dD].}d}||krt }tj |||||dqhntj |t |dt|S)Ncsg|]}|qSr rDr7rHr r r;5sz%_AlltoAll.forward..r$css|]}|VqdSr+rQrFr r r rI9rJz$_AlltoAll.forward..) rr<r%r=input_tensor_size_listr&r?rWrXZGLOOrLrr)r)rrUrZmy_rankr9to_sendr rHr r*2s$  z_AlltoAll.forwardcs.fdd|jD}dtj|j|gRS)Ncs g|]}tj|djdqS)rdevice)r5emptyr^)r8rErCr r r;Ksz&_AlltoAll.backward..rB)r[rrr)r)rCr@r r`r r/Is z_AlltoAll.backwardNr0r r r r r1s rc@s$eZdZeddZeddZdS)rcCs4||_||_||_||_tj|||||d|S)N)rr r)rrE input_sizerr r%r!)r)rrrr rr r r r*Ss z_AlltoAllSingle.forwardc Cs4tj|j|jd}dt|j||j|j| fS)Nr])NNNN) r5r_rar^rrrrr r>)r)r-rr r r r/bsz_AlltoAllSingle.backwardNr0r r r r rRs rc@s$eZdZeddZeddZdS)r"cCs(||_||_|}tj|||d|SrN)rrr(r%r#)r)rrrr r r r*qs z_AllReduce.forwardcCsdt|j|j|fSrA)r"rrrrMr r r r/ysz_AllReduce.backwardNr0r r r r r"ps r")r5Ztorch.distributed distributedr%Ztorch.autogradrrrZWORLDr rrr,rrrrr!r#rr rrrrrrr"r r r r s0    !