a yµÿf•Fã@sdZddlZddlZddlmZddlmmZddlm Z m Z ddl m Z ddl mZmZmZdZGdd „d ejƒZGd d „d eƒZGd d „d ejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZdS)zTransformer modules.éN)Ú constant_Úxavier_uniform_é)ÚConv)Ú _get_clonesÚinverse_sigmoidÚ#multi_scale_deformable_attn_pytorch) ÚTransformerEncoderLayerÚTransformerLayerÚTransformerBlockÚMLPBlockÚ LayerNorm2dÚAIFIÚDeformableTransformerDecoderÚ!DeformableTransformerDecoderLayerÚ MSDeformAttnÚMLPcs\eZdZdZddde ¡df‡fdd„ Zedd d „ƒZdd d „Z dd d„Z ddd„Z ‡Z S)r z2Defines a single layer of the transformer encoder.ééçFcsžtƒ ¡ddlm}|s"tdƒ‚tj|||dd|_t ||¡|_ t ||¡|_ t  |¡|_ t  |¡|_ t |¡|_t |¡|_t |¡|_||_||_dS)zAInitialize the TransformerEncoderLayer with specified parameters.é)Ú TORCH_1_9z]TransformerEncoderLayer() requires torch>=1.9 to use nn.MultiheadAttention(batch_first=True).T)ÚdropoutZ batch_firstN)ÚsuperÚ__init__Zutils.torch_utilsrÚModuleNotFoundErrorÚnnÚMultiheadAttentionÚmaÚLinearÚfc1Úfc2Ú LayerNormÚnorm1Únorm2ÚDropoutrÚdropout1Údropout2ÚactÚnormalize_before)ÚselfÚc1ÚcmÚ num_headsrr(r)r©Ú __class__©ú^/var/www/html/django/DPS/env/lib/python3.9/site-packages/ultralytics/nn/modules/transformer.pyrs   ÿ     z TransformerEncoderLayer.__init__NcCs|dur |S||S)z2Add position embeddings to the tensor if provided.Nr0©ZtensorÚposr0r0r1Úwith_pos_embed6sz&TransformerEncoderLayer.with_pos_embedc Cst| ||¡}}|j|||||dd}|| |¡}| |¡}| | | | |¡¡¡¡}|| |¡}|  |¡S)z.Performs forward pass with post-normalization.©ÚvalueÚ attn_maskZkey_padding_maskr) r4rr&r#r!rr(r r'r$)r*ÚsrcÚsrc_maskÚsrc_key_padding_maskr3ÚqÚkÚsrc2r0r0r1Ú forward_post;s z$TransformerEncoderLayer.forward_postc Cst| |¡}| ||¡}}|j|||||dd}|| |¡}| |¡}| | | | |¡¡¡¡}||  |¡S)z-Performs forward pass with pre-normalization.r5r) r#r4rr&r$r!rr(r r')r*r8r9r:r3r=r;r<r0r0r1Ú forward_preEs  z#TransformerEncoderLayer.forward_precCs&|jr| ||||¡S| ||||¡S)z8Forward propagates the input through the encoder module.)r)r?r>)r*r8r9r:r3r0r0r1ÚforwardOszTransformerEncoderLayer.forward)N)NNN)NNN)NNN) Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚGELUrÚ staticmethodr4r>r?r@Ú __classcell__r0r0r.r1r s  r csJeZdZdZddde ¡df‡fdd„ Z‡fdd „Zedd d „ƒZ ‡Z S)rz#Defines the AIFI transformer layer.rrrFcstƒ ||||||¡dS)z7Initialize the AIFI instance with specified parameters.N)rr)r*r+r,r-rr(r)r.r0r1rYsz AIFI.__init__csp|jdd…\}}}| |||¡}tƒj| d¡ ddd¡|j|j|jdd}| ddd¡  d|||g¡  ¡S)z,Forward pass for the AIFI transformer layer.rNér)ÚdeviceÚdtype)r3éÿÿÿÿ) ÚshapeÚ"build_2d_sincos_position_embeddingrr@ÚflattenÚpermuteÚtorIrJÚviewÚ contiguous)r*ÚxÚcÚhÚwZ pos_embedr.r0r1r@]s.z AIFI.forwardéçˆÃ@c CsÈ|ddksJdƒ‚tj|tjd}tj|tjd}tj||dd\}}|d}tj|tjd|}d||}| ¡d|d }| ¡d|d } t t |¡t |¡t | ¡t | ¡gd ¡d S) z)Builds 2D sine-cosine position embedding.érzHEmbed dimension must be divisible by 4 for 2D sin-cos position embedding©rJZij)Zindexinggð?©.NNr)ÚtorchÚarangeÚfloat32ZmeshgridrNÚcatÚsinÚcos) rVrUÚ embed_dimZ temperatureZgrid_wZgrid_hZpos_dimÚomegaZout_wZout_hr0r0r1rMes z'AIFI.build_2d_sincos_position_embedding)rWrX) rArBrCrDrrErr@rFrMrGr0r0r.r1rVs  rcs(eZdZdZ‡fdd„Zdd„Z‡ZS)r zeTransformer layer https://arxiv.org/abs/2010.11929 (LayerNorm layers removed for better performance).csxtƒ ¡tj||dd|_tj||dd|_tj||dd|_tj||d|_tj||dd|_ tj||dd|_ dS)z]Initializes a self-attention mechanism using linear transformations and multi-head attention.F)Úbias)rbr-N) rrrrr;r<Úvrrr r!)r*rTr-r.r0r1rys zTransformerLayer.__init__cCs<| | |¡| |¡| |¡¡d|}| | |¡¡|S)z?Apply a transformer block to the input x and return the output.r)rr;r<rer!r ©r*rSr0r0r1r@ƒs(zTransformerLayer.forward©rArBrCrDrr@rGr0r0r.r1r vs r cs(eZdZdZ‡fdd„Zdd„Z‡ZS)r z4Vision Transformer https://arxiv.org/abs/2010.11929.cs\tƒ ¡d|_|ˆkr$t|ˆƒ|_t ˆˆ¡|_tj‡‡fdd„t|ƒDƒŽ|_ ˆ|_ dS)zaInitialize a Transformer module with position embedding and specified number of heads and layers.Nc3s|]}tˆˆƒVqdS©N)r )Ú.0Ú_©Úc2r-r0r1Ú “óz,TransformerBlock.__init__..) rrÚconvrrrÚlinearZ SequentialÚrangeÚtrrl)r*r+rlr-Ú num_layersr.rkr1rŒs   zTransformerBlock.__init__cCsb|jdur| |¡}|j\}}}}| d¡ ddd¡}| || |¡¡ ddd¡ ||j||¡S)z;Forward propagates the input through the bottleneck module.NrHrr)rorLrNrOrrrpZreshaperl)r*rSÚbrjrVrUÚpr0r0r1r@–s   zTransformerBlock.forwardrgr0r0r.r1r ‰s r cs:eZdZdZejf‡fdd„ Zejejdœdd„Z ‡Z S)r z6Implements a single block of a multi-layer perceptron.cs2tƒ ¡t ||¡|_t ||¡|_|ƒ|_dS)zcInitialize the MLPBlock with specified embedding dimension, MLP dimension, and activation function.N)rrrrÚlin1Úlin2r()r*Z embedding_dimZmlp_dimr(r.r0r1r¢s zMLPBlock.__init__)rSÚreturncCs| | | |¡¡¡S)zForward pass for the MLPBlock.)rwr(rvrfr0r0r1r@©szMLPBlock.forward) rArBrCrDrrErr\ZTensorr@rGr0r0r.r1r Ÿsr cs0eZdZdZejdf‡fdd„ Zdd„Z‡ZS)rz=Implements a simple multi-layer perceptron (also called FFN).FcsXtƒ ¡||_|g|d}t dd„t|g|||gƒDƒ¡|_||_|ƒ|_dS)zXInitialize the MLP with specified input, hidden, output dimensions and number of layers.rcss|]\}}t ||¡VqdSrh)rr)riÚnr<r0r0r1rm¶rnzMLP.__init__..N) rrrsrZ ModuleListÚzipÚlayersÚsigmoidr()r*Z input_dimÚ hidden_dimZ output_dimrsr(r|rUr.r0r1r±s  (z MLP.__init__cCsZt|jƒD]6\}}||jdkr8t|dt ¡ƒ||ƒƒn||ƒ}q t|ddƒrV| ¡S|S)z Forward pass for the entire MLP.rr(r|F)Ú enumerater{rsÚgetattrrÚReLUr|)r*rSÚiÚlayerr0r0r1r@ºs0z MLP.forward) rArBrCrDrr€rr@rGr0r0r.r1r®s rcs*eZdZdZd‡fdd„ Zdd„Z‡ZS)r a1 2D Layer Normalization module inspired by Detectron2 and ConvNeXt implementations. Original implementations in https://github.com/facebookresearch/detectron2/blob/main/detectron2/layers/batch_norm.py and https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py. çíµ ÷Æ°>cs8tƒ ¡t t |¡¡|_t t |¡¡|_||_ dS)z1Initialize LayerNorm2d with the given parameters.N) rrrÚ Parameterr\ZonesÚweightZzerosrdÚeps)r*Z num_channelsr†r.r0r1rËs zLayerNorm2d.__init__cCsj|jddd}|| d¡jddd}||t ||j¡}|jdd…ddf||jdd…ddfS)z0Perform forward pass for 2D layer normalization.rT©ZkeepdimrHN)ÚmeanÚpowr\Úsqrtr†r…rd)r*rSÚuÚsr0r0r1r@ÒszLayerNorm2d.forward)rƒrgr0r0r.r1r Ás r cs4eZdZdZd ‡fdd„ Zdd„Zd d d „Z‡ZS)rz× Multiscale Deformable Attention Module based on Deformable-DETR and PaddleDetection implementations. https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/ops/modules/ms_deform_attn.py rWrYrcs¼tƒ ¡||dkr*td|›d|›ƒ‚||}|||ksFJdƒ‚d|_||_||_||_||_t  ||||d¡|_ t  ||||¡|_ t  ||¡|_ t  ||¡|_ | ¡dS)z2Initialize MSDeformAttn with the given parameters.rz.d_model must be divisible by n_heads, but got z and z(`d_model` must be divisible by `n_heads`é@rHN)rrÚ ValueErrorZ im2col_stepÚd_modelÚn_levelsÚn_headsÚn_pointsrrÚsampling_offsetsÚattention_weightsÚ value_projÚ output_projÚ_reset_parameters)r*rrr‘r’Z _d_per_headr.r0r1rás  zMSDeformAttn.__init__cCsVt|jjjdƒtj|jtjddtj |j}t  |  ¡|  ¡gd¡}||  ¡jdddd |jddd ¡ d|j|jd¡}t|jƒD],}|d d …d d …|d d …f|d9<qŒt ¡$t | d¡¡|j_Wd ƒn1sì0Yt|jjjdƒt|jjjdƒt|jjjƒt|jjjdƒt|jjjƒt|jjjdƒd S) zReset module parameters.rrZg@rKTr‡rrrHN)rr“r…Údatar\r]r‘r^ÚmathÚpiÚstackrar`ÚabsÚmaxrQÚrepeatrr’rqZno_gradrr„rdr”rr•r–)r*ZthetasZ grid_initrr0r0r1r—øs&" ÿ þÿ* 2zMSDeformAttn._reset_parametersNc CsÜ|jdd…\}}|jd}tdd„|Dƒƒ|ks6J‚| |¡}|dur\| |dtdƒ¡}| |||j|j|j¡}| |¡ |||j|j |j d¡} |  |¡ |||j|j |j ¡} t   | d¡ |||j|j |j ¡} |jd} | dkrNtj||j|jd  d¡} | | ddddd…ddd…f} |dd…dd…ddd…ddd…f| }nv| d kr´| |j |dd…dd…ddd…ddd…fd } |dd…dd…ddd…ddd…f| }ntd | ›d ƒ‚t|||| ƒ}| |¡S)a Perform forward pass for multiscale deformable attention. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py Args: query (torch.Tensor): [bs, query_length, C] refer_bbox (torch.Tensor): [bs, query_length, n_levels, 2], range in [0, 1], top-left (0,0), bottom-right (1, 1), including padding area value (torch.Tensor): [bs, value_length, C] value_shapes (List): [n_levels, 2], [(H_0, W_0), (H_1, W_1), ..., (H_{L-1}, W_{L-1})] value_mask (Tensor): [bs, value_length], True for non-padding elements, False for padding elements Returns: output (Tensor): [bs, Length_{query}, C] NrHrcss|]}|d|dVqdS)rrNr0)rirŒr0r0r1rm rnz'MSDeformAttn.forward..r[rrK)rJrIrYgà?z5Last dim of reference_points must be 2 or 4, but got Ú.)rLÚsumr•Z masked_fillÚfloatrQr‘rr“rr’r”ÚFZsoftmaxr\Z as_tensorrJrIÚfliprŽrr–)r*ÚqueryÚ refer_bboxr6Z value_shapesZ value_maskÚbsÚlen_qZlen_vr“r”Z num_pointsZoffset_normalizerÚaddZsampling_locationsÚoutputr0r0r1r@ s*        * 2*zMSDeformAttn.forward)rWrYrrY)N)rArBrCrDrr—r@rGr0r0r.r1rÚsrcsReZdZdZdddde ¡ddf‡fdd„ Zed d „ƒZd d „Z ddd„Z ‡Z S)raK Deformable Transformer Decoder Layer inspired by PaddleDetection and Deformable-DETR implementations. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/deformable_transformer.py rWrirrYcs¦tƒ ¡tj|||d|_t |¡|_t |¡|_t ||||ƒ|_ t |¡|_ t |¡|_ t  ||¡|_||_t |¡|_t  ||¡|_t |¡|_t |¡|_dS)zKInitialize the DeformableTransformerDecoderLayer with the given parameters.)rN)rrrrÚ self_attnr%r&r"r#rÚ cross_attnr'r$rÚlinear1r(Údropout3Úlinear2Údropout4Únorm3)r*rr‘Zd_ffnrr(rr’r.r0r1r@s       z*DeformableTransformerDecoderLayer.__init__cCs|dur |S||S)z;Add positional embeddings to the input tensor, if provided.Nr0r2r0r0r1r4Vsz0DeformableTransformerDecoderLayer.with_pos_embedc Cs4| | | | |¡¡¡¡}|| |¡}| |¡S)zHPerform forward pass through the Feed-Forward Network part of the layer.)r®r­r(r¬r¯r°)r*ÚtgtZtgt2r0r0r1Ú forward_ffn[sz-DeformableTransformerDecoderLayer.forward_ffnNc Cs | ||¡}} |j| dd¡|  dd¡| dd¡|dd dd¡} || | ¡}| |¡}| | ||¡| d¡|||¡} || | ¡}| |¡}|  |¡S)z:Perform the forward pass through the entire decoder layer.rr)r7rH) r4rªZ transposer&r#r«Z unsqueezer'r$r²) r*Úembedr¥ÚfeatsÚshapesÚ padding_maskr7Z query_posr;r<r±r0r0r1r@as(ÿþ ÿ z)DeformableTransformerDecoderLayer.forward)NNN) rArBrCrDrr€rrFr4r²r@rGr0r0r.r1r8s   rcs,eZdZdZd‡fdd„ Zd dd„Z‡ZS) rzÌ Implementation of Deformable Transformer Decoder based on PaddleDetection. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py rKcs<tƒ ¡t||ƒ|_||_||_|dkr.|n|||_dS)zFInitialize the DeformableTransformerDecoder with the given parameters.rN)rrrr{rsr}Úeval_idx)r*r}Z decoder_layerrsr·r.r0r1r}s   z%DeformableTransformerDecoder.__init__Nc  Csü|} g} g} d} | ¡}t|jƒD]Ä\}}|| |||| |||ƒƒ} ||| ƒ}t |t|ƒ¡}|jr¦|  ||| ƒ¡|dkrŒ|  |¡qÐ|  t |t| ƒ¡¡n*||jkrÐ|  ||| ƒ¡|  |¡qè|} |jrâ| ¡n|}q"t  | ¡t  | ¡fS)z4Perform the forward pass through the entire decoder.Nr) r|r~r{r\rZtrainingÚappendr·Údetachr›)r*r³r¥r´rµZ bbox_headZ score_headZpos_mlpr7r¶r©Z dec_bboxesZdec_clsZlast_refined_bboxrr‚ZbboxZ refined_bboxr0r0r1r@…s*     z$DeformableTransformerDecoder.forward)rK)NNrgr0r0r.r1rvs ör)rDr™r\Ztorch.nnrZtorch.nn.functionalZ functionalr¢Z torch.nn.initrrrorÚutilsrrrÚ__all__ÚModuler rr r r rr rrrr0r0r0r1Ús$  : ^>