a yf@sddlmZmZmZmZddlZddlmZddlmm Z ddl m Z ddl mZmZmZmZmZmZmZmZGdddejZGdd d ejZGd d d ejZGd d d ejZGdddejZGdddejZdS))ListOptionalTupleTypeN) LayerNorm2d)BlockCXBlockFuserMaskDownSamplerMultiScaleBlock PatchEmbedPositionEmbeddingRandomPositionEmbeddingSinecseZdZdZddddddddd ejejd d d d d feeeeeeeee e ej e ej e e e ee ed fddfdd Z ejejdddZZS)ImageEncoderViTa1 An image encoder using Vision Transformer (ViT) architecture for encoding images into a compact latent space. This class processes images by splitting them into patches, applying transformer blocks, and generating a final encoded representation through a neck module. Attributes: img_size (int): Dimension of input images, assumed to be square. patch_embed (PatchEmbed): Module for patch embedding. pos_embed (nn.Parameter | None): Absolute positional embedding for patches. blocks (nn.ModuleList): List of transformer blocks for processing patch embeddings. neck (nn.Sequential): Neck module to further process the output. Methods: forward: Processes input through patch embedding, positional embedding, blocks, and neck. Examples: >>> import torch >>> encoder = ImageEncoderViT(img_size=224, patch_size=16, embed_dim=768, depth=12, num_heads=12) >>> input_image = torch.randn(1, 3, 224, 224) >>> output = encoder(input_image) >>> print(output.shape) i g@TFr.N)img_size patch_sizein_chans embed_dimdepth num_heads mlp_ratio out_chansqkv_bias norm_layer act_layer use_abs_pos use_rel_posrel_pos_zero_init window_sizeglobal_attn_indexesreturncst||_t||f||f||d|_d|_| rTtt d||||||_t |_ t |D]D}t |||| | | | |||vr|nd||||fd }|j |qfttj||dddt|tj||dddd t||_dS) a Initializes an ImageEncoderViT instance for encoding images using Vision Transformer architecture. Args: img_size (int): Input image size, assumed to be square. patch_size (int): Size of image patches. in_chans (int): Number of input image channels. embed_dim (int): Dimension of patch embeddings. depth (int): Number of transformer blocks. num_heads (int): Number of attention heads in each block. mlp_ratio (float): Ratio of MLP hidden dimension to embedding dimension. out_chans (int): Number of output channels from the neck module. qkv_bias (bool): If True, adds learnable bias to query, key, value projections. norm_layer (Type[nn.Module]): Type of normalization layer to use. act_layer (Type[nn.Module]): Type of activation layer to use. use_abs_pos (bool): If True, uses absolute positional embeddings. use_rel_pos (bool): If True, adds relative positional embeddings to attention maps. rel_pos_zero_init (bool): If True, initializes relative positional parameters to zero. window_size (int): Size of attention window for windowed attention blocks. global_attn_indexes (Tuple[int, ...]): Indices of blocks that use global attention. Attributes: img_size (int): Dimension of input images. patch_embed (PatchEmbed): Module for patch embedding. pos_embed (nn.Parameter | None): Absolute positional embedding for patches. blocks (nn.ModuleList): List of transformer blocks. neck (nn.Sequential): Neck module for final processing. Examples: >>> encoder = ImageEncoderViT(img_size=224, patch_size=16, embed_dim=768, depth=12, num_heads=12) >>> input_image = torch.randn(1, 3, 224, 224) >>> output = encoder(input_image) >>> print(output.shape) ) kernel_sizestriderrNrr) dimrrrr r!r#r$r%Z input_sizeF)r(biasr)r(paddingr+)super__init__rr patch_embed pos_embednn Parametertorchzeros ModuleListblocksrangerappend SequentialConv2drneck)selfrrrrrrrrrr r!r"r#r$r%r&iblock __class__rc/var/www/html/django/DPS/env/lib/python3.9/site-packages/ultralytics/models/sam/modules/encoders.pyr.0sV5     zImageEncoderViT.__init__xr'cCs||}|jdurZ|jdkrLtj|jdddd|jddddddn|j}||}|jD] }||}q`||ddddS)zcProcesses input through patch embedding, positional embedding, transformer blocks, and neck module.Nrrrr) scale_factor)r/r0rF interpolatepermuter6r;)r<rCr0blkrrrAforwards  0  zImageEncoderViT.forward)__name__ __module__ __qualname____doc__r1Z LayerNormGELUintfloatboolrModulerr.r3TensorrJ __classcell__rrr?rArsJ frcseZdZdZejfeeeefeeefeeej ddfdd Z e j dddZ e j e j ee j d d d Ze j e j d d dZe j e j dddZeeee j e j fee j ee j edddZe jdddZeee j e j fee j ee j ee j e j fdddZZS) PromptEncodera: Encodes different types of prompts for input to SAM's mask decoder, producing sparse and dense embeddings. Attributes: embed_dim (int): Dimension of the embeddings. input_image_size (Tuple[int, int]): Size of the input image as (H, W). image_embedding_size (Tuple[int, int]): Spatial size of the image embedding as (H, W). pe_layer (PositionEmbeddingRandom): Module for random position embedding. num_point_embeddings (int): Number of point embeddings for different types of points. point_embeddings (nn.ModuleList): List of point embeddings. not_a_point_embed (nn.Embedding): Embedding for points that are not part of any label. mask_input_size (Tuple[int, int]): Size of the input mask. mask_downscaling (nn.Sequential): Neural network for downscaling the mask. no_mask_embed (nn.Embedding): Embedding for cases where no mask is provided. Methods: get_dense_pe: Returns the positional encoding used to encode point prompts. forward: Embeds different types of prompts, returning both sparse and dense embeddings. Examples: >>> prompt_encoder = PromptEncoder(256, (64, 64), (1024, 1024), 16) >>> points = (torch.rand(1, 5, 2), torch.randint(0, 4, (1, 5))) >>> boxes = torch.rand(1, 2, 2) >>> masks = torch.rand(1, 1, 256, 256) >>> sparse_embeddings, dense_embeddings = prompt_encoder(points, boxes, masks) >>> print(sparse_embeddings.shape, dense_embeddings.shape) torch.Size([1, 7, 256]) torch.Size([1, 256, 64, 64]) N)rimage_embedding_sizeinput_image_size mask_in_chans activationr'c st|_||_||_td|_d|_fddt|jD}t ||_ t d|_ d|dd|df|_t t jd|ddddt|d|t j|d|dddt||t j|dd|_t d|_d S) a Initializes the PromptEncoder module for encoding various types of prompts. This module encodes different types of prompts (points, boxes, masks) for input to SAM's mask decoder, producing both sparse and dense embeddings. Args: embed_dim (int): The dimension of the embeddings. image_embedding_size (Tuple[int, int]): The spatial size of the image embedding as (H, W). input_image_size (Tuple[int, int]): The padded size of the input image as (H, W). mask_in_chans (int): The number of hidden channels used for encoding input masks. activation (Type[nn.Module]): The activation function to use when encoding input masks. Attributes: embed_dim (int): Dimension of the embeddings. input_image_size (Tuple[int, int]): Size of the input image as (H, W). image_embedding_size (Tuple[int, int]): Spatial size of the image embedding as (H, W). pe_layer (PositionEmbeddingRandom): Module for random position embedding. num_point_embeddings (int): Number of point embeddings for different types of points. point_embeddings (nn.ModuleList): List of point embeddings. not_a_point_embed (nn.Embedding): Embedding for points that are not part of any label. mask_input_size (Tuple[int, int]): Size of the input mask. mask_downscaling (nn.Sequential): Neural network for downscaling the mask. Examples: >>> prompt_encoder = PromptEncoder(256, (64, 64), (1024, 1024), 16) >>> points = (torch.rand(1, 5, 2), torch.randint(0, 4, (1, 5))) >>> boxes = torch.rand(1, 2, 2) >>> masks = torch.rand(1, 1, 256, 256) >>> sparse_embeddings, dense_embeddings = prompt_encoder(points, boxes, masks) >>> print(sparse_embeddings.shape, dense_embeddings.shape) torch.Size([1, 7, 256]) torch.Size([1, 256, 64, 64]) rDcsg|]}tdqSr)r1 Embedding).0_rrrA z*PromptEncoder.__init__..rr)r(r)r(N)r-r.rrXrWrpe_layerZnum_point_embeddingsr7r1r5point_embeddingsr]not_a_point_embedZmask_input_sizer9r:rmask_downscaling no_mask_embed)r<rrWrXrYrZrer?r`rAr.s()    zPromptEncoder.__init__)r'cCs||jdS)a Returns the dense positional encoding used for encoding point prompts. This method generates a positional encoding for a dense set of points matching the shape of the image encoding. The encoding is used to provide spatial information to the model when processing point prompts. Returns: (torch.Tensor): Positional encoding tensor with shape (1, embed_dim, H, W), where H and W are the height and width of the image embedding size, respectively. Examples: >>> prompt_encoder = PromptEncoder(256, (64, 64), (1024, 1024), 16) >>> dense_pe = prompt_encoder.get_dense_pe() >>> print(dense_pe.shape) torch.Size([1, 256, 64, 64]) r)rdrWZ unsqueezer<rrrA get_dense_peszPromptEncoder.get_dense_pe)pointslabelspadr'cCs|d}|rhtj|jdddf|jd}tj|jddf|jd }tj||gdd}tj||gdd}|j||j}d||dk<||dk|j j 7<||dk|j dj 7<||dk|j dj 7<||dk|j dj 7<||d k|j d j 7<|S) zSEmbeds point prompts by applying positional encoding and label-specific embeddings.?rrrDdevicer*r) r3r4shaperpZonescatrdforward_with_coordsrXrfweightre)r<rkrlrmZ padding_pointZ padding_labelZpoint_embeddingrrrA _embed_pointss zPromptEncoder._embed_points)boxesr'cCsv|d}|ddd}|j||j}|dddddf|jdj7<|dddddf|jdj7<|S)zPEmbeds box prompts by applying positional encoding and adding corner embeddings.rnrsrDNrrr)reshaperdrvrXrerw)r<rycoordsZcorner_embeddingrrrA _embed_boxes's &&zPromptEncoder._embed_boxes)masksr'cCs ||S)zNEmbeds mask inputs by downscaling and processing through convolutional layers.)rg)r<r}rrrA _embed_masks0szPromptEncoder._embed_masks)rkryr}r'cCsB|dur|djdS|dur(|jdS|dur:|jdSdSdS)zLGets the batch size of the output given the batch size of the input prompts.Nrr)rt)rkryr}rrrA_get_batch_size4s  zPromptEncoder._get_batch_sizecCs|jdjjS)z@Returns the device of the first point embedding's weight tensor.r)rerwrprirrrA _get_deviceDszPromptEncoder._get_devicec Cs||||}tj|d|jf|d}|dur^|\}}|j|||dud}tj||gdd}|dur||} tj|| gdd}|dur||} n,|j j dddd |d|j d|j d} || fS)a? Embeds different types of prompts, returning both sparse and dense embeddings. Args: points (Tuple[torch.Tensor, torch.Tensor] | None): Point coordinates and labels to embed. The first tensor contains coordinates with shape (B, N, 2), and the second tensor contains labels with shape (B, N). boxes (torch.Tensor | None): Boxes to embed with shape (B, M, 2, 2), where M is the number of boxes. masks (torch.Tensor | None): Masks to embed with shape (B, 1, H, W). Returns: (Tuple[torch.Tensor, torch.Tensor]): A tuple containing: - sparse_embeddings (torch.Tensor): Sparse embeddings for points and boxes with shape (B, N, embed_dim). - dense_embeddings (torch.Tensor): Dense embeddings for masks of shape (B, embed_dim, embed_H, embed_W). Examples: >>> encoder = PromptEncoder(256, (64, 64), (1024, 1024), 16) >>> points = (torch.rand(1, 5, 2), torch.randint(0, 4, (1, 5))) >>> boxes = torch.rand(1, 2, 2, 2) >>> masks = torch.rand(1, 1, 256, 256) >>> sparse_emb, dense_emb = encoder(points, boxes, masks) >>> print(sparse_emb.shape, dense_emb.shape) torch.Size([1, 7, 256]) torch.Size([1, 256, 64, 64]) rroN)rmrrqrs)rr3emptyrrrxrur|r~rhrwrzexpandrW) r<rkryr}bsZsparse_embeddingsr{rlreZbox_embeddingsZdense_embeddingsrrrArJHs  zPromptEncoder.forward)rKrLrMrNr1rOrPrrrSr.r3rTrjrRrxr|r~ staticmethodrrrprrJrUrrr?rArVs4#  @ rVcsHeZdZdZd fdd Zd ejejeeejejfdddZ Z S) MemoryEncodera Encodes pixel features and masks into a memory representation for efficient image segmentation. This class processes pixel-level features and masks, fusing them to generate encoded memory representations suitable for downstream tasks in image segmentation models like SAM (Segment Anything Model). Attributes: mask_downsampler (MaskDownSampler): Module for downsampling input masks. pix_feat_proj (nn.Conv2d): Convolutional layer for projecting pixel features. fuser (Fuser): Module for fusing pixel features and masks. position_encoding (PositionEmbeddingSine): Module for adding positional encoding to features. out_proj (nn.Module): Output projection layer, either nn.Identity or nn.Conv2d. Methods: forward: Processes input pixel features and masks to generate encoded memory representations. Examples: >>> import torch >>> encoder = MemoryEncoder(out_dim=256, in_dim=256) >>> pix_feat = torch.randn(1, 256, 64, 64) >>> masks = torch.randn(1, 1, 64, 64) >>> encoded_feat, pos = encoder(pix_feat, masks) >>> print(encoded_feat.shape, pos.shape) torch.Size([1, 256, 64, 64]) torch.Size([1, 128, 64, 64]) rcstttdddd|_tj||dd|_ttdddd|_ t d d |_ t |_ ||krptj||dd|_ d S) z`Initializes the MemoryEncoder for encoding pixel features and masks into memory representations.rrDr)r(r)r,rcrrq)Z num_layers@Z num_pos_featsN)r-r.r mask_downsamplerr1r: pix_feat_projr r fuserrposition_encodingZIdentityout_proj)r<Zout_dimZin_dimr?rrAr.s   zMemoryEncoder.__init__F)pix_featr}skip_mask_sigmoidr'cCsh|st|}||}||j}||}||}||}||}|||j }||gdS)z_Processes pixel features and masks to generate encoded memory representations for segmentation.)vision_featuresvision_pos_enc) rFZsigmoidrtorprrrrdtype)r<rr}rrCposrrrArJs      zMemoryEncoder.forward)r)F) rKrLrMrNr.r3rTrRrrJrUrrr?rArzsrcs@eZdZdZd ejejedfdd Zej dddZ Z S) ImageEncodera Encodes images using a trunk-neck architecture, producing multiscale features and positional encodings. This class combines a trunk network for feature extraction with a neck network for feature refinement and positional encoding generation. It can optionally discard the lowest resolution features. Attributes: trunk (nn.Module): The trunk network for initial feature extraction. neck (nn.Module): The neck network for feature refinement and positional encoding generation. scalp (int): Number of lowest resolution feature levels to discard. Methods: forward: Processes the input image through the trunk and neck networks. Examples: >>> trunk = SomeTrunkNetwork() >>> neck = SomeNeckNetwork() >>> encoder = ImageEncoder(trunk, neck, scalp=1) >>> image = torch.randn(1, 3, 224, 224) >>> output = encoder(image) >>> print(output.keys()) dict_keys(['vision_features', 'vision_pos_enc', 'backbone_fpn']) r)trunkr;scalpcsNt||_||_||_|jj|jjksJJd|jjd|jjddS)z`Initializes the ImageEncoder with trunk and neck networks for feature extraction and refinement.zChannel dims of trunk z and neck z do not match.N)r-r.rr;r channel_listbackbone_channel_list)r<rr;rr?rrAr.s zImageEncoder.__init__)samplecCsT|||\}}|jdkr@|d|j |d|j }}|d}|||dS)zaEncodes input through patch embedding, positional embedding, transformer blocks, and neck module.rNrs)rrZ backbone_fpn)r;rr)r<rfeaturesrsrcrrrArJs "zImageEncoder.forward)r) rKrLrMrNr1rSrPr.r3rTrJrUrrr?rArsrc sVeZdZdZd eeeeeeeeeeedfdd Zee j d d d Z Z S)FpnNecka A Feature Pyramid Network (FPN) neck variant for multiscale feature fusion in object detection models. This FPN variant removes the output convolution and uses bicubic interpolation for feature resizing, similar to ViT positional embedding interpolation. Attributes: position_encoding (PositionEmbeddingSine): Sinusoidal positional encoding module. convs (nn.ModuleList): List of convolutional layers for each backbone level. backbone_channel_list (List[int]): List of channel dimensions from the backbone. fpn_interp_model (str): Interpolation mode for FPN feature resizing. fuse_type (str): Type of feature fusion, either 'sum' or 'avg'. fpn_top_down_levels (List[int]): Levels to have top-down features in outputs. Methods: forward: Performs forward pass through the FPN neck. Examples: >>> backbone_channels = [64, 128, 256, 512] >>> fpn_neck = FpnNeck(256, backbone_channels) >>> inputs = [torch.rand(1, c, 32, 32) for c in backbone_channels] >>> outputs, positions = fpn_neck(inputs) >>> print(len(outputs), len(positions)) 4 4 rrbilinearsumN)d_modelrr(r)r,fpn_interp_model fuse_typefpn_top_down_levelsc sttdd|_t|_||_|D]4} t} | dtj | ||||d|j | q*||_ |dvsrJ||_ |durtt|j}t||_dS)a Initializes a modified Feature Pyramid Network (FPN) neck. This FPN variant removes the output convolution and uses bicubic interpolation for feature resizing, similar to ViT positional embedding interpolation. Args: d_model (int): Dimension of the model. backbone_channel_list (List[int]): List of channel dimensions from the backbone. kernel_size (int): Kernel size for the convolutional layers. stride (int): Stride for the convolutional layers. padding (int): Padding for the convolutional layers. fpn_interp_model (str): Interpolation mode for FPN feature resizing. fuse_type (str): Type of feature fusion, either 'sum' or 'avg'. fpn_top_down_levels (Optional[List[int]]): Levels to have top-down features in outputs. Examples: >>> backbone_channels = [64, 128, 256, 512] >>> fpn_neck = FpnNeck(256, backbone_channels) >>> print(fpn_neck) rrconv)Z in_channelsZ out_channelsr(r)r,>ravgN)r-r.rrr1r5convsrr9Z add_moduler:r8rrr7lenlistr) r<rrr(r)r,rrrr*currentr?rrAr.s.     zFpnNeck.__init__)xsc Csdgt|j}dgt|j}t|t|jks6Jd}t|jd}t|ddD]}||}|j|||}||jvr|durtj|jtjdd|j |j dkrdnddd} || }|j d kr|d }n|}|} | ||<| | | j ||<qT||fS) aZ Performs forward pass through the Feature Pyramid Network (FPN) neck. This method processes a list of input tensors from the backbone through the FPN, applying lateral connections and top-down feature fusion. It generates output feature maps and corresponding positional encodings. Args: xs (List[torch.Tensor]): List of input tensors from the backbone, each with shape (B, C, H, W). Returns: (Tuple[List[torch.Tensor], List[torch.Tensor]]): A tuple containing: - out (List[torch.Tensor]): List of output feature maps after FPN processing, each with shape (B, d_model, H, W). - pos (List[torch.Tensor]): List of positional encodings corresponding to each output feature map. Examples: >>> fpn_neck = FpnNeck(d_model=256, backbone_channel_list=[64, 128, 256, 512]) >>> inputs = [torch.rand(1, c, 32, 32) for c in [64, 128, 256, 512]] >>> outputs, positions = fpn_neck(inputs) >>> print(len(outputs), len(positions)) 4 4 Nrrs)r@ZnearestF)rEmodeZ align_cornersZ antialiasrrD) rrr7rrFrGrr3Zfloat32rrrr) r<routrZ prev_featuresnr=rCZlateral_featuresZtop_down_featuresZx_outrrrArJOs0   zFpnNeck.forward)rrrrrN) rKrLrMrNrPrstrrr.r3rTrJrUrrr?rArs$ ?rcseZdZdZdeeeeeeefeed feeeeefeed feed fd fdd Zeeefej dddZ ej e ej dddZ Z S)Hieraa Hierarchical vision transformer for efficient multiscale feature extraction in image processing tasks. This class implements a Hiera model, which is a hierarchical vision transformer architecture designed for efficient multiscale feature extraction. It uses a series of transformer blocks organized into stages, with optional pooling and global attention mechanisms. Attributes: window_spec (Tuple[int, ...]): Window sizes for each stage. q_stride (Tuple[int, int]): Downsampling stride between stages. stage_ends (List[int]): Indices of the last block in each stage. q_pool_blocks (List[int]): Indices of blocks where pooling is applied. return_interm_layers (bool): Whether to return intermediate layer outputs. patch_embed (PatchEmbed): Module for patch embedding. global_att_blocks (Tuple[int, ...]): Indices of blocks with global attention. window_pos_embed_bkg_spatial_size (Tuple[int, int]): Spatial size for window positional embedding background. pos_embed (nn.Parameter): Positional embedding for the background. pos_embed_window (nn.Parameter): Positional embedding for the window. blocks (nn.ModuleList): List of MultiScaleBlock modules. channel_list (List[int]): List of output channel dimensions for each stage. Methods: _get_pos_embed: Generates positional embeddings by interpolating and combining window and background embeddings. forward: Performs the forward pass through the Hiera model. Examples: >>> model = Hiera(embed_dim=96, num_heads=1, stages=(2, 3, 16, 3)) >>> input_tensor = torch.randn(1, 3, 224, 224) >>> output_features = model(input_tensor) >>> for feat in output_features: ... print(feat.shape) `rrrrrDrDrDrrrrrr[rrrT.) rrdrop_path_rateq_poolq_stridestagesdim_mulhead_mul!window_pos_embed_bkg_spatial_size window_specglobal_att_blocksc  sttt| ksJ| _t} |_fddtdtdD_d|krvtjddks|nJddjddDd|_| _ t |dd d d _ | _ | _ ttjd|gj R_ttd|jdjd_d dtd|| D}d}t_t| D]}|}j|d}j durh|j vrddn|}|djvrt||}t||}|d7}t||||||jvrjnd|d }|}j|q2| rfddjdddDn jdjg_dS)zZInitializes the Hiera model, configuring its hierarchical vision transformer architecture.cs g|]}td|dqS)Nr)rr^r=)rrrArarbz"Hiera.__init__..rrNrscSsg|] }|dqSr\rr^rCrrrArarb)rr)r[r[)rr)rr(r)r,cSsg|] }|qSr)itemrrrrArarb)r*dim_outrZ drop_pathrr%csg|]}j|jqSr)r6rrrirrArarb)r-r.rrrrr7 stage_endsZ q_pool_blocksreturn_interm_layersr r/rrr1r2r3r4r0pos_embed_windowZlinspacer5r6rPr r8rr)r<rrrrrrrrrrrrrZdprZ cur_stager=rr%r>r?)r<rrAr.sZ "("$      " zHiera.__init__)hwr'cCsZ|\}}|j}tj|j||fdd}||ddt|j|jD}|dddd}|S) z`Generates positional embeddings by interpolating and combining window and background embeddings.Zbicubic)sizercSsg|]\}}||qSrr)r^rCyrrrArarbz(Hiera._get_pos_embed..rrDrr)rrFrGr0ZtileziprtrH)r<rhwZ window_embedr0rrrA_get_pos_embeds "zHiera._get_pos_embedrBcCs~||}|||jdd}g}t|jD]H\}}||}||jdks^||jvr0|jr0|dddd}||q0|S)z\Performs forward pass through Hiera model, extracting multiscale features from input images.rrrsrrD) r/rrt enumerater6rrrHr8)r<rCoutputsr=rIZfeatsrrrArJ s  z Hiera.forward) rrrrrrrrrrrrT)rKrLrMrNrPrQrr.r3rTrrrJrUrrr?rArs8#     [ r)typingrrrrr3Ztorch.nnr1Ztorch.nn.functionalZ functionalrFZultralytics.nn.modulesrr6rr r r r r rrrSrrVrrrrrrrrAs  ( VD7