a yfd@sdddlmZmZmZmZddlZddlmZddlmZm Z Gdddej Z Gdddej Z dS) )ListOptionalTupleTypeN)nn)MLP LayerNorm2dc seZdZdZdejddfeejeeejeeddfdd Z e j e j e j e j e e e j e j fdd d Ze j e j e j e j e e j e j fd d d ZZS) MaskDecodera Decoder module for generating masks and their associated quality scores using a transformer architecture. This class predicts masks given image and prompt embeddings, utilizing a transformer to process the inputs and generate mask predictions along with their quality scores. Attributes: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. iou_token (nn.Embedding): Embedding for the IoU token. num_mask_tokens (int): Number of mask tokens. mask_tokens (nn.Embedding): Embedding for the mask tokens. output_upscaling (nn.Sequential): Neural network sequence for upscaling the output. output_hypernetworks_mlps (nn.ModuleList): Hypernetwork MLPs for generating masks. iou_prediction_head (nn.Module): MLP for predicting mask quality. Methods: forward: Predicts masks given image and prompt embeddings. predict_masks: Internal method for mask prediction. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> masks, iou_pred = decoder( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, multimask_output=True ... ) >>> print(f"Predicted masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") N)transformer_dim transformernum_multimask_outputs activationiou_head_depthiou_head_hidden_dimreturnc st|_||_||_td|_|d|_t|j|_ t tj ddddt d|tj ddddd||_ tfddt|jD|_t||j||_dS) ay Initializes the MaskDecoder module for generating masks and their quality scores. Args: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder) Z kernel_sizeZstridecsg|]}tddqSrr r.0_r c/var/www/html/django/DPS/env/lib/python3.9/site-packages/ultralytics/models/sam/modules/decoders.py Tz(MaskDecoder.__init__..N)super__init__r r rr Embedding iou_tokennum_mask_tokens mask_tokens SequentialConvTranspose2droutput_upscaling ModuleListrangeoutput_hypernetworks_mlpsriou_prediction_head)selfr r rrrr __class__rrr#)s$   zMaskDecoder.__init__)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_outputrc Csb|j||||d\}}|r$tddntdd}|dd|ddddf}|dd|f}||fS)a Predicts masks given image and prompt embeddings. Args: image_embeddings (torch.Tensor): Embeddings from the image encoder. image_pe (torch.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (torch.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (torch.Tensor): Embeddings of the mask inputs. multimask_output (bool): Whether to return multiple masks or a single mask. Returns: (Tuple[torch.Tensor, torch.Tensor]): A tuple containing: - masks (torch.Tensor): Batched predicted masks. - iou_pred (torch.Tensor): Batched predictions of mask quality. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> image_emb = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_emb = torch.rand(1, 2, 256) >>> dense_emb = torch.rand(1, 256, 64, 64) >>> masks, iou_pred = decoder(image_emb, image_pe, sparse_emb, dense_emb, multimask_output=True) >>> print(f"Masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") )r2r3r4r5rNr) predict_masksslice) r/r2r3r4r5r6masksiou_predZ mask_slicerrrforwardYs  zMaskDecoder.forward)r2r3r4r5rcs\tjjjjjgdd}|d|jddd}tj||fdd}tj||jddd}||}tj||jddd}|j\} } } }  |||\} }| dddddf}| ddddj ddf| dd | | | | } |}fddtj D}tj|dd}|j\} } } } || | | | |  | d| | }|}||fS) zaPredicts masks and quality scores using image and prompt embeddings via transformer architecture.rdimrNrcs,g|]$}j|dd|ddfqSNr-rimask_tokens_outr/rrr sz-MaskDecoder.predict_masks..)torchcatr%weightr' unsqueezeexpandshaperepeat_interleaver r& transposeviewr*r,stackr.)r/r2r3r4r5 output_tokenstokenssrcpos_srcbchwhs iou_token_outupscaled_embedding hyper_in_listhyper_inr9r:rrCrr7s(    " zMaskDecoder.predict_masks)__name__ __module__ __qualname____doc__rGELUintModulerr#rETensorboolrr;r7 __classcell__rrr0rr s6!2 1r c seZdZdZdejddddddddddf eejeeejeee e e e dd fd d Z de j e j e j e j e e e ee j ee j e j fd d d Zde j e j e j e j e e ee j ee j e j fdddZddZddZZS)SAM2MaskDecodera Transformer-based decoder for predicting instance segmentation masks from image and prompt embeddings. This class extends the functionality of the MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Attributes: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. iou_token (nn.Embedding): Embedding for IOU token. num_mask_tokens (int): Total number of mask tokens. mask_tokens (nn.Embedding): Embedding for mask tokens. pred_obj_scores (bool): Whether to predict object scores. obj_score_token (nn.Embedding): Embedding for object score token. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. output_upscaling (nn.Sequential): Upscaling layers for output. use_high_res_features (bool): Whether to use high-resolution features. conv_s0 (nn.Conv2d): Convolutional layer for high-resolution features (s0). conv_s1 (nn.Conv2d): Convolutional layer for high-resolution features (s1). output_hypernetworks_mlps (nn.ModuleList): List of MLPs for output hypernetworks. iou_prediction_head (MLP): MLP for IOU prediction. pred_obj_score_head (nn.Linear | MLP): Linear layer or MLP for object score prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. Methods: forward: Predicts masks given image and prompt embeddings. predict_masks: Predicts instance segmentation masks from image and prompt embeddings. _get_stability_scores: Computes mask stability scores based on IoU between thresholds. _dynamic_multimask_via_stability: Dynamically selects the most stable mask output. Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) r r Fg?g\(\?N) r r rrrruse_high_res_featurespred_obj_scorespred_obj_scores_mlpuse_multimask_token_for_obj_ptrrc s^t|_||_||_td|_|d|_t|j|_ | |_ |j r^td|_ ||_ t tjddddtd|tjddddd||_||_|rtjdddd|_tjdddd|_tfddt|jD|_t||j||d|_|j rHtd|_| rHtdd |_| |_| |_| |_d S) a Initializes the SAM2MaskDecoder module for predicting instance segmentation masks. This decoder extends the functionality of MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Args: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. use_high_res_features (bool): Whether to use high-resolution features. iou_prediction_use_sigmoid (bool): Whether to use sigmoid for IOU prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. pred_obj_scores (bool): Whether to predict object scores. pred_obj_scores_mlp (bool): Whether to use MLP for object score prediction. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = SAM2MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder) rrrrrcsg|]}tddqSrrrrrrr %r!z,SAM2MaskDecoder.__init__..)Zsigmoidr N)r"r#r r rrr$r%r&r'rhobj_score_tokenrjr(r)rr*rgZConv2dZconv_s0Zconv_s1r+r,r-rr.ZLinearpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)r/r r rrrrrgZiou_prediction_use_sigmoidrmrnrorhrirjr0rrr#sN,   zSAM2MaskDecoder.__init__)r2r3r4r5r6 repeat_imagehigh_res_featuresrc Cs|j||||||d\}} } } |rX|ddddddddf}| ddddf} nR|jrv|jsv||| \}} n4|ddddddddf}| ddddf} |r|jr| ddddf} n| ddddf} || | | fS)a Predicts masks given image and prompt embeddings. Args: image_embeddings (torch.Tensor): Embeddings from the image encoder with shape (B, C, H, W). image_pe (torch.Tensor): Positional encoding with the shape of image_embeddings (B, C, H, W). sparse_prompt_embeddings (torch.Tensor): Embeddings of the points and boxes with shape (B, N, C). dense_prompt_embeddings (torch.Tensor): Embeddings of the mask inputs with shape (B, C, H, W). multimask_output (bool): Whether to return multiple masks or a single mask. repeat_image (bool): Flag to repeat the image embeddings. high_res_features (List[torch.Tensor] | None): Optional high-resolution features. Returns: (Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]): A tuple containing: - masks (torch.Tensor): Batched predicted masks with shape (B, N, H, W). - iou_pred (torch.Tensor): Batched predictions of mask quality with shape (B, N). - sam_tokens_out (torch.Tensor): Batched SAM token for mask output with shape (B, N, C). - object_score_logits (torch.Tensor): Batched object score logits with shape (B, 1). Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) )r2r3r4r5rprqNrr)r7rmZtraining _dynamic_multimask_via_stabilityrj) r/r2r3r4r5r6rprqr9r:rDobject_score_logitsZsam_tokens_outrrrr;:s&'     zSAM2MaskDecoder.forward)r2r3r4r5rprqrcs`d}jr0tjjjjjjjgdd}d}ntjjjjjgdd}|d| ddd}tj||fdd} |rtj || j ddd} n|j d| j dksJ|} | |} | ddksJdtj || j ddd} | j \} } }} | | | \}} |dd|ddf}|dd|d|dj ddf| dd| | ||} jsl| }n<j\}}}}}|\}}|||| |}||||}fdd tj D}tj|dd}|j \} } }}||| | ||| d||}|}jr>|dks J|dddddf}nd ||j dd}|||fS) zZPredicts instance segmentation masks from image and prompt embeddings using a transformer.rr<rr>z@image_pe should have size 1 in batch dim (from `get_dense_pe()`)Nrcs,g|]$}j|dd|ddfqSr?r@rArCrrr sz1SAM2MaskDecoder.predict_masks..g$@)rhrErFrkrGr%r'rHrIsizerKrJr r&rLrMrgr*r,rNr.rlZnew_ones)r/r2r3r4r5rprqsrOrPrQrRrSrTrUrVrWrXrYZdc1Zln1Zact1Zdc2Zact2Zfeat_s0Zfeat_s1rZr[r9r:rsrrCrr7sV (  " zSAM2MaskDecoder.predict_maskscCsT|d}|j}tj||kdd}tj|| kdd}t|dk||dS)zOComputes mask stability scores based on IoU between upper and lower thresholds.r>r<rg?)flattenrnrEsumfloatwhere)r/Z mask_logitsZstability_deltaZarea_iZarea_urrr_get_stability_scoress  z%SAM2MaskDecoder._get_stability_scorescCs|ddddddddf}|ddddf}tj|dd}tj|d|jd}|||f}|d}|||f}|d}|ddddddddf} |ddddf} || } | |jk} t| d | | |} t| | | |}| |fS)a Dynamically selects the most stable mask output based on stability scores and IoU predictions. This method is used when outputting a single mask. If the stability score from the current single-mask output (based on output token 0) falls below a threshold, it instead selects from multi-mask outputs (based on output tokens 1-3) the mask with the highest predicted IoU score. This ensures a valid mask for both clicking and tracking scenarios. Args: all_mask_logits (torch.Tensor): Logits for all predicted masks, shape (B, N, H, W) where B is batch size, N is number of masks (typically 4), and H, W are mask dimensions. all_iou_scores (torch.Tensor): Predicted IoU scores for all masks, shape (B, N). Returns: (Tuple[torch.Tensor, torch.Tensor]): - mask_logits_out (torch.Tensor): Selected mask logits, shape (B, 1, H, W). - iou_scores_out (torch.Tensor): Selected IoU scores, shape (B, 1). Examples: >>> decoder = SAM2MaskDecoder(...) >>> all_mask_logits = torch.rand(2, 4, 256, 256) # 2 images, 4 masks each >>> all_iou_scores = torch.rand(2, 4) >>> mask_logits, iou_scores = decoder._dynamic_multimask_via_stability(all_mask_logits, all_iou_scores) >>> print(mask_logits.shape, iou_scores.shape) torch.Size([2, 1, 256, 256]) torch.Size([2, 1]) Nrr>r<r)device).NN) rEZargmaxZarangertr|rHr{rorzZ expand_as)r/Zall_mask_logitsZall_iou_scoresZmultimask_logitsZmultimask_iou_scoresZbest_scores_indsZ batch_indsZbest_multimask_logitsZbest_multimask_iou_scoresZsinglemask_logitsZsinglemask_iou_scoresZstability_scoresZ is_stableZmask_logits_outZiou_scores_outrrrrrs.         z0SAM2MaskDecoder._dynamic_multimask_via_stability)N)N)r\r]r^r_rr`rarbrrdr#rErcrrrr;r7r{rrrerrr0rrfsb0e  N  Grf) typingrrrrrErZultralytics.nn.modulesrrrbr rfrrrrs  &