a ==ic*@szddlmZddlmZddlmZddlmZdddZdd d Z dd d Z dddZ dddZ dddZ d ddZd S)!)scope) ParameterTags) caffe2_pb2) initializersNCHWFc Ksd|vr|dnt}|dup*|jtjk}|rB|sBd|d<|}n|d|dg}|jj||fd|i|} |rx|sx| S| dSdS) LRNZ device_optionNCUDNNengine_Z_scaleorderr)rZCurrentDeviceScopeZ device_typerZCPUnetr) modelblob_inblob_outr use_cudnnkwargsdevZis_cpuZ blobs_outlrnrt/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/caffe2/python/helpers/normalization.pyrs&rNcKsB|r d|d<|dur*|jj||fi|S|jj|fi|SdS)zSoftmax.rr N)r ZSoftmax)r rrrrrrrsoftmax%s rc  sp jfdd}|dd|dd}}|tj|tjddg} d |vr|d rjj|||ggfd |i|} | Sjj|||g| fd |i|} | d SdS) Ncsjjgd|g|dS)Nr )shapevalue)Zparam_init_net ConstantFill)rsuffixrdim_inr rr init_blob6sz instance_norm..init_blob?sb_sm_sivis_testr r)r NextNameZ AddParameterrWEIGHTBIASZ InstanceNorm) r rrrr rrscalebias blob_outs blob_outputsrrr instance_norm/s.  r,rr c Ks|p |j}|jrdd|if} dd|if} dddif}dddif}t|| dif}t|| dif}t||dif}t| |dif} n t}t}t}t} |j|d|g|tjd}|j|d|g|tj d}|j|d|g|tj d}|j|d |g| tj d}||||d |d g}d | vrr| d rr|jj ||||d |dg|gfd| i| }|S|jj ||||d |dg|fd| i| }|dSdS)Nrrr r_s param_namerZ initializertags_b_rm_rivr"r#r$r r) r r% init_paramsrupdate_initializerExternalInitializer create_paramrr&r'COMPUTED_PARAMZ SpatialBN)r rrr init_scale init_biasScaleInitializerBiasInitializerRunningMeanInitializerRunningVarianceInitializerr r scale_init bias_initrm_initriv_initr(r) running_meanrunning_inv_varr*r+rrr spatial_bnKs          rGc Ks|p |j}|jrRdd|if} dd|if} t|| dif}t|| dif}nt}t}|j|d|g|tjd}|j|d|g|tj d}||d|dg}|jj |||g|fi| }|dS) zK Group normalizes the input, cf. https://arxiv.org/abs/1803.08494. rrr-r.r1_mean_stdr) r r%r6rr7r8r9rr&r'Z GroupNorm)r rrrr;r<r=r>r?r@r rrArBr(r)r*r+rrr spatial_gnsD      rJr4-C6?c Cs|jd|t|tr|gn|tjd|dtjd}|jd|t|trP|gn|tjd|dtjd} |j j ||| g||d|dg||dd \} } } | | | fS) a> Layer normalizes the input, cf. https://arxiv.org/pdf/1607.06450.pdf. Args: blob_in: The input blob to layer normalize. blob_out: The layer normalized output blob. dim_in: The dimension of the scale and bias. For example, if blob_in is a 2D design matrix and axis is 1, this would be the number of columns. axis: (optional) The axis to normalize. Typically the feature axis. Defaults to 1. epsilon: (optional) A small value used for numerical stability in calculation. Defaults to 1e-4. initial_scale: (optional) The initial value for the learned scale parameter. Defaults to 1.0 initial_bias: (optional) The initial value for the learned bias parameter of the layerwise standard deviation. Defaults to 0.0. Returns: A 3-tuple consisting of: - The layer normalized input blob. - The mean of the input blob across the given axis. - The standard deviation of the input blob acress the given axis. z{}_scaler)rr.z{}_biasrHrIT)axisepsilonZelementwise_affine) r9format isinstanceintrZ Initializerrr&r'r Z LayerNorm) r rrrrLrMZ initial_scaleZ initial_biasr(r) normalizedmeanZstdrrr layer_norms4$   rScKs|jrDdddif}dddif} t||dif}t|| dif}nt}t}|j|d|g|tjd} |j|d|g|tjd} |d|d g} |d kr|jj|g| fgd |d d |} n,|dkr|jj|g| fgd|d d |} | S)Nrrr rr2r.r3r"Z_svr)rr5F)Zaxesr ZkeepdimsZNHWC)rr4r5) r6rr7r8r9rr:r ZMoments)r rrrr?r@r rrCrDrErFr*r+rrrmoments_with_running_statssV    rU)rF)NF)r)rr NNNNr)rr NNNNr)r4rKrr )NNr)Z caffe2.pythonrZ%caffe2.python.modeling.parameter_inforZ caffe2.protorZcaffe2.python.modelingrrrr,rGrJrSrUrrrrs0       R < D