a 8Sic0@szddlZddlZddlmZddlZddlZddlmZddlm Z GdddeZ e d ddZ Gd d d e Z dS) N)deepcopy)Module) _LRSchedulercs2eZdZdZd fdd ZddZdd ZZS) AveragedModelaImplements averaged model for Stochastic Weight Averaging (SWA). Stochastic Weight Averaging was proposed in `Averaging Weights Leads to Wider Optima and Better Generalization`_ by Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov and Andrew Gordon Wilson (UAI 2018). AveragedModel class creates a copy of the provided module :attr:`model` on the device :attr:`device` and allows to compute running averages of the parameters of the :attr:`model`. Args: model (torch.nn.Module): model to use with SWA device (torch.device, optional): if provided, the averaged model will be stored on the :attr:`device` avg_fn (function, optional): the averaging function used to update parameters; the function must take in the current value of the :class:`AveragedModel` parameter, the current value of :attr:`model` parameter and the number of models already averaged; if None, equally weighted average is used (default: None) use_buffers (bool): if ``True``, it will compute running averages for both the parameters and the buffers of the model. (default: ``False``) Example: >>> loader, optimizer, model, loss_fn = ... >>> swa_model = torch.optim.swa_utils.AveragedModel(model) >>> scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, >>> T_max=300) >>> swa_start = 160 >>> swa_scheduler = SWALR(optimizer, swa_lr=0.05) >>> for i in range(300): >>> for input, target in loader: >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() >>> if i > swa_start: >>> swa_model.update_parameters(model) >>> swa_scheduler.step() >>> else: >>> scheduler.step() >>> >>> # Update bn statistics for the swa_model at the end >>> torch.optim.swa_utils.update_bn(loader, swa_model) You can also use custom averaging functions with `avg_fn` parameter. If no averaging function is provided, the default is to compute equally-weighted average of the weights. Example: >>> # Compute exponential moving averages of the weights and buffers >>> ema_avg = lambda averaged_model_parameter, model_parameter, num_averaged:\ 0.1 * averaged_model_parameter + 0.9 * model_parameter >>> swa_model = torch.optim.swa_utils.AveragedModel(model, avg_fn=ema_avg, use_buffers=True) .. note:: When using SWA with models containing Batch Normalization you may need to update the activation statistics for Batch Normalization. This can be done either by using the :meth:`torch.optim.swa_utils.update_bn` or by setting :attr:`use_buffers` to `True`. The first approach updates the statistics in a post-training step by passing data through the model. The second does it during the parameter update phase by averaging all buffers. Empirical evidence has shown that updating the statistics in normalization layers increases accuracy, but you may wish to empirically test which approach yields the best results in your problem. .. note:: :attr:`avg_fn` is not saved in the :meth:`state_dict` of the model. .. note:: When :meth:`update_parameters` is called for the first time (i.e. :attr:`n_averaged` is `0`) the parameters of `model` are copied to the parameters of :class:`AveragedModel`. For every subsequent call of :meth:`update_parameters` the function `avg_fn` is used to update the parameters. .. _Averaging Weights Leads to Wider Optima and Better Generalization: https://arxiv.org/abs/1803.05407 .. _There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average: https://arxiv.org/abs/1806.05594 .. _SWALP: Stochastic Weight Averaging in Low-Precision Training: https://arxiv.org/abs/1904.11943 .. _Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well: https://arxiv.org/abs/2001.02312 NFcshtt|t||_|dur.|j||_|dtjdtj |d|durXdd}||_ ||_ dS)N n_averagedr)dtypedevicecSs||||dSN)Zaveraged_model_parameterZmodel_parameterZ num_averagedr r Q/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/optim/swa_utils.pyavg_fnjsz&AveragedModel.__init__..avg_fn) superr__init__rmoduletoregister_buffertorchtensorlongr use_buffers)selfmodelrr r __class__r r rbs zAveragedModel.__init__cOs|j|i|SN)r)rargskwargsr r r forwardpszAveragedModel.forwardc Cs|jrt|j|jn|}|jr@t||n|}t||D]Z\}}|j}| |}|j dkr| |qR| | |||j |qR|j d7_ dS)Nrr ) r itertoolschainr parametersbuffersziprdetachrrcopy_r )rrZ self_paramZ model_paramZp_swaZp_modelrZp_model_r r r update_parametersss"  zAveragedModel.update_parameters)NNF)__name__ __module__ __qualname____doc__rrr& __classcell__r r rr r sVrcCsi}|D]<}t|tjjjjr t|j|_t|j |_ |j ||<q |sRdS|j }| | D]}d|_ |jd9_qh|D]4}t|ttfr|d}|dur||}||q| D]}|||_ q| |dS)acUpdates BatchNorm running_mean, running_var buffers in the model. It performs one pass over data in `loader` to estimate the activation statistics for BatchNorm layers in the model. Args: loader (torch.utils.data.DataLoader): dataset loader to compute the activation statistics on. Each data batch should be either a tensor, or a list/tuple whose first element is a tensor containing data. model (torch.nn.Module): model for which we seek to update BatchNorm statistics. device (torch.device, optional): If set, data will be transferred to :attr:`device` before being passed into :attr:`model`. Example: >>> loader, model = ... >>> torch.optim.swa_utils.update_bn(loader, model) .. note:: The `update_bn` utility assumes that each data batch in :attr:`loader` is either a tensor or a list or tuple of tensors; in the latter case it is assumed that :meth:`model.forward()` should be called on the first element of the list or tuple corresponding to the data batch. Nr)modules isinstancernn batchnorm _BatchNorm zeros_like running_mean ones_like running_varmomentumtrainingtrainkeysnum_batches_trackedlisttupler)loaderrrZmomentarZ was_traininginputZ bn_moduler r r update_bns,       r>csZeZdZdZdfdd ZeddZed d Zed d Zed dZ ddZ Z S)SWALRaAnneals the learning rate in each parameter group to a fixed value. This learning rate scheduler is meant to be used with Stochastic Weight Averaging (SWA) method (see `torch.optim.swa_utils.AveragedModel`). Args: optimizer (torch.optim.Optimizer): wrapped optimizer swa_lrs (float or list): the learning rate value for all param groups together or separately for each group. annealing_epochs (int): number of epochs in the annealing phase (default: 10) annealing_strategy (str): "cos" or "linear"; specifies the annealing strategy: "cos" for cosine annealing, "linear" for linear annealing (default: "cos") last_epoch (int): the index of the last epoch (default: -1) The :class:`SWALR` scheduler is can be used together with other schedulers to switch to a constant learning rate late in the training as in the example below. Example: >>> loader, optimizer, model = ... >>> lr_lambda = lambda epoch: 0.9 >>> scheduler = torch.optim.lr_scheduler.MultiplicativeLR(optimizer, >>> lr_lambda=lr_lambda) >>> swa_scheduler = torch.optim.swa_utils.SWALR(optimizer, >>> anneal_strategy="linear", anneal_epochs=20, swa_lr=0.05) >>> swa_start = 160 >>> for i in range(300): >>> for input, target in loader: >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() >>> if i > swa_start: >>> swa_scheduler.step() >>> else: >>> scheduler.step() .. _Averaging Weights Leads to Wider Optima and Better Generalization: https://arxiv.org/abs/1803.05407 coscs|||}t||jD]\}}||d<q|dvrBtd|n"|dkrT|j|_n|dkrd|j|_t|trv|dkrtd|||_ t t | ||dS)Nswa_lr)rAlinearz>anneal_strategy must by one of 'cos' or 'linear', instead got rArDrz3anneal_epochs must be equal or greater than 0, got ) _format_paramr# param_groups ValueError_cosine_anneal anneal_func_linear_annealr-int anneal_epochsrr?r)r optimizerrCrLanneal_strategy last_epochswa_lrsgrouprr r rs    zSWALR.__init__cCsVt|ttfrBt|t|jkr>tdt|dt|j|S|gt|jSdS)NzGswa_lr must have the same length as optimizer.param_groups: swa_lr has z, optimizer.param_groups has )r-r:r;lenrFrG)rMrPr r r rEszSWALR._format_paramcCs|Srr tr r r rJszSWALR._linear_annealcCsdttj|dS)Nr )mathrApirSr r r rHszSWALR._cosine_annealcCs |dkr |S|||d|Sr r )lrrCalphar r r _get_initial_lr szSWALR._get_initial_lrcsjstdtjd}jdkr0td|}tdtd|dtdj}|fddj j D}tdtd|tdj}|fddt j j |DS)NzTTo get the last learning rate computed by the scheduler, please use `get_last_lr()`.r rcs"g|]}|d|dqS)rXrC)rZ).0rQ) prev_alpharr r sz SWALR.get_lr..cs(g|] \}}|d|dqS)rCr r )r[rQrX)rYr r r]s) _get_lr_called_within_stepwarningswarn UserWarning _step_countrLmaxminrIrMrFr#)rstepZprev_tZprev_lrsrTr )rYr\rr get_lrs"         z SWALR.get_lr)r@rArB) r'r(r)r*r staticmethodrErJrHrZrfr+r r rr r?s)    r?)N)rrVcopyrr_rtorch.nnrZtorch.optim.lr_schedulerrrno_gradr>r?r r r r s   | 6