a 8Sic@sTddlZddlZddlmZddlmZddlZddlZddlm Z ddl m Z ddl m Z dZGd d d eZGd d d eZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdd d eZGd!d"d"eZGd#d$d$eZGd%d&d&eZdS)'N)inf)wraps)Counter) bisect_right) OptimizeraThe epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.c@sJeZdZdddZddZddZd d Zd d ZdddZdddZ d S) _LRSchedulerFcCst|tstdt|j||_|dkrJ|jD]}|d|dq2n*t |jD]\}}d|vrTt d|qTdd|jD|_ ||_ dd }||jj |j_ d |j_d |_||_| dS) N{} is not an Optimizerr initial_lrlrzRparam 'initial_lr' is not specified in param_groups[{}] when resuming an optimizercSsg|] }|dqS)r .0groupr r T/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/optim/lr_scheduler.py )z)_LRScheduler.__init__..csNt|ddr|St|j|jj~tfdd}d|_|S)N _with_counterFcs.}|jd7_|}||i|S)Nr) _step_count__get__)argskwargsinstancewrappedclsfuncZ instance_refr rwrapper<s z<_LRScheduler.__init__..with_counter..wrapperT)getattrweakrefref__self____func__ __class__rr)methodrr rr with_counter/s  z+_LRScheduler.__init__..with_counterr) isinstancer TypeErrorformattype__name__ optimizer param_groups setdefault enumerateKeyErrorbase_lrs last_epochsteprverbose)selfr,r2r4rir&r r r__init__s*  z_LRScheduler.__init__cCsdd|jDS)zReturns the state of the scheduler as a :class:`dict`. It contains an entry for every variable in self.__dict__ which is not the optimizer. cSsi|]\}}|dkr||qSr,r rkeyvaluer r r Urz+_LRScheduler.state_dict..__dict__itemsr5r r r state_dictOsz_LRScheduler.state_dictcCs|j|dS)Loads the schedulers state. Args: state_dict (dict): scheduler state. Should be an object returned from a call to :meth:`state_dict`. N)r>updater5rAr r rload_state_dictWsz_LRScheduler.load_state_dictcCs|jS)zB Return last computed learning rate by current scheduler. )_last_lrr@r r r get_last_lr`sz_LRScheduler.get_last_lrcCstdSN)NotImplementedErrorr@r r rget_lresz_LRScheduler.get_lrNcCsJ|rF|durtd||n(t|tr,dnd|}td|||dS)z+Display the current learning rate. Nz.Adjusting learning rate of group {} to {:.4e}.%.2f%.5dz8Epoch {}: adjusting learning rate of group {} to {:.4e}.)printr)r'float)r5Z is_verboserr epoch epoch_strr r rprint_lrisz_LRScheduler.print_lrcCs&|jdkr>t|jjds&tdtn|jjdkr>tdt|jd7_Gddd}||^|dur|jd7_|}n.tt t||_t|dr| }n|}Wdn1s0Yt t |jj |D]*\}}|\}}||d<||j|||qd d |jj D|_dS) NrraSeems like `optimizer.step()` has been overridden after learning rate scheduler initialization. Please, make sure to call `optimizer.step()` before `lr_scheduler.step()`. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rateaxDetected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-ratec@s$eZdZddZddZddZdS)z._LRScheduler.step.._enable_get_lr_callcSs ||_dSrHor5rSr r rr7sz7_LRScheduler.step.._enable_get_lr_call.__init__cSs d|j_|SNTrS_get_lr_called_within_stepr@r r r __enter__sz8_LRScheduler.step.._enable_get_lr_call.__enter__cSs d|j_dSNFrVr5r*r; tracebackr r r__exit__sz7_LRScheduler.step.._enable_get_lr_call.__exit__Nr+ __module__ __qualname__r7rXr\r r r r_enable_get_lr_callsr`_get_closed_form_lrr cSsg|] }|dqSr r rr r rrrz%_LRScheduler.step..)rhasattrr,r3warningswarn UserWarningr2rJEPOCH_DEPRECATION_WARNINGrar/zipr-rQr4rF)r5rOr`valuesr6data param_groupr r r rr3ws2       &z_LRScheduler.step)r F)N)N) r+r^r_r7rArErGrJrQr3r r r rrs 7  rcs:eZdZdZd fdd ZddZdd Zd d ZZS) LambdaLRaSets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. lr_lambda (function or list): A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> # Assuming optimizer has two groups. >>> lambda1 = lambda epoch: epoch // 30 >>> lambda2 = lambda epoch: 0.95 ** epoch >>> scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2]) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() r Fcs|||_t|ts.t|ts.|gt|j|_n6t|t|jkrZtdt|jt|t||_t t | |||dSNz"Expected {} lr_lambdas, but got {}) r,r'listtuplelenr- lr_lambdas ValueErrorr)superrlr7r5r,Z lr_lambdar2r4r$r rr7s zLambdaLR.__init__cCs^dd|jD}dgt|j|d<t|jD]&\}}t|tjs2|j|d|<q2|S)aReturns the state of the scheduler as a :class:`dict`. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer. cSsi|]\}}|dvr||qS)r,rqr r9r r rr<rz'LambdaLR.state_dict..Nrq r>r?rprqr/r'types FunctionTypecopyr5rAidxfnr r rrAs  zLambdaLR.state_dictcCsN|d}|j|||d<t|D]"\}}|dur&|j|j|q&dS)a$Loads the schedulers state. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer. Args: state_dict (dict): scheduler state. Should be an object returned from a call to :meth:`state_dict`. rqNpopr>rCr/rqr5rArqr|r}r r rrEs  zLambdaLR.load_state_dictcs,jstdfddtjjDS)NTTo get the last learning rate computed by the scheduler, please use `get_last_lr()`.csg|]\}}||jqSr r2)rlmbdabase_lrr@r rrsz#LambdaLR.get_lr..)rWrdrerhrqr1r@r r@rrJs    zLambdaLR.get_lr)r F r+r^r___doc__r7rArErJ __classcell__r r rurrls  rlcs:eZdZdZd fdd ZddZdd Zd d ZZS) MultiplicativeLRaYMultiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. lr_lambda (function or list): A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> lmbda = lambda epoch: 0.95 >>> scheduler = MultiplicativeLR(optimizer, lr_lambda=lmbda) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() r Fcs|||_t|ts.t|ts.|gt|j|_n6t|t|jkrZtdt|jt|t||_t t | |||dSrm) r,r'rnrorpr-rqrrr)rsrr7rtrur rr7s zMultiplicativeLR.__init__cCs^dd|jD}dgt|j|d<t|jD]&\}}t|tjs2|j|d|<q2|S)a2Returns the state of the scheduler as a :class:`dict`. It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. cSsi|]\}}|dvr||qSrvr r9r r rr<*rz/MultiplicativeLR.state_dict..Nrqrwr{r r rrA"s  zMultiplicativeLR.state_dictcCsN|d}|j|||d<t|D]"\}}|dur&|j|j|q&dS)rBrqNr~rr r rrE3s   z MultiplicativeLR.load_state_dictcsPjstdtjdkr:fddtjjjDSddjjDSdS)Nrrcs"g|]\}}|d|jqSrbr)rrrr@r rrJsz+MultiplicativeLR.get_lr..cSsg|] }|dqSrbr rr r rrMr) rWrdrerfr2rhrqr,r-r@r r@rrJDs  zMultiplicativeLR.get_lr)r Frr r rurrs  rcs2eZdZdZd fdd ZddZd d ZZS) StepLRa3Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. step_size (int): Period of learning rate decay. gamma (float): Multiplicative factor of learning rate decay. Default: 0.1. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.05 if epoch < 30 >>> # lr = 0.005 if 30 <= epoch < 60 >>> # lr = 0.0005 if 60 <= epoch < 90 >>> # ... >>> scheduler = StepLR(optimizer, step_size=30, gamma=0.1) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() 皙?r Fcs$||_||_tt||||dSrH) step_sizegammarsrr7)r5r,rrr2r4rur rr7lszStepLR.__init__csTjstdtjdks,jjdkr>ddjjDSfddjjDS)NrrcSsg|] }|dqSrbr rr r rrwrz!StepLR.get_lr..csg|]}|djqSrbrrr@r rrxs)rWrdrerfr2rr,r-r@r r@rrJqs z StepLR.get_lrcsfddjDS)Ncs"g|]}|jjjqSr )rr2rrrr@r rr|sz.StepLR._get_closed_form_lr..r1r@r r@rra{s zStepLR._get_closed_form_lr)rr Fr+r^r_rr7rJrarr r rurrPs rcs2eZdZdZd fdd ZddZd d ZZS) MultiStepLRaUDecays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. milestones (list): List of epoch indices. Must be increasing. gamma (float): Multiplicative factor of learning rate decay. Default: 0.1. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.05 if epoch < 30 >>> # lr = 0.005 if 30 <= epoch < 80 >>> # lr = 0.0005 if epoch >= 80 >>> scheduler = MultiStepLR(optimizer, milestones=[30,80], gamma=0.1) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() rr Fcs(t||_||_tt||||dSrH)r milestonesrrsrr7)r5r,rrr2r4rur rr7s zMultiStepLR.__init__csFjstdtjjvr0ddjjDSfddjjDS)NrcSsg|] }|dqSrbr rr r rrrz&MultiStepLR.get_lr..cs&g|]}|djjjqSrb)rrr2rr@r rrs)rWrdrerfr2rr,r-r@r r@rrJs  zMultiStepLR.get_lrcs(ttjfddjDS)Ncs"g|]}|jtjqSr )rrr2rrr5r rrsz3MultiStepLR._get_closed_form_lr..)rnsortedrelementsr1r@r rrras zMultiStepLR._get_closed_form_lr)rr Frr r rurrs rcs2eZdZdZd fdd Zdd Zd d ZZS) ConstantLRaDecays the learning rate of each parameter group by a small constant factor until the number of epoch reaches a pre-defined milestone: total_iters. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. factor (float): The number we multiply learning rate until the milestone. Default: 1./3. total_iters (int): The number of steps that the scheduler decays the learning rate. Default: 5. last_epoch (int): The index of the last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.025 if epoch == 0 >>> # lr = 0.025 if epoch == 1 >>> # lr = 0.025 if epoch == 2 >>> # lr = 0.025 if epoch == 3 >>> # lr = 0.05 if epoch >= 4 >>> scheduler = ConstantLR(self.opt, factor=0.5, total_iters=4) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() UUUUUU?r Fcs<|dks|dkrtd||_||_tt||||dS)N?rz>Constant multiplicative factor expected to be between 0 and 1.)rrfactor total_itersrsrr7)r5r,rrr2r4rur rr7s zConstantLR.__init__csjstdtjdkr2fddjjDSjjksJjjkr\ddjjDSjjkr~fddjjDSdS)Nrrcsg|]}|djqSrbrrr@r rrrz%ConstantLR.get_lr..cSsg|] }|dqSrbr rr r rrrcsg|]}|ddjqS)r rrrr@r rrrrWrdrerfr2r,r-rr@r r@rrJs    zConstantLR.get_lrcsfddjDS)Ncs,g|]$}|jjjkdjqS)r)rr2rrr@r rrsz2ConstantLR._get_closed_form_lr..rr@r r@rras zConstantLR._get_closed_form_lr)rrr Frr r rurrsrcs2eZdZdZd fdd Zd d Zd d ZZS)LinearLRaDecays the learning rate of each parameter group by linearly changing small multiplicative factor until the number of epoch reaches a pre-defined milestone: total_iters. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. start_factor (float): The number we multiply learning rate in the first epoch. The multiplication factor changes towards end_factor in the following epochs. Default: 1./3. end_factor (float): The number we multiply learning rate at the end of linear changing process. Default: 1.0. total_iters (int): The number of iterations that multiplicative factor reaches to 1. Default: 5. last_epoch (int): The index of the last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.025 if epoch == 0 >>> # lr = 0.03125 if epoch == 1 >>> # lr = 0.0375 if epoch == 2 >>> # lr = 0.04375 if epoch == 3 >>> # lr = 0.05 if epoch >= 4 >>> scheduler = LinearLR(self.opt, start_factor=0.5, total_iters=4) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() rrrr FcsZ|dks|dkrtd|dks(|dkr0td||_||_||_tt||||dS)Nrrz>Starting multiplicative factor expected to be between 0 and 1.z.cSsg|] }|dqSrbr rr r rr rc sFg|]>}|ddjjjjjdjjqS)r rr)rrrr2rr@r rr"s rr@r r@rrJs   zLinearLR.get_lrcsfddjDS)Ncs6g|].}|jjjtjjjqSr )rrminrr2rr@r rr'sz0LinearLR._get_closed_form_lr..rr@r r@rra&s zLinearLR._get_closed_form_lr)rrrr Frr r rurrs   rcs2eZdZdZd fdd ZddZdd ZZS) ExponentialLRaDecays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Optimizer): Wrapped optimizer. gamma (float): Multiplicative factor of learning rate decay. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. r Fcs||_tt||||dSrH)rrsrr7)r5r,rr2r4rur rr78szExponentialLR.__init__csDjstdtjdkr.ddjjDSfddjjDS)NrrcSsg|] }|dqSrbr rr r rrBrz(ExponentialLR.get_lr..csg|]}|djqSrbrrr@r rrCs)rWrdrerfr2r,r-r@r r@rrJ<s  zExponentialLR.get_lrcsfddjDS)Ncsg|]}|jjqSr )rr2rr@r rrGsz5ExponentialLR._get_closed_form_lr..rr@r r@rraFs z!ExponentialLR._get_closed_form_lr)r Frr r rurr,s  rc@s2eZdZdZd ddZddZdd Zd d Zd S) SequentialLRaReceives the list of schedulers that is expected to be called sequentially during optimization process and milestone points that provides exact intervals to reflect which scheduler is supposed to be called at a given epoch. Args: optimizer (Optimizer): Wrapped optimizer. schedulers (list): List of chained schedulers. milestones (list): List of integers that reflects milestone points. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): Does nothing. Example: >>> # Assuming optimizer uses lr = 1. for all groups >>> # lr = 0.1 if epoch == 0 >>> # lr = 0.1 if epoch == 1 >>> # lr = 0.9 if epoch == 2 >>> # lr = 0.81 if epoch == 3 >>> # lr = 0.729 if epoch == 4 >>> scheduler1 = ConstantLR(self.opt, factor=0.1, total_iters=2) >>> scheduler2 = ExponentialLR(self.opt, gamma=0.9) >>> scheduler = SequentialLR(self.opt, schedulers=[scheduler1, scheduler2], milestones=[2]) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() r FcCstt|D]L}||j|kr.td|d||j|djkr tddd|dq t|t|dkrtdt|t|||_||_|d|_||_|d|_ dS)NzjSequential Schedulers expects all schedulers to belong to the same optimizer, but got schedulers at index z. to be different than the optimizer passed in.rz and z to be different.rzSequential Schedulers expects number of schedulers provided to be one more than the number of milestone points, but got number of schedulers {} and the number of milestones to be equal to {}) rangerpr,rrr) _schedulers _milestonesr2rGrF)r5r, schedulersrr2r4 scheduler_idxr r rr7gs4  zSequentialLR.__init__cCsl|jd7_t|j|j}|dkrJ|j|d|jkrJ|j|dn|j||j||_dS)Nrr)r2rrrr3rGrF)r5r|r r rr3s zSequentialLR.stepcCsPdd|jD}dgt|j|d<t|jD]\}}||d|<q2|S)Returns the state of the scheduler as a :class:`dict`. It contains an entry for every variable in self.__dict__ which is not the optimizer. The wrapped scheduler states will also be saved. cSsi|]\}}|dvr||qS)r,rr r9r r rr<rz+SequentialLR.state_dict..Nrr>r?rprr/rAr5rAr|sr r rrAs zSequentialLR.state_dictcCsD|d}|j|||d<t|D]\}}|j||q&dSrBrNrr>rCr/rrEr5rArr|rr r rrEs   zSequentialLR.load_state_dictN)r Fr+r^r_rr7r3rArEr r r rrKs   rcs2eZdZdZd fdd ZddZd d ZZS) CosineAnnealingLRaSet the learning rate of each parameter group using a cosine annealing schedule, where :math:`\eta_{max}` is set to the initial lr and :math:`T_{cur}` is the number of epochs since the last restart in SGDR: .. math:: \begin{aligned} \eta_t & = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right), & T_{cur} \neq (2k+1)T_{max}; \\ \eta_{t+1} & = \eta_{t} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 - \cos\left(\frac{1}{T_{max}}\pi\right)\right), & T_{cur} = (2k+1)T_{max}. \end{aligned} When last_epoch=-1, sets initial lr as lr. Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes: .. math:: \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right) It has been proposed in `SGDR: Stochastic Gradient Descent with Warm Restarts`_. Note that this only implements the cosine annealing part of SGDR, and not the restarts. Args: optimizer (Optimizer): Wrapped optimizer. T_max (int): Maximum number of iterations. eta_min (float): Minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. .. _SGDR\: Stochastic Gradient Descent with Warm Restarts: https://arxiv.org/abs/1608.03983 rr Fcs$||_||_tt||||dSrH)T_maxeta_minrsrr7)r5r,rrr2r4rur rr7szCosineAnnealingLR.__init__csjstdtjdkr.ddjjDSjdkr`jdkr`fddtj jjDSjdj dj dkrfddtj jjDSfd djjDS) NrrcSsg|] }|dqSrbr rr r rrrz,CosineAnnealingLR.get_lr..rc s@g|]8\}}j|jdtjtjjdqSr)rmathcosr2pirrrrr@r rrs  rc s<g|]4\}}|d|jdttjjdqS)r rr)rrrrrrr@r rrs c s\g|]T}dttjjjdttjjdj|djjqS)rr )rrrr2rrrr@r rrs ) rWrdrerfr2r,r-rrhr1rr@r r@rrJs"    zCosineAnnealingLR.get_lrcsfddjDS)Nc s<g|]4}j|jdttjjjdqSr)rrrrr2rrr@r rrs  z9CosineAnnealingLR._get_closed_form_lr..rr@r r@rras z%CosineAnnealingLR._get_closed_form_lr)rr Frr r rurrs'rc@s0eZdZdZddZddZddZdd Zd S) ChainedSchedulera\Chains list of learning rate schedulers. It takes a list of chainable learning rate schedulers and performs consecutive step() functions belong to them by just one call. Args: schedulers (list): List of chained schedulers. Example: >>> # Assuming optimizer uses lr = 1. for all groups >>> # lr = 0.09 if epoch == 0 >>> # lr = 0.081 if epoch == 1 >>> # lr = 0.729 if epoch == 2 >>> # lr = 0.6561 if epoch == 3 >>> # lr = 0.59049 if epoch >= 4 >>> scheduler1 = ConstantLR(self.opt, factor=0.1, total_iters=2) >>> scheduler2 = ExponentialLR(self.opt, gamma=0.9) >>> scheduler = ChainedScheduler([scheduler1, scheduler2]) >>> for epoch in range(100): >>> train(...) >>> validate(...) >>> scheduler.step() cCsltdt|D](}||j|djkrtdd|qt||_|dj|_dd|jdjjD|_dS)Nrrz~ChainedScheduler expects all schedulers to belong to the same optimizer, but got schedulers at index {} and {} to be differentcSsg|] }|dqSrbr rr r rrrz-ChainedScheduler.__init__..r ) rrpr,rrr)rnrr-rF)r5rrr r rr7 s  zChainedScheduler.__init__cCs2|jD] }|qdd|jdjjD|_dS)NcSsg|] }|dqSrbr rr r rrrz)ChainedScheduler.step..r )rr3r,r-rF)r5Z schedulerr r rr3s  zChainedScheduler.stepcCsPdd|jD}dgt|j|d<t|jD]\}}||d|<q2|S)rcSsi|]\}}|dvr||qSrr r9r r rr<!rz/ChainedScheduler.state_dict..Nrrrr r rrAs zChainedScheduler.state_dictcCsD|d}|j|||d<t|D]\}}|j||q&dSrrrr r rrE)s   z ChainedScheduler.load_state_dictNrr r r rrs  rc @s`eZdZdZdd d Zd d ZdddZddZeddZ ddZ ddZ ddZ ddZ dS)ReduceLROnPlateaua Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler reads a metrics quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced. Args: optimizer (Optimizer): Wrapped optimizer. mode (str): One of `min`, `max`. In `min` mode, lr will be reduced when the quantity monitored has stopped decreasing; in `max` mode it will be reduced when the quantity monitored has stopped increasing. Default: 'min'. factor (float): Factor by which the learning rate will be reduced. new_lr = lr * factor. Default: 0.1. patience (int): Number of epochs with no improvement after which learning rate will be reduced. For example, if `patience = 2`, then we will ignore the first 2 epochs with no improvement, and will only decrease the LR after the 3rd epoch if the loss still hasn't improved then. Default: 10. threshold (float): Threshold for measuring the new optimum, to only focus on significant changes. Default: 1e-4. threshold_mode (str): One of `rel`, `abs`. In `rel` mode, dynamic_threshold = best * ( 1 + threshold ) in 'max' mode or best * ( 1 - threshold ) in `min` mode. In `abs` mode, dynamic_threshold = best + threshold in `max` mode or best - threshold in `min` mode. Default: 'rel'. cooldown (int): Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 0. min_lr (float or list): A scalar or a list of scalars. A lower bound on the learning rate of all param groups or each group respectively. Default: 0. eps (float): Minimal decay applied to lr. If the difference between new and old lr is smaller than eps, the update is ignored. Default: 1e-8. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = ReduceLROnPlateau(optimizer, 'min') >>> for epoch in range(10): >>> train(...) >>> val_loss = validate(...) >>> # Note that step should be called after validate() >>> scheduler.step(val_loss) rr -C6?relr:0yE>Fc Cs|dkrtd||_t|ts4tdt|j||_t|t sNt|t rt |t |j krztdt |j t |t ||_ n|gt |j |_ ||_| |_||_d|_||_||_||_d|_d|_d|_| |_d|_|j|||d|dS)NrzFactor should be < 1.0.r zexpected {} min_lrs, got {}rmode thresholdthreshold_mode)rrrr'rr(r)r*r+r,rnrorpr-min_lrspatiencer4cooldowncooldown_counterrrrbestnum_bad_epochs mode_worseepsr2_init_is_better_reset) r5r,rrrrrrmin_lrrr4r r rr7ks>  zReduceLROnPlateau.__init__cCs|j|_d|_d|_dS)z3Resets num_bad_epochs counter and cooldown counter.rN)rrrrr@r r rrszReduceLROnPlateau._resetNcCst|}|dur|jd}n ttt||_|||jrJ||_d|_n|jd7_|j rr|j d8_ d|_|j|j kr| ||j |_ d|_dd|jjD|_dS)NrrcSsg|] }|dqSrbr rr r rrrz*ReduceLROnPlateau.step..)rNr2rdrergrf is_betterrr in_cooldownrr _reduce_lrrr,r-rF)r5ZmetricsrOcurrentr r rr3s"    zReduceLROnPlateau.stepcCst|jjD]n\}}t|d}t||j|j|}|||jkr ||d<|jr t |tr`dnd|}t d |||q dS)Nr rKrLz7Epoch {}: reducing learning rate of group {} to {:.4e}.) r/r,r-rNmaxrrrr4r'rMr))r5rOr6rkZold_lrZnew_lrrPr r rrs zReduceLROnPlateau._reduce_lrcCs |jdkS)Nr)rr@r r rrszReduceLROnPlateau.in_cooldowncCs|jdkr*|jdkr*d|j}|||kS|jdkrL|jdkrL|||jkS|jdkrv|jdkrv|jd}|||kS|||jkSdS)Nrrrabsr)rrr)r5arZ rel_epsilonr r rrs    zReduceLROnPlateau.is_bettercCs^|dvrtd|d|dvr0td|d|dkr@t|_nt |_||_||_||_dS)N>rrzmode z is unknown!>rrzthreshold mode r)rrrrrrr)r5rrrr r rrsz!ReduceLROnPlateau._init_is_bettercCsdd|jDS)NcSsi|]\}}|dkr||qSr8r r9r r rr<rz0ReduceLROnPlateau.state_dict..r=r@r r rrAszReduceLROnPlateau.state_dictcCs&|j||j|j|j|jddS)Nr)r>rCrrrrrDr r rrEs z!ReduceLROnPlateau.load_state_dict) rrrrrrrrF)N)r+r^r_rr7rr3rpropertyrrrrArEr r r rr:s0 &   rc sJeZdZdZdfd d ZddZddZddZddZddZ Z S)CyclicLRa!Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper `Cyclical Learning Rates for Training Neural Networks`_. The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis. Cyclical learning rate policy changes the learning rate after every batch. `step` should be called after a batch has been used for training. This class has three built-in policies, as put forth in the paper: * "triangular": A basic triangular cycle without amplitude scaling. * "triangular2": A basic triangular cycle that scales initial amplitude by half each cycle. * "exp_range": A cycle that scales initial amplitude by :math:`\text{gamma}^{\text{cycle iterations}}` at each cycle iteration. This implementation was adapted from the github repo: `bckenstler/CLR`_ Args: optimizer (Optimizer): Wrapped optimizer. base_lr (float or list): Initial learning rate which is the lower boundary in the cycle for each parameter group. max_lr (float or list): Upper learning rate boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_lr - base_lr). The lr at any cycle is the sum of base_lr and some scaling of the amplitude; therefore max_lr may not actually be reached depending on scaling function. step_size_up (int): Number of training iterations in the increasing half of a cycle. Default: 2000 step_size_down (int): Number of training iterations in the decreasing half of a cycle. If step_size_down is None, it is set to step_size_up. Default: None mode (str): One of {triangular, triangular2, exp_range}. Values correspond to policies detailed above. If scale_fn is not None, this argument is ignored. Default: 'triangular' gamma (float): Constant in 'exp_range' scaling function: gamma**(cycle iterations) Default: 1.0 scale_fn (function): Custom scaling policy defined by a single argument lambda function, where 0 <= scale_fn(x) <= 1 for all x >= 0. If specified, then 'mode' is ignored. Default: None scale_mode (str): {'cycle', 'iterations'}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: 'cycle' cycle_momentum (bool): If ``True``, momentum is cycled inversely to learning rate between 'base_momentum' and 'max_momentum'. Default: True base_momentum (float or list): Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is 'base_momentum' and learning rate is 'max_lr'. Default: 0.8 max_momentum (float or list): Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). The momentum at any cycle is the difference of max_momentum and some scaling of the amplitude; therefore base_momentum may not actually be reached depending on scaling function. Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is 'max_momentum' and learning rate is 'base_lr' Default: 0.9 last_epoch (int): The index of the last batch. This parameter is used when resuming a training job. Since `step()` should be invoked after each batch instead of after each epoch, this number represents the total number of *batches* computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1 verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.01, max_lr=0.1) >>> data_loader = torch.utils.data.DataLoader(...) >>> for epoch in range(10): >>> for batch in data_loader: >>> train_batch(...) >>> scheduler.step() .. _Cyclical Learning Rates for Training Neural Networks: https://arxiv.org/abs/1506.01186 .. _bckenstler/CLR: https://github.com/bckenstler/CLR N triangularrcycleT皙??r Fcst|tstdt|j||_|d||}| dkrXt||j D]\}}||d<qF|d|||_ t |}|durt |n|}|||_ ||j |_ |dvr|durtd||_||_|dur|jdkr|j|_d |_n4|jd kr|j|_d |_n|jd kr$|j|_d |_n ||_| |_| |_| rd |jvrDtd|d|| }| dkr|t||j D]\}}||d <qhdd|j D|_|d|| |_tt||| |||_dS)Nr rr r max_lr)r triangular2 exp_rangez$mode is invalid and scale_fn is Nonerrrr iterationsmomentumDoptimizer must support momentum with `cycle_momentum` option enabled base_momentumcSsg|] }|dqS)rr rr r rrrz%CyclicLR.__init__.. max_momentum)r'rr(r)r*r+r, _format_paramrhr-max_lrsrN total_size step_ratiorrrr_triangular_scale_fnscale_fn scale_mode_triangular2_scale_fn_exp_range_scale_fncycle_momentumdefaultsbase_momentums max_momentumsrsrr7r1)r5r,rrZ step_size_upZstep_size_downrrrrrrrr2r4r1r rrrrur rr7EsX           zCyclicLR.__init__cCsTt|ttfr@t|t|jkr= 1, but got {}) r'intrrr)T_0T_iT_multrT_currsrr7)r5r,rrrr2r4rur rr7sz$CosineAnnealingWarmRestarts.__init__cs&jstdtfddjDS)Nrc s<g|]4}j|jdttjjjdqSr)rrrrrrrr@r rrsz6CosineAnnealingWarmRestarts.get_lr..)rWrdrerfr1r@r r@rrJs z"CosineAnnealingWarmRestarts.get_lrNcCs|dur|jdkrd}|dur^|jd}|jd|_|j|jkr\|j|j|_|j|j|_n|dkrttd|||jkr|jdkr||j|_n\tt ||j|jdd|j}||j|j|d|jd|_|j|j||_n|j|_||_t ||_Gddd}||Tt t |j j|D],\}}|\}}||d<||j|||q>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult) >>> iters = len(dataloader) >>> for epoch in range(20): >>> for i, sample in enumerate(dataloader): >>> inputs, labels = sample['inputs'], sample['labels'] >>> optimizer.zero_grad() >>> outputs = net(inputs) >>> loss = criterion(outputs, labels) >>> loss.backward() >>> optimizer.step() >>> scheduler.step(epoch + i / iters) This function can be called in an interleaved way. Example: >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult) >>> for epoch in range(20): >>> scheduler.step() >>> scheduler.step(26) >>> scheduler.step() # scheduler.step(27), instead of scheduler(20) Nrrz'Expected non-negative epoch, but got {}c@s$eZdZddZddZddZdS)z=CosineAnnealingWarmRestarts.step.._enable_get_lr_callcSs ||_dSrHrRrTr r rr7.szFCosineAnnealingWarmRestarts.step.._enable_get_lr_call.__init__cSs d|j_|SrUrVr@r r rrX1szGCosineAnnealingWarmRestarts.step.._enable_get_lr_call.__enter__cSs d|j_|SrYrVrZr r rr\5szFCosineAnnealingWarmRestarts.step.._enable_get_lr_call.__exit__Nr]r r r rr`,sr`r cSsg|] }|dqSrbr rr r rr?rz4CosineAnnealingWarmRestarts.step..)r2rrrrrr)rrrlogrr/rhr,r-rJrQr4rF)r5rOnr`r6rjrkr r r rr3s4     &$  6z CosineAnnealingWarmRestarts.step)rrr F)N)r+r^r_rr7rJr3rr r rurrs rc sBeZdZdZdfd d ZddZddZddZddZZ S) OneCycleLRaSets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper `Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates`_. The 1cycle learning rate policy changes the learning rate after every batch. `step` should be called after a batch has been used for training. This scheduler is not chainable. Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence): #. A value for total_steps is explicitly provided. #. A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided. In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch. The default behaviour of this scheduler follows the fastai implementation of 1cycle, which claims that "unpublished work has shown even better results by using only two phases". To mimic the behaviour of the original paper instead, set ``three_phase=True``. Args: optimizer (Optimizer): Wrapped optimizer. max_lr (float or list): Upper learning rate boundaries in the cycle for each parameter group. total_steps (int): The total number of steps in the cycle. Note that if a value is not provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None epochs (int): The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None steps_per_epoch (int): The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None pct_start (float): The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.3 anneal_strategy (str): {'cos', 'linear'} Specifies the annealing strategy: "cos" for cosine annealing, "linear" for linear annealing. Default: 'cos' cycle_momentum (bool): If ``True``, momentum is cycled inversely to learning rate between 'base_momentum' and 'max_momentum'. Default: True base_momentum (float or list): Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is 'base_momentum' and learning rate is 'max_lr'. Default: 0.85 max_momentum (float or list): Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is 'max_momentum' and learning rate is 'base_lr' Default: 0.95 div_factor (float): Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25 final_div_factor (float): Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4 three_phase (bool): If ``True``, use a third phase of the schedule to annihilate the learning rate according to 'final_div_factor' instead of modifying the second phase (the first two phases will be symmetrical about the step indicated by 'pct_start'). last_epoch (int): The index of the last batch. This parameter is used when resuming a training job. Since `step()` should be invoked after each batch instead of after each epoch, this number represents the total number of *batches* computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1 verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. Example: >>> data_loader = torch.utils.data.DataLoader(...) >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10) >>> for epoch in range(10): >>> for batch in data_loader: >>> train_batch(...) >>> scheduler.step() .. _Super-Convergence\: Very Fast Training of Neural Networks Using Large Learning Rates: https://arxiv.org/abs/1708.07120 N333333?rT333333?ffffff?9@@Fr cst|tstdt|j||_|durF|durF|durFtdnz|durv|dks`t|tsntd|||_ nJ|dkst|tstd||dkst|tstd||||_ | rt ||j ddd d d d t d ||j d d dd d d |j dddd d d g|_ n6t ||j ddd d d d |j dd dd d d g|_ |dkst|dkstt|t std||dvrtd|n&|dkr|j |_ n|dkr|j|_ |d |j|}|dkr t|jjD]6\}}||| |d<|||d <|d| |d<q||_|jrd|jjvrRd|jjvrRtdd|jjv|_|d || }|d || }|dkrt|||jD]F\}}}|jr|d\}}||f|d<n||d<||d <||d <qtt||||dS)Nr zBYou must define either total_steps OR (epochs AND steps_per_epoch)rz1Expected positive integer total_steps, but got {}z,Expected positive integer epochs, but got {}z5Expected positive integer steps_per_epoch, but got {}rr rrr)end_stepstart_lrend_lrstart_momentum end_momentumrrz4Expected float between 0 and 1 pct_start, but got {})rlinearz@anneal_strategy must by one of 'cos' or 'linear', instead got {}rrr rbetasr)r'rr(r)r*r+r,rrr total_stepsrN_schedule_phases_annealing_cos anneal_func_annealing_linearrr/r-rr use_beta1rhrsrr7)r5r,rrZepochsZsteps_per_epochZ pct_startZanneal_strategyrrrZ div_factorZfinal_div_factorZ three_phaser2r4rr|rrrZ m_momentumZ b_momentum_beta2rur rr7s             zOneCycleLR.__init__cCsTt|ttfr@t|t|jkrs8      UP009C^IH.cy