a 7SicF@sddlZddlZddlmZmZmZmZmZmZeedfeej dfdddZ eeddddZ eee eej fd d d Z dd d d ZGdddejjZddedddZddZdddZdS)N)AnyDictIterableListOptionalTuple.)inputsreturncCsft|trRg}|D]6}t|tjs.||q|}|j|_||qt|Stdt|j dS)Nz@Only tuple of tensors is supported. Got Unsupported input type: ) isinstancetupletorchTensorappenddetach requires_grad RuntimeErrortype__name__)routinpxrR/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/utils/checkpoint.pydetach_variables     rcCs tdd|DstddS)Ncss |]}t|tjr|jVqdSN)r r r r.0rrrr z*check_backward_validity..zBNone of the inputs have requires_grad=True. Gradients will be None)anywarningswarn)rrrrcheck_backward_validitysr")r c Gshttdd|D}g}|D]@}tj| |tjWdq1sT0Yq||fS)Ncss(|] }t|tjr|jr|VqdSr)r r r is_cuda get_devicerargrrrr&sz$get_device_states..)listsetr cudadevicer get_rng_state)argsfwd_gpu_devicesfwd_gpu_statesr*rrrget_device_states#s 0r/c CsPt||D]@\}}tj|tj|Wdq 1s@0Yq dSr)zipr r)r* set_rng_state)devicesstatesr*staterrrset_device_states1sr5c@s$eZdZeddZeddZdS)CheckpointFunctioncGst|||_||_tttd|_tt td|_ |rzt |_ d|_ tjjrzd|_ t|\|_|_g|_g|_g}t|D]B\}}t|r|||j||jdq|j|q|j|t||}Wdn1s0Y|S)N)enableddtype cache_enabledFT)r" run_functionpreserve_rng_stater is_autocast_enabledget_autocast_gpu_dtypeis_autocast_cache_enabledgpu_autocast_kwargsis_autocast_cpu_enabledget_autocast_cpu_dtypecpu_autocast_kwargsr+ fwd_cpu_statehad_cuda_in_fwdr) _initializedr/r-r.rtensor_indices enumerate is_tensorrsave_for_backwardno_grad)ctxr:r;r, tensor_inputsir&outputsrrrforward9s<      (zCheckpointFunction.forwardc Gs tjstdt|j}|j}|j}t|D]\}}||||<q0g}|j r\|j r\|j }tj j ||j d|j rt|j|j rt|j |jtt|}ttjjjfi|jPtjjjfi|j|j|} Wdn1s0YWdn1s0YWdn1s:0YWdn1sZ0Yt| tjrx| f} g} g} tt| D]>}t | |r| |j!r| "| || "||qt| dkrtdtj#| | tdd|D} d| S)NzCheckpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward(). Please use .backward() and do not pass its `inputs` argument.r2r7rzInone of output has requires_grad=True, this checkpoint() is not necessarycss$|]}t|tjr|jndVqdSr)r r r gradrrrrrsz.CheckpointFunction.backward..)NN)$r autograd_is_checkpoint_validrr'rrF saved_tensorsrGr;rDr-randomfork_rngr1rCr5r.rr enable_gradr)ampautocastr?cpurBr:r r rangelenrHrrbackward) rKr,rrFtensorsrMidx rng_devicesZdetached_inputsrNZoutputs_with_gradZargs_with_gradgradsrrrr]csR      zCheckpointFunction.backwardN)r __module__ __qualname__ staticmethodrOr]rrrrr67s )r6T) use_reentrantcOsZ|dd}|r,tdddd|D|rDtj||g|RSt||g|RSdS)aCheckpoint a model or part of the model Checkpointing works by trading compute for memory. Rather than storing all intermediate activations of the entire computation graph for computing backward, the checkpointed part does **not** save intermediate activations, and instead recomputes them in backward pass. It can be applied on any part of a model. Specifically, in the forward pass, :attr:`function` will run in :func:`torch.no_grad` manner, i.e., not storing the intermediate activations. Instead, the forward pass saves the inputs tuple and the :attr:`function` parameter. In the backwards pass, the saved inputs and :attr:`function` is retrieved, and the forward pass is computed on :attr:`function` again, now tracking the intermediate activations, and then the gradients are calculated using these activation values. The output of :attr:`function` can contain non-Tensor values and gradient recording is only performed for the Tensor values. Note that if the output consists of nested structures (ex: custom objects, lists, dicts etc.) consisting of Tensors, these Tensors nested in custom structures will not be considered as part of autograd. .. warning:: If :attr:`function` invocation during backward does anything different than the one during forward, e.g., due to some global variable, the checkpointed version won't be equivalent, and unfortunately it can't be detected. .. warning:: If ``use_reentrant=True`` is specified, then if the checkpointed segment contains tensors detached from the computational graph by `detach()` or `torch.no_grad()`, the backward pass will raise an error. This is because `checkpoint` makes all the outputs require gradients which causes issues when a tensor is defined to have no gradient in the model. To circumvent this, detach the tensors outside of the `checkpoint` function. Note that the checkpointed segment can contain tensors detached from the computational graph if ``use_reentrant=False`` is specified. .. warning:: If ``use_reentrant=True`` is specified, at least one of the inputs needs to have :code:`requires_grad=True` if grads are needed for model inputs, otherwise the checkpointed part of the model won't have gradients. At least one of the outputs needs to have :code:`requires_grad=True` as well. Note that this does not apply if ``use_reentrant=False`` is specified. .. warning:: If ``use_reentrant=True`` is specified, checkpointing currently only supports :func:`torch.autograd.backward` and only if its `inputs` argument is not passed. :func:`torch.autograd.grad` is not supported. If ``use_reentrant=False`` is specified, checkpointing will work with :func:`torch.autograd.grad`. Args: function: describes what to run in the forward pass of the model or part of the model. It should also know how to handle the inputs passed as the tuple. For example, in LSTM, if user passes ``(activation, hidden)``, :attr:`function` should correctly use the first input as ``activation`` and the second input as ``hidden`` preserve_rng_state(bool, optional, default=True): Omit stashing and restoring the RNG state during each checkpoint. use_reentrant(bool, optional, default=True): Use checkpointing implementation that requires re-entrant autograd. If ``use_reentrant=False`` is specified, ``checkpoint`` will use an implementation that does not require re-entrant autograd. This allows ``checkpoint`` to support additional functionality, such as working as expected with ``torch.autograd.grad``. Note that future versions of PyTorch will default to ``use_reentrant=False``. args: tuple containing inputs to the :attr:`function` Returns: Output of running :attr:`function` on :attr:`*args` r;TUnexpected keyword arguments: ,css|] }|VqdSrrr%rrrrrzcheckpoint..N)pop ValueErrorjoinr6apply_checkpoint_without_reentrant)functionrer,kwargspreserverrr checkpointsM rpc Ks|dd}|r,tdddd|Ddd}t|tjjrNt|}t ||}d }t d ||d |D]&}||d }t ||||||d }qr||d t |d ||S) aA helper function for checkpointing sequential models. Sequential models execute a list of modules/functions in order (sequentially). Therefore, we can divide such a model in various segments and checkpoint each segment. All segments except the last will run in :func:`torch.no_grad` manner, i.e., not storing the intermediate activations. The inputs of each checkpointed segment will be saved for re-running the segment in the backward pass. See :func:`~torch.utils.checkpoint.checkpoint` on how checkpointing works. .. warning:: Checkpointing currently only supports :func:`torch.autograd.backward` and only if its `inputs` argument is not passed. :func:`torch.autograd.grad` is not supported. .. warning: At least one of the inputs needs to have :code:`requires_grad=True` if grads are needed for model inputs, otherwise the checkpointed part of the model won't have gradients. .. warning: Since PyTorch 1.4, it allows only one Tensor as the input and intermediate outputs, just like :class:`torch.nn.Sequential`. Args: functions: A :class:`torch.nn.Sequential` or the list of modules or functions (comprising the model) to run sequentially. segments: Number of chunks to create in the model input: A Tensor that is input to :attr:`functions` preserve_rng_state(bool, optional, default=True): Omit stashing and restoring the RNG state during each checkpoint. Returns: Output of running :attr:`functions` sequentially on :attr:`*inputs` Example: >>> model = nn.Sequential(...) >>> input_var = checkpoint_sequential(model, chunks, input_var) r;Trfrgcss|] }|VqdSrrr%rrrr rz(checkpoint_sequential..csfdd}|S)Ncs$tdD]}||}q|SN)r[)inputjend functionsstartrrrO#sz.run_function..forwardr)rxrvrwrOrrurr:"sz+checkpoint_sequential..run_functionrrr)r;) rhrirjr r nn Sequentialr'childrenr\r[rp) rwsegmentsrsrnror:Z segment_sizervrxrrrcheckpoint_sequentials*    r~c str0tdtjjr0dt\i dfdd} f dd}tjj||0}tjjrrst dWd n1s0Y|S) a}Checkpointining without re-entrant autograd Args: function: describes what to run in the forward pass of the model or part of the model. It should also know how to handle the inputs passed as the tuple. For example, in LSTM, if user passes ``(activation, hidden)``, :attr:`function` should correctly use the first input as ``activation`` and the second input as ``hidden`` preserve_rng_state(bool, optional, default=True): Omit stashing and restoring the RNG state during each checkpoint. *args: Arguments to pass in to the given ``function``. FTrcsd7dSrqr)r)counterrrpackRsz+_checkpoint_without_reentrant..packc s0dt dkr fdd}dd}g}r8r8}tjj|drftrfttvtjj Htj j ||}Wdn1s0YWdn1s0YWdn1s0YWdn1s 0Y| vr&t d |S)Nrcs|<d7dSrqr)inner)storageunpack_counterrr inner_pack]szA_checkpoint_without_reentrant..unpack..inner_packcSs tddS)NzRYou are calling backwards on a tensor that is never exposed. Please open an issue.)r)packedrrr inner_unpackcszC_checkpoint_without_reentrant..unpack..inner_unpackrPzAttempt to retrieve a tensor saved by autograd multiple times without checkpoint recomputation being triggered in between, this is not currently supported. Please open an issue with details on your use case so that we can prioritize adding this.)r\r rUrVr1r5rWr)rXrYrRgraphsaved_tensors_hooksrrh)rrrr`Z_unused) r,rmrCr-r.had_autocast_in_fwdrDr;r)rrunpackYs(   z-_checkpoint_without_reentrant..unpackzPyTorch's CUDA state was initialized in the forward pass of a Checkpoint, which is not allowed. Please open an issue if you need this feature.N) r r<r+r)rEr/rRrrr)rmr;r,rroutputr) r,rrmrCr-r.rrDr;rrrl5s$   %"rl)T)r r typingrrrrrrr rr"intr/r5rRFunctionr6boolrpr~rlrrrrs " b[A