a 7Sicf6 @s|ddlZddlZddlmZddlmZmZmZmZmZm Z ddl m Z m Z m Z ddlZeeefZeeeedffZe eee edddZeeeedffed d d Zeeegefed d dZeeeeeeefdddZd%eeeedffeeeeeedddZeeddddZeeddddZedddZd&eeeedd d!Zd'eeeeed"d#d$ZdS)(N)Tensor)AnyCallableOptionalTupleUnionList) tree_flattentree_unflatten_broadcast_to_and_flatten.) flat_in_dims flat_argsreturncsFddt||Dr>tfddDr>tdddS)NcSs"g|]\}}|dur||qSN)size.0in_dimargrQ/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/_vmap_internals.py s z0_validate_and_get_batch_size..csg|]}|dkqS)rr)rr batch_sizesrrrzTvmap: Expected all tensors to have the same size in the mapped dimension, got sizes z for the mapped dimensionr)zipany ValueError)r r rrr_validate_and_get_batch_size sr)batched_outputsrcCst|trt|SdS)N) isinstancetuplelen)rrrr _num_outputss r$)value num_elementserror_message_lambdarcCs.t|ts|f|St||kr*t||Sr)r!r"r#r)r%r&r'rrr _as_tuples     r()in_dimsargs vmap_levelfuncrc st|ts8t|ts8tdt|d|dt|dt|dkrXtdt|dt|\}}t||}|durtdt|d|dt|dd |d t ||D]\}}t|ts|durtdt|d|d |d t|tr*t|t s*tdt|d|d |d t|d |dur|dksH|| krtdt|d|d |d| d| d qt ||} fddt ||D} t | || fS)Nvmap(z , in_dims=zv, ...)(): expected `in_dims` to be int or a (potentially nested) tuple matching the structure of inputs, got: .rz)(): got no inputs. Maybe you forgot to add inputs, or you are trying to vmap over a function with no inputs. The latter is unsupported.zb, ...)(): in_dims is not compatible with the structure of `inputs`. in_dims has structure r z but inputs has structure z, ...)(): Got in_dim=zE for an input but in_dim must be either an integer dimension or None.z' for an input but the input is of type zT. We cannot vmap over non-Tensor arguments, please use None as the respective in_dimz> for some input, but that input is a Tensor of dimensionality z- so expected in_dim to satisfy 0 <= in_dim < cs*g|]"\}}|dur|n t||qSr)torchZ_add_batch_dimrr+rrrRs z*_create_batched_inputs..)r!intr"r _get_nametyper#r r rrdimrr ) r)r*r+r,r Z args_specr rr batch_sizebatched_inputsrr0r_create_batched_inputs'sd      r7F)rout_dimsr+r5r,allow_none_pass_throughrcst|tfdd}t|trB|d}t||S|rdtfddt||DStfddt||DSdS)Nc s&dtdddtd S)Nr-, ..., out_dims=z0): `out_dims` must have one dim per output (got z outputs) of r.)r2r)r, num_outputsr8rr_sz!_unwrap_batched..rc3s.|]&\}}|dur"t||ndVqdSrr/_remove_batch_dimroutout_dimr5r+rr isz"_unwrap_batched..c3s"|]\}}t||VqdSrr=r?rBrrrCls)r$r(r!rr/r>r"r)rr8r+r5r,r9Zout_dims_as_tuplerAr)r5r,r;r8r+r_unwrap_batchedXs rD)outputsr,rc Cst|trdSt|ts@tdt|dt|dt|dt|D]B\}}t|tr\qHtdt|dt|dt|d|d qHdS)Nr-z , ...): `z%` must only return Tensors, got type z as the return.z for return r.)r!rr"rr2r3 enumerate)rEr,idxoutputrrr_validate_outputsss    rI)r8r,rcCsHt|trdSt|tr*tdd|DsDtdt|d|ddS)NcSsg|]}t|tqSr)r!r1)rrArrrrrz7_check_out_dims_is_int_or_int_tuple..r-r:zu): `out_dims` must be an int or a tuple of int representing where in the outputs the vmapped dimension should appear.)r!r1r"allrr2)r8r,rrr#_check_out_dims_is_int_or_int_tuples  rKr,cCst|dr|jSt|S)N__name__)hasattrrMreprrLrrrr2s r2)r,r)r8rcCstjdddt|||S)aN vmap is the vectorizing map. Returns a new function that maps `func` over some dimension of the inputs. Semantically, vmap pushes the map into PyTorch operations called by `func`, effectively vectorizing those operations. vmap is useful for handling batch dimensions: one can write a function `func` that runs on examples and then lift it to a function that can take batches of examples with `vmap(func)`. vmap can also be used to compute batched gradients when composed with autograd. .. note:: We have moved development of vmap to `functorch. `_ functorch's vmap is able to arbitrarily compose with gradient computation and contains significant performance improvements. Please give that a try if that is what you're looking for. Furthermore, if you're interested in using vmap for your use case, please `contact us! `_ We're interested in gathering feedback from early adopters to inform the design. .. warning:: torch.vmap is an experimental prototype that is subject to change and/or deletion. Please use at your own risk. Args: func (function): A Python function that takes one or more arguments. Must return one or more Tensors. in_dims (int or nested structure): Specifies which dimension of the inputs should be mapped over. `in_dims` should have a structure like the inputs. If the `in_dim` for a particular input is None, then that indicates there is no map dimension. Default: 0. out_dims (int or Tuple[int]): Specifies where the mapped dimension should appear in the outputs. If `out_dims` is a Tuple, then it should have one element per output. Default: 0. Returns: Returns a new "batched" function. It takes the same inputs as `func`, except each input has an extra dimension at the index specified by `in_dims`. It takes returns the same outputs as `func`, except each output has an extra dimension at the index specified by `out_dims`. .. warning: vmap works best with functional-style code. Please do not perform any side-effects in `func`, with the exception of in-place PyTorch operations. Examples of side-effects include mutating Python data structures and assigning values to variables not captured in `func`. One example of using `vmap` is to compute batched dot products. PyTorch doesn't provide a batched `torch.dot` API; instead of unsuccessfully rummaging through docs, use `vmap` to construct a new function. >>> torch.dot # [D], [D] -> [] >>> batched_dot = torch.vmap(torch.dot) # [N, D], [N, D] -> [N] >>> x, y = torch.randn(2, 5), torch.randn(2, 5) >>> batched_dot(x, y) `vmap` can be helpful in hiding batch dimensions, leading to a simpler model authoring experience. >>> batch_size, feature_size = 3, 5 >>> weights = torch.randn(feature_size, requires_grad=True) >>> >>> def model(feature_vec): >>> # Very simple linear model with activation >>> return feature_vec.dot(weights).relu() >>> >>> examples = torch.randn(batch_size, feature_size) >>> result = torch.vmap(model)(examples) `vmap` can also help vectorize computations that were previously difficult or impossible to batch. One example is higher-order gradient computation. The PyTorch autograd engine computes vjps (vector-Jacobian products). Computing a full Jacobian matrix for some function f: R^N -> R^N usually requires N calls to `autograd.grad`, one per Jacobian row. Using `vmap`, we can vectorize the whole computation, computing the Jacobian in a single call to `autograd.grad`. >>> # Setup >>> N = 5 >>> f = lambda x: x ** 2 >>> x = torch.randn(N, requires_grad=True) >>> y = f(x) >>> I_N = torch.eye(N) >>> >>> # Sequential approach >>> jacobian_rows = [torch.autograd.grad(y, x, v, retain_graph=True)[0] >>> for v in I_N.unbind()] >>> jacobian = torch.stack(jacobian_rows) >>> >>> # vectorized gradient computation >>> def get_vjp(v): >>> return torch.autograd.grad(y, x, v) >>> jacobian = torch.vmap(get_vjp)(I_N) .. note:: vmap does not provide general autobatching or handle variable-length sequences out of the box. zPlease use functorch.vmap instead of torch.vmap (https://github.com/pytorch/functorch). We've moved development on torch.vmap over to functorch; functorch's vmap has a multitude of significant performance and functionality improvements.) stacklevel)warningswarn_vmap)r,r)r8rrrvmaps erU)r,r)r8r9rcs tfdd}|S)Ncsnttj}zHt||\}}|}s>t|t|||dWtjStj0dS)N)r9)rKr/_C_vmapmode_increment_nestingr7rIrD_vmapmode_decrement_nesting)r*r+r6r5rr9r,r)r8rrwrapped s     z_vmap..wrapped) functoolswraps)r,r)r8r9rZrrYrrTs rT)F)rr)rrF)r/r[rtypingrrrrrrZtorch.utils._pytreer r r rRr1Z in_dims_tZ out_dims_trr$strr(r7boolrDrIrKr2rUrTrrrrs2       4    o