a 7Sic /@s,ddlZddlZddlmZmZmZmZmZmZm Z m Z ddl m Z m Z ddlmZddlmZmZgdZe dd d Ze d ZGd d d eeZGdddeeZGdddee edfZGdddeeZGdddeZGdddeeZe feeeeeeeeedddZdS)N)GenericIterableIteratorListOptionalSequenceTupleTypeVar)default_generatorrandperm) _accumulate) GeneratorTensor)DatasetIterableDataset TensorDataset ConcatDataset ChainDatasetSubset random_splitT_coT) covariantTc@s.eZdZdZedddZddddd Zd S) raAn abstract class representing a :class:`Dataset`. All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite :meth:`__getitem__`, supporting fetching a data sample for a given key. Subclasses could also optionally overwrite :meth:`__len__`, which is expected to return the size of the dataset by many :class:`~torch.utils.data.Sampler` implementations and the default options of :class:`~torch.utils.data.DataLoader`. .. note:: :class:`~torch.utils.data.DataLoader` by default constructs a index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided. returncCstdSNNotImplementedErrorselfindexr"T/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/utils/data/dataset.py __getitem__2szDataset.__getitem__z Dataset[T_co]zConcatDataset[T_co])otherrcCs t||gSr)rr r%r"r"r#__add__5szDataset.__add__N)__name__ __module__ __qualname____doc__rr$r'r"r"r"r#r"src@s4eZdZdZeedddZeedddZdS) raAn iterable Dataset. All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream. All subclasses should overwrite :meth:`__iter__`, which would return an iterator of samples in this dataset. When a subclass is used with :class:`~torch.utils.data.DataLoader`, each item in the dataset will be yielded from the :class:`~torch.utils.data.DataLoader` iterator. When :attr:`num_workers > 0`, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. :func:`~torch.utils.data.get_worker_info`, when called in a worker process, returns information about the worker. It can be used in either the dataset's :meth:`__iter__` method or the :class:`~torch.utils.data.DataLoader` 's :attr:`worker_init_fn` option to modify each copy's behavior. Example 1: splitting workload across all workers in :meth:`__iter__`:: >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example code only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... worker_info = torch.utils.data.get_worker_info() ... if worker_info is None: # single-process data loading, return the full iterator ... iter_start = self.start ... iter_end = self.end ... else: # in a worker process ... # split workload ... per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... iter_start = self.start + worker_id * per_worker ... iter_end = min(iter_start + per_worker, self.end) ... return iter(range(iter_start, iter_end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [3, 4, 5, 6] >>> # Mult-process loading with two worker processes >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [3, 5, 4, 6] >>> # With even more workers >>> print(list(torch.utils.data.DataLoader(ds, num_workers=20))) [3, 4, 5, 6] Example 2: splitting workload across all workers using :attr:`worker_init_fn`:: >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example code only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... return iter(range(self.start, self.end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [3, 4, 5, 6] >>> >>> # Directly doing multi-process loading yields duplicate data >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [3, 3, 4, 4, 5, 5, 6, 6] >>> # Define a `worker_init_fn` that configures each dataset copy differently >>> def worker_init_fn(worker_id): ... worker_info = torch.utils.data.get_worker_info() ... dataset = worker_info.dataset # the dataset copy in this worker process ... overall_start = dataset.start ... overall_end = dataset.end ... # configure the dataset to only process the split workload ... per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... dataset.start = overall_start + worker_id * per_worker ... dataset.end = min(dataset.start + per_worker, overall_end) ... >>> # Mult-process loading with the custom `worker_init_fn` >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn))) [3, 5, 4, 6] >>> # With even more workers >>> print(list(torch.utils.data.DataLoader(ds, num_workers=20, worker_init_fn=worker_init_fn))) [3, 4, 5, 6] rcCstdSrrr r"r"r#__iter__szIterableDataset.__iter__)r%cCs t||gSr)rr&r"r"r#r'szIterableDataset.__add__N) r(r)r*r+rrr-rr'r"r"r"r#r=serc@sBeZdZUdZeedfed<eddddZdd Zd d Z dS) rzDataset wrapping tensors. Each sample will be retrieved by indexing tensors along the first dimension. Args: *tensors (Tensor): tensors that have the same size of the first dimension. .tensorsN)r.rcs(tfddDsJd|_dS)Nc3s&|]}dd|dkVqdS)rN)size.0tensorr.r"r# z)TensorDataset.__init__..zSize mismatch between tensors)allr.)r r.r"r3r#__init__szTensorDataset.__init__cstfdd|jDS)Nc3s|]}|VqdSrr"r0r!r"r#r4r5z,TensorDataset.__getitem__..)tupler.rr"r8r#r$szTensorDataset.__getitem__cCs|jddSNr)r.r/r,r"r"r#__len__szTensorDataset.__len__) r(r)r*r+rr__annotations__r7r$r;r"r"r"r#rs r.csreZdZUdZeeeed<eeed<e ddZ e eddfdd Z d d Z d d ZeddZZS)rzDataset as a concatenation of multiple datasets. This class is useful to assemble different existing datasets. Args: datasets (sequence): List of datasets to be concatenated datasetscumulative_sizescCs6gd}}|D]"}t|}|||||7}q|Sr:)lenappend)sequencerselr"r"r#cumsums   zConcatDataset.cumsumNr=rcs^tt|t||_t|jdks.Jd|jD]}t|tr4Jdq4||j|_ dS)Nrz(datasets should not be an empty iterablez.ConcatDataset does not support IterableDataset) superrr7listr=r? isinstancerrFr>)r r=d __class__r"r#r7s   zConcatDataset.__init__cCs |jdS)N)r>r,r"r"r#r;szConcatDataset.__len__cCsf|dkr*| t|krtdt||}t|j|}|dkrF|}n||j|d}|j||S)Nrz8absolute value of index should not exceed dataset length)r? ValueErrorbisect bisect_rightr>r=)r idxZ dataset_idxZ sample_idxr"r"r#r$s zConcatDataset.__getitem__cCstjdtdd|jS)Nz:cummulative_sizes attribute is renamed to cumulative_sizes) stacklevel)warningswarnDeprecationWarningr>r,r"r"r#cummulative_sizesszConcatDataset.cummulative_sizes)r(r)r*r+rrrr<int staticmethodrFrr7r;r$propertyrY __classcell__r"r"rLr#rs    rcs<eZdZdZeeddfdd ZddZdd ZZ S) ra_Dataset for chaining multiple :class:`IterableDataset` s. This class is useful to assemble different existing dataset streams. The chaining operation is done on-the-fly, so concatenating large-scale datasets with this class will be efficient. Args: datasets (iterable of IterableDataset): datasets to be chained together NrGcstt|||_dSr)rHrr7r=)r r=rLr"r#r7szChainDataset.__init__ccs2|jD]&}t|tsJd|D] }|Vq qdS)N*ChainDataset only supports IterableDataset)r=rJr)r rKxr"r"r#r-s zChainDataset.__iter__cCs2d}|jD]"}t|ts Jd|t|7}q |S)Nrr^)r=rJrr?)r totalrKr"r"r#r;s  zChainDataset.__len__) r(r)r*r+rrr7r-r;r]r"r"rLr#rs rc@sTeZdZUdZeeed<eeed<eeeeddddZ dd Z d d Z dS) rz Subset of a dataset at specified indices. Args: dataset (Dataset): The whole Dataset indices (sequence): Indices in the whole set selected for subset datasetindicesN)rarbrcCs||_||_dSrrarb)r rarbr"r"r#r7szSubset.__init__cs2t|tr"jfdd|DSjj|S)Ncsg|]}j|qSr")rb)r1ir,r"r# !r5z&Subset.__getitem__..)rJrIrarb)r rSr"r,r#r$s zSubset.__getitem__cCs t|jSr)r?rbr,r"r"r#r;$szSubset.__len__) r(r)r*r+rrr<rrZr7r$r;r"r"r"r#rs   r)ralengths generatorrcsJt|tkrtdtt||dfddtt||DS)a Randomly split a dataset into non-overlapping new datasets of given lengths. Optionally fix the generator for reproducible results, e.g.: >>> random_split(range(10), [3, 7], generator=torch.Generator().manual_seed(42)) Args: dataset (Dataset): Dataset to be split lengths (sequence): lengths of splits to be produced generator (Generator): Generator used for the random permutation. zDSum of input lengths does not equal the length of the input dataset!)rgcs&g|]\}}t|||qSr")r)r1offsetlengthrcr"r#re:r5z random_split..)sumr?rPr tolistzipr )rarfrgr"rcr#r(sr)rQrVtypingrrrrrrrr torchr r Z torch._utilsr rr__all__rrrrrrrrrZrr"r"r"r#s$(   p2