a
PSicS @ s` d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZm Z m
Z
m Z m Z m
Z
d dlmZ d dlZd dlZd dlmZ ddlmZmZmZmZ ddlmZ G dd d eZ G d
d d e Z G d d
d
e Z G dd de Z G dd de Z e!e"dddZ#ej$ej%ej&ej'ej(ej)dZ*d e+e,ej-dddZ.e+ej-ddd Z/e+ej-dd d Z0dS ) N)AnyCallableDictListOptionalTuple)URLError)Image ) download_and_extract_archiveextract_archiveverify_str_argcheck_integrity)
VisionDatasetc s0 e Zd ZdZddgZg dZdZdZg dZe dd Z
e d
d Z e d d
Z e dd Z
d.eeee ee edd fdd
Zdd Zdd Zdd Zeeeef d d d Zed d d!Ze ed d"d#Ze ed d$d%Z e e eef d d&d'Z ed d(d)Z dd d*d+Z ed d,d-Z! Z"S )/MNISTa] `MNIST `_ Dataset.
Args:
root (string): Root directory of dataset where ``MNIST/raw/train-images-idx3-ubyte``
and ``MNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z!http://yann.lecun.com/exdb/mnist/z.https://ossci-datasets.s3.amazonaws.com/mnist/))train-images-idx3-ubyte.gzZ f68b3c2dcbeaaa9fbdd348bbdeb94873)train-labels-idx1-ubyte.gzZ d53e105ee54ea40749a09fcbcd1e9432)t10k-images-idx3-ubyte.gzZ 9fb629c4189551a2d022fa330f9573f3)t10k-labels-idx1-ubyte.gzZ ec29112dd5afa0611ce80d1b7f02629cz training.ptztest.pt
z0 - zeroz1 - onez2 - twoz 3 - threez4 - fourz5 - fivez6 - sixz 7 - sevenz 8 - eightz9 - ninec C s t d | jS )Nz%train_labels has been renamed targetswarningswarntargetsself r V/var/www/html/django/DPS/env/lib/python3.9/site-packages/torchvision/datasets/mnist.py train_labels? s
zMNIST.train_labelsc C s t d | jS )Nz$test_labels has been renamed targetsr r r r r test_labelsD s
zMNIST.test_labelsc C s t d | jS )Nz train_data has been renamed datar r datar r r r
train_dataI s
zMNIST.train_datac C s t d | jS )Nz test_data has been renamed datar r r r r test_dataN s
zMNIST.test_dataTNF)roottrain transformtarget_transformdownloadreturnc sd t j|||d || _| r4| \| _| _d S |r@| | sPt d|
\| _| _d S )N)r& r' z;Dataset not found. You can use download=True to download it) super__init__r% _check_legacy_exist_load_legacy_datar! r r(
_check_exists RuntimeError
_load_data)r r$ r% r& r' r( __class__r r r+ S s zMNIST.__init__c s4 t j j}|sdS t fdd j jfD S )NFc 3 s" | ]}t tj j|V qd S N)r ospathjoinprocessed_folder.0filer r r o s z,MNIST._check_legacy_exist..)r4 r5 existsr7 all
training_file test_file)r Zprocessed_folder_existsr r r r, j s
zMNIST._check_legacy_existc C s( | j r | jn| j}ttj| j|S r3 ) r% r> r? torchloadr4 r5 r6 r7 )r data_filer r r r- s s zMNIST._load_legacy_datac C sX | j r
dnd d}ttj| j|}| j r2dnd d}ttj| j|}||fS )Nr% Zt10k-images-idx3-ubyte-labels-idx1-ubyte)r% read_image_filer4 r5 r6
raw_folderread_label_file)r
image_filer! Z
label_filer r r r r0 y s
zMNIST._load_dataindexr) c C s\ | j | t| j| }}tj| dd}| jdur@| |}| jdurT| |}||fS )z
Args:
index (int): Index
Returns:
tuple: (image, target) where target is index of the target class.
LmodeN)r! intr r fromarraynumpyr& r' r rJ imgtargetr r r __getitem__ s
zMNIST.__getitem__r) c C s
t | jS r3 )lenr! r r r r __len__ s z
MNIST.__len__c C s t j| j| jjdS )Nrawr4 r5 r6 r$ r2 __name__r r r r rF s zMNIST.raw_folderc C s t j| j| jjdS )N processedrY r r r r r7 s zMNIST.processed_folderc C s dd t | jD S )Nc S s i | ]\}}||qS r r )r9 i_classr r r
z&MNIST.class_to_idx..) enumerateclassesr r r r class_to_idx s zMNIST.class_to_idxc s t fdd jD S )Nc
3 s: | ]2\}}t tj jtjtj|d V qdS )r N)r r4 r5 r6 rF splitextbasename)r9 url_r r r r; s z&MNIST._check_exists..)r= resourcesr r r r r. s zMNIST._check_existsc C s | r dS tj| jdd | jD ]\}}| jD ]}| | }zvz$td| t|| j||d W nF ty } z.td| W Y d}~W t q0W Y d}~n
d}~0 0 W t nt 0 q"q0t d| q"dS )z4Download the MNIST data if it doesn't exist already.NTexist_okz Downloading )
download_rootfilenamemd5z"Failed to download (trying next):
zError downloading )
r. r4 makedirsrF rg mirrorsprintr r r/ )r rk rl mirrorre errorr r r r( s"
zMNIST.downloadc C s | j du rdnd}d| S )NTTrainTestSplit: )r% )r splitr r r
extra_repr s zMNIST.extra_repr)TNNF)#rZ
__module__ __qualname____doc__rn rg r> r? ra propertyr r r" r# strboolr r r+ r, r- r0 rN r r rT rW rF r7 r rb r. r( rv
__classcell__r r r1 r r sT
r c @ s& e Zd ZdZdgZg dZg dZdS ) FashionMNISTa `Fashion-MNIST `_ Dataset.
Args:
root (string): Root directory of dataset where ``FashionMNIST/raw/train-images-idx3-ubyte``
and ``FashionMNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z;http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/))r Z 8d4fb7e6c68d591d4c3dfef9ec88bf0d)r Z 25c81989df183df01b3e8a0aad5dffbe)r Z bef4ecab320f06d8554ea6380940ec79)r Z bb300cfdad3c16e7a12a480ee83cd310)
z T-shirt/topZTrouserZPulloverZDressZCoatZSandalZShirtZSneakerZBagz
Ankle bootNrZ rw rx ry rn rg ra r r r r r~ s r~ c @ s& e Zd ZdZdgZg dZg dZdS )KMNISTak `Kuzushiji-MNIST `_ Dataset.
Args:
root (string): Root directory of dataset where ``KMNIST/raw/train-images-idx3-ubyte``
and ``KMNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z-http://codh.rois.ac.jp/kmnist/dataset/kmnist/))r Z bdb82020997e1d708af4cf47b453dcf7)r Z e144d726b3acfaa3e44228e80efcd344)r Z 5c965bf0a639b31b8f53240b1b52f4d7)r Z 7320c461ea6c1c855c0b718fb2a4b134)
okiZsuZtsunahamaZyarewoNr r r r r r s r c s e Zd ZdZdZdZdZh dZee j
e j Z e
ee e
ee e e
ee e dgee j ee j
ee j
dZeeedd fd d
Zeed d d
Zeed ddZeed ddZeed ddZeed ddZdd Ze d ddZ dd ddZ Z S ) EMNISTa8 `EMNIST `_ Dataset.
Args:
root (string): Root directory of dataset where ``EMNIST/raw/train-images-idx3-ubyte``
and ``EMNIST/raw/t10k-images-idx3-ubyte`` exist.
split (string): The dataset has 6 different splits: ``byclass``, ``bymerge``,
``balanced``, ``letters``, ``digits`` and ``mnist``. This argument specifies
which one to use.
train (bool, optional): If True, creates dataset from ``training.pt``,
otherwise from ``test.pt``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z:https://www.itl.nist.gov/iaui/vip/cs_links/EMNIST/gzip.zipZ 58c8d27c78d21e728a6bc7b3cc06412e)ZbyclassZbymergeZbalancedlettersdigitsmnist> mr wzxr\ plcsvyjkuzN/AN)r$ ru kwargsr) c sN t |d| j| _| || _| || _t j|fi | | j | j | _
d S )Nru ) r
splitsru _training_filer>
_test_filer? r* r+ classes_split_dictra )r r$ ru r r1 r r r+ % s
zEMNIST.__init__rU c C s d| dS )NZ training_.ptr ru r r r r , s zEMNIST._training_filec C s d| dS )Ntest_r r r r r r r 0 s zEMNIST._test_filec C s d| j d| jrdnd S )Nzemnist--r% test)ru r% r r r r _file_prefix4 s zEMNIST._file_prefixc C s t j| j| j dS )NrC r4 r5 r6 rF r r r r r images_file8 s zEMNIST.images_filec C s t j| j| j dS )NrD r r r r r labels_file< s zEMNIST.labels_filec C s t | jt| jfS r3 )rE r rG r r r r r r0 @ s zEMNIST._load_datac C s t dd | j| jfD S )Nc s s | ]}t |V qd S r3 r r8 r r r r; D r_ z'EMNIST._check_exists..r= r r r r r r r. C s zEMNIST._check_existsc C s~ | r dS tj| jdd t| j| j| jd tj| jd}t |D ]$}|
drJt tj||| j qJt
| dS )z5Download the EMNIST data if it doesn't exist already.NTrh )rj rl gzipz.gz)r. r4 rm rF r re rl r5 r6 listdirendswithr shutilrmtree)r Z gzip_folderZ gzip_filer r r r( F s
zEMNIST.download) rZ rw rx ry re rl r Z_merged_classessetstringr
ascii_lettersZ _all_classessortedlistascii_lowercaser r{ r r+ staticmethodr r rz r r r r0 r| r. r( r} r r r1 r r s6
r c s e Zd ZU dZddddddZddgdd gd
d gd Zeeee eef f e
d
< g dZ d$ee e e
e
edd fdd
ZeedddZeedddZe
dddZdd Zddd d Zee eef d d d!Zedd"d#Z ZS )%QMNISTaP `QMNIST `_ Dataset.
Args:
root (string): Root directory of dataset whose ``raw``
subdir contains binary files of the datasets.
what (string,optional): Can be 'train', 'test', 'test10k',
'test50k', or 'nist' for respectively the mnist compatible
training set, the 60k qmnist testing set, the 10k qmnist
examples that match the mnist testing set, the 50k
remaining qmnist testing examples, or all the nist
digits. The default is to select 'train' or 'test'
according to the compatibility argument 'train'.
compat (bool,optional): A boolean that says whether the target
for each example is class number (for compatibility with
the MNIST dataloader) or a torch vector containing the
full qmnist information. Default=True.
download (bool, optional): If True, downloads the dataset from
the internet and puts it in root directory. If dataset is
already downloaded, it is not downloaded again.
transform (callable, optional): A function/transform that
takes in an PIL image and returns a transformed
version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform
that takes in the target and transforms it.
train (bool,optional,compatibility): When argument 'what' is
not specified, this boolean decides whether to load the
training set ot the testing set. Default: True.
r% r nist)r% r test10ktest50kr )zbhttps://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-train-images-idx3-ubyte.gzZ ed72d4157d28c017586c42bc6afe6370)z`https://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-train-labels-idx2-int.gzZ 0058f8dd561b90ffdd0f734c6a30e5e4)zahttps://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-test-images-idx3-ubyte.gzZ 1394631089c404de565df7b7aeaf9412)z_https://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-test-labels-idx2-int.gzZ 5b5b05890a5e13444e108efe57b788aa)z[https://raw.githubusercontent.com/facebookresearch/qmnist/master/xnist-images-idx3-ubyte.xzZ 7f124b3b8ab81486c9d8c2749c17f834)zYhttps://raw.githubusercontent.com/facebookresearch/qmnist/master/xnist-labels-idx2-int.xzZ 5ed0e788978e45d4a8bd4b7caec3d79d)r% r r rg r NT)r$ whatcompatr% r r) c sf |d u r|rdnd}t |dt| j | _|| _|d | _| j| _| j| _t j
||fi | d S )Nr% r r r ) r
tuplesubsetskeysr r rB r> r? r* r+ )r r$ r r r% r r1 r r r+ s
zQMNIST.__init__rU c C s> | j | j| j \\}}}tj| jtjtj|d S Nr rg r r r4 r5 r6 rF rc rd )r re rf r r r r s zQMNIST.images_filec C s> | j | j| j \}\}}tj| jtjtj|d S r r )r rf re r r r r s zQMNIST.labels_filec C s t dd | j| jfD S )Nc s s | ]}t |V qd S r3 r r8 r r r r; r_ z'QMNIST._check_exists..r r r r r r. s zQMNIST._check_existsc C s t | j}|jtjkr&td|j | dkr:tdt | j }| dkrftd| | j
dkr|ddd d d d f }|ddd d f }n@| j
d kr|dd d d d d f }|dd d d f }||fS )
Nz/data should be of dtype torch.uint8 instead of z r_ z1read_sn3_pascalvincent_tensor..little)r offset)openreadr SN3_PASCALVINCENT_TYPEMAPranger@ iinfobitssys byteorder
frombuffer bytearrayflipshapenpprodview) r5 r fmagicndtyZ
torch_typer Znum_bytes_per_valueZneeds_byte_reversalparsedr r r r s &
r )r5 r) c C sN t | dd}|jtjkr(td|j | dkrFtd| | S )NFr ,x should be of dtype torch.uint8 instead of r
z%x should have 1 dimension instead of )r r r@ r r r r r r5 r r r r rG s rG c C sJ t | dd}|jtjkr(td|j | dkrFtd| |S )NFr r r z%x should have 3 dimension instead of )r r r@ r r r r r r r r rE s rE )T)1r r4 os.pathr r r r typingr r r r r r urllib.errorr rP r r@ PILr utilsr r r
r visionr r r~ r r r bytesrN r r int8int16int32float32float64r r{ r| Tensorr rG rE r r r r s> 8 T