a [=ic @sdZddlmZmZmZmZddlmZddlm Z ddlm Z ddlm Z ddl m Z ddl mZdd l mZdd lmZee jd d d Zee jd ddZeeedZdeee jfeeeeege jffeeeeeeeeeefeeeeeefee jd ddZdS)z7Library of Cloud TPU helper functions for data loading.)CallableOptionalTextUnion)interleave_ops) dataset_ops) iterator_ops)readers)dtypes)function)ops)functional_ops)filenamereturncCsd}tj||d}|SNi buffer_size)r ZTextLineDatasetrrZdatasetro/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/tensorflow/python/tpu/datasets.py_TextLineDatasetsrcCsd}tj||d}|Sr)r ZTFRecordDatasetrrrr_TFRecordDataset#sr)tfrecordZtextlinetextNT) filesfiletypefile_reader_job worker_job num_epochsfilename_shuffle_buffer_sizenum_parallel_readsbatch_transfer_sizesloppyrc s|dur d}t|trD|tvr:td|dttt|} n(t|rR|} ntd|dt|dprd|pzd }|durd }|pd }|durd }dkrd } nd} t | t|trt j |n t|t j r|ntd||rj|dtj| ||d||r8|dt } | Wdn1sj0Yttjfddfdd} t d|Rt j dj| |rdndd} | d} |r| d} Wdn1s0Y| S)a StreamingFilesDataset constructs a dataset to stream from workers (GCE VM). Because Cloud TPUs are allocated over the network, a Cloud TPU cannot read files local to your GCE VM. In order to train using files stored on your local VM (e.g. on local SSD for extreme performance), use the StreamingFilesDataset helper to generate a dataset to feed your Cloud TPU with files from your GCE VM. The resulting dataset may return an OutOfRangeError if there are no files found as a result of the fileglob expansion. Note: StreamingFilesDataset assumes that the session is using a TPUClusterResolver and has therefore a worker and a coordinator job. File loading will be done on the coordinator job. Args: files: A string glob to match files, or a `tf.data.Dataset` generating file names. filetype: A string (one of 'tfrecord', or 'textline') or a single-argument TensorFlow function that when given a filename returns a dataset. file_reader_job: An optional string that corresponds to the job that should perform the file reads. worker_job: An optional string that corresponds to the job that should process the tensors (i.e. your GPU or TPU worker). num_epochs: The number of epochs through the training set that should be generated. By default, it will repeat infinitely. filename_shuffle_buffer_size: An optional integer whose value controls the shuffling of the file names. If you would like to read from the files in the same order, set to 0 or False. num_parallel_reads: An optional integer controlling the number of files to read from concurrently. (Set to 1 for no parallelism.) batch_transfer_size: An optional integer controlling the batching used to amortize the remote function invocation overhead. Set to a very large number to increase throughput. Set to a very small number to reduce memory consumption. Set to False to skip batching. sloppy: (Optional.) If `False`, read input data while maintaining a deterministic order. (This may have significant performance impacts.) sloppy defaults to: True. Returns: A `tf.data.Dataset` with an infinite stream of elements generated by a parallel interleaving of the set of files matched (or generated) by `files` with a type is the output of the dataset specified by `filetype`. Raises: ValueError: if any argument is not of the expected type. NrzUnexpected filetype. Received: z. Expected one of z@Argument `filetype` should be a string or a callable. Received: z of type .Z coordinatorZworkeriz/job:coordinator/task:0z/job:%szOArgument `files` should be a string or a `tf.data.Dataset` instance. Received: r)Z cycle_lengthr"cs$tj|tt}|S)N)rIteratorZfrom_string_handlerget_legacy_output_typesZget_legacy_output_shapesZget_next)hZremote_iterator)source_datasetrr LoadingFuncs  z*StreamingFilesDataset..LoadingFunccsnt}t|tjr|g}nt|ttfr2|}ntdtj g|dd}t |dkrf|dS|SdS)NzbSource dataset has invalid output types. Only list/tuples or TensorFlow tensor types are accepted.z/job:%s/replica:0/task:0/cpu:0)argsZToutftargetr&r) rr( isinstancer ZDTypelisttuple ValueErrorr Z remote_calllen)Z unused_inputZsource_dataset_output_typesZ output_typesZ remote_callsr+rr*Z source_handlerrMapFns"  z$StreamingFilesDataset..MapFn)Znum_parallel_calls)r/str _FILETYPE_MAPr2r0keyscallabletyper ZdevicerDatasetZ list_filesZ DatasetV2shuffleapplyrZparallel_interleaverepeatbatchZprefetchZmake_one_shot_iteratorZ string_handler ZDefunr stringrangemapZunbatch)rrrrrrr r!r"Z reader_fnZfile_reader_deviceZsource_iteratorr5Zoutput_datasetrr4rStreamingFilesDataset0s~9           (  .rE)NNNNNNNT)__doc__typingrrrrZ'tensorflow.python.data.experimental.opsrZtensorflow.python.data.opsrrr Ztensorflow.python.frameworkr r r Ztensorflow.python.opsr r=rrr9intboolrErrrrsH