a .SicN  @sdZddlmZddlmZddlmZddlmZddlmZddlmZddl m Z dd l m Z dd l m Z dd l mZdd l mZdd l mZddl mZddl mZddl mZddlmZddlmZddlmZddlmZddlmZejZ ej!Z"ej#Z$edddgde%ddddZ&edgde'dddddZ(ed gde'dd!dd$d%Z)ed&gde'dd'dd(d)Z*ed*gde'dd+dd,d-Z+ed.gde'dd/dd0d1Z,d2d3Z-Gd4d5d5e.Z/d6d7Z0d8d9Z1d:d;Z2ddd?Z4d@dAZ5dBdCZ6dDdEZ7dFdGZ8dHdIZ9dJdKZ:dLdMZ;dNdOZdTdUZ?ddXdYZ@ddZd[ZAdd\d]ZBdd^d_ZCed`gde'ddaddbdcZDeddgde'ddeddfdgZEedhgde'ddiddjdkZFedlgde'ddmddndoZGedpgde'ddqddrdsZHedtgde'dduddvdwZIedxgde'ddyddzd{ZJed|gde'dd}dd~dZKdS)zzInput pipeline. Please see the [reading data how-to](https://tensorflow.org/api_guides/python/reading_data) for context. )context) constant_op)dtypes)ops) sparse_tensor) tensor_shape)utils) array_ops)control_flow_ops) data_flow_ops)io_ops)math_ops) random_ops) sparse_ops)variable_scope)summary) queue_runner) deprecation)collections_abc) tf_exportzio.match_filenames_onceztrain.match_filenames_once)v1NcCsTt|d|g0}tj|t|ddtjjgdWdS1sF0YdS)a\Save the list of files matching pattern, so it is only computed once. NOTE: The order of the files returned is deterministic. Args: pattern: A file pattern (glob), or 1D tensor of file patterns. name: A name for the operations (optional). Returns: A variable that is initialized to the list of files matching the pattern(s). Zmatching_filenamesF)name initial_value trainablevalidate_shape collectionsN)r name_scopevsvariabler matching_files GraphKeysLOCAL_VARIABLES)patternrr#\/var/www/html/django/DPS/env/lib/python3.9/site-packages/tensorflow/python/training/input.pymatch_filenames_once5s  r%ztrain.limit_epochsz{Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.c Cs|dur |S|dkr td|t|d|g}tjdtjd}tj|ddtj j gd}| |}t |g,t j||d WdWdS1s0YWdn1s0YdS) aReturns tensor `num_epochs` times and then raises an `OutOfRange` error. Note: creates local counter `epochs`. Use `local_variables_initializer()` to initialize local variables. Args: tensor: Any `Tensor`. num_epochs: A positive integer (optional). If specified, limits the number of steps the output tensor may be evaluated. name: A name for the operations (optional). Returns: tensor or `OutOfRange`. Raises: ValueError: if `num_epochs` is invalid. Nrznum_epochs must be > 0 not %d. limit_epochsdtypeepochsF)rrrr) ValueErrorrrrconstantrint64rrr r! count_up_tocontrol_dependenciesr identity)tensor num_epochsrZzero64r)counterr#r#r$r&Ls  r&ztrain.input_producerzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.T c Cstrtdt|d|gtj|dd}|jdd|}|sTt d|rft j ||d}t ||}t j||jjg|g||d } | |g} ttj| | g| d |durt|t| tjd || WdS1s0YdS) aOutput the rows of `input_tensor` to a queue for an input pipeline. Note: if `num_epochs` is not `None`, this function creates local counter `epochs`. Use `local_variables_initializer()` to initialize local variables. Args: input_tensor: A tensor with the rows to produce. Must be at least one-dimensional. Must either have a fully-defined shape, or `element_shape` must be defined. element_shape: (Optional.) A `TensorShape` representing the shape of a row of `input_tensor`, if it cannot be inferred. num_epochs: (Optional.) An integer. If specified `input_producer` produces each row of `input_tensor` `num_epochs` times before generating an `OutOfRange` error. If not specified, `input_producer` can cycle through the rows of `input_tensor` an unlimited number of times. shuffle: (Optional.) A boolean. If true, the rows are randomly shuffled within each epoch. seed: (Optional.) An integer. The seed to use if `shuffle` is true. capacity: (Optional.) The capacity of the queue to be used for buffering the input. shared_name: (Optional.) If set, this queue will be shared under the given name across multiple sessions. summary_name: (Optional.) If set, a scalar summary for the current queue size will be generated, using this name as part of the tag. name: (Optional.) A name for queue. cancel_op: (Optional.) Cancel op for the queue Returns: A queue with the output rows. A `QueueRunner` for the queue is added to the current `QUEUE_RUNNER` collection of the current graph. Raises: ValueError: If the shape of the input cannot be inferred from the arguments. RuntimeError: If called with eager execution enabled. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility Input pipelines based on Queues are not supported when eager execution is enabled. Please use tf.data to ingest data into your model instead.input_producer input_tensorr*NzZEither `input_tensor` must have a fully defined shape or `element_shape` must be specified)seed)capacityrshapes shared_namer) cancel_op?)rexecuting_eagerly RuntimeErrorrrconvert_to_tensorshape merge_withis_fully_definedr+rrandom_shuffler&r FIFOQueuer( base_dtype enqueue_manyradd_queue_runner QueueRunnerrscalarr castsizerfloat32) r7 element_shaper2shuffler9r:r< summary_namerr=qZenqr#r#r$r6ps89  r6ztrain.string_input_producerzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.c Csd}t|tjs|st|t|d|g}tj|tjd}tt t t |d|ggt |}Wdn1s0Yt|g||||||d||d WdS1s0YdS)aOutput strings (e.g. filenames) to a queue for an input pipeline. Note: if `num_epochs` is not `None`, this function creates local counter `epochs`. Use `local_variables_initializer()` to initialize local variables. Args: string_tensor: A 1-D string tensor with the strings to produce. num_epochs: An integer (optional). If specified, `string_input_producer` produces each string from `string_tensor` `num_epochs` times before generating an `OutOfRange` error. If not specified, `string_input_producer` can cycle through the strings in `string_tensor` an unlimited number of times. shuffle: Boolean. If true, the strings are randomly shuffled within each epoch. seed: An integer (optional). Seed used if shuffle == True. capacity: An integer. Sets the queue capacity. shared_name: (optional). If set, this queue will be shared under the given name across multiple sessions. All sessions open to the device which has this queue will be able to access it via the shared_name. Using this in a distributed setting means each name will only be seen by one of the sessions which has access to this operation. name: A name for the operations (optional). cancel_op: Cancel op for the queue (optional). Returns: A queue with the output strings. A `QueueRunner` for the Queue is added to the current `Graph`'s `QUEUE_RUNNER` collection. Raises: ValueError: If the string_tensor is a null Python list. At runtime, will fail with an assertion if string_tensor becomes a null tensor. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility z6string_input_producer requires a non-null input tensorr6r'rNfraction_of_%d_full) r7rOr2rPr9r:r<rrQr=) isinstancerTensorr+rrArstringr/r Assertr greaterr rMr0r6) string_tensorr2rPr9r:r<rr=Z not_null_errr#r#r$string_input_producers03(rZztrain.range_input_producerzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.range(limit).shuffle(limit).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.c CsZt|d|g6}t|}t|g|||||d|| WdS1sL0YdS)aProduces the integers from 0 to limit-1 in a queue. Note: if `num_epochs` is not `None`, this function creates local counter `epochs`. Use `local_variables_initializer()` to initialize local variables. Args: limit: An int32 scalar tensor. num_epochs: An integer (optional). If specified, `range_input_producer` produces each integer `num_epochs` times before generating an OutOfRange error. If not specified, `range_input_producer` can cycle through the integers an unlimited number of times. shuffle: Boolean. If true, the integers are randomly shuffled within each epoch. seed: An integer (optional). Seed used if shuffle == True. capacity: An integer. Sets the queue capacity. shared_name: (optional). If set, this queue will be shared under the given name across multiple sessions. name: A name for the operations (optional). Returns: A Queue with the output integers. A `QueueRunner` for the Queue is added to the current `Graph`'s `QUEUE_RUNNER` collection. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility r6rSN)rrr ranger6)limitr2rPr9r:r<rZ range_tensorr#r#r$range_input_producers #   r]ztrain.slice_input_producerzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(tuple(tensor_list)).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.c st|d|jt|}|s&tdt|dd}t||||||d}|fdd|D} | WdS1s~0YdS)aProduces a slice of each `Tensor` in `tensor_list`. Implemented using a Queue -- a `QueueRunner` for the Queue is added to the current `Graph`'s `QUEUE_RUNNER` collection. Args: tensor_list: A list of `Tensor` objects. Every `Tensor` in `tensor_list` must have the same size in the first dimension. num_epochs: An integer (optional). If specified, `slice_input_producer` produces each slice `num_epochs` times before generating an `OutOfRange` error. If not specified, `slice_input_producer` can cycle through the slices an unlimited number of times. shuffle: Boolean. If true, the integers are randomly shuffled within each epoch. seed: An integer (optional). Seed used if shuffle == True. capacity: An integer. Sets the queue capacity. shared_name: (optional). If set, this queue will be shared under the given name across multiple sessions. name: A name for the operations (optional). Returns: A list of tensors, one for each element of `tensor_list`. If the tensor in `tensor_list` has shape `[N, a, b, .., z]`, then the corresponding output tensor will have shape `[a, b, ..., z]`. Raises: ValueError: if `slice_input_producer` produces nothing from `tensor_list`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility r6z7Expected at least one tensor in slice_input_producer().r)r2rPr9r:r<csg|]}t|qSr#)r gather.0tindexr#r$ rz(slice_input_producer..N)rr%convert_n_to_tensor_or_indexed_slicesr+r rBr]dequeue) tensor_listr2rPr9r:r<rZ range_sizequeueoutputr#rbr$slice_input_producer=s) rkcCsdd|DS)NcSsg|]}|D]}|q qSr#r#)r`rhr1r#r#r$rdzrez_flatten..r#tensor_list_listr#r#r$_flattenysrnc@s\eZdZdZddZddZddZdd Zd d Ze d d Z e ddZ e ddZ dS)_SparseMetaDatazDStore information about the Tensor: Is it sparse?, map_op, and rank.cCs||_||_t||_dS)a?Create the metadata. Args: sparse: Python boolean. map_op: The `Operation` that created the `SparseTensorsMap` in question. This Op contains information about the underlying Map object and the dtype of the original data. rank: The statically known rank of the `SparseTensor`. N)_sparse_map_opr as_dimension_rank)selfsparsemap_oprankr#r#r$__init__s z_SparseMetaData.__init__cCsX|j|jkrdS|jsdS|jdu|jdukr2dS|j|jkrBdS|j|jsTdSdS)NFT)rurvrwis_compatible_withrtotherr#r#r$__eq__s  z_SparseMetaData.__eq__cCs || SN)r|rzr#r#r$__ne__sz_SparseMetaData.__ne__cCsd|j|jj|jfS)Nz[SparseMetaData(%s, %s, %s)])rurvrrwrtr#r#r$__str__s z_SparseMetaData.__str__cCs0||krtd||f|jr,|j|j|S)Nz2SparseMetaData objects are incompatible: %s vs. %s)r+rurwrCrzr#r#r$rCsz_SparseMetaData.merge_withcCs|jSr})rqrr#r#r$rvsz_SparseMetaData.map_opcCs|jSr})rprr#r#r$rusz_SparseMetaData.sparsecCs|jSr})rsrr#r#r$rwsz_SparseMetaData.rankN) __name__ __module__ __qualname____doc__rxr|r~rrCpropertyrvrurwr#r#r#r$ro}s  rocs,ttr$fddttdDSSdS)Ncsg|] }|qSr#r#)r`ktensorsr#r$rdrez#_as_tensor_list..key)rTdictsortedstrrr#rr$_as_tensor_lists rcCsj|s tdt|dtrbt|d}|ddD]}t||kr6tdq6dd|DS|SdS)Nz$Expected at least one set of tensorsrr8z8All dictionaries in tensors_list must have the same keyscSsg|] }t|qSr#)r)r`rr#r#r$rdrez(_as_tensor_list_list..)r+rTrsetkeys) tensors_list expected_keysrr#r#r$_as_tensor_list_lists rcsBt|tr:t|dkrgfddtt|tdDSSdS)Nr8csi|]\}}||qSr#r#)r`irrhr#r$ sz%_as_original_type..r)rTrlen enumeraterr)original_tensorsrhr#rr$_as_original_types   rcs|pdgt|}fddfddfddt||D}dd fd dt|||D}d dt||D}||fS) aStore SparseTensors for feeding into batch, etc. If `shared_map_ops` is provided, the underlying `SparseTensorsMap` objects are reused (shared). This argument is useful for, e.g., `batch_join` where multiple enqueue operations write to the same Queue component, and another (dequeue) thread reads from that same location and must then restore the associated `SparseTensor` objects. In this case, the sparse restore must have a single `SparseTensorMap` from which to read out the handles; so a single `SparseTensorMap` must be shared for storing across the multiple enqueue operations. This sharing is performed by calling `_store_sparse_tensors` the first time with `shared_map_ops=None`, and then in subsequent times with this value set to the list of `Operation` objects created in the first call. Args: tensor_list: List of `Tensor` and `SparseTensor` objects. enqueue_many: Python `Boolean`. keep_input: Must be a scalar bool Tensor (not a Python bool). If False, don't store. shared_map_ops: (optional) List of `Operation` objects from a previous call to `_store_sparse_tensors`. If not `None`, the op types should be one of `AddSparseToTensorsMap` or `AddManySparseToTensorsMap` in the locations corresponding to `SparseTensors` in `tensor_list`. Returns: A tuple `(stored_list, sparse_info_list)` where `stored_list` is a list of `Tensor` objects (same length as `tensor_list`) and `sparse_info_list` is a list of the same length of `_SparseMetaData` objects. NcsJt|tjstdddS|jjdjd}r8|d8}td|pB||dS)NFr8rT)rurvrw)rTr SparseTensorro dense_shaperB with_rankdims)raZ storing_oprvrw)rHr#r$_sparse_meta_datas   z0_store_sparse_tensors.._sparse_meta_datacsxt|tjs|S|r|jnd}dd}dd}dd}jjdkr^t|||}d d }nrh|}n|}|||S) z"Store Sparse tensor, if necessary.Ncst|fddddS)z+Conditionally store a single sparse Tensor.cs tdSN)r<) _store_sparser# map_op_namerar#r$rezZ_store_sparse_tensors.._maybe_store.._maybe_store_sparse..cSstdtjS)N)rr,rr-r#r#r#r$rre)r smart_cond)rar keep_inputr#rr$_maybe_store_sparse s  zH_store_sparse_tensors.._maybe_store.._maybe_store_sparsecs0t|fddfdd}|dg|S)z,Conditionally store multiple sparse Tensors.cs tdSr_store_many_sparser#rr#r$rrez__store_sparse_tensors.._maybe_store.._maybe_store_many_sparse..cs dttddtjS)Nrrr8)r onesrBrr-r#)rar#r$rreN)rr set_shape)rarr out_tensorr#rr$_maybe_store_many_sparses   zM_store_sparse_tensors.._maybe_store.._maybe_store_many_sparsecSs|jdddf}t||S)z9Convert a per-row `keep_input` vector to a per-value one.Nr)indicesr r^)rar row_valuesr#r#r$_sparse_values_to_keepszK_store_sparse_tensors.._maybe_store.._sparse_values_to_keepr8cSs t||dSrr)rar_r#r#r$r rez=_store_sparse_tensors.._maybe_store..)rTrrrrBndimsr sparse_retain)ra shared_map_oprrrrZstore_f)rHrr#r$ _maybe_stores   z+_store_sparse_tensors.._maybe_storecsg|]\}}||qSr#r#)r`rar)rr#r$rd'sz)_store_sparse_tensors..cSs*|jjD]}|jjdvr|jSq|jS)N)AddSparseToTensorsMapAddManySparseToTensorsMap)opinputstype)storedr7r#r#r$ _sparse_op,s   z)_store_sparse_tensors.._sparse_opcs"g|]\}}}|||qSr#r#)r`rarr)rrr#r$rd4scSs(g|] \}}|jr t|dgn|qS)r)rur expand_dims)r`sZs_infor#r#r$rd9s)rzip)rhrHrZshared_map_opsZmaybe_shared_map_ops stored_listsparse_info_listr#)rrrrHrr$_store_sparse_tensorss "   rc Cst|d||\}}|g}|ddD]Z}t|||dd|D\}}||krbtd|d|fddt||D}||q&||fS)z5Store SparseTensors for feeding into batch_join, etc.rr8NcSsg|] }|jqSr#)rv)r`str#r#r$rdGrez._store_sparse_tensors_join..z*Inconsistent SparseTensors list: %s vs. %scSsg|]\}}||qSr#)rC)r`info candidater#r#r$rdKs)rr+rappend) rmrHrs0rZstored_list_listrhrZsparse_info_candidater#r#r$_store_sparse_tensors_join?s$    rcst|tj}|s|f}ddt||D}tdd|D}|rndd|Dfddfdd|D}n|}|rz|S|d S) z>Restore SparseTensors after dequeue in batch, batch_join, etc.c Ss@g|]8\}}|jr8t|jt|dgt|jddn|qS)r8) sparse_map_opsparse_handlesrw)ru_restore_sparservr squeezerdimension_valuerw)r`rrr#r#r$rdXs z+_restore_sparse_tensors..css|]}t|tjVqdSr})rTrrr`xr#r#r$ ^rez*_restore_sparse_tensors..cSs"g|]}t|tjr|jn|qSr#)rTrrvaluesrr#r#r$rd`scs t|Sr})r with_dependencies)r)t_valuesr#r$rdrez)_restore_sparse_tensors..csBg|]:}t|tjr6tj|j|j|jdn|qS))rrr)rTrrrrrr) with_depsr#r$rdes r)rTrSequencerany)rrZreceived_sequencerZhas_stZensure_restore_tensorsr#)rrr$_restore_sparse_tensorsSs"   rcCst|}|std|S)Nz(Expected at least one tensor in batch().)rrfr+rr#r#r$ _validateqs rcCsdd|D}|std|S)NcSsg|]}t|qSr#)rrfr`tlr#r#r$rdysz"_validate_join..z,Expected at least one input in batch_join().)r+rlr#r#r$_validate_joinxs rcCsNt|}|jjdurtd|s6|jjdkr6td|jjdkrJtd|S)zAValidate `keep_input` argument to conditional batching functions.Nz<`keep_input` dimensions must be known at graph construction.r8z:`keep_input` cannot be a vector when `enqueue_many=False`.z'`keep_input` must be 0 or 1 dimensions.)rrArBrr+)rrHr#r#r$_validate_keep_inputs   rcCsddd|D}|d}|ddD]<}||kr"tdddd|Ddd d|Dfq"|S) NcSsg|]}dd|DqS)cSsg|] }|jqSr#r'r_r#r#r$rdrez&_dtypes...r#rr#r#r$rdrez_dtypes..rr8z+Expected types to be consistent: %s vs. %s.z, css|] }|jVqdSr}r*rr#r#r$rrez_dtypes..css|] }|jVqdSr}r*rr#r#r$rre) TypeErrorjoin)rm all_typestypesZ other_typesr#r#r$_dtypessrcCsLdd|D}|r dd|D}|d}|ddD]}||q4|S)NcSsg|]}t|qSr#)ras_shaper`rr#r#r$rdrez!_merge_shapes..cSsg|]}|dddqS)r8N)with_rank_at_leastrr#r#r$rdrerr8)rCas_list) shape_listrHZ merged_shaperr#r#r$ _merge_shapess rcsh|durdtd}D]2}t|D]$}||jjdur$td||q$qfddt|D}|S)aCalculate and merge the shapes of incoming tensors. Args: tensor_list_list: List of tensor lists. shapes: List of shape tuples corresponding to tensors within the lists. enqueue_many: Boolean describing whether shapes will be enqueued as batches or individual entries. Returns: A list of shapes aggregating shape inference info from `tensor_list_list`, or returning `shapes` if it is not `None`. Raises: ValueError: If any of the inferred shapes in `tensor_list_list` lack a well defined rank. NrzCannot infer Tensor's rank: %scs$g|]tfddDqS)csg|]}|jqSr#)rBrrrr#r$rdsz&_shapes...)rr`rHrmrr$rds  z_shapes..)rr[rBrr+)rmr;rHZlen0rrr#rr$_shapess   rcs$t|tjfdd|D}|S)z>Select which examples to enqueue based on vector `keep_input`.cs g|]}tj|dddqS))num_partitionsr8)r dynamic_partitionrZselect_ir#r$rdsz,_select_which_to_enqueue..)r rLrint32)rhrr#rr$_select_which_to_enqueues  rcs^|r |jn|jjjdkr4fdd|D}nfdd|D}tt||dS)z&Enqueue `tensor_list_list` in `queue`.r8csg|]}t|qSr#)rr enqueue_fnrr#r$rdsz!_enqueue_join..cs&g|]tfddtjqS)csSr}r#r#)rrr#r$rrez*_enqueue_join...)rrr no_oprr)rr$rds  N)rHenqueuerBrrrIrJ)rirmrHr enqueue_opsr#rr$ _enqueue_joins   rcsj|r |jn|j|jjdkr4t|g|}n t|fddtjg|}t t ||dS)z!Enqueue `tensor_list` in `queue`.r8csSr}r#r#rrhr#r$rrez_enqueue..N) rHrrBrrrrr rrrIrJ)rirhthreadsrHrrr#rr$_enqueues    rcCs|r tjStjSr})r PaddingFIFOQueuerF) dynamic_padr#r#r$ _which_queues rr8Fc Cstrtdt|} t| dt| |g} t| } t||}t | ||\} } t | g} t | g||}t ||| || d}t || |||td|t|tjd||r|j|| d}n|j|| d}t|| }t||WdS1s0YdS)z.Helper function for `batch` and `maybe_batch`.r5batchr:rr;r<rSr>r*N)rr?r+rrrlistrrrrrrrrrKr rLrMrrN dequeue_up_to dequeue_manyrr)r batch_sizer num_threadsr:rHr;rallow_smaller_final_batchr<rrh sparse_inforridequeuedr#r#r$_batchs4   r c Cstrtdt|} t| dt| |g} t| } t||}t | ||\} } t | } t | ||}t ||| ||d} t | | ||td|t| tjd||r| j|| d}n| j|| d}t|| }t|d|WdS1s0YdS) z8Helper function for `batch_join` and `maybe_batch_join`.r5 batch_joinrrSr>r*rN)rr?r+rrrrnrrrrrrrrrKr rLrMrrNrrrr)rrrr:rHr;rrr<rrmrrrirr#r#r$ _batch_joins:    r c  CsNtrtdt|} t| dt| |g} ||krLtd||ft| } t||}t | ||\} } t | g}t | g||}t j |||||| d}t|| |||ttd||tjd||}d|||f}t||| r|j|| d}n|j|| d}t|| }t||Wd S1s@0Yd S) z>Helper function for `shuffle_batch` and `maybe_shuffle_batch`.r5 shuffle_batchz5capacity %d must be bigger than min_after_dequeue %d.r:min_after_dequeuer9rr;r<rr>fraction_over_%d_of_%d_fullr*N)rr?r+rrrrrrrrrr RandomShuffleQueuerr rLmaximumrMrrNrrKrrrr)rrr:rrrr9rHr;rr<rrhrrrifullrQrr#r#r$_shuffle_batch@sP       rc  Cs0trtdt|} t| dt| |g} t| } t||}t | ||\} } t | } t | ||}t j |||| || d}t|| ||ttd||tjd||}d|||f}t|||r|j|| d}n|j|| d}t|| }t|d|WdS1s"0YdS) zHHelper function for `shuffle_batch_join` and `maybe_shuffle_batch_join`.r5shuffle_batch_joinr rr>rr*N)rr?r+rrrrnrrrrrr rrr rLrrMrrNrrKrrrr)rrr:rrr9rHr;rr<rrmrrrirrQrr#r#r$_shuffle_batch_joinlsH       rz train.batchzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).c Cst||d|||||||| d S)aWCreates batches of tensors in `tensors`. The argument `tensors` can be a list or a dictionary of tensors. The value returned by the function will be of the same type as `tensors`. This function is implemented using a queue. A `QueueRunner` for the queue is added to the current `Graph`'s `QUEUE_RUNNER` collection. If `enqueue_many` is `False`, `tensors` is assumed to represent a single example. An input tensor with shape `[x, y, z]` will be output as a tensor with shape `[batch_size, x, y, z]`. If `enqueue_many` is `True`, `tensors` is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of `tensors` should have the same size in the first dimension. If an input tensor has shape `[*, x, y, z]`, the output will have shape `[batch_size, x, y, z]`. The `capacity` argument controls the how long the prefetching is allowed to grow the queues. The returned operation is a dequeue operation and will throw `tf.errors.OutOfRangeError` if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself. *N.B.:* If `dynamic_pad` is `False`, you must ensure that either (i) the `shapes` argument is passed, or (ii) all of the tensors in `tensors` must have fully-defined shapes. `ValueError` will be raised if neither of these conditions holds. If `dynamic_pad` is `True`, it is sufficient that the *rank* of the tensors is known, but individual dimensions may have shape `None`. In this case, for each enqueue the dimensions with value `None` may have a variable length; upon dequeue, the output tensors will be padded on the right to the maximum shape of the tensors in the current minibatch. For numbers, this padding takes value 0. For strings, this padding is the empty string. See `PaddingFIFOQueue` for more info. If `allow_smaller_final_batch` is `True`, a smaller batch value than `batch_size` is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the `shape` property will have a first `Dimension` value of `None`, and operations that depend on fixed batch_size would fail. Args: tensors: The list or dictionary of tensors to enqueue. batch_size: The new batch size pulled from the queue. num_threads: The number of threads enqueuing `tensors`. The batching will be nondeterministic if `num_threads > 1`. capacity: An integer. The maximum number of elements in the queue. enqueue_many: Whether each tensor in `tensors` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensors`. dynamic_pad: Boolean. Allow variable dimensions in input shapes. The given dimensions are padded upon dequeue so that tensors within a batch have the same shapes. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (Optional). If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the same types as `tensors` (except if the input is a list of one element, then it returns a tensor, not a list). Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensors`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility T) rrr:rHr;rrr<rr ) rrrr:rHr;rrr<rr#r#r$rsUrztrain.maybe_batchzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.filter(...).batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).c Cst|||||||||| | d S)aConditionally creates batches of tensors based on `keep_input`. See docstring in `batch` for more details. Args: tensors: The list or dictionary of tensors to enqueue. keep_input: A `bool` Tensor. This tensor controls whether the input is added to the queue or not. If it is a scalar and evaluates `True`, then `tensors` are all added to the queue. If it is a vector and `enqueue_many` is `True`, then each example is added to the queue only if the corresponding value in `keep_input` is `True`. This tensor essentially acts as a filtering mechanism. batch_size: The new batch size pulled from the queue. num_threads: The number of threads enqueuing `tensors`. The batching will be nondeterministic if `num_threads > 1`. capacity: An integer. The maximum number of elements in the queue. enqueue_many: Whether each tensor in `tensors` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensors`. dynamic_pad: Boolean. Allow variable dimensions in input shapes. The given dimensions are padded upon dequeue so that tensors within a batch have the same shapes. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (Optional). If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the same types as `tensors`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensors`. )rr:rHr;rrr<rr) rrrrr:rHr;rrr<rr#r#r$ maybe_batchs+rztrain.batch_joinzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.interleave(...).batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).c Cst||d|||||||d S)aRuns a list of tensors to fill a queue to create batches of examples. The `tensors_list` argument is a list of tuples of tensors, or a list of dictionaries of tensors. Each element in the list is treated similarly to the `tensors` argument of `tf.compat.v1.train.batch()`. WARNING: This function is nondeterministic, since it starts a separate thread for each tensor. Enqueues a different list of tensors in different threads. Implemented using a queue -- a `QueueRunner` for the queue is added to the current `Graph`'s `QUEUE_RUNNER` collection. `len(tensors_list)` threads will be started, with thread `i` enqueuing the tensors from `tensors_list[i]`. `tensors_list[i1][j]` must match `tensors_list[i2][j]` in type and shape, except in the first dimension if `enqueue_many` is true. If `enqueue_many` is `False`, each `tensors_list[i]` is assumed to represent a single example. An input tensor `x` will be output as a tensor with shape `[batch_size] + x.shape`. If `enqueue_many` is `True`, `tensors_list[i]` is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of `tensors_list[i]` should have the same size in the first dimension. The slices of any input tensor `x` are treated as examples, and the output tensors will have shape `[batch_size] + x.shape[1:]`. The `capacity` argument controls the how long the prefetching is allowed to grow the queues. The returned operation is a dequeue operation and will throw `tf.errors.OutOfRangeError` if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself. *N.B.:* If `dynamic_pad` is `False`, you must ensure that either (i) the `shapes` argument is passed, or (ii) all of the tensors in `tensors_list` must have fully-defined shapes. `ValueError` will be raised if neither of these conditions holds. If `dynamic_pad` is `True`, it is sufficient that the *rank* of the tensors is known, but individual dimensions may have value `None`. In this case, for each enqueue the dimensions with value `None` may have a variable length; upon dequeue, the output tensors will be padded on the right to the maximum shape of the tensors in the current minibatch. For numbers, this padding takes value 0. For strings, this padding is the empty string. See `PaddingFIFOQueue` for more info. If `allow_smaller_final_batch` is `True`, a smaller batch value than `batch_size` is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the `shape` property will have a first `Dimension` value of `None`, and operations that depend on fixed batch_size would fail. Args: tensors_list: A list of tuples or dictionaries of tensors to enqueue. batch_size: An integer. The new batch size pulled from the queue. capacity: An integer. The maximum number of elements in the queue. enqueue_many: Whether each tensor in `tensor_list_list` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensor_list_list[i]`. dynamic_pad: Boolean. Allow variable dimensions in input shapes. The given dimensions are padded upon dequeue so that tensors within a batch have the same shapes. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (Optional) If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the same number and types as `tensors_list[i]`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensor_list_list`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility T)rr:rHr;rrr<rr ) rrr:rHr;rrr<rr#r#r$r 5sar ztrain.maybe_batch_joinzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.interleave(...).filter(...).batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).c Cst|||||||||| d S)aRuns a list of tensors to conditionally fill a queue to create batches. See docstring in `batch_join` for more details. Args: tensors_list: A list of tuples or dictionaries of tensors to enqueue. keep_input: A `bool` Tensor. This tensor controls whether the input is added to the queue or not. If it is a scalar and evaluates `True`, then `tensors` are all added to the queue. If it is a vector and `enqueue_many` is `True`, then each example is added to the queue only if the corresponding value in `keep_input` is `True`. This tensor essentially acts as a filtering mechanism. batch_size: An integer. The new batch size pulled from the queue. capacity: An integer. The maximum number of elements in the queue. enqueue_many: Whether each tensor in `tensor_list_list` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensor_list_list[i]`. dynamic_pad: Boolean. Allow variable dimensions in input shapes. The given dimensions are padded upon dequeue so that tensors within a batch have the same shapes. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (Optional) If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the same number and types as `tensors_list[i]`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensor_list_list`. )r:rHr;rrr<rr) rrrr:rHr;rrr<rr#r#r$maybe_batch_joins,rztrain.shuffle_batchzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.shuffle(min_after_dequeue).batch(batch_size)`.c Cs t||||d|||||| | d S)a Creates batches by randomly shuffling tensors. This function adds the following to the current `Graph`: * A shuffling queue into which tensors from `tensors` are enqueued. * A `dequeue_many` operation to create batches from the queue. * A `QueueRunner` to `QUEUE_RUNNER` collection, to enqueue the tensors from `tensors`. If `enqueue_many` is `False`, `tensors` is assumed to represent a single example. An input tensor with shape `[x, y, z]` will be output as a tensor with shape `[batch_size, x, y, z]`. If `enqueue_many` is `True`, `tensors` is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of `tensors` should have the same size in the first dimension. If an input tensor has shape `[*, x, y, z]`, the output will have shape `[batch_size, x, y, z]`. The `capacity` argument controls the how long the prefetching is allowed to grow the queues. The returned operation is a dequeue operation and will throw `tf.errors.OutOfRangeError` if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself. For example: ```python # Creates batches of 32 images and 32 labels. image_batch, label_batch = tf.compat.v1.train.shuffle_batch( [single_image, single_label], batch_size=32, num_threads=4, capacity=50000, min_after_dequeue=10000) ``` *N.B.:* You must ensure that either (i) the `shapes` argument is passed, or (ii) all of the tensors in `tensors` must have fully-defined shapes. `ValueError` will be raised if neither of these conditions holds. If `allow_smaller_final_batch` is `True`, a smaller batch value than `batch_size` is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the `shape` property will have a first `Dimension` value of `None`, and operations that depend on fixed batch_size would fail. Args: tensors: The list or dictionary of tensors to enqueue. batch_size: The new batch size pulled from the queue. capacity: An integer. The maximum number of elements in the queue. min_after_dequeue: Minimum number elements in the queue after a dequeue, used to ensure a level of mixing of elements. num_threads: The number of threads enqueuing `tensor_list`. seed: Seed for the random shuffling within the queue. enqueue_many: Whether each tensor in `tensor_list` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensor_list`. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (Optional) If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the types as `tensors`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensors`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility T)rrr9rHr;rr<rr) rrr:rrr9rHr;rr<rr#r#r$r sXr ztrain.maybe_shuffle_batchzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.filter(...).shuffle(min_after_dequeue).batch(batch_size)`.c Cs t|||||||||| | | d S)aiCreates batches by randomly shuffling conditionally-enqueued tensors. See docstring in `shuffle_batch` for more details. Args: tensors: The list or dictionary of tensors to enqueue. batch_size: The new batch size pulled from the queue. capacity: An integer. The maximum number of elements in the queue. min_after_dequeue: Minimum number elements in the queue after a dequeue, used to ensure a level of mixing of elements. keep_input: A `bool` Tensor. This tensor controls whether the input is added to the queue or not. If it is a scalar and evaluates `True`, then `tensors` are all added to the queue. If it is a vector and `enqueue_many` is `True`, then each example is added to the queue only if the corresponding value in `keep_input` is `True`. This tensor essentially acts as a filtering mechanism. num_threads: The number of threads enqueuing `tensor_list`. seed: Seed for the random shuffling within the queue. enqueue_many: Whether each tensor in `tensor_list` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensor_list`. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (Optional) If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the types as `tensors`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensors`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility )rr9rHr;rr<rr) rrr:rrrr9rHr;rr<rr#r#r$maybe_shuffle_batchCs1rztrain.shuffle_batch_joinzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.interleave(...).shuffle(min_after_dequeue).batch(batch_size)`.c Cst||||d|||||| d S)a0Create batches by randomly shuffling tensors. The `tensors_list` argument is a list of tuples of tensors, or a list of dictionaries of tensors. Each element in the list is treated similarly to the `tensors` argument of `tf.compat.v1.train.shuffle_batch()`. This version enqueues a different list of tensors in different threads. It adds the following to the current `Graph`: * A shuffling queue into which tensors from `tensors_list` are enqueued. * A `dequeue_many` operation to create batches from the queue. * A `QueueRunner` to `QUEUE_RUNNER` collection, to enqueue the tensors from `tensors_list`. `len(tensors_list)` threads will be started, with thread `i` enqueuing the tensors from `tensors_list[i]`. `tensors_list[i1][j]` must match `tensors_list[i2][j]` in type and shape, except in the first dimension if `enqueue_many` is true. If `enqueue_many` is `False`, each `tensors_list[i]` is assumed to represent a single example. An input tensor with shape `[x, y, z]` will be output as a tensor with shape `[batch_size, x, y, z]`. If `enqueue_many` is `True`, `tensors_list[i]` is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of `tensors_list[i]` should have the same size in the first dimension. If an input tensor has shape `[*, x, y, z]`, the output will have shape `[batch_size, x, y, z]`. The `capacity` argument controls the how long the prefetching is allowed to grow the queues. The returned operation is a dequeue operation and will throw `tf.errors.OutOfRangeError` if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself. If `allow_smaller_final_batch` is `True`, a smaller batch value than `batch_size` is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the `shape` property will have a first `Dimension` value of `None`, and operations that depend on fixed batch_size would fail. Args: tensors_list: A list of tuples or dictionaries of tensors to enqueue. batch_size: An integer. The new batch size pulled from the queue. capacity: An integer. The maximum number of elements in the queue. min_after_dequeue: Minimum number elements in the queue after a dequeue, used to ensure a level of mixing of elements. seed: Seed for the random shuffling within the queue. enqueue_many: Whether each tensor in `tensor_list_list` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensors_list[i]`. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (optional). If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the same number and types as `tensors_list[i]`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensors_list`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility T)rr9rHr;rr<rr) rrr:rr9rHr;rr<rr#r#r$rsTrztrain.maybe_shuffle_batch_joinzQueue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.interleave(...).filter(...).shuffle(min_after_dequeue).batch(batch_size)`.c Cst|||||||||| | d S)ayCreate batches by randomly shuffling conditionally-enqueued tensors. See docstring in `shuffle_batch_join` for more details. Args: tensors_list: A list of tuples or dictionaries of tensors to enqueue. batch_size: An integer. The new batch size pulled from the queue. capacity: An integer. The maximum number of elements in the queue. min_after_dequeue: Minimum number elements in the queue after a dequeue, used to ensure a level of mixing of elements. keep_input: A `bool` Tensor. This tensor controls whether the input is added to the queue or not. If it is a scalar and evaluates `True`, then `tensors` are all added to the queue. If it is a vector and `enqueue_many` is `True`, then each example is added to the queue only if the corresponding value in `keep_input` is `True`. This tensor essentially acts as a filtering mechanism. seed: Seed for the random shuffling within the queue. enqueue_many: Whether each tensor in `tensor_list_list` is a single example. shapes: (Optional) The shapes for each example. Defaults to the inferred shapes for `tensors_list[i]`. allow_smaller_final_batch: (Optional) Boolean. If `True`, allow the final batch to be smaller if there are insufficient items left in the queue. shared_name: (optional). If set, this queue will be shared under the given name across multiple sessions. name: (Optional) A name for the operations. Returns: A list or dictionary of tensors with the same number and types as `tensors_list[i]`. Raises: ValueError: If the `shapes` are not specified, and cannot be inferred from the elements of `tensors_list`. @compatibility(eager) Input pipelines based on Queues are not supported when eager execution is enabled. Please use the `tf.data` API to ingest data under eager execution. @end_compatibility )r9rHr;rr<rr) rrr:rrr9rHr;rr<rr#r#r$maybe_shuffle_batch_joins2r)N)NN) NNTNr4NNNN)NTNr4NNN)NTNr4NN)NTNr4NN)N)r8r4FNFFNN)r4FNFFNN)r8NFNFNN)NFNFNN)r8r4FNFFNN)r8r4FNFFNN)r4FNFFNN)r4FNFFNN)r8NFNFNN)r8NFNFNN)NFNFNN)NFNFNN)Lrtensorflow.python.eagerrtensorflow.python.frameworkrrrrrtensorflow.python.layersrtensorflow.python.opsr r r r r rrrrtensorflow.python.summaryrtensorflow.python.trainingrtensorflow.python.utilrZtensorflow.python.util.compatr tensorflow.python.util.tf_exportr_add_sparse_to_tensors_mapr_add_many_sparse_to_tensors_mapr"_take_many_sparse_from_tensors_maprdeprecated_endpointsr% deprecatedr&r6rZr]rkrnobjectrorrrrrrrrrrrrrrrrr r rrrrr rr rrrr#r#r#r$sb                      R E % 6; f  !  ( $ - - ^ 4 i 4 c ; ]