a d=icK@s&dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZeejejd Zd Zd Zd ZdZdZdZdZdZ dZ!e"Z#Gddde$Z%dddZ&Gddde'Z(Gddde)Z*ddZ+dS)z/Downloads experiment data from TensorBoard.dev.N)blob_pb2)experiment_pb2)export_service_pb2)util) grpc_util) tb_logging) tensor_utilz-_lz metadata.jsonz scalars.jsonz tensors.jsonzblob_sequences.jsontensorsZblobsZblob_z.binc@sZeZdZdZddZdddZddZd d Zd d Zd dZ ddZ ddZ ddZ dS)TensorBoardExporteraExports all of the user's experiment data from TensorBoard.dev. Data is exported into a directory, with one file per experiment. Each experiment file is a sequence of time series, represented as a stream of JSON objects, one per line. Each JSON object includes a run name, tag name, `tensorboard.compat.proto.summary_pb2.SummaryMetadata` proto (base64-encoded, standard RFC 4648 alphabet), and set of points. Points are stored in three equal-length lists of steps, wall times (as seconds since epoch), and scalar values, for storage efficiency. Such streams of JSON objects may be conveniently processed with tools like jq(1). For example one line of an experiment file might read (when pretty-printed): { "points": { "steps": [0, 5], "values": [4.8935227394104, 2.5438034534454346], "wall_times": [1563406522.669238, 1563406523.0268838] }, "run": "lr_1E-04,conv=1,fc=2", "summary_metadata": "CgkKB3NjYWxhcnMSC3hlbnQveGVudF8x", "tag": "xent/xent_1" } This is a time series with two points, both logged on 2019-07-17, one about 0.36 seconds after the other. c Csx||_||_tj|j}|r,tj|ddzt|jWn6tyr}z|jtj kr^t WYd}~n d}~00dS)a`Constructs a TensorBoardExporter. Args: reader_service_client: A TensorBoardExporterService stub instance. output_directory: Path to a directory into which to write data. The directory must not exist, to avoid stomping existing or concurrent output. Its ancestors will be created if needed. T)exist_okN) _api_outdirospathdirnamemakedirsmkdirOSErrorerrnoEEXISTOutputDirectoryExistsError)selfZreader_service_clientZoutput_directory parent_dirern/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/tensorboard/uploader/exporter.py__init__is  zTensorBoardExporter.__init__Nc #s|durt}tjddddd}t|j||d}|D]}|j}|j|jt |j t |j d}t |j |ttjt}t|d*}tj||dd|dWdn1s0Yz|||} tfd d tttfD} ttjtttjt| D]4\} } | | }tj| |dd|d|q*|VWdn1s|0YWq6t j!y} z*| "t j#j$krt%|nWYd} ~ q6d} ~ 00q6dS) aeExecutes the export flow. Args: read_time: A fixed timestamp from which to export data, as float seconds since epoch (like `time.time()`). Optional; defaults to the current time. Yields: After each experiment is successfully downloaded, the ID of that experiment, as a string. NT) create_time update_timename description) fieldmask read_time)rr rrx) sort_keys c s(i|] }|ttj|dqS)r#) enter_contextopenrrjoin).0filenameexperiment_dirstackrr sz.TensorBoardExporter.export..)&timerZExperimentMasklist_experimentsr experiment_idrr rZformat_time_absoluterr_experiment_directoryr rrrr(_FILENAME_METADATAr'jsondumpwrite_request_json_data contextlib ExitStack_FILENAME_SCALARS_FILENAME_TENSORS_FILENAME_BLOB_SEQUENCES_DIRNAME_TENSORS_DIRNAME_BLOBSflushgrpcRpcErrorcodeZ StatusCode CANCELLEDGrpcTimeoutException)rr"Zexperiment_metadata_mask experiments experimentr1Zexperiment_metadataZmetadata_filepathoutfiledataZ file_handlesblockr*rrr+rexport~sb     (       * zTensorBoardExporter.exportc cst}||_t|j||jj|t d}|D]}t |j d}|j|j|d}d}|dr||j|d<t}nB|dr||j||d<t}n |dr||j||d<t}|r4||fVq4dS)a3Given experiment id, generates JSON data and destination file name. The JSON data describes the run, tag, metadata, in addition to - Actual data in the case of scalars - Pointer to binary files in the case of blob sequences. For the case of blob sequences, this method has the side effect of downloading the contents of the blobs and writing them to files in a subdirectory of the experiment directory. Args: experiment_id: The id of the experiment to request data for. read_time: A fixed timestamp from which to export data, as float seconds since epoch (like `time.time()`). Optional; defaults to the current time. Yields: (JSON-serializable data, destination file name) tuples. metadataascii)runtagZsummary_metadataNpointsr blob_sequences)rZStreamExperimentDataRequestr1r set_timestampread_timestampr ZStreamExperimentDatarversion_metadatabase64 b64encodeZ tag_metadataZSerializeToStringdecoderun_nameZtag_nameZHasField_process_scalar_pointsrPr:_process_tensor_pointsr r;_process_blob_sequence_pointsrQr<) rr1r"requeststreamresponserLZ json_datar*rrrr7sD   z&TensorBoardExporter._request_json_datacCs(dd|jD}t|j|t|jdS)aProcess scalar data points. Args: points: `export_service_pb2.StreamExperimentDataResponse.ScalarPoints` proto. Returns: A JSON-serializable `dict` for the steps, wall_times and values of the scalar data points. cSsg|]}|dqSgeAZ ToNanosecondsr)trrr z>TensorBoardExporter._process_scalar_points..)steps wall_timesvalues)rflistrerg)rrPrfrrrrYs z*TensorBoardExporter._process_scalar_pointscsdd|jD}t|j|dd}|ds.|Stj|}||dd}dd|jD}fd d|D}tjt j ||g|R||d <|S) aProcess tensor data points. Args: points: `export_service_pb2.StreamExperimentDataResponse.TensorPoints` proto. experiment_id: ID of the experiment that the `TensorPoints` is a part of. Returns: A JSON-serializable `dict` for the steps, wall_times and the path to the .npz files that contain the saved tensor values. cSsg|]}|dqSr_r`rarrrrcrdz>TensorBoardExporter._process_tensor_points..N)rerftensors_file_pathrerfrcSsg|]}t|qSr)rZ make_ndarray)r)Z tensor_protorrrrc+scsg|]}|qSr)_fix_string_types)r)r#rrrrc/rdri) rfrhrer2r _get_tensor_file_pathrgnpZsavezrrr()rrPr1rf json_objectr,riZndarraysrrkrrZs$   z*TensorBoardExporter._process_tensor_pointscCs|jtjkr|S|dSdS)a%Change the dtype of text arrays to String rather than Object. np.savez ends up pickling np.object arrays, while it doesn't pickle strings. The downside is that it needs to pad the length of each string in the array to the maximal length string. We only want to do this type override in this final step of the export path. Args: ndarray: a tensor converted to an ndarray Returns: The original ndarray if not np.object, dtype converted to String if np.object. z|SN)ZdtypermZobject_Zastype)rZndarrayrrrrj4s z%TensorBoardExporter._fix_string_typescCsRd}tjtd||rd|ndd}tjtj||sD|S|d7}qdS)a#Get a nonexistent path for a tensor value. Args: experiment_dir: Experiment directory. wall_time: Timestamp of the tensor (seconds since the epoch in double). Returns: A nonexistent path for the tensor, relative to the experiemnt_dir. rz%.6fz_%dz.npzN)rrr(r=exists)rr,Z wall_timeindexZtensor_file_pathrrrrlHs  z)TensorBoardExporter._get_tensor_file_pathc Csdd|jD}t|j|gd}|d}|jD]T}g}|jD]:}|jjtjj krn| |jj |} | | q>| dq>| |q0|S)aAProcess blob sequence points. As a side effect, also downloads the binary contents of the blobs to respective files. The paths to the files relative to the experiment directory is encapsulated in the returned JSON object. Args: blob_sequences: `export_service_pb2.StreamDataResponse.BlobSequencePoints` proto. Returns: A JSON-serializable `dict` for the steps and wall_times, as well as the blob_file_paths, which are the relative paths to the downloaded blob contents. cSsg|]}|dqSr_r`rarrrrcnszETensorBoardExporter._process_blob_sequence_points..)rerfblob_file_pathsrsN) rfrhrergentriesZblobstaterZ BlobStateZBLOB_STATE_CURRENT_download_blobblob_idappend) rrQr1rfrnrsZblobseqZseq_blob_file_pathsentryZ blob_pathrrrr[^s&     z1TensorBoardExporter._process_blob_sequence_pointsc Cst|j|}tj|d}tj|tt|t }t |d~}z*|j j |t dD]}||jqRWnBtjy}z(td||WYd}~WddSd}~00Wdn1s0Ytj||S)ahDownload the blob via rpc. Args: blob_id: Id of the blob. experiment_id: Id of the experiment that the blob belongs to. Returns: If the blob is downloaded successfully: The path of the downloaded blob file relative to the experiment directory. Else: `None`. )rwxbrKz2Omitting blob (id: %s) due to download failure: %sN)r2r rZStreamBlobDataRequestrrr(r>_FILENAME_BLOBS_PREFIX_FILENAME_BLOBS_SUFFIXr'r ZStreamBlobDatarrTr6rHr@rAloggererrorrelpath) rrwr1r,r\Z blob_abspathfr^Z rpc_errorrrrrvs*     Bz"TensorBoardExporter._download_blob)N) __name__ __module__ __qualname____doc__rrJr7rYrZrjrlr[rvrrrrr Is F<"&r ccs|durt}tjtd}t|j||r:|j||j |t d}|D]F}|j rn|j D] }|Vq`qP|j rtdt|j fqPtd|qPdS)acYields all of the calling user's experiments. Args: api_client: A TensorBoardExporterService stub instance. fieldmask: An optional `experiment_pb2.ExperimentMask` value. read_time: A fixed timestamp from which to export data, as float seconds since epoch (like `time.time()`). Optional; defaults to the current time. Yields: For each experiment owned by the user, an `experiment_pb2.Experiment` value. Raises: RuntimeError: If the server returns experiment IDs but no experiments, as in an old, unsupported version of the protocol. N)limitrKz4Server sent experiment_ids without experiments: <%r>zAStreamExperiments RPC returned response with no experiments: <%r>)r/rZStreamExperimentsRequest _MAX_INT64rrRrSZexperiments_maskZCopyFromZStreamExperimentsrrTrEZexperiment_ids RuntimeErrorrhr}warning)Z api_clientr!r"r\r]r^rFrrrr0s.     r0c@s eZdZdS)rN)rrrrrrrrsrcseZdZfddZZS)rDcstt||||_dS)N)superrDrr1)rr1 __class__rrrszGrpcTimeoutException.__init__)rrrr __classcell__rrrrrDsrDcCs8t|t}|r&tdjt||dtj|d|S)Nz>Unexpected characters ({bad_chars!r}) in experiment ID {eid!r}) bad_charsZeidz experiment_%s) frozenset_FILENAME_SAFE_CHARSrformatsortedrrr()base_dirr1rrrrr2s r2)NN),rrUr8rr@r4rstringr/numpyrmZtensorboard.uploader.protorrrZtensorboard.uploaderrZtensorboard.utilrrrr ascii_lettersdigitsrrr3r:r;r<r=r>r{r|Z get_loggerr}objectr r0 ValueErrorr ExceptionrDr2rrrrsD       d ,