a ==icD@s<ddlmZmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZddlmZmZmZddlmZdd lmZdd lmZmZmZdd lm Z dd l!Z"dd l#m$Z%dd l&Z&dd l'Z'dd l(Z(dd l)Z)dddZ*dddZ+GdddeZ,GdddeZ-GdddeZ.GdddeZ/d S))CompositeReaderCompositeReaderBuilder ReaderBuilderReaderWithDelayReaderWithLimitReaderWithTimeLimit)Dataset) DBFileReader)pipe)Struct NewRecord FeedRecord) LocalSession) TaskGroup final_output WorkspaceType)TestCase) CachedReader)core workspaceschema)opsNdc Cs|pd}td|}t|Ntdtt|||f}t||}t ||d}t |||Wdn1sv0Y| ||S)Nsrc{}_initlabelname) rNetformat NameScoper nparrayranger rr run)wssizeoffsetrZsrc_initZ src_valuesZ src_blobssrc_dsr)j/home/droni/.local/share/virtualenvs/DPS-5Je3_V2c/lib/python3.9/site-packages/caffe2/python/dataio_test.pymake_source_dataset!s   * r+cCsf|pd}td|}t|&t||d}||Wdn1sN0Y|||S)Ndstrr)rrrr rZ init_emptyr$)r%rrZdst_initdst_dsr)r)r*make_destination_dataset-s  ( r.c@s,eZdZddZddZddZddZd S) TestReaderBuildercCs0tdtf|_||_||_||_d|_dS)Nr)rr ZScalar_schema_name_size_offset_src_ds)selfrr&r'r)r)r*__init__8s zTestReaderBuilder.__init__cCs|jSN)r0r5r)r)r*rAszTestReaderBuilder.schemacCst||j|j|jd|_iS)Nr'r&r)r+r3r2r1r4)r5r%r)r)r*setupDs zTestReaderBuilder.setupcKs|jSr7)r4)r5kwargsr)r)r* new_readerIszTestReaderBuilder.new_readerN)__name__ __module__ __qualname__r6rr:r<r)r)r)r*r/7s r/c@sDeZdZeejddddZeejddddZ dS)TestCompositeReaderZ JENKINS_URLzFlaky test on Jenkinscs`tjt}d}ddt|D}dfddt|D}fddt||D}fdd|D}t||D]\}}t|t||qxtj ddt||D} t | } t 6} t |d d|D} t | | dd Wdn1s0Y|| t|D]@} tt| || }t|| |d | qdS) NcSsg|]}d|qSzsrc_{}r.0ir)r)r* Sz=TestCompositeReader.test_composite_reader..rcsg|] }|qSr)r)rDr&r)r*rGUrHcs g|]\}}t||dqSr9)r+rErr'r&r%r)r*rGVscs g|]}t|jdqS)r) fetch_blobstrZ field_blobs)rErr%r)r*rGYrHcSs g|]\}}||fqSr))content clone_schema)rErr(r)r)r*rG`scSsg|] }|qSr))reader)rEr(r)r)r*rGirHnum_runtime_threadszi: {})rC Workspacerr#zipnptassert_array_equalrr r.rrr writerr$sortedrMrNrPrr)r5sessionnum_srcsnamesoffsetsZsrc_dsesdatadr' dst_ds_schemar-tgrRrF written_datar)rLr*test_composite_readerNs:    0  z)TestCompositeReader.test_composite_readercs.tj}t|}d}ddt|D}dfddt|D}fddt||D}tjddt||D}t||}t <} t ||} | j |dt | |dd Wdn1s0Y|| t||D]D\} } t|t|| } tt| | | d | qdS) NrAcSsg|]}d|qSrBrCrDr)r)r*rGwrHzETestCompositeReader.test_composite_reader_builder..rcsg|] }|qSr)r)rDrIr)r*rGyrHcsg|]\}}t||dqSrJ)r/rKrIr)r*rGzscSsg|]\}}||fqSr))r)rErZsrc_ds_builderr)r)r*rGsrOrSzname: {})rrUrVrr#rWrr r.rrr:r r<rZr$r[rMrNrPrrXrYr)r5r%r\r]r^r_Zsrc_ds_buildersrbr-rcZreader_builderrr'rdr)rIr*test_composite_reader_builderrs>    $ z1TestCompositeReader.test_composite_reader_builderN) r=r>r?unittestskipIfosenvirongetrerfr)r)r)r*r@Ms #r@c@s\eZdZddZddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ dS)TestReaderWithLimitcsVtj}t|}t|}dgdfdd}t"}t|d|dWdn1s^0Y||| d d| d d| d  dtF}t|d d }tt |d d dd }t||d dWdn1s0Y||| d d | d d | d  d dS)NrAc slt:tgdg}tgdg}tgdg}Wdn1sH0Yttgdg}Wdn1s~0Yt|t|tTtt|t|Wdn1s0Yt|Wdn1s0YtFt t|d<t t|d<t t|d<Wdn1s^0Y|S)NZglobal_counterZglobal_counter2Zglobal_counter3 task_counterr) rZ task_initZ CreateCounterZtask_instance_initZCountUpZtask_instance_exitloopZ RetrieveCountZ task_exitr)ZrecZcounter1Zcounter2Zcounter3rmZtotalsr)r*procs" , ,   (( 2z6TestReaderWithLimit.test_runtime_threads..proc)rT processorrrrnrorS)num_iter)rtrT) rrUrVrr+rr rRr$ assertEqualfetchr)r5r%r\r(rrrcq1q2r)rqr*test_runtime_threadss,   0 . z(TestReaderWithLimit.test_runtime_threadscCs<tj}t|}t||d}t||}||||fS)NrI)rrUrVrr+r.rPrQ)r5r&r%r\r(r-r)r)r*_test_limit_reader_init_shareds   z2TestReaderWithLimit._test_limit_reader_init_sharedcKs$||\} } } } ttjdZ} |dkrD|t| |fi|}n|| fi|}t|| |dWdn1s~0Y| | t t | j t | }|||||||||t | j t | tt||| j t ||dS)N)Zworkspace_typerrS)r}rrGLOBALrrRr rZr$lenr[blobsrNrPrryZassertGreaterEqualZassertLessEqualrxlistr#Z data_finished)r5Z reader_classr&expected_read_lenexpected_read_len_thresholdexpected_finish num_threads read_delayZ limiter_argsr%r\r(r-rcrRread_lenr)r)r*_test_limit_reader_shareds8  0 " z-TestReaderWithLimit._test_limit_reader_sharedc Cs|jtdddddddddS)NrrTrsr&rrrrrrvrrr8r)r)r*%test_count_limit_reader_without_limitsz9TestReaderWithLimit.test_count_limit_reader_without_limitc Cs|jtdddddddddS)NrrFrsrrr8r)r)r*'test_count_limit_reader_with_zero_limitsz;TestReaderWithLimit.test_count_limit_reader_with_zero_limitc Cs|jtdddddddddS)Nr rFrsrrr8r)r)r*&test_count_limit_reader_with_low_limitsz:TestReaderWithLimit.test_count_limit_reader_with_low_limitc Cs|jtdddddddddS)NrrTrsnrrr8r)r)r*'test_count_limit_reader_with_high_limitsz;TestReaderWithLimit.test_count_limit_reader_with_high_limitc Cs|jtdddddddddS)NrrTrsg?r&rrrrrdurationrrr8r)r)r*$test_time_limit_reader_without_limit&sz8TestReaderWithLimit.test_time_limit_reader_without_limitc CsVd}d}d}d}tt|||}|d|}|jt||d|dd|||ddS)N2g?rnroFr)introundrr)r5r&rZsleep_durationrrr)r)r*'test_time_limit_reader_with_short_limit1s z;TestReaderWithLimit.test_time_limit_reader_with_short_limitc Cs|jtdddddddddS)NrrTrg?rrrr8r)r)r*&test_time_limit_reader_with_long_limitJsz:TestReaderWithLimit.test_time_limit_reader_with_long_limitN) r=r>r?r|r}rrrrrrrrr)r)r)r*rls2 $     rlc@seZdZddZddZeddZddZed d Zed d Z e d e j vdddZe d e j vdddZdS)TestDBFileReadercCs g|_dSr7) temp_pathsr8r)r)r*setUpYszTestDBFileReader.setUpcCs|jD]}||qdSr7)r _delete_path)r5pathr)r)r*tearDown\s zTestDBFileReader.tearDowncCs2tj|rt|ntj|r.t|dSr7)ririsfileremoveisdirshutilrmtree)rr)r)r*ras   zTestDBFileReader._delete_pathcCs>t}|j}Wdn1s$0Y|j||Sr7)tempfileNamedTemporaryFilerrappend)r5f temp_pathr)r)r*_make_temp_pathhs $ z TestDBFileReader._make_temp_pathcCst||}|Sr7)r+rR)r%r&r(r)r)r*_build_source_readeros z%TestDBFileReader._build_source_readercCsnt||}t"}t||ddWdn1s@0Y|||jt|  S)NrsrS) r.rrQrr rZr$rrNrPrry)r%rRr\r-rcr)r)r*_read_all_datats 0 zTestDBFileReader._read_all_dataLevelDBz Need LevelDBc Cstj}t|}|}t||d|dd}|}||| |||}| t |t t dt||d|}|}||| |||}| t |t t d||t||d|}|}||| |||}| t |t t d||dS)NrF)Z loop_overi,)rrUrVrrrrbuild_cache_stepr$rrxr[rr#r) r5r%r\db_pathZcached_reader1rr`Zcached_reader2Zcached_reader3r)r)r*test_cached_reader~s4       z#TestDBFileReader.test_cached_readercCstj}t|}|}t||d|dd}|}||t |dd}| |||}| t |t td||dS)Nrr)rZdb_type)rrUrVrrrrrr$r rrxr[rr#r)r5r%r\rZ cached_readerrZdb_file_readerr`r)r)r*test_db_file_readers"   z$TestDBFileReader.test_db_file_readerN)r=r>r?rr staticmethodrrrrrgrhrrUZregistered_dbsrrr)r)r)r*rXs    'r)rrN)N)0Zcaffe2.python.dataiorrrrrrZcaffe2.python.datasetrZcaffe2.python.db_file_readerr Zcaffe2.python.pipeliner Zcaffe2.python.schemar r r Zcaffe2.python.sessionrZcaffe2.python.taskrrrZcaffe2.python.test_utilrZcaffe2.python.cached_readerrZ caffe2.pythonrrrZcaffe2.python.net_builderrnumpyr!Z numpy.testingtestingrXrirrgrr+r.r/r@rlrr)r)r)r*s.          JB