a NSicf%@sLzddlmZmZWney.edYn0ddlZddlZddlZddlmZddl m Z ddl m Z ddlmZmZmZmZddlmZiad d Zee eefd d d Zd&eeedddZddZddZedddZedddZedddZ efeeddd Z!efeedd!d"Z"ed#e!ed$e"ed%edS)')urlparse urlunparsezEurllib cannot be found, urlparse from python2 is no longer supported.N) timedelta)Dict) FileStore PrefixStoreStoreTCPStore)default_pg_timeoutcCs"|tvrtd||t|<dS)aRegisters a new rendezvous handler. Before we can run collective algorithms, participating processes need to find each other and exchange information to be able to communicate. We call this process rendezvous. The outcome of the rendezvous process is a triplet containing a shared key/value store, the rank of the process, and the total number of participating processes. If none of the bundled rendezvous methods apply to your execution environment you can opt to register your own rendezvous handler. Pick a unique name and use the URL scheme to identify it when calling the `rendezvous()` function. Args: scheme (str): URL scheme to identify your rendezvous handler. handler (function): Handler that is invoked when the `rendezvous()` function is called with a URL that uses the corresponding scheme. It must be a generator function that yields the triplet. z/Rendezvous handler for {}:// already registeredN)_rendezvous_handlers RuntimeErrorformat)schemehandlerrX/var/www/html/django/DPS/env/lib/python3.9/site-packages/torch/distributed/rendezvous.pyregister_rendezvous_handlers r)queryreturncCs(tddddtd|dDDS)Ncss|]}|d|dfVqdS)rr Nr.0pairrrr 8z!_query_to_dict..css|]}|dVqdS)=N)splitrrrrr8r&)dictfilterrrrrr_query_to_dict7sr!)urlrank world_sizecKst|tjs tdt||t|tjs:td|t|tjsTtd|t|}|dksl|dkrt |j }d|vrd|vsJdj|d|dkr||d<|dkr||d<|j d d d d | Dd }t|}|jtvrtd|jt|j|fi|S)Nz`url` must be a string. {}: {}z`rank` must be an integer. {}z#`world_size` must be an integer. {}r"r$r%zEThe url: {url} has node-specific arguments(rank, world_size) already.r#{}rcSsg|]\}}d||qSz{}={}rrkvrrr Trzrendezvous..r zNo rendezvous handler for {}://) isinstancesixstring_classesr rtypenumbersIntegralrr!r_replacejoinitemsrrr )r#r$r%kwargsresult query_dictrrr rendezvous:s<     r:cCst|j}d}|jdkr8tjd|}ttjd|}t|j}|dkrVt ||d<t ||d<|j d d d d | Dd }t|}|jtvrtd |jtt|j|\}}}|S) Nr"envRANK WORLD_SIZEr$r%r'rcSsg|]\}}d||qSr(r)r*rrrr-prz._create_store_from_options..r zNo handler for {}://)r init_methodrosenvirongetintr!rstrr4rr5r6rr r next)Zbackend_optionsr$r8r%r9r#store_rrr_create_store_from_options]s&      rGcCs td|S)Nz+Error initializing torch.distributed using ) ValueErrormsgrrr_rendezvous_errorzsrKr&c ksdd}t|}|j}tjdkrPddl}|j|j}|j|}|rPtj |}|s\|dt |j }d|vrv|dd|vr|d t |d}t |d} t || } | || fVtd dS) NcSs td|S)Nzfile:// rendezvous: rKrIrrr_errorsz(_file_rendezvous_handler.._errorwin32rz path missingr$rank parameter missingr%world size parameter missingz3Unable to perform rerendezvous using file:// method)rpathsysplatformurllib.requestnetlocrequest url2pathnamer?normpathr!rrBrr ) r#r7rMr8rQurllib full_pathr9r$r%rErrr_file_rendezvous_handler~s*         r[)rcCstjddtdkS)NZTORCHELASTIC_USE_AGENT_STORET)r?r@rArCrrrr_torchelastic_use_agent_storesr\cCsvd|krdks&ntd|dtrVtjd}t|||d|}td||S|dk}t|||||dd Sd S) a Smartly creates a c10d Store object on ``rank`` based on whether we need to re-use agent store. The TCPStore server is assumed to be hosted on ``hostname:port``. If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that the agent leader (node rank 0) hosts the TCPStore server (for which the endpoint is specified by the given ``hostname:port``). Hence ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname and port are correctly passed via ``hostname`` and ``port``. All non-zero ranks will create and return a TCPStore client. riz-port must have value from 0 to 65535 but was .ZTORCHELASTIC_RESTART_COUNTFz/worker/attempt_T)Z multi_tenantN)rHr\r?r@r r)hostnameportr$r%timeoutattemptZ tcp_storeZ start_daemonrrr_create_c10d_stores  rb)r#r`c ksdd}t|}|js|dt|j}d|vr8|dd|vrH|dt|d}t|d}|jdusnJt|j|j|||}|||fVtddS) NcSs td|S)Nztcp:// rendezvous: rLrIrrrrMsz'_tcp_rendezvous_handler.._errorzport number missingr$rOr%rPz3Unable to perform re-rendezvous using tcp:// method)rr_r!rrBr^rbr ) r#r`r7rMr8r9r$r%rErrr_tcp_rendezvous_handlers    rcc +sddfddttdfdd }t|}t|j}d|vrPt|d}n t|d }d |vrrt|d }n t|d }|d }t|d } t|| |||} | ||fVtddS)NcSs td|S)Nzenv:// rendezvous: rLrIrrrrMsz'_env_rendezvous_handler.._errorcs d|S)Nz-environment variable %s expected, but not setr)var)rMrr _env_errorsz+_env_rendezvous_handler.._env_error)env_varrcs$tj|d}|s|n|SdS)N)r?r@rA)rfZenv_val)rerr_get_env_or_raises z2_env_rendezvous_handler.._get_env_or_raiser$r<r%r=Z MASTER_ADDRZ MASTER_PORTz3Unable to perform re-rendezvous using env:// method)rCrr!rrBrbr ) r#r`r7rgr8r9r$r%Z master_addrZ master_portrEr)rerMr_env_rendezvous_handlers       rhtcpr;file)r"r")# urllib.parserr ImportErrorr2r?rRdatetimertypingr torch._six_sixr/Ztorch.distributedrrrr constantsr r rrCr!rBr:rGrKr[boolr\rbrcrhrrrrsB       #    -