a lc$E@s`ddlZddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZmZmZmZmZmZmZerddlmZmZGd d d eej e!Z"Gd d d e"e!Z#Gd dde"e!Z$dZ%ej&dddkrRe%ej'ej(f7Z%ej&dddkrRe%ej)ej*ej+ej,ej-ej.f7Z%dddZ/dS)N)Module) IterableIteratorListOptionalTupleAnycast TYPE_CHECKINGType)xrange) LineNumbers)Token match_tokenis_non_coding_tokenpatched_generate_tokens last_stmtannotate_fstring_nodesgenerate_tokens)AstNode TokenInfoc@s6eZdZddZejddZd ddZd dd Zd S) ASTTextBasecCs$||_t|}||_t||_dSN) _filenamesix ensure_text_textr _line_numbers)self source_textfilenamer"O/var/www/html/django/DPS/env/lib/python3.9/site-packages/asttokens/asttokens.py__init__"s zASTTextBase.__init__cCstdS) Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node. If the positions can't be determined, or the nodes don't correspond to any particular text, returns ``(1, 0)`` for both. ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``. This means that if ``padded`` is True, the start position will be adjusted to include leading whitespace if ``node`` is a multiline statement. N)NotImplementedErrorrnodepaddedr"r"r#get_text_positions1s zASTTextBase.get_text_positionsTcCs(|||\}}|jj||jj|fS)a Returns the (startpos, endpos) positions in source text corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r*rline_to_offsetrr(r)startendr"r"r#get_text_range?s  zASTTextBase.get_text_rangecCs|||\}}|j||S)z Returns the text corresponding to the given node. Returns '' for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r/rr,r"r"r#get_textMszASTTextBase.get_textN)T)T) __name__ __module__ __qualname__r$abcabstractmethodr*r/r0r"r"r"r#r!s  rcseZdZdZd%fdd ZddZd d Zed d Zed dZ eddZ eddZ ddZ ddZ ddZd&ddZd'ddZd(ddZd)dd Zd*d!d"Zd#d$ZZS)+ ASTTokensa ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and as tokens, and is used to mark and access token and position information. ``source_text`` must be a unicode or UTF8-encoded string. If you pass in UTF8 bytes, remember that all offsets you'll get are to the unicode text, which is available as the ``.text`` property. If ``parse`` is set, the ``source_text`` will be parsed with ``ast.parse()``, and the resulting tree marked with token info and made available as the ``.tree`` property. If ``tree`` is given, it will be marked and made available as the ``.tree`` property. In addition to the trees produced by the ``ast`` module, ASTTokens will also mark trees produced using ``astroid`` library . If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST tree created separately. FN csrtt||||r"t||n||_|dur:t|j}t| ||_ dd|j D|_ |jrn| |jdS)NcSsg|] }|jqSr")startpos).0tokr"r"r# |z&ASTTokens.__init__..) superr6r$astparse_treerrlist_translate_tokens_tokens_token_offsets mark_tokens)rr r?treer!tokens __class__r"r#r$ms zASTTokens.__init__cCsddlm}|||dS)ap Given the root of the AST or Astroid tree produced from source_text, visits all nodes marking them with token and position information by adding ``.first_token`` and ``.last_token``attributes. This is done automatically in the constructor when ``parse`` or ``tree`` arguments are set, but may be used manually with a separate AST or Astroid tree. r ) MarkTokensN)rErJZ visit_tree)rZ root_noderJr"r"r#rEs zASTTokens.mark_tokensc csdtt|D]R\}}|\}}}}}t|||||||j|d|d|j|d|dVq dS)zS Translates the given standard library tokens into our own representation. rr N) enumeraterrrr+) rZoriginal_tokensindexr:tok_typetok_strr-r.liner"r"r#rBs zASTTokens._translate_tokenscCs|jS)z,The source code passed into the constructor.)rrr"r"r#textszASTTokens.textcCs|jS)zIThe list of tokens corresponding to the source code from the constructor.)rCrPr"r"r#rGszASTTokens.tokenscCs|jS)zTThe root of the AST tree passed into the constructor or parsed from the source code.)r@rPr"r"r#rFszASTTokens.treecCs|jS)zThe filename that was parsed)rrPr"r"r#r!szASTTokens.filenamecCs|jt|j|dS)z Returns the token containing the given character offset (0-based position in source text), or the preceeding token if the position is between tokens. r )rCbisectrD)roffsetr"r"r#get_token_from_offsetszASTTokens.get_token_from_offsetcCs||j||S)z Returns the token containing the given (lineno, col_offset) position, or the preceeding token if the position is between tokens. )rTrr+rlineno col_offsetr"r"r# get_tokens zASTTokens.get_tokencCs|||j||S)zd Same as get_token(), but interprets col_offset as a UTF8 offset, which is what `ast` uses. )rXr from_utf8_colrUr"r"r#get_token_from_utf8szASTTokens.get_token_from_utf8cCs2|jd}|s(t|j|jr(|d7}q|j|S)z Returns the next token after the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r rLrrCtyperr: include_extrair"r"r# next_tokens   zASTTokens.next_tokencCs2|jd}|s(t|j|jr(|d8}q|j|S)z Returns the previous token before the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r r[r]r"r"r# prev_tokens   zASTTokens.prev_tokencCs>|}|r|jn|j}t|||s:t|js:||dd}q|S)z Looks for the first token, starting at start_token, that matches tok_type and, if given, the token string. Searches backwards if reverse is True. Returns ENDMARKER token if not found (you can check it with `token.ISEOF(t.type)`. Tr^)rar`rtokenISEOFr\)rZ start_tokenrMrNreversetadvancer"r"r# find_tokens zASTTokens.find_tokenccs<t|j|jdD]$}|s*t|j|js|j|VqdS)z Yields all tokens in order from first_token through and including last_token. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. r N)r rLrrCr\)r first_token last_tokenr^r_r"r"r# token_ranges zASTTokens.token_rangecCs|j|j|j|dS)z Yields all tokens making up the given node. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. rb)rkrirj)rr(r^r"r"r# get_tokensszASTTokens.get_tokenscCsNt|dsdS|jj}|jj}|rFtdd||DrF|ddf}||fS)r%rir rrncss|]}t|tjVqdSr)rrcNEWLINE)r9rfr"r"r# r<z/ASTTokens.get_text_positions..r)hasattrrir-rjr.anyrlr,r"r"r#r*s  zASTTokens.get_text_positions)FNr7N)F)F)NF)F)F)r1r2r3__doc__r$rErBpropertyrQrGrFr!rTrXrZr`rarhrkrlr* __classcell__r"r"rHr#r6Ys,          r6csJeZdZdZdfdd ZeddZedd Zd d Zd d Z Z S)ASTTexta Supports the same ``get_text*`` methods as ``ASTTokens``, but uses the AST to determine the text positions instead of tokens. This is faster than ``ASTTokens`` as it requires less setup work. It also (sometimes) supports nodes inside f-strings, which ``ASTTokens`` doesn't. Astroid trees are not supported at all and will raise an error. Some node types and/or Python versions are not supported. In these cases the ``get_text*`` methods will fall back to using ``ASTTokens`` which incurs the usual setup cost the first time. If you want to avoid this, check ``supports_tokenless(node)`` before calling ``get_text*`` methods. Nr7csRt|tjtdfstdtt|||||_|jdurHt |jd|_ dS)NzASTText only supports AST trees) isinstancer>ASTr\r&r=rvr$r@r _asttokens)rr rFr!rHr"r#r$+s  zASTText.__init__cCs,|jdur&t|j|j|_t|j|jSr)r@r>r?rrrrPr"r"r#rF;s  z ASTText.treecCs&|jdur t|j|j|jd|_|jS)N)rFr!)ryr6rrFrrPr"r"r# asttokensCs zASTText.asttokensc Cstjdddkrtdt|tjrrrZoffset_to_linelenrrqgetattrrrVrYrWr int end_linenoend_col_offset) rr(r)Z decoratorsZ start_nodeZstart_col_offsetr-Zend_noderrr.r"r"r#_get_text_positions_tokenlessNs*       z%ASTText._get_text_positions_tokenlesscCs2t|ddrdSt|r$|||S|j||S)r%Z_broken_positionsNrm)rsupports_tokenlessrrzr*r'r"r"r#r*~s  zASTText.get_text_positions)Nr7) r1r2r3rsr$rtrFrzrr*rur"r"rHr#rvs  0rvr"r{r|cCs@t|tjtdfo>t|t o>tjdddko>dtjvS)aQ Returns True if the Python version and the node (if given) are supported by the ``get_text*`` methods of ``ASTText`` without falling back to ``ASTTokens``. See ``ASTText`` for why this matters. The following cases are not supported: - Python 3.7 and earlier - PyPy - Astroid nodes (``get_text*`` methods of ``ASTText`` will raise an error) - ``ast.arguments`` and ``ast.withitem`` - The following nodes in Python 3.8 only: - ``ast.arg`` - ``ast.Starred`` - ``ast.Slice`` - ``ast.ExtSlice`` - ``ast.Index`` - ``ast.keyword`` Nr{r|pypy) rwr>rxr\_unsupported_tokenless_typesrrversionlower)r(r"r"r#rs  r)N)0r4r>rRrrcrtypingrrrrrrr r r rZ six.movesr line_numbersrutilrrrrrrrrrwith_metaclassABCMetaobjectrr6rvrr argumentswithitemargStarredSliceExtSliceIndexkeywordrr"r"r"r#s6 ,  $8Dx