a lc8@sdZddlmZddlmZmZmZmZmZm Z ddl m Z m Z edZ Gdddee ZGdd d ZGd d d ee ZGd d d ZddZddZddZddZeedddZddZddZddZdS) a This module defines the data structures used to represent a grammar. Specifying grammars in pgen is possible with this grammar:: grammar: (NEWLINE | rule)* ENDMARKER rule: NAME ':' rhs NEWLINE rhs: items ('|' items)* items: item+ item: '[' rhs ']' | atom ['+' | '*'] atom: '(' rhs ')' | NAME | STRING This grammar is self-referencing. This parser generator (pgen2) was created by Guido Rossum and used for lib2to3. Most of the code has been refactored to make it more Pythonic. Since this was a "copy" of the CPython Parser parser "pgen", there was some work needed to make it more readable. It should also be slightly faster than the original pgen2, because we made some optimizations. ) literal_eval)TypeVarGenericMappingSequenceSetUnion) GrammarParserNFAState _TokenTypeTc@s6eZdZdZeeeedfeedfdddZdS)Grammara Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The only important part in this parsers are dfas and transitions between dfas. zDFAState[_TokenTypeT]ReservedString)start_nonterminal rule_to_dfasreserved_syntax_stringscCs||_||_||_dSN)nonterminal_to_dfasrr)selfrrrrQ/var/www/html/django/DPS/env/lib/python3.9/site-packages/parso/pgen2/generator.py__init__/szGrammar.__init__N)__name__ __module__ __qualname____doc__strrrrrrrrr %s   r c@s0eZdZdZgfdeddddZddZdS) DFAPlanzj Plans are used for the parser to create stack nodes and do the proper DFA state transitions. DFAStatenext_dfa dfa_pushescCs||_||_dSrr)rrr rrrr=szDFAPlan.__init__cCsd|jj|j|jfS)Nz %s(%s, %s)) __class__rrr rrrr__repr__AszDFAPlan.__repr__N)rrrrrrr#rrrrr8src@sFeZdZdZeeeedddZddZddZ d d Z d d Z d S)raa The DFAState object is the core class for pretty much anything. DFAState are the vertices of an ordered graph while arcs and transitions are the edges. Arcs are the initial edges, where most DFAStates are not connected and transitions are then calculated to connect the DFA state machines that have different nonterminals. ) from_rulenfa_setfinalcCs^t|tsJttt|ts$Jt|ts2J||_||_i|_i|_i|_ ||v|_ dSr) isinstancesetnextiterr r$r%arcsnonterminal_arcs transitionsis_final)rr$r%r&rrrrOszDFAState.__init__cCs8t|tsJ||jvsJt|ts*J||j|<dSr)r'rr+r)rnext_labelrrradd_arc`szDFAState.add_arccCs*|jD]\}}||ur ||j|<q dSr)r+items)roldnewr0r/rrr unifystatefszDFAState.unifystatecCsdt|tsJ|j|jkrdSt|jt|jkr6dS|jD]\}}||j|ur@dSq@dS)NFT)r'rr.lenr+r2get)rotherr0r/rrr__eq__ks zDFAState.__eq__cCsd|jj|j|jfS)Nz<%s: %s is_final=%s>)r!rr$r.r"rrrr#yszDFAState.__repr__N) rrrrrrr rr1r5r9r#rrrrrEs  rc@s&eZdZdZedddZddZdS)r z Most grammars will have certain keywords and operators that are mentioned in the grammar as strings (e.g. "if") and not token types (e.g. NUMBER). This class basically is the former. valuecCs ||_dSrr:)rr;rrrrszReservedString.__init__cCsd|jj|jfS)Nz%s(%s))r!rr;r"rrrr#szReservedString.__repr__N)rrrrrrr#rrrrr sr cCspd}|rld}t|D]T\}}t|dt|D]8}||}||kr.||=|D]}|||qLd}qq.qqdS)a This is not theoretically optimal, but works well enough. Algorithm: repeatedly look for two states that have the same set of arcs (same labels pointing to the same nodes) and unify them, until things stop changing. dfas is a list of DFAState instances TFN) enumerateranger6r5)dfaschangesiZstate_ijZstate_jstaterrr_simplify_dfass rDc st|tsJt|tsJfddt}||t|j||g}|D]}i}|jD]6}|jD]*}|jdurd||jt}|j |qdqZ| D]D\} }|D]} | j|krqqt|j||} | | | | | qqL|S)z Uses the powerset construction algorithm to create DFA states from sets of NFA states. Also does state reduction if some states are not needed. csJt|tsJ||vrdS|||jD]}|jdur*|j|q*dSr)r'r addr+nonterminal_or_stringr)) nfa_state base_nfa_setnfa_arc addclosurerrrKs   z_make_dfas..addclosureN) r'r r(rr$r%r+rF setdefaultr)r2appendr1) startfinishrHZstatesrCr+rGrIr%rFZ nested_staterrJr _make_dfass*        rPc Cstd|j|g}t|D]\}}td|||ur4dp6d|jD]^}|j|j}}||vrf||}nt|}|||durtd|q@td||fq@qdS)NzDump of NFA for State(final)z -> %d %s -> %d) printr$r=r+rFr)indexr6rM) rNrOtodorArCarcr0r/rBrrr _dump_nfas    rYcCsftd|djt|D]H\}}td||jr0dp2d|jD]\}}td|||fq@qdS)NzDump of DFA forrrQrRrSrT)rUr$r=r.r+r2rV)r?rArC nonterminalr/rrr _dump_dfass r[) bnf_grammarreturnc Csi}d}t|D]2\}}t||}t||||j<|dur|j}qi}|D]T\}}|D]F} | jD]6\} } | |vr| | j| <qnt||| } t | | j | <qnq`qTt |t |||S)a ``bnf_text`` is a grammar in extended BNF (using * for repetition, + for at-least-once repetition, [] for optional parts, | for alternatives and () for grouping). It's not EBNF according to ISO/IEC 14977. It's a dialect Python uses in its own parser. N) r parserPrDr$r2r+r,_make_transitionrr-_calculate_tree_traversalr ) r\token_namespacerrZnfa_aZnfa_zr?Zreserved_stringsrZ dfa_stateZterminal_or_nonterminalr transitionrrrgenerate_grammars,    rdcCs|drt||S|ddvs*J||ds>|drBJt|}z ||WStyzt|}||<|YS0dS)z Creates a reserved string ("if", "for", "*", ...) or returns the token type (NUMBER, STRING, ...) for a given grammar terminal. r)"'z"""z'''N)isalphagetattr startswithrKeyErrorr )rarr0r;rrrrr_s    r_c Csi}t|}||D]}||vrt|||q|D]}|D]}|j}|jD]\}}||D]r\}} ||vr||} t| j r| j dj n| j j | r| dj n|j g} t d|j |ft | t|| ||<qnqZqFq>dS)z By this point we know how dfas can move around within a stack node, but we don't know how we can add a new stack node (nonterminal transitions). rzZRule %s is ambiguous; given a %s token, we can't determine if we should evaluate %s or %s.N)listkeyssort_calculate_first_plansvaluesr-r,r2sortedr r$r ValueErrortupler) r first_plansZ nonterminalsrZr?rbr-rrcpushesZ prev_planchoicesrrrr`.s>     r`c Cs||}i}d||<|d}|jD]\}}|jg||<q&|jD]h\}}z ||} Wntyxt|||} Yn0| durtd|| D]\} } |g| || <qqF|||<|S)z Calculates the first plan in the first_plans dictionary for every given nonterminal. This is going to be used to know when to create stack nodes. Nrzleft recursion for rule %r)r-r2rr,rjrorr) rrtrZr?Znew_first_plansrCrcr/Z nonterminal2Z first_plans2trurrrro`s"   roN)rastrtypingrrrrrrZparso.pgen2.grammar_parserr r r r rrr rDrPrYr[rrdr_r`rorrrrs    :2*2