Your IP : 216.73.216.213
B
Z��Z� � @ s| d dl mZ d dlZddlmZ ddlmZ ddlmZ ddlmZ ddlmZ dZ d Z
G d
d � d e �Z G d d
� d
e �Z
dS )� )�absolute_importN� )�Actions)�DFA)�Errors)�Machines)�Regexps� c @ s e Zd ZdZdZdZdd� ZdS )�Statez�
This class is used as part of a Plex.Lexicon specification to
introduce a user-defined state.
Constructor:
State(name, token_specifications)
Nc C s || _ || _d S )N)�name�tokens)�selfr r � r �G/opt/alt/python37/lib64/python3.7/site-packages/Cython/Plex/Lexicons.py�__init__% s zState.__init__)�__name__�
__module__� __qualname__�__doc__r r r r r r r r
s r
c @ s: e Zd ZdZdZdZd dd�Zdd� Zdd � Zd
d � Z dS )
�Lexicona}
Lexicon(specification) builds a lexical analyser from the given
|specification|. The specification consists of a list of
specification items. Each specification item may be either:
1) A token definition, which is a tuple:
(pattern, action)
The |pattern| is a regular axpression built using the
constructors defined in the Plex module.
The |action| is the action to be performed when this pattern
is recognised (see below).
2) A state definition:
State(name, tokens)
where |name| is a character string naming the state,
and |tokens| is a list of token definitions as
above. The meaning and usage of states is described
below.
Actions
-------
The |action| in a token specication may be one of three things:
1) A function, which is called as follows:
function(scanner, text)
where |scanner| is the relevant Scanner instance, and |text|
is the matched text. If the function returns anything
other than None, that value is returned as the value of the
token. If it returns None, scanning continues as if the IGNORE
action were specified (see below).
2) One of the following special actions:
IGNORE means that the recognised characters will be treated as
white space and ignored. Scanning will continue until
the next non-ignored token is recognised before returning.
TEXT causes the scanned text itself to be returned as the
value of the token.
3) Any other value, which is returned as the value of the token.
States
------
At any given time, the scanner is in one of a number of states.
Associated with each state is a set of possible tokens. When scanning,
only tokens associated with the current state are recognised.
There is a default state, whose name is the empty string. Token
definitions which are not inside any State definition belong to
the default state.
The initial state of the scanner is the default state. The state can
be changed in one of two ways:
1) Using Begin(state_name) as the action of a token.
2) Calling the begin(state_name) method of the Scanner.
To change back to the default state, use '' as the state name.
N� c C s� t |t�st�d��|r.ddlm} d}|� }t�� }|�d�} d}
x||D ]t} t | t �r�|�| j
�} xX| j D ] }
| � || |
|
� |
d7 }
qlW qJt | t
�r�| � || | |
� |
d7 }
qJt�|
d��qJW |r�|� }||| }|� }|r�|d@ r�|�d� |�|� tj||d@ dk�o|d �}|�r2|� }||| }|�rV|d
@ �rV|�d � |�|� |�r�|�d || � |�d
|| � |�d| � || _d S )Nz Scanner definition is not a listr )�timeg � z5Expected a token definition (tuple) or State instancez
============= NFA ===========
� )�debugr z
============= DFA ===========
zConstructing NFA : %5.2f
zConverting to DFA: %5.2f
zTOTAL : %5.2f
)�
isinstance�listr ZInvalidScannerZTimingr r ZMachineZnew_initial_stater
r r �add_token_to_machine�tuple� InvalidToken�write�dumpr Z
nfa_to_dfa�machine)r
Zspecificationsr Z debug_flagsZtimingsr Z
total_timeZtime1ZnfaZdefault_initial_state� token_number�specZuser_initial_state�tokenZtime2Ztime3ZdfaZtime4r r r r u sT
zLexicon.__init__c
C s� y�| � |�\}}t|tj�r"|}n4y
|j W n tk
rJ t�|�}Y n X t�|�}|�� }|j |||ddd� |j
|| d� W n6 t j k
r� } z| �
d|| f ��W d d } ~ X Y nX d S )Nr r )Z match_bolZnocase)�priorityzToken number %d: %s)�parse_token_definitionr r ZAction�__call__�AttributeError�Return�CallZ new_stateZ
build_machineZ
set_actionr Z PlexError� __class__)
r
r" Z
initial_state�
token_specr# �reZ action_spec�actionZ final_state�er r r r � s
z Lexicon.add_token_to_machinec C sP t |t�st�d��t|�dkr*t�d��|\}}t |tj�sHt�d��||fS )Nz Token definition is not a tupler z)Wrong number of items in token definitionz Pattern is not an RE instance)r r r r �lenr ZRE)r
r- �patternr/ r r r r' � s
z Lexicon.parse_token_definitionc C s | j �|�S )N)r" �get_initial_state)r
r r r r r3 � s zLexicon.get_initial_state)Nr N)
r r r r r" Ztablesr r r' r3 r r r r r * s F
.
r )�
__future__r �typesr r r r r r ZDUMP_NFAZDUMP_DFA�objectr
r r r r r �<module> s