|
|
b00f53050b
|
Clean up Decoder result handling
|
2023-03-06 20:32:01 -05:00 |
|
|
|
2e5083d823
|
Add explicit base types to the enums
|
2023-03-06 19:58:24 -05:00 |
|
|
|
c6ea4f83c2
|
Store parser values according to the rule/pattern type
|
2022-11-13 22:20:30 -05:00 |
|
|
|
8dc27686aa
|
Rename result_type to ptype, parser result to pvalue
|
2022-11-12 13:29:10 -05:00 |
|
|
|
bca0a14371
|
Allow storing a result value for a token from a lexer code block
|
2022-10-16 21:40:25 -04:00 |
|
|
|
ca8a360c0e
|
Provide matched text to lexer user code block
|
2022-10-15 13:37:34 -04:00 |
|
|
|
623c644e74
|
Work on real D strings instead of ubyte pointer and length
Also fix a couple UTF-8 decoder bugs!
|
2022-10-15 13:32:33 -04:00 |
|
|
|
727c8cd1ea
|
Execute rule user code blocks when reducing the rule
|
2022-10-13 04:56:50 -04:00 |
|
|
|
74beaf7ed8
|
Store final parse result and give access with .result parser property
|
2022-10-12 21:33:09 -04:00 |
|
|
|
31970522de
|
Store parse result; add result_type grammar keyword
|
2022-10-12 20:56:14 -04:00 |
|
|
|
02be6de48e
|
Add lexer modes and $mode() code expansion
|
2022-10-09 22:49:01 -04:00 |
|
|
|
6bd9d4a09b
|
Remove _TOKEN_NONE and use _TOKEN_COUNT instead
|
2022-10-03 21:40:34 -04:00 |
|
|
|
ced5a56424
|
Support global user code blocks
|
2022-10-03 21:24:13 -04:00 |
|
|
|
66d654b6b9
|
Add $token() user code block expansion
|
2022-10-02 10:43:47 -04:00 |
|
|
|
f46b5b3f4d
|
Remove TOKEN_EOF; define EOF token and start rule in Generator
|
2022-10-02 10:07:44 -04:00 |
|
|
|
e7e30c4f28
|
Add pattern statement
|
2022-09-30 21:05:18 -04:00 |
|
|
|
a4c5546876
|
Disable parser debug output
|
2022-09-27 12:33:23 -04:00 |
|
|
|
48704295bb
|
Rename lexer state "accepts" to "token"
|
2022-09-25 14:57:46 -04:00 |
|
|
|
4d716f6c10
|
Remove Token#c_name; use given token case in token constants
|
2022-09-25 14:48:49 -04:00 |
|
|
|
01c9340819
|
Rename lexer variables to reflect trying to get longest match
|
2022-09-25 14:44:44 -04:00 |
|
|
|
672098ad32
|
Execute user code blocks assigned to tokens
|
2022-09-24 17:31:40 -04:00 |
|
|
|
2fbe13e071
|
Do not consume lookahead token when reducing
|
2022-06-25 21:35:54 -04:00 |
|
|
|
f2cc5b112e
|
Handle shifting states after reducing
|
2022-06-25 16:16:20 -04:00 |
|
|
|
84c4a16ce6
|
Start on Parser.parse()
|
2022-06-21 23:03:00 -04:00 |
|
|
|
f17efe8c82
|
Add RuleSet#id to use when reducing
Parser will know what state to go to after reducing a Rule based on the
RuleSet ID.
Start on Parser class.
|
2022-06-21 20:07:27 -04:00 |
|
|
|
ca6a93a4c5
|
Move generation logic to new Generator class
|
2022-06-05 14:57:58 -04:00 |
|
|
|
f37801ec9e
|
Store tokens and drop tokens separately
|
2022-06-05 14:36:19 -04:00 |
|
|
|
6f1ce32775
|
Move Lexer::DFA#build_tables to Lexer
|
2022-06-05 14:18:35 -04:00 |
|
|
|
7598c589fe
|
Detect other invalid UTF-8 encodings
|
2022-05-31 22:26:09 -04:00 |
|
|
|
f3ed678fe1
|
Store tokens in Hash by name
|
2021-09-27 21:40:12 -04:00 |
|
|
|
00016f16b3
|
Combine Grammar and Generator into top-level Imbecile class
|
2021-08-22 21:04:46 -04:00 |
|
|
|
f295acb593
|
Generator builds a Lexer, not a Lexer::DFA
|
2021-08-19 13:11:12 -04:00 |
|
|
|
3158e51059
|
Add length field to LexedToken
|
2021-07-06 11:59:35 -04:00 |
|
|
|
d9e4f64d2e
|
Fix returning TOKEN_EOF when lexing at EOF
|
2021-07-06 11:55:44 -04:00 |
|
|
|
ec2dcf9a72
|
Fix not progressing through input while lexing a token
|
2021-07-06 11:47:33 -04:00 |
|
|
|
230c324209
|
Fix iterating through all transitions in a state
|
2021-07-06 11:09:13 -04:00 |
|
|
|
24fab8515d
|
Decoder.decode_code_point returns struct with code point and length together
|
2021-07-06 10:50:32 -04:00 |
|
|
|
1dcdd87a28
|
Generate token constants and names to top-level parser class
|
2021-07-06 10:28:35 -04:00 |
|
|
|
8aec7ec0de
|
Lexer class can be used standalone
|
2021-07-06 10:15:07 -04:00 |
|
|
|
748c219625
|
Do not return dropped tokens from Lexer.lex_token()
|
2021-07-05 22:53:58 -04:00 |
|
|
|
71ee7de9f9
|
Remove obsolete lex() and lex_token() methods
|
2021-07-05 22:49:50 -04:00 |
|
|
|
2121acc87e
|
Complete Lexer.lex_token()
|
2021-07-05 22:41:09 -04:00 |
|
|
|
f2563cf255
|
Work on Lexer.lex_token()
|
2021-07-05 22:02:27 -04:00 |
|
|
|
24d12be3b9
|
Add TOKEN enum entries for EOF, decode error, drop, and none
|
2021-07-05 20:11:55 -04:00 |
|
|
|
91d6ee25ea
|
Add Lexer class
|
2021-07-05 19:13:41 -04:00 |
|
|
|
2f1cb47bea
|
Add Decoder class to decode code points
|
2021-07-05 18:47:10 -04:00 |
|
|
|
651461c570
|
Start on decode_code_point()
|
2021-06-29 23:17:44 -04:00 |
|
|
|
3ce54bd303
|
Start on lex()/lex_token()
|
2021-06-29 23:10:40 -04:00 |
|
|
|
15454f926a
|
Add TokenNames array
|
2021-06-29 22:54:24 -04:00 |
|
|
|
aae7bc188c
|
Use unsigned literals
|
2021-06-26 18:11:20 -04:00 |
|
