josh/propane
1
0 Fork 0
Code Issues 3 Pull Requests Projects Releases Wiki Activity
212 Commits 6 Branches 26 Tags
3f92ae46c43875b09854cc07a54695c840d8eb65
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
josh 3f92ae46c4 Map rule components to Token/Rule references 2021-08-22 21:21:41 -04:00
josh 00016f16b3 Combine Grammar and Generator into top-level Imbecile class 2021-08-22 21:04:46 -04:00
josh 9273bfccf6 Move Token/Rule out of Grammar class 2021-08-19 20:00:40 -04:00
josh f295acb593 Generator builds a Lexer, not a Lexer::DFA 2021-08-19 13:11:12 -04:00
josh 51a31317a6 Move FA#build_tables to Lexer::DFA 2021-08-19 11:55:34 -04:00
josh 9459883e74 Add Lexer class; Move LexerDFA to Lexer::DFA 2021-08-18 17:09:45 -04:00
josh 28591907c1 Move FA class out of Regex class 2021-08-18 17:05:03 -04:00
josh 37d6917b49 Rework Rule constructor 2021-07-27 21:22:46 -04:00
josh 2685c05360 Change rule syntax 2021-07-19 21:55:08 -04:00
josh c0c3353fd7 Test lexing empty null string returns EOF 2021-07-06 12:06:07 -04:00
josh 3158e51059 Add length field to LexedToken 2021-07-06 11:59:35 -04:00
josh d9e4f64d2e Fix returning TOKEN_EOF when lexing at EOF 2021-07-06 11:55:44 -04:00
josh ec2dcf9a72 Fix not progressing through input while lexing a token 2021-07-06 11:47:33 -04:00
josh 578e165e2d Fix off-by-one error in state IDs 2021-07-06 11:44:03 -04:00
josh e8df4296cc Begin testing lexer 2021-07-06 11:09:39 -04:00
josh 230c324209 Fix iterating through all transitions in a state 2021-07-06 11:09:13 -04:00
josh 1271e19b50 Test multi-byte code point decoding 2021-07-06 11:02:43 -04:00
josh 12e11399af Add decoder tests 2021-07-06 10:57:06 -04:00
josh 24fab8515d Decoder.decode_code_point returns struct with code point and length together 2021-07-06 10:50:32 -04:00
josh 1dcdd87a28 Generate token constants and names to top-level parser class 2021-07-06 10:28:35 -04:00
josh 8aec7ec0de Lexer class can be used standalone 2021-07-06 10:15:07 -04:00
josh c96d55b031 Fix class name 2021-07-06 10:14:14 -04:00
josh ca7d4862f9 Run test executable; build with unit tests 2021-07-06 10:03:42 -04:00
josh 3c874ae4c1 Compile generated parser with a test file 2021-07-05 23:05:55 -04:00
josh 748c219625 Do not return dropped tokens from Lexer.lex_token() 2021-07-05 22:53:58 -04:00
josh 71ee7de9f9 Remove obsolete lex() and lex_token() methods 2021-07-05 22:49:50 -04:00
josh 2121acc87e Complete Lexer.lex_token() 2021-07-05 22:41:09 -04:00
josh f2563cf255 Work on Lexer.lex_token() 2021-07-05 22:02:27 -04:00
josh 24d12be3b9 Add TOKEN enum entries for EOF, decode error, drop, and none 2021-07-05 20:11:55 -04:00
josh 91d6ee25ea Add Lexer class 2021-07-05 19:13:41 -04:00
josh 2f1cb47bea Add Decoder class to decode code points 2021-07-05 18:47:10 -04:00
josh 651461c570 Start on decode_code_point() 2021-06-29 23:17:44 -04:00
josh 3ce54bd303 Start on lex()/lex_token() 2021-06-29 23:10:40 -04:00
josh 15454f926a Add TokenNames array 2021-06-29 22:54:24 -04:00
josh 4beb3d2016 Add some token constants 2021-06-27 23:09:42 -04:00
josh aae7bc188c Use unsigned literals 2021-06-26 18:11:20 -04:00
josh a716dedeb6 Start on test framework to compile and run generated parser 2021-06-26 16:17:24 -04:00
josh 93cb25df62 Do not generate token names for drop tokens 2021-06-26 16:16:18 -04:00
josh 61dd5bc5a0 Move imbecile_spec to lexer_dfa_spec 2021-06-26 16:01:49 -04:00
josh 10a8ef5eb4 Update generated lexer state and transition tables 2021-06-26 15:58:36 -04:00
josh 98584ce07a Add FA#build_tables 2021-06-24 15:06:10 -04:00
josh 2122ca02fe Start generating lexer states and transitions 2021-06-23 23:15:02 -04:00
josh 5881f13380 Generate enum of token identifiers 2021-06-23 22:22:45 -04:00
josh ebc1d8f001 Fix FA#to_s to show correct destination state 2021-06-23 22:21:53 -04:00
josh 5fecd5c6a2 Refactor into FA#enumerate 2021-06-22 22:01:39 -04:00
josh 5b688b090d Add some attr_readers 2021-06-21 22:52:27 -04:00
josh f77218801f Error if Start rule not found 2021-06-21 22:48:17 -04:00
josh 70118dd019 Check for duplicate token/rule names in Generator 2021-06-21 22:34:43 -04:00
josh d552f2a540 CLI: accept --log option 2021-06-19 12:06:02 -04:00
josh d2fac07249 Add Generator class 2021-06-15 16:51:36 -04:00
josh a34272dfd6 Add Grammar::Rule class 2021-06-14 22:49:43 -04:00
josh 9d05861819 Parse grammar input by multiline regex 2021-06-12 22:57:32 -04:00
josh 03035a25a5 Update spec task to accept an example pattern 2021-06-12 22:46:13 -04:00
josh db70f8b94d Add "drop" grammar keyword to drop patterns 2021-06-09 22:48:30 -04:00
josh f67dd62b20 Add \s to expand to whitespace characters 2021-06-09 22:37:00 -04:00
josh c6bac6d3a1 Rename TokenDFA -> LexerDFA 2021-06-08 13:54:46 -04:00
josh aa92970c31 Add some lexer tests 2021-06-07 22:21:52 -04:00
josh b8282e748e Start on a test lexer for lexer specs 2021-06-07 17:17:37 -04:00
josh 930ac56148 Do not accept 0-length tokens 2021-06-06 15:29:30 -04:00
josh 7f54778ba8 Rename Regex::DFA to TokenDFA 2021-06-06 15:18:21 -04:00
josh 701903def2 Token should build its own NFA 2021-06-06 14:09:28 -04:00
josh afea886ecb Add Grammar::Token class 2021-06-06 14:04:33 -04:00
josh 03b2e87186 Grammar takes in input string instead of file name 2021-06-06 10:09:53 -04:00
josh e4370cac62 Print accepting token in FA#to_s 2021-06-06 09:59:28 -04:00
josh ed3f599e25 Create common FA/State/Transition classes across NFA/DFA 2021-06-06 09:41:23 -04:00
josh 1228a76c55 Fix MultiplicityUnit#to_nfa again 2021-05-26 10:17:03 -04:00
josh 538e360cb3 Fix MultiplicityUnit#to_nfa 2021-05-25 16:59:22 -04:00
josh e7f8c3726c Fix NFA#to_s 2021-05-25 16:14:19 -04:00
josh b6e3a5c151 Record accepting token in DFA state 2021-05-25 16:00:25 -04:00
josh 35ef94dbd3 Print out DFA to test 2021-05-25 15:52:47 -04:00
josh 37e1252ded Continue building DFA 2021-05-25 15:44:23 -04:00
josh 214ece7d90 Add NFA::Transition, start on DFA construction 2021-05-23 21:41:50 -04:00
josh 8473df421a Add specs for CodePointRange 2021-05-23 20:41:40 -04:00
josh 3987f08cd7 Add CodePointRange class 2021-05-23 17:52:20 -04:00
josh 3a1650906e Show non-printable characters better in NFA#to_s 2021-05-21 14:39:02 -04:00
josh 952bffc33c Move DFA#nil_transition_states to NFA::State 2021-05-21 14:27:42 -04:00
josh f64f3683c6 Add NFA#to_s 2021-05-21 14:24:16 -04:00
josh 43f5caf449 Fix some NFA creation 2021-05-20 17:34:18 -04:00
josh f38a7456e9 Add DFA#nil_transition_states 2021-05-20 17:08:34 -04:00
josh c77c81bf25 Mark regex NFA end state as accepting the token 2021-05-18 16:34:26 -04:00
josh 7196a0605a Add DFA class 2021-05-18 16:31:16 -04:00
josh 24054461a2 Merge Regex::Parser into Regex, move Unit to its own file 2021-05-18 16:14:42 -04:00
josh 89a5976064 Make Regex::Parser build a NFA after parsing 2021-05-18 16:07:39 -04:00
josh d3df67be1e Update rake 2021-05-18 16:03:14 -04:00
josh 791340b292 Build NFA for each token pattern 2021-05-17 22:57:18 -04:00
josh cf8718b69c Allow token definition with no pattern 2021-05-17 22:40:23 -04:00
josh 39f164a7db Parse . in a regex 2021-05-17 17:20:56 -04:00
josh 70b3e56de2 Store all characters as ranges; add CharacterClassUnit#to_nfa 2021-05-14 13:52:03 -04:00
josh 2e8e72a1e8 Add CharacterClassUnit and use it instead of AlternatesUnit 2021-05-14 12:32:53 -04:00
josh ea27baa630 Add #to_nfa for other regex unit types 2021-05-13 15:57:09 -04:00
josh d8dd64d860 Add NFA class 
Start converting units to NFAs
 2021-05-13 00:01:12 -04:00
josh 54cefda186 Use Parser 2021-05-11 16:52:28 -04:00
josh 201a38fb51 Add Parser specs 2021-05-11 15:29:40 -04:00
josh 33f9d01883 Rename start/end to min/max for CharacterRangeUnit 2021-05-11 15:28:45 -04:00
josh 9b09625c8a Fix parsing - at beginning of negated character class 2021-05-11 14:57:16 -04:00
josh 6119d860bc Fix character class parsing into an AlternatesUnit 2021-05-11 14:57:01 -04:00
josh 611ebeeddd Fix max multiplicity count parsing 2021-05-11 11:37:46 -04:00
josh 449eec4982 Fix multiplicity count parsing 2021-05-11 11:33:10 -04:00
josh 8cd648fc8f Create spec file for Parser 2021-05-07 16:58:38 -04:00
josh 885ef6c151 Rename Regex::Unit -> Regex::Parser 2021-05-07 16:57:05 -04:00