diff --git a/doc/user_guide.md b/doc/user_guide.md index 973f0ed..56c374c 100644 --- a/doc/user_guide.md +++ b/doc/user_guide.md @@ -52,6 +52,23 @@ The generated source code will be written to the output file. If a log file path is specified, Propane will write a log file containing detailed information about the parser states and transitions. +##> Target language selection + +Propane determines the target output language from the extension of the output +file name given on the command line: + + * `.c` selects the C target. + A header file with a `.h` extension is generated alongside the + implementation file. + * `.cc`, `.cpp`, or `.cxx` selects the C++ target. + A header file with a `.h` extension is generated alongside the + implementation file. + * `.d` selects the D target. + * `.rs` selects the Rust target. + +Propane reports an error if the target language cannot be determined from the +output file name. + #> Propane Grammar File A Propane grammar file provides Propane with the patterns, tokens, grammar @@ -129,6 +146,14 @@ import std.stdio; >> ``` +Rust example: + +``` +<< +use std::collections::HashMap; +>> +``` + Standalone code blocks are emitted early in the output file as top-level code outside the context of any function. Standalone code blocks are a good place to include/import any other necessary @@ -219,6 +244,32 @@ token integer /\d+/ << >> ``` +#### Rust lexer code block arguments + +The lexer code block is passed the following arguments: + + * `match_` (`&[u8]`) - a slice containing the text matched by the lexer pattern. + * `match_length` (`usize`) - length of the matched text. + +The argument is named `match_` rather than `match` because `match` is a Rust +keyword. +It is a byte slice rather than a string; use `std::str::from_utf8()` or +`String::from_utf8_lossy()` to view the matched text as a string. + +``` +ptype i64; + +token integer /\d+/ << + let mut v: i64 = 0; + for c in match_ + { + v *= 10; + v += (c - b'0') as i64; + } + $$ = v; +>> +``` + ### Parser rule code blocks Example: @@ -267,8 +318,9 @@ positions. In this case the position should be checked for validity before its `row` and `col` fields are used (see `${#p_position_valid}`). For C targets this can be accomplished with -`if (p_position_valid(${$.position}))` and for D targets this can be -accomplished with `if (${$.position}.valid)`. +`if (p_position_valid(${$.position}))`, for D targets with +`if (${$.position}.valid)`, and for Rust targets with +`if ${$.position}.valid()`. In tree generation mode, a full parse tree is automatically constructed in memory for user code to traverse after parsing is complete. @@ -304,6 +356,11 @@ accessors on a node handle: C-style functions and macros above are also available. * D: `@property` accessors, e.g. `node.field`, `node.valid`, `node.position`, `node.token`, `node.pvalue`. + * Rust: handle methods called with `()`, e.g. `node.field()`, `node.valid()`, + `node.position()`, `node.end_position()`, `node.n_fields()`, + `node.token()`, `node.pvalue()`, `node.data()` (a reference to the node + record, for token user fields), and `node.node_id()` (for identity + comparison). The positional position expansions (`${$.position}`, `${N.position}`, etc...) are not available in tree generation mode; use the position accessors above @@ -324,6 +381,21 @@ Assignment -> ident equals Expr << >> ``` +Rust example: + +``` +tree; + +Assignment -> ident equals Expr << + /* $$ is the Assignment tree node, $1 is the ident Token node, and $3 is + * the Expr rule node. */ + println!("assignment on row {}, col {}", + $$.position().row, $$.position().col); + println!("target identifier ends on row {}, col {}", + $1.end_position().row, $1.end_position().col); +>> +``` + ##> `context_user_fields` statement - adding custom fields to the context Propane uses a context structure for lexer and parser operations. @@ -355,6 +427,28 @@ drop /#(.*)\n/ << >> ``` +Rust example: + +``` +context_user_fields << + pub comments: String, +>> +drop /#(.*)\n/ << + /* Accumulate comments before the next parser tree node. */ + ${context.comments} += std::str::from_utf8(match_).unwrap(); +>> +``` + +For the Rust target, the code block contents are inserted directly into the +generated `p_context_t` struct definition. +Each field must therefore be written as a Rust struct field, terminated with a +comma. +The generated `p_context_t` derives `Default`, so each user context field type +must implement `Default`. +Mark a field `pub` if code outside of the generated module needs to access it; +grammar user code blocks are emitted into the generated module itself and can +access a field regardless. + If a pointer to any allocated memory is stored in a user-defined context field, it is up to the user to free any memory when the program is finished using the context structure. @@ -416,9 +510,39 @@ free_token_node << >> ``` +Example freeing `pvalue` (Rust): + +``` +tree; +free_token_node << + if !${token.pvalue}.is_null() + { + unsafe { drop(Box::from_raw(${token.pvalue})); } + } +>> +ptype *mut i32; +token a << + $$ = Box::into_raw(Box::new(1)); +>> +token b << + $$ = Box::into_raw(Box::new(2)); +>> +Start -> a:a b:b; +``` + The `free_token_node` statement user code block is not emitted for D language since D has a garbage collector. +The code block is emitted for the Rust target, where it runs from +`p_context_delete()`. +A `ptype` or token user field which owns its memory (a `String`, a `Vec`, a +`Box`, and so on) is released when the context is dropped and does not need a +`free_token_node` code block. +The statement is only needed for memory which Rust does not track, such as a +raw pointer obtained from `Box::into_raw()`. +Note that the generated `p_context_t` does not implement `Drop`, so a +`free_token_node` code block only runs if `p_context_delete()` is called. + ##> `lex_fn` statement - specifying a custom lexer function Propane generates both a lexer and a parser. @@ -460,6 +584,38 @@ static size_t mylexfn(p_context_t * context, p_token_info_t * out_token_info) lex_fn mylexfn; ``` +Example (Rust): + +``` +<< +fn mylexfn(context: &mut p_context_t, out_token_info: &mut p_token_info_t) -> usize +{ + let mut result = P_SUCCESS; + if context.count > 0 + { + out_token_info.token = TOKEN_a; + out_token_info.pvalue = p_value(context.count); + context.count -= 1; + } + else + { + result = p_lex(context, out_token_info); + if out_token_info.token == TOKEN_c + { + context.count = 3; + } + } + result +} +>> + +context_user_fields << + count: usize, +>> + +lex_fn mylexfn; +``` + The `lex_fn` statement takes one argument specifying the name of the custom lexer function. The user must supply a value for the `token` field of the `p_token_info_t` @@ -542,6 +698,11 @@ generated output without any surrounding `#line` directives. This can be useful when debugging the generated parser itself, or when the `#line` directives interfere with other tooling. +The `noline` statement only affects the C, C++, and D targets. +Rust has no `#line` directive equivalent, so `#line` directives are never +emitted into Rust output and the `noline` statement has no effect for the Rust +target. + ##> `on_tree_node` statement - custom initialization of a token tree node The `on_token_node` statement can be used to provide code that initializes @@ -566,6 +727,24 @@ drop /#(.*)\n/ << >> ``` +For example (Rust): + +``` +context_user_fields << + pub comments: String, +>> +token_user_fields << + pub comments: String, +>> +on_token_node << + ${token.comments} = std::mem::take(&mut ${context.comments}); +>> +drop /#(.*)\n/ << + /* Accumulate comments before the next parser tree node. */ + ${context.comments} += std::str::from_utf8(match_).unwrap(); +>> +``` + ##> `prefix` statement - specifying the generated API prefix By default the public API (types, constants, and functions) of the generated @@ -645,6 +824,30 @@ Lexer user code blocks may assign a parse value to the generated `Token` node by assigning to `$$` within a lexer code block. The type of the parse value `$$` is given by the global `ptype` type. +### Rust `ptype` requirements + +For the Rust target, every `ptype` type must implement both `Clone` and +`Default`. +When tree generation mode is not active, the generated `p_value_t` is an enum +with one variant per defined `ptype`. +The generated accessor functions clone the held value, and return +`Default::default()` when the variant currently held is not the one requested. + +A user-defined `ptype` type therefore normally needs a +`#[derive(Clone, Default)]`: + +``` +<< +#[derive(Clone, Default)] +pub struct Value +{ + pub n: i64, +} +>> + +ptype Value; +``` + ##> `start` statement - specifying the parser start rule name The start rule can be changed from the default of `Start` by using the `start` @@ -754,6 +957,19 @@ token_user_fields << >> ``` +Example (Rust): + +``` +token_user_fields << + pub mytokenval: String, +>> +``` + +For the Rust target, the code block contents are inserted directly into the +generated token tree node struct, so each field must be written as a Rust +struct field terminated with a comma. +Each user token field type must implement `Clone` and `Default`. + The `on_token_node` statement can be used to provide code that initializes any token user fields when a token tree node instance is created. @@ -781,6 +997,9 @@ the `free_token_node` statement can be used to supply a code block which will be executed immediately before the token node is freed. For C++, the `delete` statement is used to free the token tree node, so the destructor for any custom token user fields will be called. +For Rust, a token user field which owns its memory is dropped along with the +context, so a `free_token_node` code block is only needed for memory which Rust +does not track. ##> `tree` statement - tree generation mode @@ -874,6 +1093,28 @@ assert_eq(22, itemsmore.item.pToken1.pvalue); assert(itemsmore.pItemsMore is null); ``` +The equivalent traversal for a Rust target, where tree node fields are accessor +methods and an absent child is a handle whose `valid()` method returns `false`: + +``` +let mut context = p_context_new(b"a, ((b)), b"); +assert_eq!(P_SUCCESS, p_parse(&mut context)); +let start = p_result(&context); +let items = start.pItems(); +assert!(items.valid()); +assert!(items.item().valid()); +assert_eq!(TOKEN_a, items.item().pToken1().token()); +assert_eq!(11, items.item().pToken1().pvalue()); +let mut itemsmore = items.pItemsMore(); +assert!(itemsmore.valid()); +assert_eq!(TOKEN_b, itemsmore.item().item().item().pToken1().token()); +assert_eq!(22, itemsmore.item().item().item().pToken1().pvalue()); +itemsmore = itemsmore.pItemsMore(); +assert_eq!(TOKEN_b, itemsmore.item().pToken1().token()); +assert!(!itemsmore.pItemsMore().valid()); +p_context_delete(context); +``` + ## `tree_prefix` and `tree_suffix` statements In tree generation mode, structure types are defined and named based on the @@ -1212,8 +1453,8 @@ lexing based on the grammar's lexer patterns. Propane defines a `p_context_t` structure type. The structure is intended to be used opaquely and stores information related to the state of the lexer and parser. -Integrating code must define an instance of the `p_context_t` structure. -A pointer to this instance is passed to the generated functions. +A `p_context_t` instance is allocated and initialied with the `p_context_new()` +function. ### `p_position_t` @@ -1227,6 +1468,9 @@ For C targets, the `p_position_t` structure can be checked for validity by calling `p_position_valid(pos)` where `pos` is a `p_position_t` structure instance. +For Rust targets, the `p_position_t` structure can be checked for validity by +calling its `valid()` method (e.g. `if pos.valid()`). + ### `p_value_t` If tree generation mode is enabled, the `p_value_t` type is defined to be the @@ -1238,6 +1482,13 @@ In this case, the API functions `p_value()` and `p_value_XXX()` for each given `ptype` name `XXX` are generated to return `p_value_t` instances holding the corresponding `ptype`. +For Rust targets, `p_value_t` is an enum rather than a union, and every `ptype` +type must implement `Clone` and `Default` (see +${#Rust ptype requirements}). +Reading a `p_value_t` with an accessor for a `ptype` other than the one it +currently holds returns `Default::default()` rather than reinterpreting the +stored bytes. + ### `p_token_info_t` The `p_token_info_t` structure contains the following fields: @@ -1251,6 +1502,9 @@ The `p_token_info_t` structure contains the following fields: for the default value or `p_value_get_XXX(&token_info.pvalue)` for named `ptype` values. +For Rust targets, `p_token_info_t` implements `Default`, so a token info +structure to pass to `p_lex()` can be created with `p_token_info_t::default()`. + ### Tree Node Types If tree generation mode is enabled, a structure type for each rule will be @@ -1271,8 +1525,9 @@ A rule node may not have valid positions if the rule allows for an empty match. In this case the `position` structure should be checked for validity before using it. For C targets this can be accomplished with -`if (p_position_valid(node->position))` and for D targets this can be -accomplished with `if (node.position.valid)`. +`if (p_position_valid(node->position))`, for D targets with +`if (node.position.valid)`, and for Rust targets with +`if node.position().valid()`. A `Token` node has the following additional fields: @@ -1297,6 +1552,12 @@ the same name but with a positional suffix (`pItems1`) which both point to the parsed `Items` node. Their value will be null if the parsed `Items` rule was empty. +For Rust targets each of these generated fields is an accessor method returning +a tree node handle, so the fields described in this section are read as +`node.pItems()`, `node.pItems1()`, and so on. +A handle for an absent child is not null; instead its `valid()` method returns +`false`. + The `Items` structure will have fields: * `pItem` and `pItem1` which point to the parsed `Item` structure. @@ -1359,11 +1620,32 @@ D example: p_context_t * context = p_context_new(input); ``` +Rust example: + +``` +let mut context = p_context_new(b"a = 1"); +``` + +For Rust targets, `p_context_new()` accepts the input as a `&[u8]` byte slice +and copies it into the returned context. + ### `p_context_delete` The `p_context_delete()` function must be called to deinitialize and deallocate a context structure allocated by `p_context_init()`. +For Rust targets, `p_context_delete()` takes the context by value and consumes +it. +The memory owned by the context is released when the context is dropped, so the +call is only strictly required when the grammar supplies a `free_token_node` +code block, which runs from `p_context_delete()`. + +Rust example: + +``` +p_context_delete(context); +``` + ### `p_lex` The `p_lex()` function is the main entry point to the lexer. @@ -1401,6 +1683,30 @@ case P_SUCCESS: } ``` +Rust example: + +``` +let mut context = p_context_new(input); +let mut token_info = p_token_info_t::default(); +loop +{ + match p_lex(&mut context, &mut token_info) + { + P_SUCCESS => + { + if token_info.token == TOKEN___EOF + { + break; + } + println!("{} ({} bytes)", p_token_names[token_info.token as usize], + token_info.length); + } + /* P_DECODE_ERROR, P_UNEXPECTED_INPUT, or P_USER_TERMINATED. */ + _ => break, + } +} +``` + ### `p_parse` The `p_parse()` function is the main entry point to the parser. @@ -1413,6 +1719,13 @@ p_context_t * context = p_context_new(input, input_length); size_t result = p_parse(context); ``` +Rust example: + +``` +let mut context = p_context_new(input); +let result = p_parse(&mut context); +``` + When multiple start rules are specified, a separate parse function is generated for each which starts parsing at the given rule. For example, if `Statement` is specified as a start rule: @@ -1453,6 +1766,20 @@ size_t p_parse_inner_Statement(p_context_t * context, Passing `null` for the slice makes the function behave identically to `p_parse_Statement()`. +For Rust targets, the signature accepts a slice: + +``` +pub fn p_parse_inner_Statement(context: &mut p_context_t, + follow_tokens: &[p_token_t]) -> usize; +``` + +Passing an empty slice (`&[]`) makes the function behave identically to +`p_parse_Statement()`: + +``` +let result = p_parse_inner_Statement(&mut context, &[TOKEN_rbrace]); +``` + When the parse is completed via a non-EOF follow token, that follow token is **not** consumed from the input stream. The parser rewinds the input index and text position to the start of the follow @@ -1477,6 +1804,10 @@ For D targets, rather than using `p_position_valid()`, the `valid` property function of the `p_position_t` structure can be queried (e.g. `if (node.position.valid)`). +For Rust targets, rather than using `p_position_valid()`, the `valid()` method +of the `p_position_t` structure can be called +(e.g. `if node.position().valid()`). + ### `p_result` The `p_result()` function can be used to retrieve the final parse value after @@ -1512,6 +1843,20 @@ if (p_parse_Statement(context) == P_SUCCESS) In this case, the parser will start parsing with the `Statement` rule and the parse result from the `Statement` rule will be returned. +Rust example: + +``` +let mut context = p_context_new(input); +if p_parse(&mut context) == P_SUCCESS +{ + let result = p_result(&context); +} +``` + +For Rust targets in tree generation mode, `p_result()` returns a `Start` tree +node handle which borrows the context, so the handle must be dropped before the +context is passed to `p_context_delete()`. + ### `p_position` The `p_position()` function can be used to retrieve the parser position where @@ -1530,6 +1875,18 @@ if (p_parse(context) == P_UNEXPECTED_TOKEN) } ``` +Rust example: + +``` +let mut context = p_context_new(input); +if p_parse(&mut context) == P_UNEXPECTED_TOKEN +{ + let error_position = p_position(&context); + eprintln!("Error: unexpected token at row {} column {}", + error_position.row, error_position.col); +} +``` + ### `p_set_position` The `p_set_position()` function sets the current text position stored in the @@ -1549,6 +1906,14 @@ p_set_position(context, start); p_parse(context); ``` +Rust example: + +``` +let mut context = p_context_new(input); +p_set_position(&mut context, p_position_t { row: 5, col: 20 }); +p_parse(&mut context); +``` + ### `p_input_index` The `p_input_index()` function returns the current input text byte offset, @@ -1565,6 +1930,15 @@ size_t offset = p_input_index(context); /* Remaining input starts at `input + offset`. */ ``` +Rust example: + +``` +let mut context = p_context_new(input); +p_parse_inner_Statement(&mut context, follow_tokens); +let offset = p_input_index(&context); +/* Remaining input starts at `&input[offset..]`. */ +``` + ### `p_set_input_index` The `p_set_input_index()` function sets the current input text byte offset, @@ -1587,6 +1961,17 @@ p_set_input_index(context, saved_index); p_set_position(context, saved_position); ``` +Rust example: + +``` +/* Save the cursor and text position at the start of a section. */ +let saved_index = p_input_index(&context); +let saved_position = p_position(&context); +/* ... later, rewind to re-read that section. */ +p_set_input_index(&mut context, saved_index); +p_set_position(&mut context, saved_position); +``` + ### `p_user_terminate_code` The `p_user_terminate_code()` function can be used to retrieve the user @@ -1604,6 +1989,15 @@ if (p_parse(context) == P_USER_TERMINATED) } ``` +Rust example: + +``` +if p_parse(&mut context) == P_USER_TERMINATED +{ + let user_terminate_code = p_user_terminate_code(&context); +} +``` + ### `p_token` The `p_token()` function can be used to retrieve the current parse token. @@ -1620,6 +2014,15 @@ if (p_parse(context) == P_UNEXPECTED_TOKEN) } ``` +Rust example: + +``` +if p_parse(&mut context) == P_UNEXPECTED_TOKEN +{ + let unexpected_token = p_token(&context); +} +``` + ### `p_decode_code_point` The `p_decode_code_point()` function can be used to decode code points from a @@ -1640,6 +2043,19 @@ assert(code_point == 0x1F9E1u); assert(code_point_length == 4u); ``` +Rust Example: + +``` +let mut code_point: p_code_point_t = 0; +let mut code_point_length: u8 = 0; + +let result = p_decode_code_point(b"\xf0\x9f\xa7\xa1", + &mut code_point, &mut code_point_length); +assert_eq!(P_SUCCESS, result); +assert_eq!(0x1F9E1, code_point); +assert_eq!(4, code_point_length); +``` + ### `p_tree_delete` The `p_tree_delete()` function can be used to free the memory used by the tree. @@ -1678,6 +2094,12 @@ They are especially useful when tree generation mode is not active. In that case, the `p_value_t` union can hold one of several different possible value types. +Rust example: + +``` +out_token_info.pvalue = p_value(count); +``` + ### `p_value_get` The `p_value_get()` accessor functions can be used to extract a user value @@ -1694,6 +2116,16 @@ These functions are the counterpart to `p_value()` constructor functions and are useful for reading the parser value associated with a lexed token, for example `p_value_get(&token_info.pvalue)`. +Rust example: + +``` +let value = p_value_get(&token_info.pvalue); +``` + +For Rust targets these accessors return a clone of the held value. +If the `p_value_t` holds a different `ptype` than the one requested, the +accessor returns `Default::default()`. + ##> Data ### `p_token_names` @@ -1715,6 +2147,19 @@ if (p_parse(context) == P_UNEXPECTED_TOKEN) } ``` +Rust example: + +``` +let mut context = p_context_new(input); +if p_parse(&mut context) == P_UNEXPECTED_TOKEN +{ + let error_position = p_position(&context); + eprintln!("Error: unexpected token `{}' at row {} column {}", + p_token_names[p_token(&context) as usize], + error_position.row, error_position.col); +} +``` + #> License Propane is licensed under the terms of the MIT License: