josh/propane
1
0
Fork 0
Code Issues 3 Pull Requests Projects Releases Wiki Activity

Update user guide with Rust documentation and examples

Browse Source
This commit is contained in:
Josh Holtrop 2026-08-19 21:20:19 -04:00
parent 71abdcb3ac
commit e22ebc87d3
1 changed files with 451 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

457
doc/user_guide.md
View File

@ -52,6 +52,23 @@ The generated source code will be written to the output file.
If a log file path is specified, Propane will write a log file containing
detailed information about the parser states and transitions.
##> Target language selection
Propane determines the target output language from the extension of the output
file name given on the command line:
* `.c` selects the C target.
A header file with a `.h` extension is generated alongside the
implementation file.
* `.cc`, `.cpp`, or `.cxx` selects the C++ target.
A header file with a `.h` extension is generated alongside the
implementation file.
* `.d` selects the D target.
* `.rs` selects the Rust target.
Propane reports an error if the target language cannot be determined from the
output file name.
#> Propane Grammar File
A Propane grammar file provides Propane with the patterns, tokens, grammar
@ -129,6 +146,14 @@ import std.stdio;
>>
```
Rust example:
```
<<
use std::collections::HashMap;
>>
```
Standalone code blocks are emitted early in the output file as top-level code
outside the context of any function.
Standalone code blocks are a good place to include/import any other necessary
@ -219,6 +244,32 @@ token integer /\d+/ <<
>>
```
#### Rust lexer code block arguments
The lexer code block is passed the following arguments:
* `match_` (`&[u8]`) - a slice containing the text matched by the lexer pattern.
* `match_length` (`usize`) - length of the matched text.
The argument is named `match_` rather than `match` because `match` is a Rust
keyword.
It is a byte slice rather than a string; use `std::str::from_utf8()` or
`String::from_utf8_lossy()` to view the matched text as a string.
```
ptype i64;
token integer /\d+/ <<
let mut v: i64 = 0;
for c in match_
{
v *= 10;
v += (c - b'0') as i64;
}
$$ = v;
>>
```
### Parser rule code blocks
Example:
@ -267,8 +318,9 @@ positions.
In this case the position should be checked for validity before its `row` and
`col` fields are used (see `${#p_position_valid}`).
For C targets this can be accomplished with
`if (p_position_valid(${$.position}))` and for D targets this can be
accomplished with `if (${$.position}.valid)`.
`if (p_position_valid(${$.position}))`, for D targets with
`if (${$.position}.valid)`, and for Rust targets with
`if ${$.position}.valid()`.
In tree generation mode, a full parse tree is automatically constructed in
memory for user code to traverse after parsing is complete.
@ -304,6 +356,11 @@ accessors on a node handle:
C-style functions and macros above are also available.
* D: `@property` accessors, e.g. `node.field`, `node.valid`, `node.position`,
`node.token`, `node.pvalue`.
* Rust: handle methods called with `()`, e.g. `node.field()`, `node.valid()`,
`node.position()`, `node.end_position()`, `node.n_fields()`,
`node.token()`, `node.pvalue()`, `node.data()` (a reference to the node
record, for token user fields), and `node.node_id()` (for identity
comparison).
The positional position expansions (`${$.position}`, `${N.position}`, etc...)
are not available in tree generation mode; use the position accessors above
@ -324,6 +381,21 @@ Assignment -> ident equals Expr <<
>>
```
Rust example:
```
tree;
Assignment -> ident equals Expr <<
/* $$ is the Assignment tree node, $1 is the ident Token node, and $3 is
* the Expr rule node. */
println!("assignment on row {}, col {}",
$$.position().row, $$.position().col);
println!("target identifier ends on row {}, col {}",
$1.end_position().row, $1.end_position().col);
>>
```
##> `context_user_fields` statement - adding custom fields to the context
Propane uses a context structure for lexer and parser operations.
@ -355,6 +427,28 @@ drop /#(.*)\n/ <<
>>
```
Rust example:
```
context_user_fields <<
pub comments: String,
>>
drop /#(.*)\n/ <<
/* Accumulate comments before the next parser tree node. */
${context.comments} += std::str::from_utf8(match_).unwrap();
>>
```
For the Rust target, the code block contents are inserted directly into the
generated `p_context_t` struct definition.
Each field must therefore be written as a Rust struct field, terminated with a
comma.
The generated `p_context_t` derives `Default`, so each user context field type
must implement `Default`.
Mark a field `pub` if code outside of the generated module needs to access it;
grammar user code blocks are emitted into the generated module itself and can
access a field regardless.
If a pointer to any allocated memory is stored in a user-defined context field,
it is up to the user to free any memory when the program is finished using the
context structure.
@ -416,9 +510,39 @@ free_token_node <<
>>
```
Example freeing `pvalue` (Rust):
```
tree;
free_token_node <<
if !${token.pvalue}.is_null()
{
unsafe { drop(Box::from_raw(${token.pvalue})); }
}
>>
ptype *mut i32;
token a <<
$$ = Box::into_raw(Box::new(1));
>>
token b <<
$$ = Box::into_raw(Box::new(2));
>>
Start -> a:a b:b;
```
The `free_token_node` statement user code block is not emitted for D language
since D has a garbage collector.
The code block is emitted for the Rust target, where it runs from
`p_context_delete()`.
A `ptype` or token user field which owns its memory (a `String`, a `Vec`, a
`Box`, and so on) is released when the context is dropped and does not need a
`free_token_node` code block.
The statement is only needed for memory which Rust does not track, such as a
raw pointer obtained from `Box::into_raw()`.
Note that the generated `p_context_t` does not implement `Drop`, so a
`free_token_node` code block only runs if `p_context_delete()` is called.
##> `lex_fn` statement - specifying a custom lexer function
Propane generates both a lexer and a parser.
@ -460,6 +584,38 @@ static size_t mylexfn(p_context_t * context, p_token_info_t * out_token_info)
lex_fn mylexfn;
```
Example (Rust):
```
<<
fn mylexfn(context: &mut p_context_t, out_token_info: &mut p_token_info_t) -> usize
{
let mut result = P_SUCCESS;
if context.count > 0
{
out_token_info.token = TOKEN_a;
out_token_info.pvalue = p_value(context.count);
context.count -= 1;
}
else
{
result = p_lex(context, out_token_info);
if out_token_info.token == TOKEN_c
{
context.count = 3;
}
}
result
}
>>
context_user_fields <<
count: usize,
>>
lex_fn mylexfn;
```
The `lex_fn` statement takes one argument specifying the name of the custom
lexer function.
The user must supply a value for the `token` field of the `p_token_info_t`
@ -542,6 +698,11 @@ generated output without any surrounding `#line` directives.
This can be useful when debugging the generated parser itself, or when the
`#line` directives interfere with other tooling.
The `noline` statement only affects the C, C++, and D targets.
Rust has no `#line` directive equivalent, so `#line` directives are never
emitted into Rust output and the `noline` statement has no effect for the Rust
target.
##> `on_tree_node` statement - custom initialization of a token tree node
The `on_token_node` statement can be used to provide code that initializes
@ -566,6 +727,24 @@ drop /#(.*)\n/ <<
>>
```
For example (Rust):
```
context_user_fields <<
pub comments: String,
>>
token_user_fields <<
pub comments: String,
>>
on_token_node <<
${token.comments} = std::mem::take(&mut ${context.comments});
>>
drop /#(.*)\n/ <<
/* Accumulate comments before the next parser tree node. */
${context.comments} += std::str::from_utf8(match_).unwrap();
>>
```
##> `prefix` statement - specifying the generated API prefix
By default the public API (types, constants, and functions) of the generated
@ -645,6 +824,30 @@ Lexer user code blocks may assign a parse value to the generated `Token` node
by assigning to `$$` within a lexer code block.
The type of the parse value `$$` is given by the global `ptype` type.
### Rust `ptype` requirements
For the Rust target, every `ptype` type must implement both `Clone` and
`Default`.
When tree generation mode is not active, the generated `p_value_t` is an enum
with one variant per defined `ptype`.
The generated accessor functions clone the held value, and return
`Default::default()` when the variant currently held is not the one requested.
A user-defined `ptype` type therefore normally needs a
`#[derive(Clone, Default)]`:
```
<<
#[derive(Clone, Default)]
pub struct Value
{
pub n: i64,
}
>>
ptype Value;
```
##> `start` statement - specifying the parser start rule name
The start rule can be changed from the default of `Start` by using the `start`
@ -754,6 +957,19 @@ token_user_fields <<
>>
```
Example (Rust):
```
token_user_fields <<
pub mytokenval: String,
>>
```
For the Rust target, the code block contents are inserted directly into the
generated token tree node struct, so each field must be written as a Rust
struct field terminated with a comma.
Each user token field type must implement `Clone` and `Default`.
The `on_token_node` statement can be used to provide code that initializes
any token user fields when a token tree node instance is created.
@ -781,6 +997,9 @@ the `free_token_node` statement can be used to supply a code block which
will be executed immediately before the token node is freed.
For C++, the `delete` statement is used to free the token tree node, so the
destructor for any custom token user fields will be called.
For Rust, a token user field which owns its memory is dropped along with the
context, so a `free_token_node` code block is only needed for memory which Rust
does not track.
##> `tree` statement - tree generation mode
@ -874,6 +1093,28 @@ assert_eq(22, itemsmore.item.pToken1.pvalue);
assert(itemsmore.pItemsMore is null);
```
The equivalent traversal for a Rust target, where tree node fields are accessor
methods and an absent child is a handle whose `valid()` method returns `false`:
```
let mut context = p_context_new(b"a, ((b)), b");
assert_eq!(P_SUCCESS, p_parse(&mut context));
let start = p_result(&context);
let items = start.pItems();
assert!(items.valid());
assert!(items.item().valid());
assert_eq!(TOKEN_a, items.item().pToken1().token());
assert_eq!(11, items.item().pToken1().pvalue());
let mut itemsmore = items.pItemsMore();
assert!(itemsmore.valid());
assert_eq!(TOKEN_b, itemsmore.item().item().item().pToken1().token());
assert_eq!(22, itemsmore.item().item().item().pToken1().pvalue());
itemsmore = itemsmore.pItemsMore();
assert_eq!(TOKEN_b, itemsmore.item().pToken1().token());
assert!(!itemsmore.pItemsMore().valid());
p_context_delete(context);
```
## `tree_prefix` and `tree_suffix` statements
In tree generation mode, structure types are defined and named based on the
@ -1212,8 +1453,8 @@ lexing based on the grammar's lexer patterns.
Propane defines a `p_context_t` structure type.
The structure is intended to be used opaquely and stores information related to
the state of the lexer and parser.
Integrating code must define an instance of the `p_context_t` structure.
A pointer to this instance is passed to the generated functions.
A `p_context_t` instance is allocated and initialied with the `p_context_new()`
function.
### `p_position_t`
@ -1227,6 +1468,9 @@ For C targets, the `p_position_t` structure can be checked for validity by
calling `p_position_valid(pos)` where `pos` is a `p_position_t` structure
instance.
For Rust targets, the `p_position_t` structure can be checked for validity by
calling its `valid()` method (e.g. `if pos.valid()`).
### `p_value_t`
If tree generation mode is enabled, the `p_value_t` type is defined to be the
@ -1238,6 +1482,13 @@ In this case, the API functions `p_value()` and `p_value_XXX()` for each given
`ptype` name `XXX` are generated to return `p_value_t` instances holding the
corresponding `ptype`.
For Rust targets, `p_value_t` is an enum rather than a union, and every `ptype`
type must implement `Clone` and `Default` (see
${#Rust ptype requirements}).
Reading a `p_value_t` with an accessor for a `ptype` other than the one it
currently holds returns `Default::default()` rather than reinterpreting the
stored bytes.
### `p_token_info_t`
The `p_token_info_t` structure contains the following fields:
@ -1251,6 +1502,9 @@ The `p_token_info_t` structure contains the following fields:
for the default value or `p_value_get_XXX(&token_info.pvalue)` for named
`ptype` values.
For Rust targets, `p_token_info_t` implements `Default`, so a token info
structure to pass to `p_lex()` can be created with `p_token_info_t::default()`.
### Tree Node Types
If tree generation mode is enabled, a structure type for each rule will be
@ -1271,8 +1525,9 @@ A rule node may not have valid positions if the rule allows for an empty match.
In this case the `position` structure should be checked for validity before
using it.
For C targets this can be accomplished with
`if (p_position_valid(node->position))` and for D targets this can be
accomplished with `if (node.position.valid)`.
`if (p_position_valid(node->position))`, for D targets with
`if (node.position.valid)`, and for Rust targets with
`if node.position().valid()`.
A `Token` node has the following additional fields:
@ -1297,6 +1552,12 @@ the same name but with a positional suffix (`pItems1`) which both point to the
parsed `Items` node.
Their value will be null if the parsed `Items` rule was empty.
For Rust targets each of these generated fields is an accessor method returning
a tree node handle, so the fields described in this section are read as
`node.pItems()`, `node.pItems1()`, and so on.
A handle for an absent child is not null; instead its `valid()` method returns
`false`.
The `Items` structure will have fields:
* `pItem` and `pItem1` which point to the parsed `Item` structure.
@ -1359,11 +1620,32 @@ D example:
p_context_t * context = p_context_new(input);
```
Rust example:
```
let mut context = p_context_new(b"a = 1");
```
For Rust targets, `p_context_new()` accepts the input as a `&[u8]` byte slice
and copies it into the returned context.
### `p_context_delete`
The `p_context_delete()` function must be called to deinitialize and deallocate
a context structure allocated by `p_context_init()`.
For Rust targets, `p_context_delete()` takes the context by value and consumes
it.
The memory owned by the context is released when the context is dropped, so the
call is only strictly required when the grammar supplies a `free_token_node`
code block, which runs from `p_context_delete()`.
Rust example:
```
p_context_delete(context);
```
### `p_lex`
The `p_lex()` function is the main entry point to the lexer.
@ -1401,6 +1683,30 @@ case P_SUCCESS:
}
```
Rust example:
```
let mut context = p_context_new(input);
let mut token_info = p_token_info_t::default();
loop
{
match p_lex(&mut context, &mut token_info)
{
P_SUCCESS =>
{
if token_info.token == TOKEN___EOF
{
break;
}
println!("{} ({} bytes)", p_token_names[token_info.token as usize],
token_info.length);
}
/* P_DECODE_ERROR, P_UNEXPECTED_INPUT, or P_USER_TERMINATED. */
_ => break,
}
}
```
### `p_parse`
The `p_parse()` function is the main entry point to the parser.
@ -1413,6 +1719,13 @@ p_context_t * context = p_context_new(input, input_length);
size_t result = p_parse(context);
```
Rust example:
```
let mut context = p_context_new(input);
let result = p_parse(&mut context);
```
When multiple start rules are specified, a separate parse function is generated
for each which starts parsing at the given rule.
For example, if `Statement` is specified as a start rule:
@ -1453,6 +1766,20 @@ size_t p_parse_inner_Statement(p_context_t * context,
Passing `null` for the slice makes the function behave identically to
`p_parse_Statement()`.
For Rust targets, the signature accepts a slice:
```
pub fn p_parse_inner_Statement(context: &mut p_context_t,
follow_tokens: &[p_token_t]) -> usize;
```
Passing an empty slice (`&[]`) makes the function behave identically to
`p_parse_Statement()`:
```
let result = p_parse_inner_Statement(&mut context, &[TOKEN_rbrace]);
```
When the parse is completed via a non-EOF follow token, that follow token is
**not** consumed from the input stream.
The parser rewinds the input index and text position to the start of the follow
@ -1477,6 +1804,10 @@ For D targets, rather than using `p_position_valid()`, the `valid` property
function of the `p_position_t` structure can be queried
(e.g. `if (node.position.valid)`).
For Rust targets, rather than using `p_position_valid()`, the `valid()` method
of the `p_position_t` structure can be called
(e.g. `if node.position().valid()`).
### `p_result`
The `p_result()` function can be used to retrieve the final parse value after
@ -1512,6 +1843,20 @@ if (p_parse_Statement(context) == P_SUCCESS)
In this case, the parser will start parsing with the `Statement` rule and the
parse result from the `Statement` rule will be returned.
Rust example:
```
let mut context = p_context_new(input);
if p_parse(&mut context) == P_SUCCESS
{
let result = p_result(&context);
}
```
For Rust targets in tree generation mode, `p_result()` returns a `Start` tree
node handle which borrows the context, so the handle must be dropped before the
context is passed to `p_context_delete()`.
### `p_position`
The `p_position()` function can be used to retrieve the parser position where
@ -1530,6 +1875,18 @@ if (p_parse(context) == P_UNEXPECTED_TOKEN)
}
```
Rust example:
```
let mut context = p_context_new(input);
if p_parse(&mut context) == P_UNEXPECTED_TOKEN
{
let error_position = p_position(&context);
eprintln!("Error: unexpected token at row {} column {}",
error_position.row, error_position.col);
}
```
### `p_set_position`
The `p_set_position()` function sets the current text position stored in the
@ -1549,6 +1906,14 @@ p_set_position(context, start);
p_parse(context);
```
Rust example:
```
let mut context = p_context_new(input);
p_set_position(&mut context, p_position_t { row: 5, col: 20 });
p_parse(&mut context);
```
### `p_input_index`
The `p_input_index()` function returns the current input text byte offset,
@ -1565,6 +1930,15 @@ size_t offset = p_input_index(context);
/* Remaining input starts at `input + offset`. */
```
Rust example:
```
let mut context = p_context_new(input);
p_parse_inner_Statement(&mut context, follow_tokens);
let offset = p_input_index(&context);
/* Remaining input starts at `&input[offset..]`. */
```
### `p_set_input_index`
The `p_set_input_index()` function sets the current input text byte offset,
@ -1587,6 +1961,17 @@ p_set_input_index(context, saved_index);
p_set_position(context, saved_position);
```
Rust example:
```
/* Save the cursor and text position at the start of a section. */
let saved_index = p_input_index(&context);
let saved_position = p_position(&context);
/* ... later, rewind to re-read that section. */
p_set_input_index(&mut context, saved_index);
p_set_position(&mut context, saved_position);
```
### `p_user_terminate_code`
The `p_user_terminate_code()` function can be used to retrieve the user
@ -1604,6 +1989,15 @@ if (p_parse(context) == P_USER_TERMINATED)
}
```
Rust example:
```
if p_parse(&mut context) == P_USER_TERMINATED
{
let user_terminate_code = p_user_terminate_code(&context);
}
```
### `p_token`
The `p_token()` function can be used to retrieve the current parse token.
@ -1620,6 +2014,15 @@ if (p_parse(context) == P_UNEXPECTED_TOKEN)
}
```
Rust example:
```
if p_parse(&mut context) == P_UNEXPECTED_TOKEN
{
let unexpected_token = p_token(&context);
}
```
### `p_decode_code_point`
The `p_decode_code_point()` function can be used to decode code points from a
@ -1640,6 +2043,19 @@ assert(code_point == 0x1F9E1u);
assert(code_point_length == 4u);
```
Rust Example:
```
let mut code_point: p_code_point_t = 0;
let mut code_point_length: u8 = 0;
let result = p_decode_code_point(b"\xf0\x9f\xa7\xa1",
&mut code_point, &mut code_point_length);
assert_eq!(P_SUCCESS, result);
assert_eq!(0x1F9E1, code_point);
assert_eq!(4, code_point_length);
```
### `p_tree_delete`
The `p_tree_delete()` function can be used to free the memory used by the tree.
@ -1678,6 +2094,12 @@ They are especially useful when tree generation mode is not active.
In that case, the `p_value_t` union can hold one of several different possible
value types.
Rust example:
```
out_token_info.pvalue = p_value(count);
```
### `p_value_get`
The `p_value_get()` accessor functions can be used to extract a user value
@ -1694,6 +2116,16 @@ These functions are the counterpart to `p_value()` constructor functions and
are useful for reading the parser value associated with a lexed token, for
example `p_value_get(&token_info.pvalue)`.
Rust example:
```
let value = p_value_get(&token_info.pvalue);
```
For Rust targets these accessors return a clone of the held value.
If the `p_value_t` holds a different `ptype` than the one requested, the
accessor returns `Default::default()`.
##> Data
### `p_token_names`
@ -1715,6 +2147,19 @@ if (p_parse(context) == P_UNEXPECTED_TOKEN)
}
```
Rust example:
```
let mut context = p_context_new(input);
if p_parse(&mut context) == P_UNEXPECTED_TOKEN
{
let error_position = p_position(&context);
eprintln!("Error: unexpected token `{}' at row {} column {}",
p_token_names[p_token(&context) as usize],
error_position.row, error_position.col);
}
```
#> License
Propane is licensed under the terms of the MIT License: