propane/doc/user_guide.md

${remove}
WARNING: This user guide is meant to be preprocessed and rendered by a custom
script.
The markdown source file is not intended to be viewed directly and will not
include all intended content.
${/remove}

#> Overview

Propane is a LALR Parser Generator (LPG) which:

  * accepts LR(0), SLR, and LALR grammars
  * generates a built-in lexer to tokenize input
  * supports UTF-8 lexer inputs
  * generates a table-driven shift/reduce parser to parse input in linear time
  * targets C, C++, D, or Rust language outputs
  * optionally supports automatic full parse tree generation
  * supports starting parsing from multiple start rules
  * tracks input text start and end positions for all matched tokens/rules
  * is MIT-licensed
  * is distributable as a standalone Ruby script

#> Installation

Propane is designed to be distributed as a stand-alone single file script that
can be copied into and versioned in a project's source tree.
The only requirement to run Propane is that the system has a Ruby interpreter
installed.
The latest release can be downloaded from [https://github.com/holtrop/propane/releases](https://github.com/holtrop/propane/releases).
Simply copy the `propane` executable script into the desired location within
the project to be built (typically the root of the repository) and mark it
executable.

#> Command Line Usage

Propane is typically invoked from the command-line as `./propane`.

    Usage: ./propane [options] <input-file> <output-file>
    Options:
      -h, --help  Show this usage and exit.
      --log LOG   Write log file. This will show all parser states and their
                  associated shifts and reduces. It can be helpful when
                  debugging a grammar.
      --version   Show program version and exit.
      -w          Treat warnings as errors. This option will treat shift/reduce
                  conflicts as fatal errors and will print them to stderr in
                  addition to the log file.

The user must specify the path to a Propane input grammar file and a path to an
output file.
The generated source code will be written to the output file.
If a log file path is specified, Propane will write a log file containing
detailed information about the parser states and transitions.

##> Target language selection

Propane determines the target output language from the extension of the output
file name given on the command line:

  * `.c` selects the C target.
    A header file with a `.h` extension is generated alongside the
    implementation file.
  * `.cc`, `.cpp`, or `.cxx` selects the C++ target.
    A header file with a `.h` extension is generated alongside the
    implementation file.
  * `.d` selects the D target.
  * `.rs` selects the Rust target.

Propane reports an error if the target language cannot be determined from the
output file name.

#> Propane Grammar File

A Propane grammar file provides Propane with the patterns, tokens, grammar
rules, and user code blocks from which to build the generated lexer and parser.

Example grammar file:

```
<<
import std.math;
>>

# Parser values are unsigned integers.
ptype ulong;

# A few basic arithmetic operators.
token plus /\+/;
token times /\*/;
token power /\*\*/;
token integer /\d+/ <<
  ulong v;
  foreach (c; match_text)
  {
    v *= 10;
    v += (c - '0');
  }
  $$ = v;
>>
token lparen /\(/;
token rparen /\)/;
# Drop whitespace.
drop /\s+/;

Start -> E1 << $$ = $1; >>
E1 -> E2 << $$ = $1; >>
E1 -> E1 plus E2 << $$ = $1 + $3; >>
E2 -> E3 << $$ = $1; >>
E2 -> E2 times E3 << $$ = $1 * $3; >>
E3 -> E4 << $$ = $1; >>
E3 -> E3 power E4 << $$ = pow($1, $3); >>
E4 -> integer << $$ = $1; >>
E4 -> lparen E1 rparen << $$ = $2; >>
```

Grammar files can contain comment lines beginning with `#` which are ignored.
White space in the grammar file is also ignored.

It is convention to use the extension `.propane` for the Propane grammar file,
however any file name is accepted by Propane.

This user guide follows the convention of beginning a token name with a
lowercase character and beginning a rule name with an uppercase character.

##> User Code Blocks

User code blocks begin following a "<<" token and end with a ">>" token found
at the end of a line.
All text lines in the code block are copied verbatim into the output file.

### Standalone Code Blocks

C example:

```
<<
#include &lt;stdio.h>
>>
```

D example:

```
<<
import std.stdio;
>>
```

Rust example:

```
<<
use std::collections::HashMap;
>>
```

Standalone code blocks are emitted early in the output file as top-level code
outside the context of any function.
Standalone code blocks are a good place to include/import any other necessary
supporting code modules.
They can also define helper functions that can be reused by lexer or parser
user code blocks.
They are emitted in the order they are defined in the grammar file.

For a C target, the word "header" may immediately follow the "<<" token to
cause Propane to emit the code block in the generated header file rather than
the generated implementation file.
This allows including another header that may be necessary to define any types
needed by a `ptype` directive, for example:

```
<&lt;header
#include "mytypes.h"
>>
```

### Lexer pattern code blocks

Lexer code blocks appear between `<<` and `>>` markers following a `drop`,
`token`, or pattern expression.
User code in a lexer code block will be executed when the lexer matches the
given pattern.
Assignment to the `$$` symbol will associate a parser value with the lexed
token.
This parser value can then be used later in a parser rule.

The input text positions of the matched token can also be accessed from within
a lexer code block.
Each of these positions is an instance of the `p_position_t` structure (see
`${#p_position_t}`), which contains 1-based `row` and `col` fields.
The start position of the matched token is accessed with `${position}`, and
the end position of the matched token is accessed with `${end_position}`.

Example:

```
token integer /\d+/ <<
  printf("integer token on row %d, col %d\n",
      ${position}.row, ${position}.col);
  $$ = parse_integer(match_text, match_length);
>>
```

#### C/C++ lexer code block arguments

The lexer code block is passed the following arguments:

  * `match_text` (`uint8_t const *`) - points to the text matched by the lexer pattern.
  * `match_length` (`size_t`) - length of the matched text.

Example:

```
ptype long;

token integer /\d+/ <<
  long v = 0;
  for (size_t i = 0u; i < match_length; i++)
  {
    v *= 10;
    v += (match_text[i] - '0');
  }
  $$ = v;
>>
```

#### D lexer code block arguments

The lexer code block is passed the following arguments:

  * `match_text` (`string`) - a slice containing the text matched by the lexer pattern.

```
ptype ulong;

token integer /\d+/ <<
  ulong v;
  foreach (c; match_text)
  {
    v *= 10;
    v += (c - '0');
  }
  $$ = v;
>>
```

#### Rust lexer code block arguments

The lexer code block is passed the following arguments:

  * `match_text` (`&[u8]`) - a slice containing the text matched by the lexer pattern.
  * `match_length` (`usize`) - length of the matched text.

The matched text is a byte slice rather than a string; use
`std::str::from_utf8()` or `String::from_utf8_lossy()` to view it as a string.

```
ptype i64;

token integer /\d+/ <<
  let mut v: i64 = 0;
  for c in match_text
  {
      v *= 10;
      v += (c - b'0') as i64;
  }
  $$ = v;
>>
```

### Parser rule code blocks

Example:

```
E1 -> E1 plus E2 << $$ = $1 + $3; >>
```

Parser rule code blocks appear following a rule expression.
User code in a parser rule code block will be executed when the parser reduces
the given rule.
Assignment to the `$$` symbol will associate a parser value with the reduced
rule.
Parser values for the rules or tokens in the rule pattern can be accessed
positionally with tokens `$1`, `$2`, `$3`, etc...

The input text positions for the reduced rule and for the individual rule
components can also be accessed from within a parser rule code block.
Each of these positions is an instance of the `p_position_t` structure (see
`${#p_position_t}`), which contains 1-based `row` and `col` fields.

The start position of the overall reduced rule is accessed with
`${$.position}`, and the end position of the overall reduced rule is accessed
with `${$.end_position}`.

The start and end positions of an individual rule component are accessed
positionally with `${N.position}` and `${N.end_position}`, where `N` is the
1-based index of the component (`${1.position}` for the first component,
`${2.position}` for the second, and so on).

Example:

```
Assignment -> ident equals Expr <<
    printf("assignment on row %d, col %d\n",
        ${$.position}.row, ${$.position}.col);
    printf("target identifier ends on row %d, col %d\n",
        ${1.end_position}.row, ${1.end_position}.col);
    printf("expression starts on row %d, col %d\n",
        ${3.position}.row, ${3.position}.col);
>>
```

A rule or rule component that allows for an empty match may not have valid
positions.
In this case the position should be checked for validity before its `row` and
`col` fields are used (see `${#p_position_valid}`).
For C targets this can be accomplished with
`if (p_position_valid(${$.position}))`, for D targets with
`if (${$.position}.valid)`, and for Rust targets with
`if ${$.position}.valid()`.

In tree generation mode, a full parse tree is automatically constructed in
memory for user code to traverse after parsing is complete.
Parser rule code blocks are still supported in tree generation mode, but they
behave differently than when tree generation mode is not active.
The code block for a rule is executed after the rule has been matched and its
tree node has been fully formed.
Within the code block, `$$` refers to the tree node handle for the reduced
rule, and the rule components are accessed positionally with `$1`, `$2`, `$3`,
etc..., each a tree node handle for that component (a rule node or a `Token`
node).
Field aliases (see the "Specifying parser rules" section) may also be used to
reference a component tree node by name; a field alias behaves identically to
the positional reference for that component.

Tree nodes are stored in a compact arena owned by the parser context and are
referenced by lightweight handles rather than pointers. The whole tree is freed
together with the context by `p_context_delete()`; there is no separate tree
delete function, and tree node handles are only valid while the context is
alive.

Child fields, positions, and token payloads are accessed through per-language
accessors on a node handle:

  * C: field accessor functions `p_TYPE_field(node)` and tree walk macros
    `p_tree_walk_TYPE(node, field1, field2, ...)`; generic accessors
    `p_node_valid(node)`, `p_node_position(node)`, `p_node_end_position(node)`,
    `p_node_n_fields(node)`, `p_node_data(node)` (a pointer to the node record,
    for token payload and user fields), and `p_node_id(node)` (for identity
    comparison).
  * C++: handle methods called with `()`, e.g. `node.field()`, `node.valid()`,
    `node.position()`, `node.token()`, `node.pvalue()`, and `node.data()`. The
    C-style functions and macros above are also available.
  * D: `@property` accessors, e.g. `node.field`, `node.valid`, `node.position`,
    `node.token`, `node.pvalue`.
  * Rust: handle methods called with `()`, e.g. `node.field()`, `node.valid()`,
    `node.position()`, `node.end_position()`, `node.n_fields()`,
    `node.token()`, `node.pvalue()`, `node.data()` (a reference to the node
    record, for token user fields), and `node.node_id()` (for identity
    comparison).

The positional position expansions (`${$.position}`, `${N.position}`, etc...)
are not available in tree generation mode; use the position accessors above
instead.

C example:

```
tree;

Assignment -> ident equals Expr <<
    /* $$ is the Assignment tree node, $1 is the ident Token node, and $3 is
     * the Expr rule node. */
    printf("assignment on row %d, col %d\n",
        p_node_position($$).row, p_node_position($$).col);
    printf("target identifier ends on row %d, col %d\n",
        p_node_end_position($1).row, p_node_end_position($1).col);
>>
```

Rust example:

```
tree;

Assignment -> ident equals Expr <<
    /* $$ is the Assignment tree node, $1 is the ident Token node, and $3 is
     * the Expr rule node. */
    println!("assignment on row {}, col {}",
        $$.position().row, $$.position().col);
    println!("target identifier ends on row {}, col {}",
        $1.end_position().row, $1.end_position().col);
>>
```

##> `context_user_fields` statement - adding custom fields to the context

Propane uses a context structure for lexer and parser operations.
Custom fields may be added to the context structure by using the grammar
`context_user_fields` statement.
This allows lexer pattern or parser rule code blocks to access user-defined
fields within the context structure.

Example:

```
context_user_fields <<
    int mycontextval;
>>
```

Lexer user code blocks or parser user code blocks can access user-defined
context fields by using the `${context.<field>}` syntax.

C++ example:

```
context_user_fields <<
    std::string comments;
>>
drop /#(.*)\n/ <<
    /* Accumulate comments before the next parser tree node. */
    ${context.comments} += std::string((const char *)match_text, match_length);
>>
```

Rust example:

```
context_user_fields <<
    pub comments: String,
>>
drop /#(.*)\n/ <<
    /* Accumulate comments before the next parser tree node. */
    ${context.comments} += std::str::from_utf8(match_text).unwrap();
>>
```

For the Rust target, the code block contents are inserted directly into the
generated `p_context_t` struct definition.
Each field must therefore be written as a Rust struct field, terminated with a
comma.
The generated `p_context_t` derives `Default`, so each user context field type
must implement `Default`.
Mark a field `pub` if code outside of the generated module needs to access it;
grammar user code blocks are emitted into the generated module itself and can
access a field regardless.

If a pointer to any allocated memory is stored in a user-defined context field,
it is up to the user to free any memory when the program is finished using the
context structure.

##> `drop` statement - ignoring input patterns

A `drop` statement can be used to specify a lexer pattern that when matched
should result in the matched input being dropped and lexing continuing after
the matched input.

A common use for a `drop` statement would be to ignore whitespace sequences in
the user input.

Example:

```
drop /\s+/;
```

See also ${#Regular expression syntax}.

## `free_token_node` statement - freeing user-allocated memory in token node fields

If user lexer code block allocates memory to store in a token node's `pvalue`
or any custom token user fields store pointers to allocated memory, the
`free_token_node` grammar statement can be used to provide a code block which
can be used to free memory properly.

Example freeing `pvalue` (C):

```
tree;
free_token_node <<
    free(${token.pvalue});
>>
ptype int *;
token a <<
  $$ = (int *)malloc(sizeof(int));
  *$$ = 1;
>>
token b <<
  $$ = (int *)malloc(sizeof(int));
  *$$ = 2;
>>
Start -> a:a b:b;
```

Example freeing custom token user fields (C):

```
token_user_fields <<
    char * comments;
>>
on_token_node <<
    ${token.comments} = (char *)malloc(some_len);
>>
free_token_node <<
    free(${token.comments});
>>
```

Example freeing `pvalue` (Rust):

```
tree;
free_token_node <<
    if !${token.pvalue}.is_null()
    {
        unsafe { drop(Box::from_raw(${token.pvalue})); }
    }
>>
ptype *mut i32;
token a <<
  $$ = Box::into_raw(Box::new(1));
>>
token b <<
  $$ = Box::into_raw(Box::new(2));
>>
Start -> a:a b:b;
```

The `free_token_node` statement user code block is not emitted for D language
since D has a garbage collector.

The code block is emitted for the Rust target, where it runs from
`p_context_delete()`.
A `ptype` or token user field which owns its memory (a `String`, a `Vec`, a
`Box`, and so on) is released when the context is dropped and does not need a
`free_token_node` code block.
The statement is only needed for memory which Rust does not track, such as a
raw pointer obtained from `Box::into_raw()`.
Note that the generated `p_context_t` does not implement `Drop`, so a
`free_token_node` code block only runs if `p_context_delete()` is called.

##> `lex_fn` statement - specifying a custom lexer function

Propane generates both a lexer and a parser.
By default, the parser uses the generated `p_lex()` function directly to
return information for the next lexed token from the input stream.

However, the user can specify a custom lex function.
This function may or may not use the Propane generated `p_lex()` function under
the hood.
For example, a token sequence could be injected or repeated from a previously
saved macro definition.

Example (C/C++):

```
<<
static size_t mylexfn(p_context_t * context, p_token_info_t * out_token_info)
{
    static size_t count;
    size_t result = P_SUCCESS;
    if (count > 0)
    {
        out_token_info->token = TOKEN_a;
        out_token_info->pvalue = p_value(count);
        count--;
    }
    else
    {
        result = p_lex(context, out_token_info);
        if (out_token_info->token == TOKEN_c)
        {
            count = 3;
        }
    }
    return result;
}
>>

lex_fn mylexfn;
```

Example (Rust):

```
<<
fn mylexfn(context: &mut p_context_t, out_token_info: &mut p_token_info_t) -> usize
{
    let mut result = P_SUCCESS;
    if context.count > 0
    {
        out_token_info.token = TOKEN_a;
        out_token_info.pvalue = p_value(context.count);
        context.count -= 1;
    }
    else
    {
        result = p_lex(context, out_token_info);
        if out_token_info.token == TOKEN_c
        {
            context.count = 3;
        }
    }
    result
}
>>

context_user_fields <<
    count: usize,
>>

lex_fn mylexfn;
```

The `lex_fn` statement takes one argument specifying the name of the custom
lexer function.
The user must supply a value for the `token` field of the `p_token_info_t`
output structure so that the parser knows what token was lexed.

Additionally, if the parser user code makes use of the token's pvalue, then
the lexer function must supply a value for the `pvalue` field of the
`p_token_info_t` structure.
The `p_value()` generated API function could be useful for specifying
parser values to associate with the lexed token when tree generation is not
enabled.
When tree generation is enabled, the `pvalue` field can be set to an instance
of whatever type the user has defined as the `ptype` type.

The custom lex function returns a result code to the parser.
Returning `P_SUCCESS` indicates that the function has produced a token in the
`token` output field and the parse should proceed.
Returning any other result code stops the parse immediately; the parser
propagates that code out of `p_parse()` (and the `p_parse_XXX()` and
`p_parse_inner_XXX()` functions) unchanged.
This allows a custom lex function to surface an error condition, for example by
propagating a `P_UNEXPECTED_TOKEN`, `P_UNEXPECTED_INPUT`, or `P_DECODE_ERROR`
result from a nested `p_parse_inner_XXX()` or `p_lex()` call.

Observe the following contract when returning a result code other than
`P_SUCCESS`:

* Do not return `P_DROP`.
  This code is an internal lexer signal and is never returned to the parser by
  the generated `p_lex()` function.
  Returning it from a custom lex function would be reported as a spurious parse
  failure. If drop functionality is required, the custom lex function should
  loop and return the next non-drop token.
* Do not return `P_EOF` to signal the end of the input.
  The end of input is communicated to the parser by returning `P_SUCCESS` with
  the `token` field set to the end-of-input token (`TOKEN___EOF`), which is what
  the generated `p_lex()` function does.
  Returning `P_EOF` aborts the parse rather than allowing it to complete.
* The `p_token()` and `p_position()` accessors are populated by the parser only
  when the parser itself detects an unexpected token.
  When a custom lex function returns an error code, `p_token()` is not updated
  and may not reflect a meaningful token, and `p_position()` reflects the
  lexer's current text position rather than a parser-identified error location.
  A custom lex function that wants a specific reported position can set it with
  `p_set_position()` before returning.
* Returning `P_USER_TERMINATED` does not populate the user terminate code
  returned by `p_user_terminate_code()`. This user terminate code is normally
  populated by the `$terminate()` function in the user code block.

##> `module` statement - specifying the generated parser module name

The `module` statement can be used to specify the module name for a generated
D module.

```
module proj.parser;
```

If a module statement is not present, then the generated D module will not
contain a module statement and the default module name will be used.

##> `noline` statement - disabling `#line` directives

By default, Propane emits `#line` directives into the generated output around
user code blocks.
These directives instruct the compiler to report any warnings or errors in the
user code using the file name and line number of the original grammar file,
rather than the generated output file.
This makes it easier to locate the source of a compiler diagnostic in the
grammar file.

The `noline` statement disables the emission of these `#line` directives.

```
noline;
```

When the `noline` statement is present, user code blocks are copied to the
generated output without any surrounding `#line` directives.
This can be useful when debugging the generated parser itself, or when the
`#line` directives interfere with other tooling.

The `noline` statement only affects the C, C++, and D targets.
Rust has no `#line` directive equivalent, so `#line` directives are never
emitted into Rust output and the `noline` statement has no effect for the Rust
target.

##> `on_tree_node` statement -  custom initialization of a token tree node

The `on_token_node` statement can be used to provide code that initializes
any token user fields when a token tree node instance is created.

For example (C++):

```
context_user_fields <<
    std::string comments;
>>
token_user_fields <<
    std::string comments;
>>
on_token_node <<
    ${token.comments} = ${context.comments};
    ${context.comments} = "";
>>
drop /#(.*)\n/ <<
    /* Accumulate comments before the next parser tree node. */
    ${context.comments} += std::string((const char *)match_text, match_length);
>>
```

For example (Rust):

```
context_user_fields <<
    pub comments: String,
>>
token_user_fields <<
    pub comments: String,
>>
on_token_node <<
    ${token.comments} = std::mem::take(&mut ${context.comments});
>>
drop /#(.*)\n/ <<
    /* Accumulate comments before the next parser tree node. */
    ${context.comments} += std::str::from_utf8(match_text).unwrap();
>>
```

##> `prefix` statement - specifying the generated API prefix

By default the public API (types, constants, and functions) of the generated
lexer and parser uses a prefix of `p_`.

This prefix can be changed with the `prefix` statement.

Example:

```
prefix myparser_;
```

With a parser generated with this `prefix` statement, instead of calling
`p_context_new()` you would call `myparser_context_new()`.

The `prefix` statement can be optionally used if you would like to change the
prefix used by your generated lexer and parser to something other than the
default.

It can also be used when generating multiple lexers/parsers to be used in the
same program to avoid symbol collisions.

##> `ptype` statement - specifying parser value types

The `ptype` statement is used to define parser value type(s).
Example:

```
ptype void *;
```

This defines the default parser value type to be `void *` (this is, in fact,
the default parser value type if the grammar file does not specify otherwise).

Each defined lexer token type and parser rule has an associated parser value
type.
When the lexer runs, each lexed token has a parser value associated with it.
When the parser runs, each instance of a reduced rule has a parser value
associated with it.
Propane supports using different parser value types for different rules and
token types.
The example `ptype` statement above defines the default parser value type.
A parser value type name can optionally be specified following the `ptype`
keyword.
For example:

```
ptype Value;
ptype array = Value[];
ptype dict = Value[string];

Object -> lbrace rbrace << $$ = new Value(); >>

Values (array) -> Value << $$ = [$1]; >>
Values -> Values comma Value << $$ = $1 ~ [$3]; >>

KeyValue (dict) -> string colon Value << $$ = [$1: $3]; >>
```

In this example, the default parser value type is `Value`.
A parser value type named `array` is defined to mean `Value[]`.
A parser value type named `dict` is defined to mean `Value[string]`.
Any defined tokens or rules that do not specify a parser value type will have
the default parser value type associated with them.
To associate a different parser value type with a token or rule, write the
parser value type name in parentheses following the name of the token or rule.
In this example:

  * a reduced `Object`'s parser value has a type of `Value`.
  * a reduced `Values`'s parser value has a type of `Value[]`.
  * a reduced `KeyValue`'s parser value has a type of `Value[string]`.

When tree generation mode is active, the `ptype` functionality works differently.
In this mode, only one `ptype` is used by the parser.
Lexer user code blocks may assign a parse value to the generated `Token` node
by assigning to `$$` within a lexer code block.
The type of the parse value `$$` is given by the global `ptype` type.

### Rust `ptype` requirements

For the Rust target, every `ptype` type must implement both `Clone` and
`Default`.
When tree generation mode is not active, the generated `p_value_t` is an enum
with one variant per defined `ptype`.
The generated accessor functions clone the held value, and return
`Default::default()` when the variant currently held is not the one requested.

A user-defined `ptype` type therefore normally needs a
`#[derive(Clone, Default)]`:

```
<<
#[derive(Clone, Default)]
pub struct Value
{
    pub n: i64,
}
>>

ptype Value;
```

##> `start` statement - specifying the parser start rule name

The start rule can be changed from the default of `Start` by using the `start`
statement.
Example:

```
start MyStartRule;
```

Multiple start rules can be specified, either with multiple `start` statements
or one `start` statement listing multiple start rules.
Example:

```
start Module ModuleItem Statement Expression;
```

When multiple start rules are specified, multiple `p_parse_*()`,
`p_parse_inner_*()`, and `p_result_*()` functions are generated.
A default `p_parse()` and `p_result()` are generated corresponding to the first
start rule.
Additionally, each start rule causes the generation of another version of each
of these functions, for example `p_parse_Statement()`,
`p_parse_inner_Statement()`, and `p_result_Statement()`.

##> `token` statement - specifying tokens

The `token` statement allows defining a lexer token and a pattern to match that
token.
The name of the token must be specified immediately following the `token`
keyword.
A regular expression pattern may optionally follow the token name.
If a regular expression pattern is not specified, the name of the token is
taken to be the pattern.
See also: ${#Regular expression syntax}.

Example:

```
token for;
```

In this example, the token name is `for` and the pattern to match it is
`/for/`.

Example:

```
token lbrace /\{/;
```

In this example, the token name is `lbrace` and a single left curly brace will
match it.

The `token` statement can also include a user code block.
The user code block will be executed whenever the token is matched by the
lexer.

Example:

```
token if << writeln("'if' keyword lexed"); >>
```

The `token` statement is actually a shortcut statement for a combination of a
`tokenid` statement and a pattern statement.
To define a lexer token without an associated pattern to match it, use a
`tokenid` statement.
To define a lexer pattern that may or may not result in a matched token, use
a pattern statement.

##> `tokenid` statement - defining tokens without a matching pattern

The `tokenid` statement can be used to define a token without associating it
with a lexer pattern that matches it.

Example:

```
tokenid string;
```

The `tokenid` statement can be useful when defining a token that may optionally
be returned by user code associated with a pattern.

It is also useful when lexer modes and multiple lexer patterns are required to
build up a full token.
A common example is parsing a string.
See the ${#Lexer modes} chapter for more information.

##> `token_user_fields` statement - adding custom token fields

When tree generation mode is active, Propane generates a tree node structure
and a token node structure for each matching rule and token instance in the
input string.
The user may add custom fields to token tree nodes using the `token_user_fields`
statement.
The code block supplied to the `token_user_fields` is inserted in the `struct`
generated by the parser to hold a token tree node.

Example (D/C++):

```
token_user_fields <<
    string mytokenval;
>>
```

Example (Rust):

```
token_user_fields <<
    pub mytokenval: String,
>>
```

For the Rust target, the code block contents are inserted directly into the
generated token tree node struct, so each field must be written as a Rust
struct field terminated with a comma.
Each user token field type must implement `Clone` and `Default`.

The `on_token_node` statement can be used to provide code that initializes
any token user fields when a token tree node instance is created.

For example (C++):

```
context_user_fields <<
    std::string comments;
>>
token_user_fields <<
    std::string comments;
>>
on_token_node <<
    ${token.comments} = ${context.comments};
    ${context.comments} = "";
>>
drop /#(.*)\n/ <<
    /* Accumulate comments before the next parser tree node. */
    ${context.comments} += std::string((const char *)match_text, match_length);
>>
```

If a pointer to any allocated memory is stored in a user-defined context field,
the `free_token_node` statement can be used to supply a code block which
will be executed immediately before the token node is freed.
For C++, the `delete` statement is used to free the token tree node, so the
destructor for any custom token user fields will be called.
For Rust, a token user field which owns its memory is dropped along with the
context, so a `free_token_node` code block is only needed for memory which Rust
does not track.

##> `tree` statement - tree generation mode

To activate tree generation mode, place the `tree` statement in your grammar file:

```
tree;
```

It is recommended to place this statement early in the grammar.

In tree generation mode various aspects of propane's behavior are changed:

  * Only one `ptype` is allowed.
  * Parser user code blocks execute after the rule's tree node has been formed
  and access the tree nodes via `$$`, `$1`, `$2`, etc... (see the "Parser rule
  code blocks" section).
  * Structure types are generated to represent the parsed tokens and rules as
  defined in the grammar.
  * The parse result from `p_result()` is a `Start` tree node handle referring
  to the root of the parse tree for the input. If the user has changed the start
  rule with the `start` grammar statement, the name of the start structure will
  be given by the user-specified start rule instead of `Start`.

Example tree generation grammar:

```
tree;

ptype int;

token a << $$ = 11; >>
token b << $$ = 22; >>
token one /1/;
token two /2/;
token comma /,/ <<
  $$ = 42;
>>
token lparen /\(/;
token rparen /\)/;
drop /\s+/;

Start -> Items;

Items -> Item:item ItemsMore;
Items -> ;

ItemsMore -> comma Item:item ItemsMore;
ItemsMore -> ;

Item -> a;
Item -> b;
Item -> lparen Item:item rparen;
Item -> Dual;

Dual -> One Two;
Dual -> Two One;
One -> one;
Two -> two;
```

The following unit test describes the fields that will be present for an
example parse:

```
string input = "a, ((b)), b";
p_context_t * context = p_context_new(input);
assert_eq(P_SUCCESS, p_parse(context));
Start start = p_result(context);
assert(start.pItems1.valid);
assert(start.pItems.valid);
Items items = start.pItems;
assert(items.item.valid);
assert(items.item.pToken1.valid);
assert_eq(TOKEN_a, items.item.pToken1.token);
assert_eq(11, items.item.pToken1.pvalue);
assert(items.pItemsMore.valid);
ItemsMore itemsmore = items.pItemsMore;
assert(itemsmore.item.valid);
assert(itemsmore.item.item.valid);
assert(itemsmore.item.item.item.valid);
assert(itemsmore.item.item.item.pToken1.valid);
assert_eq(TOKEN_b, itemsmore.item.item.item.pToken1.token);
assert_eq(22, itemsmore.item.item.item.pToken1.pvalue);
assert(itemsmore.pItemsMore.valid);
itemsmore = itemsmore.pItemsMore;
assert(itemsmore.item.valid);
assert(itemsmore.item.pToken1.valid);
assert_eq(TOKEN_b, itemsmore.item.pToken1.token);
assert_eq(22, itemsmore.item.pToken1.pvalue);
assert(!itemsmore.pItemsMore.valid);
p_context_delete(context);
```

The equivalent traversal for a Rust target, where tree node fields are accessor
methods and an absent child is a handle whose `valid()` method returns `false`:

```
let mut context = p_context_new(b"a, ((b)), b");
assert_eq!(P_SUCCESS, p_parse(&mut context));
let start = p_result(&context);
let items = start.pItems();
assert!(items.valid());
assert!(items.item().valid());
assert_eq!(TOKEN_a, items.item().pToken1().token());
assert_eq!(11, items.item().pToken1().pvalue());
let mut itemsmore = items.pItemsMore();
assert!(itemsmore.valid());
assert_eq!(TOKEN_b, itemsmore.item().item().item().pToken1().token());
assert_eq!(22, itemsmore.item().item().item().pToken1().pvalue());
itemsmore = itemsmore.pItemsMore();
assert_eq!(TOKEN_b, itemsmore.item().pToken1().token());
assert!(!itemsmore.pItemsMore().valid());
p_context_delete(context);
```

## `tree_prefix` and `tree_suffix` statements

In tree generation mode, structure types are defined and named based on the
rules in the grammar.
Additionally, a structure type called `Token` is generated to hold parsed
token information.

These structure names can be modified by using the `tree_prefix` or `tree_suffix`
statements in the grammar file.
The field names that refer to instances of the structures are not affected by
the `tree_prefix` or `tree_suffix` values.

For example, if the following two lines were added to the example above:

```
tree_prefix ABC;
tree_suffix XYZ;
```

Then the types would be used as such instead:

```
string input = "a, ((b)), b";
p_context_t * context = p_context_new(input);
assert_eq(P_SUCCESS, p_parse(context));
ABCStartXYZ start = p_result(context);
assert(start.pItems1.valid);
assert(start.pItems.valid);
ABCItemsXYZ items = start.pItems;
assert(items.pItem.valid);
assert(items.pItem.pToken1.valid);
assert_eq(TOKEN_a, items.pItem.pToken1.token);
assert_eq(11, items.pItem.pToken1.pvalue);
assert(items.pItemsMore.valid);
ABCItemsMoreXYZ itemsmore = items.pItemsMore;
assert(itemsmore.pItem.valid);
assert(itemsmore.pItem.pItem.valid);
assert(itemsmore.pItem.pItem.pItem.valid);
assert(itemsmore.pItem.pItem.pItem.pToken1.valid);
```

##> Specifying a lexer pattern

A pattern statement is used to define a lexer pattern that can execute user
code but may not result in a matched token.

Example:

```
/foo+/ << writeln("saw a foo pattern"); >>
```

This can be especially useful with ${#Lexer modes}.

See also ${#Regular expression syntax}.

##> Regular expression syntax

A regular expression ("regex") is used to define lexer patterns in `token`,
pattern, and `drop` statements.
A regular expression begins and ends with a `/` character.

Example:

```
/#.*/
```

Regular expressions can include many special characters/sequences:

  * The `.` character matches any input character other than a newline.
  * The `*` character matches any number of the previous regex element.
  * The `+` character matches one or more of the previous regex element.
  * The `?` character matches 0 or 1 of the previous regex element.
  * The `[` character begins a character class.
  * The `(` character begins a matching group.
  * The `{` character begins a count qualifier.
  * The `\` character escapes the following character and changes its meaning:
    * The `\a` sequence matches an ASCII bell character (0x07).
    * The `\b` sequence matches an ASCII backspace character (0x08).
    * The `\d` sequence is shorthand for the `[0-9]` character class.
    * The `\D` sequence matches every code point not matched by `\d`.
    * The `\f` sequence matches an ASCII form feed character (0x0C).
    * The `\n` sequence matches an ASCII new line character (0x0A).
    * The `\r` sequence matches an ASCII carriage return character (0x0D).
    * The `\s` sequence is shorthand for the `[ \t\r\n\f\v]` character class.
    * The `\S` sequence matches every code point not matched by `\s`.
    * The `\t` sequence matches an ASCII tab character (0x09).
    * The `\v` sequence matches an ASCII vertical tab character (0x0B).
    * The `\w` sequence is shorthand for the `[a-zA-Z0-9_]` character class.
    * The `\W` sequence matches every code point not matched by `\w`.
    * Any other character matches itself.
  * The `|` character creates an alternate match.

Any other character just matches itself in the input stream.

A character class consists of a list of character alternates or character
ranges that can be matched by the character class.
For example `[a-zA-Z_]` matches any lowercase character between `a` and `z` or
any uppercase character between `A` and `Z` or the underscore `_` character.
Character classes can also be negative character classes if the first character
after the `[` is a `^` character.
In this case, the set of characters matched by the character class is the
inverse of what it otherwise would have been.
For example, `[^0-9]` matches any character other than 0 through 9.

A matching group can be used to override the pattern sequence that multiplicity
specifiers apply to.
For example, the pattern `/foo+/` matches "foo" or "foooo", while the pattern
`/(foo)+/` matches "foo" or "foofoofoo", but not "foooo".

A count qualifier in curly braces can be used to restrict the number of matches
of the preceding atom to an explicit minimum and maximum range.
For example, the pattern `\d{3}` matches exactly 3 digits 0-9.
Both a minimum and maximum multiplicity count can be specified and separated by
a comma.
For example, `/a{1,5}/` matches between 1 and 5 `a` characters.
Either the minimum or maximum count can be omitted to omit the corresponding
restriction in the number of matches allowed.

An alternate match is created with the `|` character.
For example, the pattern `/foo|bar/` matches either the sequence "foo" or the
sequence "bar".

##> Lexer modes

Lexer modes can be used to change the set of patterns that are matched by the
lexer.
A common use for lexer modes is to match strings.

Example:

```
<<
string mystringvalue;
>>

tokenid str;

# String processing
/"/ <<
  mystringvalue = "";
  $mode(string);
>>
string: /[^"]+/ << mystringvalue ~= match_text; >>
string: /"/ <<
  $mode(default);
  return $token(str);
>>
```

A lexer mode is defined by placing the name before a colon (`:`) character that
precedes a token or pattern statement.
The token or pattern statement is restricted to only applying if the named mode
is active.

By default, the active lexer mode is named `default`.
A `$mode()` call within a lexer code block can be used to change lexer modes.

In the above example, when the lexer in the default mode sees a doublequote
(`"`) character, the lexer code block will clear the `mystringvalue` variable
and will set the lexer mode to `string`.
When the lexer begins looking for patterns to match against the input, it will
now look only for patterns tagged for the `string` lexer mode.
Any non-`"` character will be appended to the `mystringvalue` string.
A `"` character will end the `string` lexer mode and return to the `default`
lexer mode.
It also returns the `str` token now that the token is complete.

Note that the token name `str` above could have been `string` instead - the
namespace for token names is distinct from the namespace for lexer modes.

Multiple modes can be specified for a token or pattern or drop statement.
For example, if the grammar wanted to only recognize an identifier following
a `.` token and not other keywords, it could switch to an `identonly` mode
when matching a `.`
The `ident` token pattern will be matched in either the `default` or
`identonly` mode.

```
ptype char;
token abc;
token def;
default, identonly: token ident /[a-z]+/ <<
  $$ = match_text[0];
  $mode(default);
  return $token(ident);
>>
token dot /\./ <<
  $mode(identonly);
>>
default, identonly: drop /\s+/;
```

##> Specifying parser rules

Rule statements create parser rules which define the grammar that will be
parsed by the generated parser.

Multiple rules with the same name can be specified.
Rules with the same name define a rule set for that name and act as
alternatives that the parser can accept when attempting to match a reference to
that rule.

The default start rule name is `Start`.
This can be changed with the `start` statement.
The grammar file must define a rule with the name of the start rule name which
will be used as the top-level starting rule that the parser attempts to reduce.

Rule statements are composed of the name of the rule, a `->` token, the fields
defining the rule pattern that must be matched, and a terminating semicolon or
user code block.

Example:

```
ptype ulong;
start Top;
token word /[a-z]+/ << $$ = match_text.length; >>
Top -> word << $$ = $1; >>
```

In the above example the `Top` rule is defined to match a single `word`
token.

Another example:

```
Start -> E1 << $$ = $1; >>
E1 -> E2 << $$ = $1; >>
E1 -> E1 plus E2 << $$ = $1 + $3; >>
E2 -> E3 << $$ = $1; >>
E2 -> E2 times E3 << $$ = $1 * $3; >>
E3 -> E4 << $$ = $1; >>
E3 -> E3 power E4 << $$ = pow($1, $3); >>
E4 -> integer << $$ = $1; >>
E4 -> lparen E1 rparen << $$ = $2; >>
```

This example uses the default start rule name of `Start`.

A parser rule has zero or more fields on the right side of its definition.
Each of these fields is either a token name or a rule name.
A field can be immediately followed by a `?` character to signify that it is
optional.
A field can optionally be followed by a `:` and then a field alias name.
If present, the field alias name is used to refer to the field value in user
code blocks, or if tree generation mode is active, the field alias name is used
as the field name in the generated tree node structure.
An optional and named field must use the format `field?:name`.
Example:

```
token public;
token private;
token int;
token ident /[a-zA-Z_][a-zA-Z_0-9]*/;
token semicolon /;/;
IntegerDeclaration -> Visibility?:visibility int ident:name semicolon;
Visibility -> public;
Visibility -> private;
```

In a parser rule code block, parser values for the right side fields are
accessible as `$1` for the first field's parser value, `$2` for the second
field's parser value, etc...
For the `IntegerDeclaration` rule, the first field value can also be referred to as `${visibility}` and the third field value can also be referred
to as `${name}`.
The `$$` symbol accesses the output parser value for this rule.
The above examples demonstrate how the parser values for the rule components
can be used to produce the parser value for the accepted rule.

In tree generation mode, parser rule code blocks access the reduced rule tree
node and its component tree nodes via `$$`, `$1`, `$2`, etc... (see the "Parser
rule code blocks" section).
Field aliases may still be used in tree generation mode to reference a component
tree node by name, behaving identically to the corresponding positional
reference.

##> User termination of the lexer or parser

Propane supports allowing lexer or parser user code blocks to terminate
execution of the parser.
Some example uses of this functionality could be to:

  * Detect integer overflow when lexing an integer literal constant.
  * Detect and report an error as soon as possible during parsing before continuing to parse any more of the input.
  * Determine whether parsing should stop and instead be retried using a different parser version.

To terminate parsing from a lexer or parser user code block, use the
`$terminate(code)` function, passing an integer expression argument.
For example:

```
NewExpression -> new Expression << $terminate(42); >>
```

The value passed to the `$terminate()` function is known as the "user terminate
code".
If the parser returns a `P_USER_TERMINATED` result code, then the user
terminate code can be accessed using the `p_user_terminate_code()` API
function.

#> Propane generated API

By default, Propane uses a prefix of `p_` when generating a lexer/parser.
This prefix is used for all publicly declared types and functions.
The uppercase version of the prefix is used for all constant values.

This section documents the generated API using the default `p_` or `P_` names.

##> Constants

Propane generates the following result code constants:

* `P_SUCCESS`: A successful decode/lex/parse operation has taken place.
* `P_DECODE_ERROR`: An error occurred when decoding UTF-8 input.
* `P_UNEXPECTED_INPUT`: Input was received by the lexer that does not match any lexer pattern.
* `P_UNEXPECTED_TOKEN`: A token was seen in a location that does not match any parser rule.
* `P_DROP`: The lexer matched a drop pattern.
* `P_EOF`: The lexer reached the end of the input string.
* `P_USER_TERMINATED`: A parser user code block has requested to terminate the parser.

Result codes are returned by the API functions `p_decode_code_point()`, `p_lex()`, and `p_parse()`.

##> Types

### `p_code_point_t`

The `p_code_point_t` type is aliased to a 32-bit unsigned integer.
It is used to store decoded code points from the input text and perform
lexing based on the grammar's lexer patterns.

### `p_context_t`

Propane defines a `p_context_t` structure type.
The structure is intended to be used opaquely and stores information related to
the state of the lexer and parser.
A `p_context_t` instance is allocated and initialied with the `p_context_new()`
function.

### `p_position_t`

The `p_position_t` structure contains two fields: `row` and `col`.
These fields contain the 1-based row and column describing a parser position.

For D targets, the `p_position_t` structure can be checked for validity by
querying the `valid` property.

For C targets, the `p_position_t` structure can be checked for validity by
calling `p_position_valid(pos)` where `pos` is a `p_position_t` structure
instance.

For Rust targets, the `p_position_t` structure can be checked for validity by
calling its `valid()` method (e.g. `if pos.valid()`).

### `p_value_t`

If tree generation mode is enabled, the `p_value_t` type is defined to be the
type given to the `ptype` statement in the grammar file.

If tree generation mode is not enabled, there could be more than one `ptype`
given, so the `p_value_t` type is a union of all possible `ptype` types.
In this case, the API functions `p_value()` and `p_value_XXX()` for each given
`ptype` name `XXX` are generated to return `p_value_t` instances holding the
corresponding `ptype`.

For Rust targets, `p_value_t` is an enum rather than a union, and every `ptype`
type must implement `Clone` and `Default` (see
${#Rust ptype requirements}).
Reading a `p_value_t` with an accessor for a `ptype` other than the one it
currently holds returns `Default::default()` rather than reinterpreting the
stored bytes.

### `p_token_info_t`

The `p_token_info_t` structure contains the following fields:

* `position` (`p_position_t`) holds the text position of the first code point in the token.
* `end_position` (`p_position_t`) holds the text position of the last code point in the token.
* `length` (`size_t`) holds the number of input bytes used by the token.
* `token` (`p_token_t`) holds the token ID of the lexed token
* `pvalue` (`p_value_t`) holds the parser value associated with the token.
  The actual user value can be extracted with `p_value_get(&token_info.pvalue)`
  for the default value or `p_value_get_XXX(&token_info.pvalue)` for named
  `ptype` values.

For Rust targets, `p_token_info_t` implements `Default`, so a token info
structure to pass to `p_lex()` can be created with `p_token_info_t::default()`.

### Tree Node Types

If tree generation mode is enabled, a structure type for each rule will be
generated.
The name of the structure type is given by the name of the rule.
Additionally a structure type called `Token` is generated to represent a
tree node which refers to a raw parser token rather than a composite rule.

#### Tree Node Fields

All tree nodes have a `position` field specifying the text position of the
beginning of the matched token or rule, and an `end_position` field specifying
the text position of the end of the matched token or rule.
Each of these fields are instances of the `p_position_t` structure.

A `Token` node will always have a valid `position` and `end_position`.
A rule node may not have valid positions if the rule allows for an empty match.
In this case the `position` structure should be checked for validity before
using it.
For C targets this can be accomplished with
`if (p_position_valid(node->position))`, for D targets with
`if (node.position.valid)`, and for Rust targets with
`if node.position().valid()`.

A `Token` node has the following additional fields:

  * `token` which specifies which token was parsed (one of `TOKEN_*`)
  * `pvalue` which specifies the parser value for the token. If a lexer user
  code block assigned to `$$`, the assigned value will be stored here.

Tree node structures for rules contain generated fields based on the
right hand side components specified for all rules of a given name.

In this example:

```
Start -> Items;

Items -> Item ItemsMore;
Items -> ;
```

The `Start` structure will have a field called `pItems` and another field of
the same name but with a positional suffix (`pItems1`) which both refer to the
parsed `Items` node.
Both will be invalid node handles if the parsed `Items` rule was empty.

Tree node fields are not data members; they are read through the per-language
accessors described in the "Parser rule code blocks" section, so this field is
read as `p_Start_pItems(node)` for C, `node.pItems()` for C++ and Rust, and
`node.pItems` for D.

The `Items` structure will have fields:

  * `pItem` and `pItem1` which refer to the parsed `Item` node.
  * `pItemsMore` and `pItemsMore2` which refer to the parsed `ItemsMore` node.

If a rule can be empty (for example in the second `Items` rule above), then the
field referring to that rule's generated tree node will be an invalid node
handle if the parser matches the empty rule pattern.

The non-positional tree node field will not be generated if there are multiple
positions in which an instance of the node it refers to could be present.
For example, in the below rules:

```
Dual -> One Two;
Dual -> Two One;
```

The generated `Dual` structure will contain `pOne1`, `pTwo2`, `pTwo1`, and
`pOne2` fields.
However, a `pOne` field and `pTwo` field will not be generated since it would
be ambiguous which one was matched.

If the first rule is matched, then `pOne1` and `pTwo2` will be valid node
handles while `pTwo1` and `pOne2` will be invalid.
If the second rule is matched instead, then the opposite would be the case.

If a field alias is present in a rule definition, an additional field will be
generated in the tree node with the field alias name.
For example:

```
Exp -> Exp:left plus ExpB:right;
```

In the generated `Exp` structure, the fields `pExp`, `pExp1`, and `left` will
all refer to the same child node (an instance of the `Exp` structure), and the
fields `pExpB`, `pExpB3`, and `right` will all refer to the same child node
(an instance of the `ExpB` structure).

##> Functions

### `p_context_new`

The `p_context_new()` function must be called to allocate and initialize the
context structure.
The input to be used for lexing/parsing is passed in when initializing the
context structure.

C example:

```
p_context_t * context = p_context_new(input, input_length);
```

D example:

```
p_context_t * context = p_context_new(input);
```

Rust example:

```
let mut context = p_context_new(b"a = 1");
```

For Rust targets, `p_context_new()` accepts the input as a `&[u8]` byte slice
and copies it into the returned context.

### `p_context_delete`

The `p_context_delete()` function must be called to deinitialize and deallocate
a context structure allocated by `p_context_new()`.

In tree generation mode, the whole parse tree is owned by the context and is
freed by `p_context_delete()`.
Tree node handles are only valid while the context is alive.
If a lexer user code block allocates memory to store in a token node's `pvalue`
or in a custom token user field, the `free_token_node` statement can be used to
provide a code block which frees that memory; if specified, the
`free_token_node` code block is executed from `p_context_delete()`.

For Rust targets, `p_context_delete()` takes the context by value and consumes
it.
The memory owned by the context is released when the context is dropped, so the
call is only strictly required when the grammar supplies a `free_token_node`
code block, which runs from `p_context_delete()`.

Rust example:

```
p_context_delete(context);
```

### `p_lex`

The `p_lex()` function is the main entry point to the lexer.
It is normally called automatically by the generated parser to retrieve the
next input token for the parser and does not need to be called by the user.
However, the user may initialize a context and call `p_lex()` to use the
generated lexer in a standalone mode.

Example:

```
p_context_t * context = p_context_new(input, input_length);
p_token_info_t token_info;
size_t result = p_lex(context, &token_info);
switch (result)
{
case P_DECODE_ERROR:
    /* UTF-8 decode error */
    break;
case P_UNEXPECTED_INPUT:
    /* Input text does not match any lexer pattern. */
    break;
case P_USER_TERMINATED:
    /* Lexer user code block requested to terminate the lexer. */
    break;
case P_SUCCESS:
    /*
     * token_info.position holds the text position of the first code point in the token.
     * token_info.end_position holds the text position of the last code point in the token.
     * token_info.length holds the number of input bytes used by the token.
     * token_info.token holds the token ID of the lexed token
     * token_info.pvalue holds the parser value associated with the token.
     */
    break;
}
```

Rust example:

```
let mut context = p_context_new(input);
let mut token_info = p_token_info_t::default();
loop
{
    match p_lex(&mut context, &mut token_info)
    {
        P_SUCCESS =>
        {
            if token_info.token == TOKEN___EOF
            {
                break;
            }
            println!("{} ({} bytes)", p_token_names[token_info.token as usize],
                token_info.length);
        }
        /* P_DECODE_ERROR, P_UNEXPECTED_INPUT, or P_USER_TERMINATED. */
        _ => break,
    }
}
```

### `p_parse`

The `p_parse()` function is the main entry point to the parser.
It must be passed a pointer to an initialized context structure.

Example:

```
p_context_t * context = p_context_new(input, input_length);
size_t result = p_parse(context);
```

Rust example:

```
let mut context = p_context_new(input);
let result = p_parse(&mut context);
```

When multiple start rules are specified, a separate parse function is generated
for each which starts parsing at the given rule.
For example, if `Statement` is specified as a start rule:

```
size_t result = p_parse_Statement(context);
```

In this case, the parser will start parsing with the `Statement` rule.

### `p_parse_inner_XXX`

For each start rule, a `p_parse_inner_XXX()` function is also generated.
This variant of the parser entry point accepts a caller-provided array of
"follow tokens" -- tokens the caller allows to appear immediately after the
start rule in some outer grammar context.
It is useful when embedding a Propane-generated sub-parser within an outer
parser and the outer parser knows which tokens naturally terminate the
sub-parse.

For C targets, the signature is (example for a rule named `Statement`):

```
size_t p_parse_inner_Statement(p_context_t * context,
    p_token_t const * follow_tokens, size_t n_follow_tokens);
```

Passing a `NULL` pointer (or a count of zero) makes the function behave
identically to `p_parse_Statement()`.

For D targets, the signature accepts a slice:

```
size_t p_parse_inner_Statement(p_context_t * context,
    const(p_token_t)[] follow_tokens);
```

Passing `null` for the slice makes the function behave identically to
`p_parse_Statement()`.

For Rust targets, the signature accepts a slice:

```
pub fn p_parse_inner_Statement(context: &mut p_context_t,
    follow_tokens: &[p_token_t]) -> usize;
```

Passing an empty slice (`&[]`) makes the function behave identically to
`p_parse_Statement()`:

```
let result = p_parse_inner_Statement(&mut context, &[TOKEN_rbrace]);
```

When the parse is completed via a non-EOF follow token, that follow token is
**not** consumed from the input stream.
The parser rewinds the input index and text position to the start of the follow
token so that a subsequent call to `p_lex()` or another parse function sees the
same token.

### `p_position_valid`

The `p_position_valid()` function is only generated for C targets.
it is used to determine whether or not a `p_position_t` structure is valid.

Example:

```
if (p_position_valid(node->position))
{
    ....
}
```

For D targets, rather than using `p_position_valid()`, the `valid` property
function of the `p_position_t` structure can be queried
(e.g. `if (node.position.valid)`).

For Rust targets, rather than using `p_position_valid()`, the `valid()` method
of the `p_position_t` structure can be called
(e.g. `if node.position().valid()`).

### `p_result`

The `p_result()` function can be used to retrieve the final parse value after
`p_parse()` returns a `P_SUCCESS` value.

Example:

```
p_context_t * context = p_context_new(input, input_length);
size_t result = p_parse(context);
if (p_parse(context) == P_SUCCESS)
{
    result = p_result(context);
}
```

If tree generation mode is active, then the `p_result()` function returns a
`Start` tree node handle referring to the root of the parse tree.

When multiple start rules are specified, a separate result function is generated
for each which returns the parse result for the corresponding rule.
For example, if `Statement` is specified as a start rule:

```
p_context_t * context = p_context_new(input, input_length);
size_t result = p_parse(context);
if (p_parse_Statement(context) == P_SUCCESS)
{
    result = p_result_Statement(context);
}
```

In this case, the parser will start parsing with the `Statement` rule and the
parse result from the `Statement` rule will be returned.

Rust example:

```
let mut context = p_context_new(input);
if p_parse(&mut context) == P_SUCCESS
{
    let result = p_result(&context);
}
```

For Rust targets in tree generation mode, `p_result()` returns a `Start` tree
node handle which borrows the context, so the handle must be dropped before the
context is passed to `p_context_delete()`.

### `p_position`

The `p_position()` function can be used to retrieve the parser position where
an error occurred.

Example:

```
p_context_t * context = p_context_new(input, input_length);
size_t result = p_parse(context);
if (p_parse(context) == P_UNEXPECTED_TOKEN)
{
    p_position_t error_position = p_position(context);
    fprintf(stderr, "Error: unexpected token at row %u column %u\n",
        error_position.row, error_position.col);
}
```

Rust example:

```
let mut context = p_context_new(input);
if p_parse(&mut context) == P_UNEXPECTED_TOKEN
{
    let error_position = p_position(&context);
    eprintln!("Error: unexpected token at row {} column {}",
        error_position.row, error_position.col);
}
```

### `p_set_position`

The `p_set_position()` function sets the current text position stored in the
context.
This is useful when performing a nested parse operation over a slice of a
larger enclosing document: by setting the initial text position to something
other than the default `(1, 1)`, any token positions or error positions
subsequently reported by the lexer/parser are relative to the enclosing
document rather than the slice.

Example:

```
p_context_t * context = p_context_new(input, input_length);
p_position_t start = { .row = 5, .col = 20 };
p_set_position(context, start);
p_parse(context);
```

Rust example:

```
let mut context = p_context_new(input);
p_set_position(&mut context, p_position_t { row: 5, col: 20 });
p_parse(&mut context);
```

### `p_input_index`

The `p_input_index()` function returns the current input text byte offset,
measured from the start of the input text passed to `p_context_new()`.
This is useful for slicing out the remaining input after a partial parse,
or for locating tokens in the original input buffer.

Example:

```
p_context_t * context = p_context_new(input, input_length);
p_parse_inner_Statement(context, follow_tokens, n_follow_tokens);
size_t offset = p_input_index(context);
/* Remaining input starts at `input + offset`. */
```

Rust example:

```
let mut context = p_context_new(input);
p_parse_inner_Statement(&mut context, follow_tokens);
let offset = p_input_index(&context);
/* Remaining input starts at `&input[offset..]`. */
```

### `p_set_input_index`

The `p_set_input_index()` function sets the current input text byte offset,
measured from the start of the input text passed to `p_context_new()`.
This moves the lexer's read cursor, which can be used together with
`p_set_position()` to rewind the input part-way through a parse in order to
re-read an earlier section of the input.
The byte offset is not validated; the caller is responsible for providing an
offset within the bounds of the input text.
A value previously returned by `p_input_index()` is a suitable argument.

Example:

```
/* Save the cursor and text position at the start of a section. */
size_t saved_index = p_input_index(context);
p_position_t saved_position = p_position(context);
/* ... later, rewind to re-read that section. */
p_set_input_index(context, saved_index);
p_set_position(context, saved_position);
```

Rust example:

```
/* Save the cursor and text position at the start of a section. */
let saved_index = p_input_index(&context);
let saved_position = p_position(&context);
/* ... later, rewind to re-read that section. */
p_set_input_index(&mut context, saved_index);
p_set_position(&mut context, saved_position);
```

### `p_user_terminate_code`

The `p_user_terminate_code()` function can be used to retrieve the user
terminate code after `p_parse()` returns a `P_USER_TERMINATED` value.
User terminate codes are arbitrary values that can be defined by the user to
be returned when the user requests to terminate parsing.
They have no particular meaning to Propane.

Example:

```
if (p_parse(context) == P_USER_TERMINATED)
{
    size_t user_terminate_code = p_user_terminate_code(context);
}
```

Rust example:

```
if p_parse(&mut context) == P_USER_TERMINATED
{
    let user_terminate_code = p_user_terminate_code(&context);
}
```

### `p_token`

The `p_token()` function can be used to retrieve the current parse token.
This is useful after `p_parse()` returns a `P_UNEXPECTED_TOKEN` value.
terminate code after `p_parse()` returns a `P_USER_TERMINATED` value to
indicate what token the parser was not expecting.

Example:

```
if (p_parse(context) == P_UNEXPECTED_TOKEN)
{
    p_token_t unexpected_token = p_token(context);
}
```

Rust example:

```
if p_parse(&mut context) == P_UNEXPECTED_TOKEN
{
    let unexpected_token = p_token(&context);
}
```

### `p_decode_code_point`

The `p_decode_code_point()` function can be used to decode code points from a
UTF-8 string.
It does not require a lexer/parser context structure and can be used as a
standalone UTF-8 decoder or from within a lexer or parser user code block.

D Example:

```
size_t result;
p_code_point_t code_point;
ubyte code_point_length;

result = p_decode_code_point("\xf0\x9f\xa7\xa1", &code_point, &code_point_length);
assert(result == P_SUCCESS);
assert(code_point == 0x1F9E1u);
assert(code_point_length == 4u);
```

Rust Example:

```
let mut code_point: p_code_point_t = 0;
let mut code_point_length: u8 = 0;

let result = p_decode_code_point(b"\xf0\x9f\xa7\xa1",
    &mut code_point, &mut code_point_length);
assert_eq!(P_SUCCESS, result);
assert_eq!(0x1F9E1, code_point);
assert_eq!(4, code_point_length);
```

### `p_value`

The `p_value(v)` function builds an instance of the `p_value_t` with the
default member set to the value of `v`.

A `p_value_XXX(v)` function is defined for each user-defined `ptype` name with
the user-given name in place of the `XXX`.

These functions are useful for custom lexer functions which need to return a
parser value corresponding to a lexed token.

They are especially useful when tree generation mode is not active.
In that case, the `p_value_t` union can hold one of several different possible
value types.

Rust example:

```
out_token_info.pvalue = p_value(count);
```

### `p_value_get`

The `p_value_get()` accessor functions can be used to extract a user value
from a `p_value_t` union.

The `p_value_get(pvalue)` function accepts a pointer to a `p_value_t` and
returns the value of its default member.

A `p_value_get_XXX(pvalue)` function is generated for each user-defined `ptype`
name with the user-given name in place of the `XXX`, returning the value of the
corresponding member.

These functions are the counterpart to `p_value()` constructor functions and
are useful for reading the parser value associated with a lexed token, for
example `p_value_get(&token_info.pvalue)`.

Rust example:

```
let value = p_value_get(&token_info.pvalue);
```

For Rust targets these accessors return a clone of the held value.
If the `p_value_t` holds a different `ptype` than the one requested, the
accessor returns `Default::default()`.

##> Data

### `p_token_names`

The `p_token_names` array contains the grammar-specified token names.
It is indexed by the token ID.

C example:

```
p_context_t * context = p_context_new(input, input_length);
size_t result = p_parse(context);
if (p_parse(context) == P_UNEXPECTED_TOKEN)
{
    p_position_t error_position = p_position(context);
    fprintf(stderr, "Error: unexpected token `%s' at row %u column %u\n",
        p_token_names[context->token],
        error_position.row, error_position.col);
}
```

Rust example:

```
let mut context = p_context_new(input);
if p_parse(&mut context) == P_UNEXPECTED_TOKEN
{
    let error_position = p_position(&context);
    eprintln!("Error: unexpected token `{}' at row {} column {}",
        p_token_names[p_token(&context) as usize],
        error_position.row, error_position.col);
}
```

#> License

Propane is licensed under the terms of the MIT License:

```
${include LICENSE.txt}
```

#> Contributing

Propane is developed on [github](https://github.com/holtrop/propane).

Issues may be submitted to [https://github.com/holtrop/propane/issues](https://github.com/holtrop/propane/issues).

Pull requests may be submitted as well:

  1. Fork it
  2. Create your feature branch (`git checkout -b my-new-feature`)
  3. Commit your changes (`git commit -am 'Add some feature'`)
  4. Push to the branch (`git push origin my-new-feature`)
  5. Create new Pull Request

#> Change Log

${changelog}