Allow user to specify custom token node fields

doc/user_guide.md

@@ -256,6 +256,95 @@ If a pointer to any allocated memory is stored in a user-defined context field,
 it is up to the user to free any memory when the program is finished using the
 context structure.
 
+### Custom token fields code blocks: the `token_user_fields` statement
+
+When tree generation mode is active, Propane generates a tree node structure
+and a token node structure for each matching rule and token instance in the
+input string.
+The user may add custom fields to token tree nodes using the `token_user_fields`
+statement.
+The code block supplied to the `token_user_fields` is inserted in the `struct`
+generated by the parser to hold a token tree node.
+
+Example (D/C++):
+
+```
+token_user_fields <<
+    string mytokenval;
+>>
+```
+
+The `on_token_node` statement can be used to provide code that initializes
+any token user fields when a token tree node instance is created.
+
+For example (C++):
+
+```
+context_user_fields <<
+    std::string comments;
+>>
+token_user_fields <<
+    std::string comments;
+>>
+on_token_node <<
+    ${token.comments} = ${context.comments};
+    ${context.comments} = "";
+>>
+drop /#(.*)\n/ <<
+    /* Accumulate comments before the next parser tree node. */
+    ${context.comments} += std::string((const char *)match, match_length);
+>>
+```
+
+If a pointer to any allocated memory is stored in a user-defined context field,
+the `free_token_user_fields` statement can be used to supply a code block which
+will be executed immediately before the token node is freed.
+For C++, the `delete` statement is used to free the token tree node, so the
+destructor for any custom token user fields will be called.
+
+### Custom initialization of a token tree node - the `on_tree_node` statement
+
+The `on_token_node` statement can be used to provide code that initializes
+any token user fields when a token tree node instance is created.
+
+For example (C++):
+
+```
+context_user_fields <<
+    std::string comments;
+>>
+token_user_fields <<
+    std::string comments;
+>>
+on_token_node <<
+    ${token.comments} = ${context.comments};
+    ${context.comments} = "";
+>>
+drop /#(.*)\n/ <<
+    /* Accumulate comments before the next parser tree node. */
+    ${context.comments} += std::string((const char *)match, match_length);
+>>
+```
+
+### Freeing allocated memory in a custom token user field - the `free_token_user_fields` statement
+
+The `free_token_user_fields` statement allows the user to provide a code block
+which will be executed immediately prior to freeing the token tree node.
+
+For example (C):
+
+```
+token_user_fields <<
+    char * comments;
+>>
+on_token_node <<
+    ${token.comments} = (char *)malloc(some_len);
+>>
+free_token_user_fields <<
+    free(${token.comments});
+>>
+```
+
 ##> Tree generation mode - the `tree` statement
 
 To activate tree generation mode, place the `tree` statement in your grammar file: