Parsing Expression Grammar Template Library (PEGTL)

PEGTL provides a vast library of rules categorized by their behavior and the character sets they target. Rules are grouped into several functional categories:

• Atomic: Basic building blocks like eof, bol (beginning of line), eot (end of text), and everything.
• ASCII: Rules specifically targeting the ASCII character set, such as alnum, digit, blank, and string<C...>.
• Unicode/ICU: Rules based on Unicode properties and ICU rules, including alphabetic, id_start, id_continue, and various grapheme properties.
• Combinators: Rules that combine other rules, such as seq<R...> (sequence), opt<R> (optional), star<R> (zero or more), and plus<R> (one or more).
• Controlling: Rules that manage the parsing state or flow, such as action<A, R...>, control<C, R...>, and state<S, R...>.
• Convenience: High-level rules for common patterns, like list<R, S>, separated<S, R...>, and rep<Num, R...>.
• Exceptional: Rules for error handling and conditional logic, such as must<R...>, raise<T>, and various try_catch variants.
• Member: Rules that operate on members of a set, such as not_one<M, U...> or string<M, U...>.

Rules and combinators in PEGTL reside in the tao::pegtl namespace (or a sub-namespace). You can change the default namespace using the TAO_PEGTL_NAMESPACE macro in tao/pegtl/config.hpp.

Key concepts for interpreting rule documentation:

• Equivalence: Some rules are documented as equivalent to other combinations. This refers to matching behavior, not necessarily the underlying implementation.
• Parameter Packs: Documentation uses R... to indicate zero-or-more or one-or-more template parameters (e.g., seq< R... >).
• End Of Line (EOL) Rules: Rules available in scan or lazy sub-namespaces are designed for end-of-line scanning. The default EOL behavior can be modified by defining TAO_PEGTL_DEFAULT_EOL before including tao/pegtl/system.hpp or by providing a custom EOL rule to the input type.
• Control: The default control is tao::pegtl::normal, which does not call control functions for rules in the tao::pegtl::internal namespace.

To use PEGTL effectively, it is important to understand its core abstractions:

• Rule: A class that models a production rule of a formal grammar or a parser combinator.
• Grammar: A set of one or more related rules, with one or more designated top-level rules as entry points.
• Input: A class representing input data (often a sequence of char) that adheres to an informal input interface.
• Action: A class used for semantic processing. It must provide a static apply() or apply0() function. Advanced actions may also provide a static match() function.
• Control: A class that manages behind-the-scenes details of a parsing run.
• States: User-defined objects passed to all rules, actions, and control functions to maintain context.
• Parsing Run: The execution of tao::pegtl::parse() (or tao::pegtl::parse_nested() for nested runs).
• Position: An object indicating a specific location in the input data, often including metadata like filename and line number.

Parsing Outcomes

• Success: Occurs when a rule's match() function returns true.
• Local Failure: Occurs when a match() function returns false, typically triggering backtracking.
• Global Failure: Occurs when a match() function throws an exception, usually aborting the entire parsing run.

To create a more compact AST-like tree, use a selector. A selector determines which grammar rules produce tree nodes.

Custom Selector

Define a template specializing for specific rules:

template< typename Rule >
struct my_selector : std::false_type {};

template<> 
struct my_selector< number > : std::true_type {};

Using tao::pegtl::parse_tree::selector

The selector class template allows combining multiple rule collections using .on< Rules... >. This allows you to specify different behaviors (like storing content or removing it) for different sets of rules.

template< typename Rule >
using my_selector = tao::pegtl::parse_tree::selector<
   Rule,
   tao::pegtl::parse_tree::store_content::on< number, identifier >,
   tao::pegtl::parse_tree::remove_content::on< plus, minus > >;

auto root = tao::pegtl::parse_tree::parse< my_grammar, my_selector >( in );

In this example:

• number and identifier nodes keep their matched input in data.
• plus and minus nodes are structural (they keep children but have empty data).
• Rules not mentioned do not produce nodes, but their selected descendants are attached to the nearest selected ancestor.