Parsimonious Documentation

To improve performance and reduce memory usage, avoid defining the same expression multiple times in your grammar. Instead, define the expression as its own rule and reference that rule wherever it is needed.

Why this matters:

• Caching: Parsimoniuos uses expression object identity for fast cache lookups. Referencing a single rule ensures better cache hits.
• RAM Efficiency: Repeating expressions can lead to increased RAM usage, as the parser may cache an integer for every character in the text for every unique expression object.

Balancing Regex vs. Rules:

• Large Regexes: Jamming more logic into a single regex executes faster because it avoids running Python code between segments.
• Broken-up Rules: Breaking expressions into smaller, reusable rules provides better cache performance if those pieces are used in multiple places.

When using quantifiers like ? (optional) and * (zero or more), place them at the highest possible level in your grammar hierarchy.

Reasoning: If quantifiers are placed at lower levels, the parser might succeed on empty matches, creating unnecessary nodes in your parse tree that do not represent actual content.

To install Parsimonious, use the standard Python package manager:

$ pip install parsimonious

To use Parsimonious, define a grammar using the Grammar class with a string representing your PEG rules. The first rule defined is the default start symbol. You can then call .parse(text) on the grammar instance to generate an Abstract Syntax Tree (AST) of Node objects.

from parsimonious.grammar import Grammar

grammar = Grammar(
    """
    bold_text  = bold_open text bold_close
    text       = ~"[A-Z 0-9]*"i
    bold_open  = "(("
    bold_close = "))"
    """
)

# Returns a tree of Node objects
print(grammar.parse('((bold stuff))'))

To turn a parse tree into a useful representation (like an AST or a string), subclass NodeVisitor.

How it works:

1. Subclass NodeVisitor.
2. Implement methods named visit_<rule_name> for each grammar rule you want to handle. The <rule_name> corresponds to the name of the expression in the grammar.
3. Each visit_<rule_name> method receives two arguments:
   • node: The current Node being visited.
   • visited_children: A list containing the results of calling visit() on each of the node's children.
4. Call visitor.visit(root_node) to start the depth-first traversal.

Note: NodeVisitor does not transform the tree in place. It returns a new representation. If a method is not implemented for a specific rule, generic_visit is called, which raises a NotImplementedError by default.

If you want to perform lexing as a separate pass (e.g., to handle indentation-based languages), use the TokenGrammar class. Instead of operating on raw text strings, TokenGrammar operates on sequences of pre-lexed tokens.

Note that Regex expressions are not supported in TokenGrammar because it operates on tokens rather than individual characters.

Parsimonious provides compound expressions to build complex rules from simpler ones.

• Sequence(*members): A concatenation operator. All members must match in order, one after another.
• OneOf(*members): An alternation operator. The first member that matches wins. It tests members in the order they are provided.

If an error occurs during the tree visitation process (e.g., a VisitationException because a method is missing), Parsimoniuos provides a detailed error report. The report includes:

1. The standard Python traceback.
2. The specific VisitationException message.
3. A visual representation of the parse tree at the point of failure, with a pointer (<-- *** We were here. ***) indicating exactly which node caused the error.

The NodeVisitor class provides an inversion-of-control framework for walking a parse tree and transforming it into a new construct (such as a string, a new tree, or another object).

Implementation Pattern: When implementing visitor methods, you can take advantage of the fact that nodes are iterable. This allows you to use tuple unpacking in your method signatures to capture child nodes directly.

For example, if a production is defined as or_term = "/" "_" term, your visitor method can unpack the children like this:

def visit_or_term(self, or_term, (slash, _, term)):
    ...