FastParse Documentation

The pythonparse.Expressions object provides a set of fastparse parsers for Python's expression grammar. These parsers are designed to handle components that can be used within larger expressions, such as literals, operators, and complex structures like list comprehensions or lambdas.

Key Characteristics:

• Whitespace/Indentation: The parsers ignore whitespace and do not rely on indentation (they are designed for expression-level parsing, not full statement/block parsing).
• AST Output: Most parsers return an Ast.expr or related type, representing the parsed structure as an Abstract Syntax Tree.
• Grammar Source: The grammar is manually transcribed from the official Python documentation.

import pythonparse.Expressions
import fastparse._

// Example of how one might use the 'test' parser to parse a Python expression
val input = "x + 1 if True else y"
val result = parse(input, Expressions.test)
// result will contain the parsed Ast.expr structure

FastParse provides several pre-defined Whitespace implementations that can be used as implicit whitespace handlers in your parsers. These strategies determine how the parser skips whitespace and handles comments.

To use one, you can provide the desired object as an implicit value in the scope where your parser is defined. Common strategies include:

• NoWhitespace: Consumes nothing.
• SingleLineWhitespace: Consumes only spaces ( ) and tabs (\t).
• MultiLineWhitespace: Consumes spaces, tabs, carriage returns (\r), and newlines (\n).
• ScriptWhitespace: Supports # line comments (like Bash or Python).
• JavaWhitespace: Supports // line comments and /* */ multiline comments (non-nesting).
• JsonnetWhitespace: Supports both # and // line comments, and /* */ multiline comments (non-nesting).
• ScalaWhitespace: Supports // line comments and /* */ multiline comments with nesting support.

import fastparse.Whitespace._

// Example: Using ScalaWhitespace implicitly
// This will allow your parser to skip spaces, newlines, // comments, and nested /* comments */
object MyParser extends RecursiveParser {
  implicit val whitespace: Whitespace = ScalaWhitespace

  val number = parsers.number
  val expression = number ~ number
}

The atom parser handles the most basic building blocks of expressions, including collections and parenthesized expressions:

• Empty collections: (), [], {}
• Parenthesized expressions: (expr)
• List comprehensions/lists: [x for x in xs] or [1, 2, 3]
• Dict/Set comprehensions/literals: {k: v for k, v in x} or {1, 2, 3}
• Tuples: (1, 2, 3)
• Strings and Names: Literal strings and variable names.

def atom[$: P]: P[Ast.expr] = {
    def empty_tuple = ("(" ~ ")").map(_ => Ast.expr.Tuple(Nil, Ast.expr_context.Load))
    def empty_list = ("[" ~ "]").map(_ => Ast.expr.List(Nil, Ast.expr_context.Load))
    def empty_dict = ("{" ~ "}").map(_ => Ast.expr.Dict(Nil, Nil))
    P(
      empty_tuple  |
      empty_list |
      empty_dict |
      "(" ~ (yield_expr | generator | tuple | test) ~ ")" |
      "[" ~ (list_comp | list) ~ "]" |
      "{" ~ dictorsetmaker ~ "}" |
      "`" ~ testlist1.map(x => Ast.expr.Repr(Ast.expr.Tuple(x, Ast.expr_context.Load))) ~ "`" |
      STRING.rep(1).map(_.mkString).map(Ast.expr.Str.apply) |
      NAME.map(Ast.expr.Name(_, Ast.expr_context.Load)) |
      NUMBER
    )
}

FastParse provides several built-in combinators for common parsing tasks:

• Start: Succeeds only if the current index is 0.
• End: Succeeds only if the parser has reached the end of the input.
• IgnoreCase(s: String): Parses the string s case-insensitively.
• Pass: A no-op parser that always succeeds without consuming any characters. Can optionally return a value: Pass(value).
• Fail: A no-op parser that always fails. Can take a custom error message: Fail("error message").
• Index: Returns the current index in the input as an Int. Useful for capturing source locations.
• AnyChar: Consumes and succeeds on any single character, provided the input is not at its end.
• SingleChar: Consumes and returns the single character parsed. Useful for one-character lookahead patterns.
• P0: A type alias for P[Unit] (parsers that return no value).

import fastparse._

// Example of using combinators in a parser
def myParser: P[Unit] = {
  Start ~ IgnoreCase("Hello") ~ SingleChar ~ End
}