chumsky

repository·main·Indexed 26 days ago

https://github.com/zesterer/chumsky

A high-performance parser combinator library for Rust designed for creating expressive and error-tolerant parsers. Suitable for compilers, binary protocols, and embedded systems, it supports `no_std` and zero-copy parsing. Key features include powerful error recovery via `recover_with` and `via_parser`, multiple built-in error types (EmptyErr, Cheap, Simple, Rich) to balance performance and diagnostics, and support for various input types through the `Input` trait.

Tokens
21.3K
Snippets
59
Records
140
Agent score
84%

What's inside chumsky

  1. Introduction to chumsky

    main
    chumsky is a parser combinator library for Rust designed for writing high-performance, expressive parsers. It can be used for both lexing and parsing, and supports advanced features like error recovery and handling recursion.
  2. Overview of Chumsky

    main
    Chumsky is a high-performance parser combinator library for Rust. It is designed for writing expressive parsers for a wide range of inputs, including compilers, binary protocols, configuration files, and complex input validation. It supports no_std for embedded environments and features zero-copy parsing to minimize allocations.
  3. Understand Chumsky's grammar classification

    main
    Chumsky is primarily a PEG (Parsing Expression Grammar) parser, capable of parsing all known context-free grammars. It also provides limited support for context-sensitive parsing, allowing previously parsed elements to inform future parsing steps. This can be achieved using Parser::ignore_with_ctx and Parser::then_with_ctx. For more complex requirements, the library can be extended via custom and ExtParser to support arbitrary grammar logic.
  4. Handle keywords and avoid ambiguity in parsers

    main

    When parsing keywords (like let or fn), use text::ascii::keyword to ensure the parser matches the exact identifier and doesn't accidentally match a longer identifier that happens to start with that keyword.

    To avoid ambiguity between keywords and identifiers (e.g., a variable named let), place the keyword parser earlier in the .or() chain than the identifier parser. Chumsky resolves ambiguity by choosing the first successful parse it encounters.

    let r#let = text::ascii::keyword("let")
        .ignore_then(ident)
        .then_ignore(just('='))
        .then(expr.clone())
        .then_ignore(just(';'))
        .then(decl)
        .map(|((name, rhs), then)| Expr::Let {
            name,
            rhs: Box::new(rhs),
            then: Box::new(then),
        });
    
    r#let.or(expr).padded()
  5. Handle recursive parsers with `recursive`

    main

    Because chumsky parsers are values rather than functions, you cannot implement recursion by having a parser function call itself directly. Doing so results in a recursive opaque type compilation error or a runtime stack overflow.

    To implement a recursive parser, use the recursive combinator. This combinator provides a closure with a parameter that represents the parser being defined, allowing you to refer to it within its own definition without infinite recursion during the parser construction phase.

    use chumsky::prelude::*;
    
    fn a_parser<'src>() -> impl Parser<'src, &'src str, i32> {
        recursive(|a_parser| {
            let int = text::int(10).map(|s: &str| s.parse().unwrap());
    
            let atom = choice((
                int,
                a_parser.delimited_by(just('('), just(')')),
            ))
                .padded();
    
            atom.clone().foldl(
                just('+').padded().ignore_then(atom).repeated(),
                |lhs, rhs| lhs + rhs,
            )
        })
    }
  6. Set up a Chumsky project

    main

    To start a new project with Chumsky, create a new Rust binary project and add chumsky as a dependency. You can use the following boilerplate to read a file from a command-line argument and print its contents.

    use chumsky::prelude::*;
    
    fn main() {
        let src = std::fs::read_to_string(std::env::args().nth(1).unwrap()).unwrap();
    
        println!("{}", src);
    }
  7. Add chumsky as a dependency

    main

    You can add chumsky to your Cargo project using either the CLI or by manually editing your Cargo.toml file.

    Minimum Supported Rust Version (MSRV): 1.65 (due to the use of Generic Associated Types).

    Note: The nightly feature is exempt from this MSRV and may require the latest nightly Rust compiler.

    $ cargo add chumsky
  8. Write pure closures for parser optimizations

    main

    Chumsky employs performance optimizations that may skip generating output values that go unused (e.g., the output of a in a.ignore_then(b)). This includes potentially optimizing away calls to combinators like Parser::map.

    Requirement: Any closures or functions used inline within a parser should be semantically pure. You should not assume a closure is called any specific number of times. While side effects are permitted, they must be irrelevant to the correct functioning of the parser.

    Example of acceptable impurity: String interning within Parser::map_with is acceptable because interning a string that is ultimately unused does not affect the parser's correctness.

  9. Optimize compilation times for large parsers

    main

    Heavy use of the type system can lead to long compilation times. Use these strategies to improve performance:

    1. Avoid long .or() chains: Long chains of Parser::or can cause exponential behavior in the Rust trait solver. Replace them with choice(), which provides identical behavior but is much easier for the compiler to process.
    2. Use .boxed() for type erasure: For long parser chains, call .boxed() at the end. This performs an allocation during parser creation (not during parsing) to reduce the complexity of the type the compiler must understand. This can significantly improve compilation times and may even improve runtime performance.

    Note: .boxed() does not impact the performance of the actual parsing process, only the creation of the parser instance.

  10. Manage shared mutable state in parsers

    main

    While Chumsky parsers are conceptually stateless pure functions, you can provide shared mutable state for tasks like string interning, arena allocation, or lossless syntax tree tracking.

    To use shared state:

    1. Define your state type (e.g., using SimpleState from chumsky::inspector).
    2. Include the state in your parser's extra type parameters using extra::Full.
    3. Use Parser::parse_with_state or Parser::check_with_state to execute the parser.
    4. Access the state within parser combinators using methods like Parser::map_with via the extra parameter.

    Warning on Purity: Do not rely on state being touched a specific number of times. Chumsky may optimize out closures or invoke them an arbitrary number of times during backtracking/rewinding. Do not use state to 'count' occurrences of patterns.

    use chumsky::{prelude::*, inspector::SimpleState};
    use std::collections::HashMap;
    
    // 1. Define the extra type parameters including the state
    type MyExtra = extra::Full<EmptyErr, SimpleState<HashMap<String, usize>>, ()>;
    
    // 2. Include the extra type in the Parser signature
    fn my_parser<'a>() -> impl Parser<'a, &'a str, Vec<usize>, MyExtra> {
        text::ident()
            .map_with(|s: &str, e| {
                // 3. Access the state via the extra parameter 'e'
                let state: &mut SimpleState<HashMap<_, _>> = e.state();
                let id = state.len();
                *state.entry(s.to_string()).or_insert(id)
            })
            .padded()
            .repeated()
            .collect()
    }
    
    // 4. Use parse_with_state to provide the initial state
    let mut intern_table = HashMap::new();
    let idents = my_parser()
        .parse_with_state("the rabbit saw the other rabbit", &mut intern_table)
        .into_result()
        .unwrap();
    
    assert_eq!(idents[0], idents[3]); // 'the' matches