chumsky
repository·main·Indexed 26 days ago
https://github.com/zesterer/chumskyA high-performance parser combinator library for Rust designed for creating expressive and error-tolerant parsers. Suitable for compilers, binary protocols, and embedded systems, it supports `no_std` and zero-copy parsing. Key features include powerful error recovery via `recover_with` and `via_parser`, multiple built-in error types (EmptyErr, Cheap, Simple, Rich) to balance performance and diagnostics, and support for various input types through the `Input` trait.
What's inside chumsky
- chumsky is a parser combinator library for Rust designed for writing high-performance, expressive parsers. It can be used for both lexing and parsing, and supports advanced features like error recovery and handling recursion.
Overview of Chumsky
mainChumsky is a high-performance parser combinator library for Rust. It is designed for writing expressive parsers for a wide range of inputs, including compilers, binary protocols, configuration files, and complex input validation. It supportsno_stdfor embedded environments and features zero-copy parsing to minimize allocations.Understand Chumsky's grammar classification
mainChumsky is primarily a PEG (Parsing Expression Grammar) parser, capable of parsing all known context-free grammars. It also provides limited support for context-sensitive parsing, allowing previously parsed elements to inform future parsing steps. This can be achieved usingParser::ignore_with_ctxandParser::then_with_ctx. For more complex requirements, the library can be extended viacustomandExtParserto support arbitrary grammar logic.Handle keywords and avoid ambiguity in parsers
mainWhen parsing keywords (like
letorfn), usetext::ascii::keywordto ensure the parser matches the exact identifier and doesn't accidentally match a longer identifier that happens to start with that keyword.To avoid ambiguity between keywords and identifiers (e.g., a variable named
let), place the keyword parser earlier in the.or()chain than the identifier parser. Chumsky resolves ambiguity by choosing the first successful parse it encounters.let r#let = text::ascii::keyword("let") .ignore_then(ident) .then_ignore(just('=')) .then(expr.clone()) .then_ignore(just(';')) .then(decl) .map(|((name, rhs), then)| Expr::Let { name, rhs: Box::new(rhs), then: Box::new(then), }); r#let.or(expr).padded()Handle recursive parsers with `recursive`
mainBecause chumsky parsers are values rather than functions, you cannot implement recursion by having a parser function call itself directly. Doing so results in a
recursive opaque typecompilation error or a runtime stack overflow.To implement a recursive parser, use the
recursivecombinator. This combinator provides a closure with a parameter that represents the parser being defined, allowing you to refer to it within its own definition without infinite recursion during the parser construction phase.use chumsky::prelude::*; fn a_parser<'src>() -> impl Parser<'src, &'src str, i32> { recursive(|a_parser| { let int = text::int(10).map(|s: &str| s.parse().unwrap()); let atom = choice(( int, a_parser.delimited_by(just('('), just(')')), )) .padded(); atom.clone().foldl( just('+').padded().ignore_then(atom).repeated(), |lhs, rhs| lhs + rhs, ) }) }Set up a Chumsky project
mainTo start a new project with Chumsky, create a new Rust binary project and add
chumskyas a dependency. You can use the following boilerplate to read a file from a command-line argument and print its contents.use chumsky::prelude::*; fn main() { let src = std::fs::read_to_string(std::env::args().nth(1).unwrap()).unwrap(); println!("{}", src); }Add chumsky as a dependency
mainYou can add chumsky to your Cargo project using either the CLI or by manually editing your
Cargo.tomlfile.Minimum Supported Rust Version (MSRV): 1.65 (due to the use of Generic Associated Types).
Note: The
nightlyfeature is exempt from this MSRV and may require the latest nightly Rust compiler.$ cargo add chumskyWrite pure closures for parser optimizations
mainChumsky employs performance optimizations that may skip generating output values that go unused (e.g., the output of
aina.ignore_then(b)). This includes potentially optimizing away calls to combinators likeParser::map.Requirement: Any closures or functions used inline within a parser should be semantically pure. You should not assume a closure is called any specific number of times. While side effects are permitted, they must be irrelevant to the correct functioning of the parser.
Example of acceptable impurity: String interning within
Parser::map_withis acceptable because interning a string that is ultimately unused does not affect the parser's correctness.Import the chumsky prelude
mainTo avoid manually importing every type, trait, and function, use the prelude to bring all commonly used items into scope:
use chumsky::prelude::*;Optimize compilation times for large parsers
mainHeavy use of the type system can lead to long compilation times. Use these strategies to improve performance:
- Avoid long
.or()chains: Long chains ofParser::orcan cause exponential behavior in the Rust trait solver. Replace them withchoice(), which provides identical behavior but is much easier for the compiler to process. - Use
.boxed()for type erasure: For long parser chains, call.boxed()at the end. This performs an allocation during parser creation (not during parsing) to reduce the complexity of the type the compiler must understand. This can significantly improve compilation times and may even improve runtime performance.
Note:
.boxed()does not impact the performance of the actual parsing process, only the creation of the parser instance.- Avoid long
View the Chumsky guide online
mainThe comprehensive Chumsky guide is hosted on docs.rs. For the most up-to-date and interactive documentation, visit the official guide page.
https://docs.rs/chumsky/latest/chumsky/guide/index.htmlManage shared mutable state in parsers
mainWhile Chumsky parsers are conceptually stateless pure functions, you can provide shared mutable state for tasks like string interning, arena allocation, or lossless syntax tree tracking.
To use shared state:
- Define your state type (e.g., using
SimpleStatefromchumsky::inspector). - Include the state in your parser's
extratype parameters usingextra::Full. - Use
Parser::parse_with_stateorParser::check_with_stateto execute the parser. - Access the state within parser combinators using methods like
Parser::map_withvia theextraparameter.
Warning on Purity: Do not rely on state being touched a specific number of times. Chumsky may optimize out closures or invoke them an arbitrary number of times during backtracking/rewinding. Do not use state to 'count' occurrences of patterns.
use chumsky::{prelude::*, inspector::SimpleState}; use std::collections::HashMap; // 1. Define the extra type parameters including the state type MyExtra = extra::Full<EmptyErr, SimpleState<HashMap<String, usize>>, ()>; // 2. Include the extra type in the Parser signature fn my_parser<'a>() -> impl Parser<'a, &'a str, Vec<usize>, MyExtra> { text::ident() .map_with(|s: &str, e| { // 3. Access the state via the extra parameter 'e' let state: &mut SimpleState<HashMap<_, _>> = e.state(); let id = state.len(); *state.entry(s.to_string()).or_insert(id) }) .padded() .repeated() .collect() } // 4. Use parse_with_state to provide the initial state let mut intern_table = HashMap::new(); let idents = my_parser() .parse_with_state("the rabbit saw the other rabbit", &mut intern_table) .into_result() .unwrap(); assert_eq!(idents[0], idents[3]); // 'the' matches- Define your state type (e.g., using