DOCSEARCHbyAlgolia
Submit Docs

LibCST Documentation

repository·main·Indexed 23 days ago

https://github.com/instagram/libcst

A lossless Concrete Syntax Tree (CST) parser and serializer for Python 3.0 through 3.15. LibCST is designed for automated refactoring (codemods) and linters, preserving original formatting, comments, and whitespace. It includes a high-performance Rust-based native extension using a PEG parser, a system for creating custom codemods via VisitorBasedCodemodCommand, and utilities for tree traversal and transformation.

Tokens
35.4K
Snippets
57
Records
179
Agent score
83%

What's inside LibCST

  1. Use the LibCST native extension for parsing

    main

    The libcst.native module is a Rust-based extension that enables high-performance parsing of Python grammar using a PEG parser. It is packaged with LibCST and is used by default by the standard LibCST APIs.

    Key features include:

    • Feature parity with the pure-python parser (supports target versions, strings, and bytes).
    • High performance (aiming for within 2x CPython performance).
    • Uses a PEG parser to closely follow the CPython grammar definition.

    While the core logic is in Rust, the Python wrapper handles input conversion (e.g., converting bytes to UTF-8 strings) before passing them to the Rust layer.

  2. What is LibCST and how does it differ from AST and CST?

    main

    LibCST is a hybrid format that combines the benefits of Abstract Syntax Trees (AST) and Concrete Syntax Trees (CST):

    • Like an AST: It parses source code into nodes that represent the semantics of the code (e.g., Call, Name, Integer).
    • Like a CST: It preserves all whitespace and formatting, allowing the tree to be reprinted exactly to match the original source.

    LibCST achieves this by using an internal whitespace parser to assign whitespace ownership to specific nodes (e.g., using a Comma node to manage the whitespace surrounding a comma). This makes complex code manipulations easier because the semantic structure and the formatting structure are both explicitly represented in the tree.

  3. What is LibCST and how does it work?

    main

    LibCST is a Concrete Syntax Tree (CST) parser and serializer library for Python (supporting Python 3.0 through 3.15).

    Unlike a standard Abstract Syntax Tree (AST), a CST is lossless: it preserves all formatting details, including comments, whitespaces, parentheses, and other non-semantic information. This makes it ideal for building automated refactoring (codemod) applications and linters where maintaining code style is critical.

    LibCST aims to provide a compromise between an AST and a traditional CST by reorganizing and naming node types and fields so that the lossless CST feels and behaves like an AST.

  4. How LibCST metadata works

    main

    LibCST provides a standardized, declarative, and type-safe way to associate arbitrary metadata with nodes in a CST while maintaining tree immutability.

    To use metadata, you follow this pattern:

    1. Wrap the module: Use libcst.metadata.MetadataWrapper to wrap your parsed CST module.
    2. Define dependencies: If using a visitor, your visitor class should extend libcst.metadata.MetadataDependent and declare its required metadata providers in the METADATA_DEPENDENCIES attribute.
    3. Access metadata: Inside your visitor methods (e.g., visit_Name), use self.get_metadata(ProviderClass, node) to retrieve the data.
    4. Execute: Call wrapper.visit(your_visitor) to trigger the metadata computation and visitation.
    class NamePrinter(cst.metadata.MetadataDependent):
    METADATA_DEPENDENCIES = (cst.metadata.PositionProvider,)

    def visit_Name(self, node: cst.Name) -> None:
        pos = self.get_metadata(cst.metadata.PositionProvider, node).start
        print(f"{node.value} found at line {pos.line}, column {pos.column}")

wrapper = cst.metadata.MetadataWrapper(cst.parse_module("x = 1"))
result = wrapper.visit(NamePrinter())

  5. How LibCST tree traversal is designed

    main

    LibCST trees are optimized for ease of traversal using three main design patterns:

    1. Flat Structure: LibCST avoids unnecessary wrapper nodes. For example, instead of an AsyncFunction wrapper around a FunctionDef, it uses a FunctionDef node with an async attribute. Parentheses are attached to the expressions they operate on rather than being separate wrapper nodes.
    2. Regularity: The tree structure is kept consistent. For example, a Module always contains a list of statements, even if that list is empty or contains only one item.
    3. High-Level Abstraction: The tree is designed to be close to the Python AST. Traversal should focus on semantic operations rather than syntactic trivia (like manually handling commas or ignoring parentheses).

  6. Understand the distinction between CSTNode methods and libcst.helpers

    main

    LibCST provides helpers to reduce boilerplate, categorized by where they are attached:

    • CSTNode methods: These are simple, read-only methods attached directly to nodes. They only require data from the direct children of the node.
    • libcst.helpers package functions: These are standalone functions used for more complex operations, such as node transformations or tasks requiring recursive traversal of the syntax tree.

  7. How parser execution order works

    main

    The parser uses a bottom-up approach (via pgen2), meaning child productions are converted before their parent productions. Within a single production, child conversion functions are called from left to right.

    For a grammar like add_expr: NUMBER ['+' add_expr], the conversion order follows the parse tree from the leaves up to the root (e.g., the first NUMBER is converted first, and the final add_expr is converted last).

  8. Understand LibCST Nodes and the CST Structure

    main

    LibCST represents Python's full grammar in a whitespace-sensitive fashion through a Concrete Syntax Tree (CST). The tree is composed of CSTNode objects and their subclasses. Unlike an Abstract Syntax Tree (AST), the CST preserves all syntactic details, including whitespace and comments.

    Key structural components include:

    • Module: The top-level node representing an entire Python module (libcst.Module).
    • Expressions: Nodes representing values (e.g., libcst.BaseExpression). These can be parsed individually using libcst.parse_expression or as part of a larger structure.
    • Statements: Nodes representing a line of code or control structures (e.g., libcst.If). Statements are categorized into Simple Statements (which contain expressions) and Compound Statements (which contain blocks of other statements).
    • Whitespace/Trivia: Nodes like libcst.Newline or libcst.Comment that encapsulate the formatting of the source code.

  9. How CST nodes are structured in the native extension

    main

    In the Rust implementation, every CST node (e.g., Foo) is managed via a dual-node pattern using the #[cst_node] proc macro:

    • DeflatedFoo: The intermediate output of the parsing phase. It contains TokenRef fields (but not whitespace) and uses two lifetime parameters: 'r (or 'input) for Token references and 'a for original input string slices. It implements the Inflate trait.
    • Foo: The public-facing CST node produced after inflation. It retains the 'a lifetime to refer to input string slices and contains whitespace fields, but does not contain TokenRefs. It implements IntoPy to allow translation back into Python objects.

    This separation allows the parser to work efficiently with references to the original input without unnecessary string allocations.

  10. Understand the difference between AST and CST

    main

    LibCST is designed to bridge the gap between Abstract Syntax Trees (AST) and traditional Concrete Syntax Trees (CST).

    Abstract Syntax Trees (AST)

    Python's built-in ast module produces trees that focus on the semantics of the code. While excellent for compilers and type checkers, ASTs are lossy. They discard information necessary to reconstruct the exact original source code, such as:

    • Comments
    • Specific newline characters (\n, \r, or \r\n)
    • Exact whitespace between tokens

    Concrete Syntax Trees (CST)

    A CST is lossless. It retains all information required to reprint the exact input code. This is achieved by storing whitespace and comment information (often in prefix properties).

    Traditional CST libraries (like lib2to3) can be difficult to use for complex code transformations because:

    • They follow the formal grammar closely, making semantic extraction harder.
    • Managing nodes like COMMA or whitespace ownership is error-prone when adding or removing elements.

    LibCST

    LibCST provides a CST that is designed to be easier to manipulate for complex operations while remaining lossless, making it suitable for tools that need to perform code transformations (refactoring) without losing formatting or comments.

  11. How Codemods work in LibCST

    main

    A Codemod is an automated refactor designed to be applied to codebases of any size. LibCST provides a framework for building higher-order transforms composed of simpler, individual transforms.

    All codemods derive from the libcst.codemod.Codemod base class, which provides:

    • A execution context
    • Automatic metadata resolution
    • Support for multi-pass transforms

    For LibCST-compatible visitors and transforms, you should use ContextAwareTransformer or ContextAwareVisitor. Note that ContextAwareTransformer is still a Codemod and must be executed via the transform_module interface.

  12. How LibCST handles node modification and whitespace

    main

    LibCST is built for safe and easy modification through several mechanisms:

    • Strong Typing and Constraints: All nodes are fully typed, and runtime constraints prevent the construction of invalid nodes (e.g., a Name node cannot be created with an invalid Python identifier). This ensures that multiple passes can be performed safely without needing to re-parse the code after every change.
    • Sane Defaults and Semantic Focus: When constructing nodes, you can specify only the semantics. You do not need to manually supply whitespace or commas unless you want to control them explicitly.
    • Intelligent Whitespace Ownership: Whitespace and comments are owned by the nodes they belong to. For example, a statement owns the comments directly above it and any trailing comments on the same line. Deleting a statement automatically removes its associated whitespace/comments.
    • Localized Changes: Syntactic trivia (like commas or spacing) is treated as a child of the node it belongs to. This allows you to change or replace a single field in a node without needing to manually fix up adjacent nodes.
    • Reparenting: Nodes are designed to be easily moved or copied from one part of the tree to another.