The rustc-dev-guide

The conversion of source code into a validated AST involves the following key processes:

1. Lexing and Parsing: Converting raw text into a structured format.
2. Macro Expansion: Expanding macros, which may require parsing the resulting output.
3. Name Resolution: Resolving names of macros and imports (required by macro expansion).
4. Conditional Compilation: Handling #[cfg] attributes and similar logic.
5. Feature-gate Checking: Ensuring the code uses allowed language features.
6. AST Validation: Final verification of the AST structure.

The rustdoc tool is split into several parts:

• librustdoc: Contains the bulk of the logic.
• src/tools/rustdoc: The binary that calls rustdoc::main.
• src/tools/rustdoc-js and src/tools/rustdoc-themes: JavaScript and CSS for documentation rendering.
• src/rustdoc-json-types: Type definitions used when generating output with --output-format=json.

The rust-lang/rust repository is a single large Cargo workspace. It is organized into three main directories that house the core components of the Rust ecosystem:

• compiler/: Contains the source code for rustc, organized into many interdependent crates.
• library/: Contains the standard libraries (core, alloc, std, proc_macro, test) and the Rust runtime (backtrace, rtstartup, lang_start).
• tests/: Contains the compiler tests.
• src/: Contains source code for tools like rustdoc, clippy, cargo, the build system, and language documentation.

Monomorphization is the process where the Rust compiler generates a unique copy of a generic function for every concrete type it is used with. This ensures high performance by allowing the compiler to work with specific types rather than pointers to heap-allocated objects (as seen in languages like Java), but it increases compile times and binary size due to the creation of multiple code copies.

For example, using both Vec<u64> and Vec<String> results in two distinct copies of the Vec implementation in the final binary.

Chalk is an experimental trait solver for Rust developed by the Types team. It aims to enable advanced trait system features and bug fixes that are difficult to implement in the current solver, such as Generic Associated Types (GATs) and specialization.

Conceptually, Chalk recasts the Rust trait system as a logic program. It "lowers" Rust code (structs, traits, and impl declarations) into logical inference rules, which are then executed via queries, similar to how a Prolog solver operates. While it uses a more expressive variant than standard Prolog Horn clauses, the underlying mechanism is based on logical inference.

Salsa is a library for incremental recomputation. It allows reusing previous computations to increase efficiency by automatically tracking dependencies between values.

Core Mechanism

1. Base Inputs: Values provided directly to the system (e.g., source files, manifests) that are not computed by Salsa.
2. Derived Values: Intermediate values produced by 'pure' functions. Salsa tracks which inputs were accessed to compute these values.
3. Dependency Graph: Salsa treats computations as a graph of nodes. When an input changes, Salsa only recomputes downstream values until it finds a node that remains unchanged (early termination), preventing unnecessary work.

This model is used extensively in rust-analyzer and chalk.

An eRFC is a variant of the RFC process used for highly complex features where the high-level need is clear, but the detailed specification is too large to settle on upfront.

Instead of a final design, an eRFC outlines a high-level strategy to authorize a period of active experimentation. This allows developers to implement the feature behind a feature gate to gather practical data, which then informs a subsequent formal RFC for stabilization.

While used for major features like coroutines, the explicit "eRFC" label is rarely used today; the project now generally prefers approving a standard RFC for an initial version and iterating via the nightly channel.

Canonicalization is the process of isolating an inference value from its context. It is used to implement [canonical queries][cq] to improve caching and detect cycles during trait resolution.

In practice, this involves replacing unbound inference variables (and free lifetimes) with canonical placeholders (e.g., ?0, ?1, ?2) in a consistent, fixed order. This ensures that two different queries that are structurally identical (even if they use different original inference variables) result in the same canonical form. For example, if ?T and ?U are distinct unbound variables, the type (?T, ?U) and (?U, ?T) both canonicalize to (?0, ?1).