Pilota

Pilota-Build is a code generation framework designed to generate Rust code from Interface Definition Language (IDL) files. It supports both Thrift and Protobuf protocols through a unified intermediate representation (IR) and a modular architecture.

Key Design Goals

• Multi-protocol support: Unified architecture for Thrift and Protobuf.
• High-performance compilation: Utilizes the Salsa incremental compilation framework to improve efficiency.
• Extensibility: A plugin system allows for custom code generation behaviors.
• Workspace support: Native support for Cargo workspace modes.
• Type safety: Uses a strongly-typed intermediate representation to ensure the correctness of generated code.

Core Features

• Incremental compilation: Only re-compiles modified parts.
• Parallel processing: Supports parallel parsing and code generation.
• Smart dependency management: Automatically handles cross-file and cross-package dependencies.
• Flexible code generation: Supports file splitting and custom naming.

The data flow within Pilota-Build follows a pipeline from raw IDL files to generated Rust files:

1. Input & Parsing

• IDL Files (.thrift or .proto) are read.
• The Parser converts IDL syntax into an Abstract Syntax Tree (AST).
• The AST is lowered into the initial IR (Intermediate Representation), generating unresolved symbol references.

2. Resolution & Transformation

• The Resolver builds a symbol table and resolves all symbol references.
• This process produces the RIR (Resolved IR).
• RIR types are converted into CodegenTy (Code Generation Types), handling special cases like recursive types and applying type tags/attributes.

3. Plugin & Generation

• Plugins are traversed; they can modify the RIR or collect additional code.
• The Codegen stage selects the appropriate Backend to generate Rust code.
• The final code is formatted and written to Rust Files.

Pilota-Build uses a multi-stage type system to transition from raw IDL definitions to concrete Rust types. This ensures lazy resolution and type safety.

Type Hierarchy

1. IDL Types: The raw types defined in the source file.
2. IR Types (Initial): Unresolved representations.
   • TyKind includes base types (String, Bool, I32), container types (Vec, Set, Map), and unresolved Path references.
3. RIR Types (Resolved): Types after symbol resolution.
   • Ty contains a TyKind and a TagId.
   • Path contains a DefKind (Type/Value/Mod) and a DefId (the unique identifier for the resolved definition).
4. CodegenTy (Code Generation Types): The final mapping to Rust.
   • Base Types: e.g., FastStr $\rightarrow$ ::pilota::FastStr, String $\rightarrow$ ::std::string::String.
   • Containers: e.g., Vec $\rightarrow$ ::std::vec::Vec<T>, Map $\rightarrow$ ::pilota::AHashMap<K,V>.
   • ADTs (Algebraic Data Types): Includes Struct, Enum, and NewType.

Key Characteristics

• Lazy Resolution: Path references are only converted to specific definitions during the resolution phase.
• Type Tagging: Supports custom attributes and behaviors via TagId.
• Recursive Type Handling: Automatically detects and manages recursive type structures.

// IR 中的类型（未解析）
pub enum TyKind {
    Path(Path),  // Path { segments: ["User", "Status"] }
    // ...
}

// RIR 中的类型（已解析）
pub struct Path {
    pub kind: DefKind,  // Type/Value/Mod
    pub did: DefId,     // 定义的唯一标识符
}

// 代码生成类型
pub enum CodegenTy {
    Adt(AdtDef),  // AdtDef { did: DefId, kind: Struct/Enum/NewType }
    // ...
}

Pilota-Build uses two levels of intermediate representation to decouple parsing from code generation:

1. IR (Initial Intermediate Representation): A language-agnostic representation of the raw IDL. Core types include ItemKind (Message, Enum, Service, NewType, Const, Mod, Use) and Path (unresolved segments).
2. RIR (Resolved Intermediate Representation): Produced by the Resolver after symbol resolution and path calculation. It represents the fully resolved structure of the IDL.

The type system also distinguishes between Ty (the RIR type) and CodegenTy (the type used during the actual code generation phase, such as String, Vec, or Map).

// Top-level items in IR
pub enum ItemKind {
    Message(Message),
    Enum(Enum),
    Service(Service),
    NewType(NewType),
    Const(Const),
    Mod(Mod),
    Use(Use),
}

// Types in RIR
pub struct Ty {
    pub kind: TyKind,
    pub tags_id: TagId,
}

// Types used during Codegen
pub enum CodegenTy {
    FastStr,
    String,
    Vec(Arc<CodegenTy>),
    Map(Arc<CodegenTy>, Arc<CodegenTy>),
    Adt(AdtDef),
}

The Pilota-Build architecture is organized into several distinct layers, each with specific responsibilities:

1. User Interface Layer: Provides a concise Builder API that hides internal complexity.
2. Extension Layer: Contains the Plugin System for user-defined behaviors.
3. Code Generation Layer: Converts the intermediate representation into target Rust code (includes Thrift Backend, Protobuf Backend, and Workspace Generator).
4. Middle Layer: Handles core business logic, including symbol resolution and the type system (Context Manager, Resolver).
5. Intermediate Representation (IR) Layer: A language-agnostic unified representation. The flow is: IR $\rightarrow$ Resolver $\rightarrow$ RIR $\rightarrow$ CodegenTy.
6. Parsing Layer: Parses IDL files into the initial IR (Thrift Parser, Protobuf Parser).
7. Data Storage Layer: Uses Salsa Database to provide incremental computation capabilities.

Pilota-Build implements incremental compilation using the Salsa framework.

• RootDatabase: A Salsa-backed database that stores nodes, files, and derived queries.
• Incremental Updates: When input files change, Salsa automatically invalidates only the affected parts of the dependency graph.
• Efficiency: This allows for parallel parsing, cached symbol tables, and lazy evaluation of type information, ensuring that only necessary re-computations occur during the build process.

#[salsa::database(RirDatabase)]
pub struct RootDatabase {
    storage: salsa::Storage<RootDatabase>,
}

pub trait RirDatabase {
    #[salsa::input]
    fn nodes(&self) -> Arc<FxHashMap<DefId, rir::Node>>;
    
    #[salsa::input]
    fn files(&self) -> Arc<FxHashMap<FileId, Arc<rir::File>>>;
    
    fn node(&self, def_id: DefId) -> Option<rir::Node>;
    fn item(&self, def_id: DefId) -> Option<Arc<rir::Item>>;
    fn codegen_ty(&self, def_id: DefId) -> CodegenTy;
}