Apache DataFusion

The create-wasm-app template provides a pre-configured environment for WebAssembly development including:

• Bundling: Configured with webpack, webpack-cli, and webpack-dev-server via webpack.config.js.
• Boilerplate: Includes index.html (bare bones HTML), index.js (example of importing/using a WASM package), and .gitignore.
• Scripts: A start script is defined in package.json to run the webpack-dev-server.
• Licensing: Includes LICENSE-APACHE and LICENSE-MIT templates.

The datasource submodule provides common components used for data source integration in Apache DataFusion. It includes definitions for types such as FileScanConfig and FileCompression.

Note for users: Most developers should use the main [datafusion] crate directly, as it re-exports this module. You only need to depend on this submodule specifically if you have a specialized requirement that necessitates direct access to these components outside of the standard datafusion crate interface.

The datafusion-ffi crate provides a stable interface for interoperability between different versions of DataFusion or between DataFusion and other libraries. This is necessary because Rust lacks a stable ABI (Application Binary Interface). By using this FFI, you can load libraries at runtime (e.g., community-led extensions or proprietary data sources) without requiring all code to be compiled into a single executable.

Key Use Cases:

• Providing external services like a TableProvider to datafusion-python without requiring the full Python codebase as a dependency.
• Creating modular interfaces for runtime loading of community extensions (e.g., datafusion-contrib).

DataFusion supports several types of User Defined Functions (UDFs) that can be integrated into the execution engine. Depending on your use case, you can implement one of the following:

• Scalar: Takes a row of data and returns a single value.
• Scalar (async): A scalar function designed for performing async operations, such as network or I/O calls.
• Window: Takes a row of data and returns a single value, with access to surrounding rows.
• Aggregate: Takes a group of rows and returns a single value.
• Table: Takes parameters and returns a TableProvider to be used within a query plan.

The session module defines the session-related APIs and extension points for the DataFusion query engine. A session acts as the runtime context for query execution, managing:

• Configuration
• Runtime environment
• Function registry
• Planning

While this crate defines the shared interfaces, concrete query-engine implementations are found in higher-level DataFusion crates.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It provides a full query planner and a columnar, streaming, multi-threaded, vectorized execution engine.

Key capabilities include:

• APIs: SQL and DataFrame APIs.
• Built-in Formats: Support for CSV, Parquet, JSON, and Avro.
• Extensibility: Customization of data sources, query languages, functions, and operators.

Related subprojects for end-users:

• DataFusion Python: Python interface for SQL and DataFrame queries.
• DataFusion Java: Java interface for SQL and DataFrame queries.
• DataFusion Comet: Accelerator for Apache Spark.
• DataFusion Ballista: Distributed query execution engine for clusters.

DataFusion provides an extension system to customize SQL parsing and planning without modifying the core codebase. This allows developers to:

• Support custom operators from other SQL dialects (e.g., PostgreSQL's -> for JSON).
• Add custom data types not natively supported.
• Implement custom SQL constructs such as TABLESAMPLE, PIVOT/UNPIVOT, or MATCH_RECOGNIZE.