Polars DataFrame Library

Polars is a high-performance DataFrame library designed for manipulating structured data. It is written in Rust and provides interfaces for Python, R, and NodeJS.

Key capabilities include:

• Parallel Execution: Automatically utilizes all available CPU cores.
• Query Optimization: Uses an internal optimizer to determine the most efficient execution path for your queries.
• Out-of-Core Processing: Supports a streaming API to process datasets larger than available RAM.
• Apache Arrow Support: Can consume and produce Arrow data, often using zero-copy operations.
• GPU Support: Optional support for running queries on NVIDIA GPUs.
• Versatile I/O: Supports local storage, cloud storage, and databases.

The polars.DataFrame class is the primary data structure in Polars. It provides a comprehensive suite of methods for data manipulation, organized into several functional categories:

• Aggregation: Methods for summarizing data (e.g., sums, means, counts).
• Attributes: Accessing metadata about the DataFrame (e.g., shape, column names).
• Computation: Performing mathematical or logical operations on the data.
• Descriptive: Statistical summaries of the data distribution.
• Export: Saving the DataFrame to various formats (e.g., CSV, Parquet, JSON).
• Group By: Splitting data into groups for specialized operations.
• Modify/Select: Reshaping the DataFrame by selecting, adding, or removing columns.
• Miscellaneous: General utility methods.
• Plot: Visualizing the data.
• Style: Formatting the DataFrame for display.

Polars is a fast analytical query engine for DataFrames written in Rust. It is designed for high performance using multi-threaded, vectorized (SIMD) execution.

Key capabilities include:

• Lazy & Eager Execution: Supports both immediate execution and lazy query optimization.
• Larger-than-RAM Processing: Uses a streaming engine to process datasets that exceed available memory.
• Expressive API: Allows composing complex queries using powerful expressions.
• Interoperability: Built on the Apache Arrow Columnar Format for zero-copy data sharing.
• Extensibility: Supports custom I/O and Expression plugins.
• Multi-language Support: Available for Python, Rust, Node.js, R, and SQL.
• GPU Acceleration: Optional support for NVIDIA GPUs.

Polars provides a wide range of SQL functions categorized by their purpose. When writing SQL queries against Polars DataFrames, you can use functions from the following groups:

• Aggregate: Functions for summarizing data (e.g., SUM, AVG, COUNT).
• Array: Functions for manipulating array/list types.
• Bitwise: Functions for bitwise operations.
• Conditional: Logic-based functions (e.g., CASE WHEN).
• Mathematical: Standard arithmetic and math functions.
• String: Functions for text manipulation.
• Temporal: Functions for date and time operations.
• Trigonometry: Trigonometric functions.
• Types: Functions for type casting and conversion.
• Window: Functions used with window clauses (e.g., RANK, ROW_NUMBER).

The Polars Python API is organized into several core modules. All classes and functions exposed in the polars.* namespace are public and can be used to build data processing pipelines. The API is categorized into the following functional areas:

• Core Data Structures: DataFrame, LazyFrame, and Series.
• Data Manipulation: expressions, selectors, datatype_expr, and general functions.
• Data Types & Schema: datatypes and schema.
• Input/Output & Integration: io, catalog, sql, and plugins.
• System & Configuration: config, api, and metadata.
• Development & Error Handling: exceptions and testing.

Polars provides a SQL interface that allows you to run SQL queries against DataFrames. The interface is organized into several key areas:

• Python API: How to interact with the SQL engine using Python.
• Query Clauses: Supported SQL clauses (e.g., SELECT, FROM, WHERE, GROUP BY).
• Set Operations: SQL operations for combining result sets (e.g., UNION, INTERSECT).
• Table Operations: Operations performed on tables/DataFrames.
• Functions: A collection of supported SQL functions.
• Operators: Supported SQL operators for comparisons, arithmetic, and logic.

There is a variety of community-maintained plugins available for specialized tasks. Notable categories include:

Various

• polars-xdt: Extra datetime-related functionality.
• polars-hash: Stable non-cryptographic and cryptographic hashing functions.

Data Science

• polars-distance: Pairwise distance functions.
• polars-ds: Numerical and string data analysis procedures.

Bioinformatics

• polars-bio: Genomics library built on Polars, Apache Arrow, and Apache DataFusion.

Geo (Geospatial)

• polars-st: Spatial operations on DataFrames, Series, and Expressions (similar to Shapely/Geopandas).
• polars-reverse-geocode: Offline reverse geocoder for finding cities from (latitude, longitude).
• polars-h3: Support for the H3 discrete global grid system (hexagonal indexing).

Polars provides a Catalog interface to interact with Unity Catalog. This allows you to list catalogs, namespaces, and tables, as well as retrieve table information and scan tables directly into Polars.

Key capabilities include:

• Discovery: Use list_catalogs(), list_namespaces(), and list_tables() to explore the Unity Catalog hierarchy.
• Metadata Inspection: Use get_table_info() to retrieve detailed metadata about a specific table.
• Data Access: Use scan_table() to create a lazy scan of a Unity Catalog table, enabling efficient query execution.

Polars provides a wide range of expression methods for performing aggregations on data. These methods are typically used within group_by contexts or when applying operations to entire columns to reduce multiple values into a single summary statistic.

Common aggregation categories include:

• Counting & Uniqueness: count, len, n_unique, approx_n_unique, null_count.
• Statistical Summaries: sum, mean, median, std, var, product, quantile.
• Extrema: min, max, nan_min, nan_max, arg_min, arg_max, min_by, max_by.
• Logical/Boolean: any, all, bitwise_and, bitwise_or, bitwise_xor, has_nulls.
• Positional: first, last, implode.