PyIceberg Documentation

The PyIceberg Expression DSL is a type-safe Domain Specific Language used to build complex row filter expressions. These expressions are passed to the row_filter argument during a scan to filter data at the source. The DSL is composed of three main building blocks:

1. Terms: References to specific fields in your data (e.g., Reference("field_name")).
2. Predicates: Expressions that evaluate to a boolean value (e.g., EqualTo, In, IsNull).
3. Logical Operators: Operators used to combine predicates (e.g., And, Or, Not).

from pyiceberg.expressions import Reference

# Create a reference to a field named "age"
age_field = Reference("age")

PyIceberg provides a pluggable auth mechanism for REST catalogs. You select the method via auth.type and provide the corresponding configuration block.

Supported types:

• noop: No authentication.
• basic: Uses auth.basic with username and password.
• oauth2: Uses auth.oauth2 with client_id, client_secret, token_url, etc.
• custom: Uses a custom AuthManager defined by auth.impl.
• google: Uses Google service account credentials via auth.google.credentials_path.
• entra: Uses Microsoft Entra ID (Azure AD) via auth.entra (requires pip install pyiceberg[entra-auth]).

catalog:
  default:
    type: rest
    uri: http://rest-catalog/ws/
    auth:
      type: oauth2
      oauth2:
        client_id: my-client-id
        client_secret: my-client-secret
        token_url: https://auth.example.com/oauth/token
        scope: read

The SimpleLocationProvider is the default provider in PyIceberg. It generates paths prefixed by {location}/data/, where {location} is retrieved from the table metadata.

• Non-partitioned tables: Files are placed directly under the data prefix (e.g., s3://bucket/ns/table/data/file.parquet).
• Partitioned tables: Uses Hive-style partition paths, where partition keys and values are included as subdirectories (e.g., s3://bucket/ns/table/data/category=orders/file.parquet).

PyIceberg uses a pluggable FileIO module to handle reading, writing, and deleting files. By default, PyIceberg automatically selects a FileIO implementation based on the URI scheme (e.g., s3://, gs://).

Default Mappings:

• s3://, s3a://, s3n://: PyArrowFileIO or FsspecFileIO (whichever is installed first)
• gs://: PyArrowFileIO
• file://: PyArrowFileIO
• hdfs://: PyArrowFileIO
• abfs://, abfss://: FsspecFileIO
• oss://: PyArrowFileIO
• hf://: FsspecFileIO

You can explicitly set the implementation using the py-io-impl configuration key. If the specified implementation cannot be loaded, PyIceberg will raise an error.

py-io-impl: pyiceberg.io.fsspec.FsspecFileIO

You can perform various comparisons and combine them using logical operators in your row filter strings.

Comparison Operators

• Basic: =, !=, >, >=, <, <=.
• Aliases: == is an alias for =, and <> is an alias for !=.
• NULL Checks: IS NULL and IS NOT NULL.
• NaN Checks (for floating-point): IS NAN and IS NOT NAN.
• Set Membership: IN (...) and NOT IN (...).
• Range: BETWEEN <val1> AND <val2> (inclusive).
• Pattern Matching: LIKE 'pattern%' and NOT LIKE 'pattern%'.

Logical Operators

• AND
• OR
• NOT
• Use parentheses () to group operations and ensure correct precedence.

Note on Precedence: The order of operations is NOT > AND > OR.

-- Combining multiple conditions
(status = 'pending' OR status = 'processing') AND NOT (priority = 'low')

Apache Iceberg uses a LocationProvider to manage file paths for a table's data files and metadata files. PyIceberg's LocationProvider is pluggable, allowing you to customize how file paths are generated for specific use cases.

Key concepts:

• Default Behavior: PyIceberg uses SimpleLocationProvider by default.
• Customization via Properties: You can customize data and metadata locations using the write.data.path and write.metadata.path table properties.
• Granular Control: You can override the new_data_location and new_metadata_location methods of a LocationProvider to implement custom path generation logic.

PyIceberg uses uv.lock to ensure cross-platform dependency consistency.

• Automatic updates: The uv-pre-commit hook automatically updates uv.lock when pyproject.toml changes. If the lockfile is updated, you must add the new uv.lock to your commit.
• Manual updates: To manually synchronize the lockfile after editing pyproject.toml, run uv lock.
• CI Enforcement: The CI system validates that uv.lock is in sync with pyproject.toml. Mismatches will cause CI builds to fail.

After running make install, you must point your IDE to the Python interpreter located at .venv/bin/python.

VS Code:

1. Press Cmd/Ctrl+Shift+P.
2. Select Python: Select Interpreter.
3. Choose .venv/bin/python.

IntelliJ IDEA:

1. Go to File -> Project Structure (⌘;).
2. Navigate to Platform Settings -> SDKs.
3. Select Add Python SDK -> Virtualenv Environment -> Existing environment.
4. Point to .venv/bin/python.

Once on the correct branch (main for major/minor, or the version-specific branch for patches), create a signed GPG tag for the Release Candidate (RC). Replace VERSION and RC with your target values.

export VERSION=0.7.0
export RC=1

export VERSION_WITH_RC=${VERSION}rc${RC}
export GIT_TAG=pyiceberg-${VERSION_WITH_RC}

git tag -s ${GIT_TAG} -m "PyIceberg ${VERSION_WITH_RC}"
git push git@github.com:apache/iceberg-python.git ${GIT_TAG}

To avoid loading entire datasets into memory, tbl.append() and tbl.overwrite() accept a pyarrow.RecordBatchReader. PyIceberg will consume the reader and microbatch it into Parquet files of approximately write.target-file-size-bytes (default 512 MiB).

Note: Streaming writes are currently only supported on unpartitioned tables. For partitioned tables, you must materialize the reader as a pa.Table first.

# reader is a pyarrow.RecordBatchReader
reader = pa.RecordBatchReader.from_batches(schema, batch_iter)
tbl.append(reader)