repository·main·Indexed 19 days ago
https://github.com/apache/iceberg-goA Golang implementation of the Apache Iceberg table specification. It provides programmatic interfaces and a CLI to interact with Iceberg tables, supporting multiple catalogs (REST, Hive, Glue, SQL, Hadoop) and filesystems (S3, GCS, Azure Blob Storage, Local). Key capabilities include metadata operations, write operations (append, rewrite, overwrite), and management of namespaces, tables, branches, and tags.
Apache Iceberg has multiple official implementations across different languages. While the table format and catalog protocols are identical across all implementations, you should choose the one that matches your runtime environment. All implementations target the same core Iceberg specification.
| Project | Language | Repository | Documentation |
|---|---|---|---|
| Apache Iceberg | Java (reference) | [apache/iceberg](https://github.com/apache/iceberg) | [iceberg.apache.org](https://iceberg.apache.org/) |
| PyIceberg | Python | [apache/iceberg-python](https://github.com/apache/iceberg-python) | [py.iceberg.apache.org](https://py.iceberg.apache.org/) |
| iceberg-rust | Rust | [apache/iceberg-rust](https://github.com/apache/iceberg-rust) | [rust.iceberg.apache.org](https://rust.iceberg.apache.org/) |
| iceberg-cpp | C++ | [apache/iceberg-cpp](https://github.com/apache/iceberg-cpp) | [cpp.iceberg.apache.org](https://cpp.iceberg.apache.org/) (early stage) |Partitioning logically divides table data to improve query performance via selective reading.
partition_name=value/).| Transform | Description |
|---|---|
identity | Use the column value directly |
bucket[n] | Hash the value into n buckets |
truncate[n] | Truncate strings to n characters |
year | Extract year from date/timestamp |
month | Extract month from date/timestamp |
day | Extract day from date/timestamp |
hour | Extract hour from timestamp |
void | Always returns null (for unpartitioned tables) |
Select an implementation based on your primary language and ecosystem:
pyiceberg-core, DataFusion-based engines, and other Rust-native systems.A term represents the left-hand side of a predicate.
iceberg.Reference("column_name") to name a column. This is the standard way to build expressions; typing is resolved later during the binding phase against a schema.Reference.Bind(schema, caseSensitive). You typically only interact with these when writing custom expression visitors.// Create an unbound term (most common use case)
term := iceberg.Reference("column_name")Iceberg allows modifying a table's schema over time while maintaining backward compatibility:
int32 to int64).The Iceberg ecosystem relies on several key abstractions to manage data and metadata:
When building filters dynamically (e.g., in a loop or conditional logic), use iceberg.AlwaysTrue{} and iceberg.AlwaysFalse{} as identity elements.
AlwaysTrue{} as the starting point when folding multiple clauses with NewAnd.AlwaysFalse{} as the starting point when folding multiple clauses with NewOr.iceberg.AlwaysTrue{}
iceberg.AlwaysFalse{}Apache Iceberg Go supports table format versions 1, 2, and 3. The maximum supported version is enforced in table/metadata.go (supportedTableFormatVersion = 3).
All V1 features are supported. V1 serves as the baseline format version.
Supported features include:
Transaction.WriteEqualityDeletes; row-level commits via Transaction.NewRowDelta.Note: ReplaceDataFiles using OpReplace is currently pending.
Supported features include:
timestamp_ns, timestamptz_ns)initial-default, write-default)_row_id, _last_updated_sequence_number)Note: Variant type shredded reader/writer, partitioned deletion vector writes, and Geometry/Geography transforms/statistics/pruning are currently in progress.
Concurrency can be controlled globally via the CLI config, or overridden per-operation in code:
max-workers in ~/.iceberg-go.yaml (Default: 5).table.WithMaxConcurrency(n int).WithMaxWriteWorkers(n int) on WriteRecords methods.WithClusteredWrite() to force single-threaded writes (mutually exclusive with WithMaxWriteWorkers).Common operations used to modify table state and create new snapshots:
The LocationProvider determines how file paths are generated for tables. Currently, there are two implementations:
simpleLocationProvider: The default implementation.objectStoreLocationProvider: Used for hashed object-storage layouts. This is automatically selected when the table property write.object-storage.enabled is set to true.Note: The LocationProvider is not currently user-pluggable; it is selected based on table properties.