DOCSEARCHbyAlgolia
Submit Docs

Vortex Documentation

repository·develop·Indexed 25 days ago

https://github.com/vortex-data/vortex

Vortex is a high-performance columnar file format and toolkit optimized for object storage, offering faster random access, scans, and writes than Apache Parquet. It includes specialized encodings like FSST for string-like data with LIKE pushdown support and OnPair for dictionary-based compression with fast per-row random access. The toolkit provides the bench-orchestrator (vx-bench) for performance comparison across engines like DataFusion and DuckDB, as well as dedicated benchmarks for compression and random access.

Tokens
82.2K
Snippets
212
Records
581
Agent score
82%

What's inside Vortex

  4. Overview of Vortex OnPair Encoding

    develop
    Vortex OnPair is an encoding designed for Binary and UTF-8 data using the OnPair short-string compression algorithm. It is a dictionary-based encoder that provides fast per-row random access. The onpair crate acts as the trainer/encoder and wraps the resulting column as a Vortex array, enabling cascading-compressor support on every integer child.

  5. Overview of the Vortex Java API

    develop

    The Vortex Java API provides bindings for the Vortex library, allowing Java applications to interact with Vortex arrays and files. The API is divided into two primary components:

    • Vortex JNI: Core JNI bindings providing direct access to Vortex functionality.
    • Vortex Spark: Integration for Apache Spark, enabling Spark to read Vortex files.

  6. Understand Vortex Layouts

    develop

    Layouts are the out-of-memory equivalent of Vortex arrays. They are hierarchical structures consisting of a vtable, metadata, dtype, children, and lazy buffers called "segments".

    Key characteristics:

    • Persistence: The tree structure can be serialized and persisted.
    • Lazy Loading: During deserialization, layouts bind to a segment source that lazily fetches data buffers.
    • Storage Agnostic: Supports efficient columnar scans over local disk, object stores, remote caches (e.g., Redis, Postgres block storage), and more.
    • File Format: The Vortex file format is essentially a serialized layout tree with data segments stored within the same file.

  7. Understand the VTable and Dispatch Pattern

    develop

    Vortex uses a VTable-based dispatch pattern to handle type erasure and polymorphism for core data types (ExtDType, Expr, Layout, and Array). This pattern avoids the overhead of dyn Trait by using a combination of a generic struct (e.g., Array<V>) and a VTable trait (e.g., ArrayVTable).

    Key components of the pattern:

    • Generic Struct (Array<V>): Holds common metadata (like dtype, len, and stats) and the encoding-specific data (array: V::Array). It provides inherent methods for operations like slice, filter, and take.
    • VTable Trait (ArrayVTable): Defines the interface for specific encodings. It includes an associated type Array which holds the actual encoding-specific data (buffers, children, etc.).
    • Erased Reference (ArrayRef): A type-erased handle (typically Arc<dyn DynArray>) used when the specific encoding is not known at compile time. It uses a thin forwarder to call methods on the underlying generic struct.
    • ZST VTable: The VTable itself is often a Zero Sized Type (ZST) used to host constructors and dispatch logic.

  8. Use vortex-proto for Protocol Buffers definitions

    develop
    The vortex-proto crate provides Protocol Buffers definitions used to serialize and deserialize messages across different crates within the Vortex workspace. Use this crate when you need to convert data structures into protobuf messages or back again.

  9. Understand Vortex Core Concepts

    develop

    Vortex is a modular ecosystem designed for compressed columnar data across in-memory, on-disk, and over-the-wire scenarios. Its architecture is built around four pillars:

    • DTypes: A logical type system (e.g., UTF8) that defines data meaning independently of physical encoding.
    • Arrays: The in-memory representation. Unlike Arrow, Vortex arrays can be compressed (e.g., bit-packed integers). They use the same representation on disk and over the wire to enable zero-copy I/O.
    • Compute: Functions that operate directly on compressed arrays by dispatching to encoding-specific kernels.
    • Layouts: Systems that organize arrays into larger-than-memory datasets, such as chunked row groups, and support various block storage backends (local disk, object stores, caches).

  10. Access Vortex Language Bindings

    develop

    Vortex provides language bindings for several environments. You can find specific API references for the following languages in the documentation directory:

    • Python: Python-specific interfaces and usage.
    • C: C language bindings.
    • Java: Java language bindings.

  12. Use Vortex files as Arrow Datasets

    develop

    Vortex files implement the Arrow Dataset interface. This allows you to use Vortex files efficiently within query engines that support Arrow, such as DuckDB and Polars.

    Key performance characteristics:

    • Predicate Pushdown: Vortex reads data proportional to the number of rows passing a filter condition.
    • Projection Pushdown: Vortex reads data proportional to the number of columns in a selection.
    • Encoding Efficiency: For most Vortex encodings, these properties hold true even when filtering for a single row.