DOCSEARCHbyAlgolia
Submit Docs

TileDB

repository·main·Indexed 24 days ago

https://github.com/tiledb-inc/tiledb

A universal storage engine for dense and sparse multi-dimensional arrays. TileDB provides an embeddable C++ library supporting cloud storage integration (AWS S3, Google Cloud Storage, Azure Blob Storage), data versioning, and parallel IO. It includes language bindings for C, C++, Python, R, Java, Go, and C#, as well as specialized packages like TileDB-BioImaging, TileDB-SOMA, TileDB-VCF, and TileDB-Vector-Search.

Tokens
36.8K
Snippets
55
Records
184
Agent score
84%

What's inside TileDB

  1. Overview of TileDB Embedded

    main

    TileDB (often referred to as TileDB Embedded) is an embeddable C++ library designed for storing and accessing dense and sparse multi-dimensional arrays. It is suitable for modeling complex data like Genomics, Geospatial, and Finance data.

    Key features include:

    • Array Types: Support for both dense and sparse arrays, including dataframes and key-value stores (via sparse arrays).
    • Storage: Support for cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage) and chunked (tiled) arrays.
    • Data Management: Data versioning (time traveling), array metadata, and array groups.
    • Performance: Fully multi-threaded implementation with parallel IO.
    • Security & Integrity: Multiple compression, encryption, and checksum filters.
    • Extensibility: Numerous APIs and integrations (Spark, Dask, MariaDB, GDAL, etc.).

  2. Understand TileDB Array Format Version History

    main

    The TileDB array format evolves with new versions to support additional features like enumerations, schema evolution, and improved metadata handling. When working with TileDB, it is important to know which version of the format your data uses, as certain features (like dimension labels or specific filter pipeline options) are only available in specific versions.

    Key milestones in the format history include:

    • Version 23 (TileDB 2.30): Added optional sections to fragment metadata footers.
    • Version 22 (TileDB 2.25): Added the _Current domain_ field to array schemas.
    • Version 20 (TileDB 2.17): Introduced support for enumerations and date/time type filters.
    • Version 18 (TileDB 2.15): Introduced dimension labels.
    • Version 10 (TileDB 2.4): Introduced schema evolution support.
    • Version 7 (TileDB 2.2): Introduced nullable attributes.
    • Version 5 (TileDB 2.0): Introduced separate data files for dimensions and variable-sized dimensions for sparse arrays.

  3. Return multiple values using std::tuple

    main
    To avoid the legacy C pattern of using 'out-arguments' (pointers or references) to return multiple values, use std::tuple. This allows a function to return multiple typed, anonymous elements as a single return value. This is preferred over returning a custom struct when the grouping is temporary or purely for the purpose of returning multiple values.

  4. How the Dictionary Encoding filter works

    main

    The Dictionary Encoding filter provides lossless compression for variable-sized string data. It works by identifying a set of unique strings (the dictionary) from the input and replacing the actual string data on disk with integer indices representing the position of each string within that dictionary.

    Key Constraints:

    • It is supported only for variable-sized strings.
    • It must be the first filter in the filter pipeline.

    Example Logic: If the input is ["HG543232", "HG543232", "HG54", "A"], the dictionary becomes ["HG543232", "HG54", "A"] and the output data becomes [0, 0, 1, 2].

    input_data = ["HG543232", "HG543232", "HG543232", "HG54", "HG54", "A", "HG543232", "HG54"]
# apply dictionary encoding ->
dictionary = ["HG543232", "HG54", "A"]
output_data = [0, 0, 0, 1, 1, 2, 0, 1]

  5. How TileDB API documentation is generated

    main

    TileDB generates its C and C++ API documentation using a pipeline of three tools:

    1. Doxygen: Extracts documentation from inline code comments into XML files.
    2. Breathe: Acts as a bridge between Doxygen's XML output and Sphinx.
    3. Sphinx: Processes the Breathe-enriched content into the final documentation format.

    Build System Integration

    The documentation build is integrated into the CMake build system via a doc target (defined in cmake/Modules/Doxygen.cmake). When the doc target is invoked, CMake generates a doxyfile.in in the build directory. This file is then used by source/Doxyfile.mk to execute Doxygen and extract docstrings into XML.

  6. Understand the Chunk on-disk format

    main

    Each chunk within a tile follows a specific serialized format. The metadata contained within a chunk depends on the sequence of filters used in the tile's filter pipeline.

    Chunk Structure:

    • Original length of chunk (uint32_t): The unfiltered number of bytes.
    • Filtered chunk length (uint32_t): The serialized/filtered number of bytes.
    • Chunk metadata length (uint32_t): The size of the metadata region.
    • Chunk metadata (uint8_t[]): Metadata bytes (varies by filter pipeline).
    • Chunk filtered data (uint8_t[]): The final bytes after the entire filter pipeline has been applied.

    Key behaviors:

    • Reading: When reading from disk, the filter pipeline is processed in reverse order.
    • Metadata: Filters typically concatenate their metadata to the chunk metadata region. Some filters (like compression) may compress the metadata produced by previous filters.
    • Data Splitting: Filters can split output byte arrays into multiple "parts". Subsequent filters in the pipeline process each part independently.
    • Cell Integrity: As of version 11, cells of arrays are not split across chunks within a tile.

  7. How the XOR filter operates on data

    main

    The XOR filter applies the XOR operation sequentially to the input data. The operation is performed in chunks of 1-4 bytes, determined by the sizeof the attribute's type representation.

    Because the operation is sequential, each element is XORed with its predecessor. The input and output data layouts are identical.

    # Conceptual example of the XOR filter logic using NumPy
data = np.random.rand(npts)
data_b = data.view(np.int64)
for i in range(1, len(data)):
  data_b[i] = data_b[i] ^ data_b[i-1]

  8. Understand the Tile on-disk structure

    main

    Tile data is divided into one or more "chunks". A tile's on-disk format consists of a count of chunks followed by the chunks themselves.

    Tile Structure:

    • Num chunks (uint64_t): Total number of chunks in the tile.
    • Chunk 1 to Chunk N: The individual chunk data blocks.

    When no filters are applied, the tile is still divided into chunks, but there are no chunk metadata bytes, and the filtered bytes are identical to the original bytes.

  9. Use the Delta Filter for integer compression

    main

    The Delta Filter is a compression transformation that reduces the storage footprint of integer type data by computing and storing the difference (delta) between consecutive elements.

    To use this filter, specify the enum value TILEDB_FILTER_DELTA (or the integer value 19) in your filter pipeline.

    Input Requirements:

    • The input must be an array of integer numbers.
    • The input type (input_t) is automatically inferred from the output type of the preceding filter in the pipeline, or from the tile's datatype if it is the first filter. You can manually override this using the Reinterpret datatype field in the filter options.

    Output Layout: When the filter is applied, the data is transformed into the following structure:

    • n (uint64_t): The total number of values in the input.
    • in_0 (input_t): The first value of the original sequence.
    • delta_1 to delta_n (input_t): The computed differences where delta_i = in_i - in_{i-1}.

  10. Use C++23-style ranges via `tiledb/stdx/ranges`

    main

    For environments where the compiler's standard library lacks C++23 range features, TileDB provides a subset of the standardized views in tiledb/stdx/ranges. This include file mimics the standard library ranges header.

    It provides implementations for:

    • zip_view
    • chunk_view

    The implementation uses a naming convention mirroring LLVM (located in the __ranges subdirectory) and employs preprocessor directives to automatically switch to the standard library implementation once the compiler supports them.

  11. Understand the TileDB Group structure

    main

    A TileDB Group is represented as a folder containing a special subdirectory named __group. This subdirectory holds one or more timestamped group files that detail the members of the group. The group folder itself can contain other files or subdirectories.

    my_group                       # Group folder
    |  ...
    |_ __group                 # Group folder
        |_ <timestamped_name>  # Timestamped group file detailing members
    |_ ...

  12. Understand the TileDB C API architecture

    main

    The TileDB C API is structured into five distinct layers that manage the transition from C-style calls to the underlying C++ core library. Understanding these layers helps in understanding how memory is managed, how errors are handled, and how objects are represented.

    The Five Layers of the C API

    1. Public API functions: These are the functions you call directly. They use extern "C" linkage and act as wrappers. Their primary responsibilities are providing uniform error processing (handling exceptions), consistent return values, and standardized logging formats.
    2. API implementation functions: These functions translate C calling sequences (like C-style pointers) into C++ calling sequences. They perform initial argument validation (e.g., checking for null pointers) and manage the memory allocation of API-visible objects.
    3. Handle classes: Derived from class api:handle, these manage the memory allocation for API-visible objects. They act as carriers for Facade instances. A handle is the primary way the C API interacts with objects, and handles forward calls to the underlying facade.
    4. Facade classes: These provide a uniform interface to core objects, regardless of their lifecycle state (e.g., whether an object was newly created via an *_alloc function or retrieved from an existing array schema). Facades allow the API to enforce rules, such as preventing modifications to immutable objects like an existing array schema.
    5. Proxy classes: These represent objects that are currently being built. They collect constructor arguments through various set_* API calls before the object is fully constructed.

    Key Relationships

    • Public API $\leftrightarrow$ Implementation: One-to-one relationship.
    • Handle $\leftrightarrow$ Facade: One-to-one relationship. Handles carry facades; facades are only created through handles.
    • Facade $\leftrightarrow$ Proxy: An optional one-to-one relationship. If an *_alloc function exists, a proxy class is typically used to collect configuration before construction.