DOCSEARCHbyAlgolia
Submit Docs

TensorStore Documentation

repository·master·Indexed 23 days ago

https://github.com/google/tensorstore

An open-source C++ and Python library for high-performance storage and manipulation of large multi-dimensional arrays. It features a uniform API for formats like Zarr and N5, supports diverse backends including local filesystems, GCS, S3, and HTTP, and provides an asynchronous API for high-throughput remote storage with ACID transaction guarantees.

Tokens
26.2K
Snippets
30
Records
163
Agent score
80%

What's inside TensorStore

  1. Overview of TensorStore capabilities

    master

    TensorStore is a C++ and Python library for storing and manipulating large multi-dimensional arrays. Key features include:

    • Advanced Indexing: Fully composable indexing operations and virtual views.
    • Uniform API: A single interface for reading and writing various array formats like zarr and N5.
    • Diverse Storage Support: Natively supports local/network filesystems, Google Cloud Storage, Amazon S3-compatible object stores, HTTP servers, and in-memory storage.
    • High Throughput: An asynchronous API designed for high-latency remote storage.
    • Data Integrity: Supports read caching and transactions with ACID (Atomicity, Isolation, Consistency, Durability) guarantees.
    • Concurrency: Supports safe, efficient access from multiple processes and machines via optimistic concurrency.

  5. Use the neuroglancer_precomputed driver

    master

    The neuroglancer_precomputed driver allows TensorStore to access volumes in the Neuroglancer Precomputed format. It can be backed by any supported key_value_store (such as Google Cloud Storage) and supports reading, writing, and creating new volumes.

    Note on Multiscale Volumes: While multiscale volumes are supported, each scale must be opened individually using specific configuration parameters to identify the desired scale.

    {
  "driver": "neuroglancer_precomputed",
  "kvstore": "gs://my-bucket/path/to/volume/"
}

  6. Apply transformations using Adapters

    master

    Adapters allow you to wrap a base driver to provide additional functionality or view the data differently without changing the underlying storage. Available adapters include:

    • cast: For changing the data type of the array.
    • downsample: For reducing the resolution of the array.
    • stack: For combining multiple arrays into a single higher-dimensional array.

    Adapters follow the TensorStoreAdapter schema.

  8. Understand the OCDBT storage format

    master

    The OCDBT (Optimized Copy-on-Write Database Tree) is represented by specific entries within a prefix/directory of an underlying key-value store. The structure consists of:

    • manifest.ocdbt: Stores the encoded manifest (database configuration and optionally the version tree).
    • manifest.{xxxxxxxxxxxxxxxx}: Used with the numbered manifest kind; stores encoded manifests where {xxxxxxxxxxxxxxxx} is a 16-digit (0-padded) lowercase hex generation number.
    • d/{xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx}: Log-structured data files containing out-of-line raw values, encoded version tree nodes, and encoded B+Tree nodes. The filename is a 128-bit random identifier in lowercase hex.

    Note on Data Prefixes: While d/ is the default prefix for writing, you can override it using the following configuration options:

    • kvstore/ocdbt.value_data_prefix
    • kvstore/ocdbt.btree_node_data_prefix
    • kvstore/ocdbt.version_tree_node_data_prefix.

    When reading, the format allows data files to have any arbitrary relative path, so these options do not affect reading.

  9. Use the cast driver for element-wise data type conversion

    master

    The cast driver provides a virtual read/write view that performs element-wise data type conversion between a base TensorStore and a view.

    Reading

    To read using the cast driver, the underlying base TensorStore must:

    1. Support reading.
    2. Support conversion from the base data type to the requested view data type.

    Writing

    To write using the cast driver, the underlying base TensorStore must:

    1. Support writing.
    2. Support conversion from the requested view data type back to the base data type.

    Transformations

    The top-level driver/cast.transform (if specified) is composed with any TensorStore.transform defined on the base TensorStore.

  10. Use NumPy-style indexing in Dimension Expressions

    master

    You can chain NumPy-style indexing to a DimExpression using dexpr[iexpr], dexpr.vindex[iexpr], or dexpr.oindex[iexpr].

    Key Differences from standard NumPy indexing:

    • Dimension Consumption: The terms in iexpr consume dimensions from the dimension selection rather than the start of the domain. Unless an Ellipsis (...) is used, iexpr must consume the entire selection.
    • Newaxis Restriction: ts.newaxis is only permitted in the first operation of a dimension expression.
    • Implicit Duplication: If iexpr is a scalar (single integer, slice, or newaxis) and the selection has multiple dimensions, iexpr is implicitly duplicated across all selected dimensions.
    • Default Mode: Using dexpr[iexpr] (the default) behaves like vindex (vectorized), where array dimensions are added as the first dimensions of the result.

  12. Integer indexing in TensorStore

    master

    Indexing with an integer selects a single position within the corresponding dimension and consumes that dimension without adding new ones to the result domain.

    Important Differences from NumPy:

    • Negative Indices: Negative values have no special meaning. They refer to actual negative positions in the domain rather than positions relative to the end of the array. To index relative to the end, you must explicitly calculate the index using the domain bounds.
    • Out-of-bounds: Specifying an index outside the explicit bounds of a dimension results in an immediate IndexError. However, indexing outside implicit bounds is permitted if the dimension is configured with them; bounds will still be checked during subsequent read/write operations.