DOCSEARCHbyAlgolia
Submit Docs

FractalBits Object Storage

repository·main·Indexed 43 days ago

https://github.com/fractalbits-labs/fractalbits

A high-performance, S3-compatible object storage system optimized for low latency and high throughput. It features a custom Fractal ART (Adaptive Radix Tree) storage engine, utilizes io_uring for asynchronous I/O, and supports native atomic rename operations for objects and directories.

Tokens
72.3K
Snippets
259
Records
331
Agent score
55%

What's inside FractalBits

  1. Overview of FractalBits

    main

    FractalBits is a high-performance, S3-compatible object storage system designed for low latency and high throughput. It is particularly suited for AI training pipelines and data analytics due to its native support for atomic rename of both objects and directories—a feature standard S3 lacks.

    Key Technical Features

    • Fractal ART (Adaptive Radix Tree) Engine: Uses a full-path name approach for metadata to avoid heavy distributed transactions, enabling high scalability and atomic renames. It also serves as an SSD-optimized local data engine.
    • High Performance: Capable of delivering >1.3M 4KB-object reads per second with P99 latency of ~3ms.
    • Technology Stack: Built with Rust (API gateway/control plane) and utilizes io_uring for asynchronous I/O in the storage engine and data plane.
    • Cost Efficiency: Designed to be more cost-effective than AWS S3 Express One Zone for small object workloads.

  2. FractalBits Technology Stack and Performance Optimizations

    main

    FractalBits achieves high performance through several key architectural choices:

    Core Technologies

    • Fractal ART (Adaptive Radix Tree): The primary storage engine for both metadata and data.
    • io_uring: Linux async I/O (custom library based on TigerBeetle's IO).
    • Actix: A thread-per-core async web framework used for the S3 gateway.
    • RPC Protocols:
      • Data Plane: Zero-copy RPC from network to disk without serialization.
      • Metadata Plane: Fixed small-size headers combined with Protobuf.

    Performance Optimizations

    • Thread-per-core architecture to avoid lock contention.
    • SIMD acceleration for storage engine operations.
    • io_uring IOPOLL mode specifically for NVMe SSDs.
    • Zero-copy for both authentication and data transfer.
    • Direct I/O to bypass the OS page cache.
    • CPU affinity pinning and header batching for RPC efficiency.

  3. NSS - Namespace Service Server (Metadata Management)

    main

    The NSS is responsible for managing the system's metadata. It is designed for high concurrency and efficient recovery using the following features:

    • Fractal ART KV Engine: An Adaptive Radix Tree-based storage engine featuring efficient checkpointing and blob compaction.
    • Atomic Operations: Enhanced tree operations support atomic directory and object renaming.
    • Crash Recovery: Uses physiological journaling for efficient recovery.
    • Performance Optimizations: Utilizes SIMD acceleration, a lightweight buffer manager (inspired by LeanStore), and lock coupling (crab-latching) for concurrent access.

  4. Perform atomic rename operations in FractalBits

    main

    Unlike traditional S3 which uses copy+delete for moves, fractal-s3 leverages FractalBits' native APIs to perform instant, atomic renames within the same bucket.

    Atomic Folder Rename

    When moving a folder (paths ending in /) within the same bucket, the tool uses the renameFolder API. This is instant regardless of the folder size.

    # Uses atomic renameFolder API
fractal-s3 mv s3://bucket/folder1/ s3://bucket/folder2/

    Atomic Object Rename

    When moving a single object within the same bucket, the tool uses the renameObject API, avoiding the copy+delete overhead.

    # Uses atomic renameObject API
fractal-s3 mv s3://bucket/old-name.txt s3://bucket/new-name.txt

    Cross-Bucket Moves

    Note that moves between different buckets are not atomic and will use the standard copy+delete method.

    # Uses copy+delete (not atomic)
fractal-s3 mv s3://bucket1/file.txt s3://bucket2/file.txt
    # Atomic folder rename
fractal-s3 mv s3://my-bucket/old-folder/ s3://my-bucket/new-folder/

  5. BSS - Blob Storage Server (Data Storage)

    main

    The BSS is the high-performance engine for storing blob data. It is optimized for NVMe storage through:

    • Fractal ART KV Engine: Used for the underlying storage engine.
    • Async I/O: Uses io_uring with IOPOLL mode for high-performance NVMe access.
    • Zero-Copy & Direct I/O: Implements zero-copy data paths and Direct I/O to bypass the page cache.
    • Scalability: Supports configurable I/O concurrency and sharding.

  7. How the FractalBits multi-tier architecture works

    main

    FractalBits is a multi-tier, S3-compatible storage system optimized for high performance. It separates concerns into four primary layers:

    1. S3 Gateway (Frontend): An Actix-based HTTP layer that handles S3 API requests (GET, PUT, DELETE, HEAD, POST), AWS Signature V4 authentication, and request routing.
    2. RSS - Root Service Server (Coordination): A high-availability layer responsible for cluster coordination, including leader election (via ETCD, DynamoDB, or FireStore), API key management, bucket management, and volume group configuration.
    3. NSS - Namespace Service Server (Metadata): Manages metadata using a Fractal ART (Adaptive Radix Tree) based KV storage engine. It handles full paths and supports infinitely splittable metadata with atomic directory/object renaming.
    4. BSS - Blob Storage Server (Data Plane): The high-performance engine for actual blob data. It uses Fractal ART for storage and leverages io_uring with IOPOLL mode for NVMe-optimized, zero-copy data paths.

    Data can be stored in various modes: AllInBss, Hybrid (using EC or replication to cloud storage), or MultiAz.

  8. S3 Gateway (S3 API Frontend)

    main

    The S3 Gateway serves as the S3-compatible HTTP entry point for clients. It handles:

    • S3 API Support: All standard requests including GET, PUT, DELETE, HEAD, and POST.
    • Security: AWS Signature V4 authentication.
    • Concurrency: Per-core request handling using dedicated worker pools via the Actix framework.
    • Efficiency: Connection pooling to backend services and request routing.

  10. System requirements for FractalBits development

    main

    To develop for FractalBits, your environment must meet the following specifications:

    • OS: Linux kernel 5.19+ (required for io_uring support). Ubuntu 24.04+ is recommended.
    • Rust: Toolchain version 1.91+.
    • Disk Space: 20GB+ free space.
    • Memory: 16GB+ recommended to avoid OOM (Out of Memory) issues during builds.

  11. Create the Fractalbits Cluster

    main

    Once your cluster.toml is ready and the bootstrap container is running on the root_server, execute the cluster creation command from your deployment machine:

    cargo xtask deploy create-cluster \
  --config ./cluster.toml \
  --bootstrap-s3-url <RSS_IP>:8080

    Note: Replace <RSS_IP> with the IP address of the root_server running the bootstrap container. If you use a custom SSH configuration, add the --ssh-config <path> flag.