DOCSEARCHbyAlgolia
Submit Docs

Fire-Flyer File System (3FS)

repository·main·Indexed 27 days ago

https://github.com/deepseek-ai/3fs

A high-performance distributed file system optimized for AI training and inference workloads. 3FS leverages modern SSDs and RDMA networks to provide strong consistency and high throughput for data loading, checkpointing, and KVCache management. The system includes components such as mgmtd, meta, and storage services, an admin CLI for cluster management, a FUSE client, and a USRBIO FIO engine plugin for benchmarking.

Tokens
42.2K
Snippets
61
Records
294
Agent score
94%

What's inside 3FS

  1. Overview of USRBIO API

    main
    User Space Ring Based IO (USRBIO) is a high-speed I/O interface for 3FS. It allows user applications to submit I/O requests directly to the 3FS I/O queue in the FUSE process, bypassing FUSE limitations such as maximum single I/O size restrictions. This enables zero-copy data exchange between the user and FUSE processes.

  2. Understand the 3FS Chunk Storage and Data Placement

    main

    3FS uses a chunk storage system designed for high bandwidth and linear scalability with SSD count. Data is replicated using Chain Replication with Apportioned Queries (CRAQ).

    Key Concepts:

    • Chain Replication: Each chunk is replicated across a chain of storage targets. Write requests are sent to the head and propagated to the tail. Read requests can be sent to any target in the chain to balance load.
    • Storage Targets: Multiple targets are created on each SSD, and targets join different chains to ensure load balancing.
    • Chain Tables: The metadata service selects a chain table for each file. Multiple tables can exist to support different requirements (e.g., one for batch/offline jobs and one for online services) using mutually exclusive nodes/SSDs.

  4. Understand 3FS file chunking and data layout

    main

    3FS divides file data into equally sized chunks and stripes them across multiple replication chains.

    Key Concepts:

    • Chunk ID Generation: Computed by concatenating the file's inode id and the chunk index.
    • Per-directory Configuration: Users can specify the chain table, chunk size, and stripe size for files on a per-directory basis.
    • Allocation Strategy: When creating a file, the metadata service uses a round-robin strategy to select consecutive replication chains from the designated chain table based on the stripe size. A random seed is then used to shuffle these chains to ensure balanced distribution across SSDs.
    • Client-side Computation: Once a client obtains the data layout from the meta service, it can independently compute chunk IDs and chains for data operations, reducing meta service overhead.

  5. Understand the Chunk Engine architecture

    main

    The Chunk Engine is composed of two primary components designed for high-performance chunk management and atomic metadata persistence:

    1. Allocator: Manages the in-memory state of chunk allocation and reclamation. It assigns chunk positions to disk space.
    2. MetaStore: Responsible for the durable persistence of allocation/reclamation events using RocksDB. It ensures atomicity via WriteBatch.

    Write Workflow:

    1. The Allocator assigns a new chunk position (in-memory).
    2. Data is written to the assigned disk position.
    3. Metadata is generated and persisted to the MetaStore using a WriteBatch to ensure the update is atomic.

    Read Safety: Read operations return an Arc<ChunkPos>, which manages ownership of the chunk position. This ensures that even if a chunk is reclaimed or deleted, the data remains valid and accessible until the Arc is dropped.

  6. Understand the 3FS Architecture

    main

    3FS is a distributed file system composed of four main components, all connected via an RDMA network (InfiniBand or RoCE):

    • Cluster Manager: Handles membership changes and distributes cluster configuration. It uses a primary/secondary election model. Configuration is typically stored in a distributed coordination service like ZooKeeper or etcd (or the same key-value store used for file metadata).
    • Metadata Service: Implements file system semantics (e.g., open, create). These services are stateless because metadata is stored in a transactional key-value store (e.g., FoundationDB).
    • Storage Service: Manages local SSDs and provides a chunk store interface. It uses Chain Replication with Apportioned Queries (CRAQ) to ensure strong consistency and high throughput.
    • Client: Provides two interfaces for applications: the FUSE client (low adoption barrier, standard POSIX) and the Native client (high performance, asynchronous zero-copy).

  7. USRBIO Core Concepts: Iov and Ior

    main

    USRBIO relies on two primary shared memory structures:

    • Iov: A large shared memory region used for zero-copy read/write operations. All read data is read into the Iov, and all write data must be written to the Iov by the user first. The FUSE process manages InfiniBand (IB) memory registration for this region.
    • Ior: A small shared memory ring used for communication between the user and FUSE processes, similar to Linux io-uring. The user enqueues requests and the FUSE process dequeues them. I/Os are executed in batches controlled by io_depth. For multi-threaded applications, it is recommended to use multiple rings to avoid synchronization overhead.

  8. Access 3fs documentation and guides

    main

    The 3fs documentation is organized into several key areas:

    • Design Notes: Detailed architectural and design documentation.
    • Setup Guide: Instructions for deploying the system.
    • USRBIO API Reference: Technical reference for the USRBIO API.
    • P Specifications: Detailed specifications for the P component.

  9. Understand 3FS metric types and storage

    main

    3FS calculates metrics within each service and stores them in ClickHouse via a monitor service. Metrics are categorized into four types, which determine how they are recorded and whether they reset after each report:

    • value (ValueRecorder): Stored in 3fs.counters. These are set directly by code (e.g., capacity) and do not necessarily reset.
    • count (CountRecorder): Stored in 3fs.counters. These typically increase and reset to zero after each report (e.g., IOPS). However, they can also be used to track 'on-flight' operations, where the value is increased at the start and decreased upon completion rather than resetting automatically.
    • distribution (DistributionRecorder): Stored in 3fs.distributions. Used to calculate percentiles like P90 and P99 for the period between reports.
    • latency (LatencyRecorder): Stored in 3fs.distributions. A specialized distribution implementation that records latency in nanoseconds (ns).

  10. Build 3FS

    main

    Build 3FS into the build directory using CMake.

    Important: Shuffle Method Compatibility Due to std::shuffle implementation differences, binaries compiled with different compiler versions (e.g., g++10 vs g++11) may be incompatible. You must specify -DSHUFFLE_METHOD to lock the algorithm:

    • Existing Clusters: Use the method corresponding to the compiler version used for the current deployment (g++10 or g++11).
    • New Clusters: Choose either g++10 or g++11, but you must remain consistent for all future builds to maintain compatibility.
    # Replace <method> with 'g++10' or 'g++11' based on your environment
cmake -S . -B build \
      -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 \
      -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
      -DSHUFFLE_METHOD=<method>
cmake --build build -j 32

  11. Install dependencies for OpenCloudOS 9 and TencentOS 4

    main

    Use the following command to install the necessary system dependencies on OpenCloudOS 9 and TencentOS 4.

    dnf install epol-release wget git meson cmake perl lld gcc gcc-c++ autoconf lz4 lz4-devel xz xz-devel \
    double-conversion-devel libdwarf-devel libunwind-devel libaio-devel gflags-devel glog-devel \
    libuv-devel gmock-devel gperftools gperftools-devel openssl-devel boost-static boost-devel mono-devel \
    libevent-devel libibverbs-devel numactl-devel python3-devel

  12. Build 3FS using Docker

    main

    If you prefer using Docker for the build environment, pull the appropriate image for your OS:

    • TencentOS-4: docker.io/tencentos/tencentos4-deepseek3fs-build:latest
    • OpenCloudOS-9: docker.io/opencloudos/opencloudos9-deepseek3fs-build:latest
    docker pull docker.io/tencentos/tencentos4-deepseek3fs-build:latest
# OR
docker pull docker.io/opencloudos/opencloudos9-deepseek3fs-build:latest