iroh-blobs

When downloading large blobs incrementally (e.g., in 16 KiB chunk groups), maintaining consistency between the .data, .outboard, and .bitfield files is challenging due to the lack of transactional guarantees in file systems.

The Consistency Model

To prevent a crash from leaving the store in an inconsistent state (where the .bitfield claims data exists that hasn't actually been flushed to the .data or .outboard files), iroh-blobs uses a specific write strategy:

1. Write Data and Outboard: The .data and .outboard files are updated as chunks arrive. These writes can happen in any order and without explicit fsync calls.
2. Update Bitfield Last: The .bitfield file (which tracks valid chunks) is only updated after the data/outboard writes.

Crucially: If a crash occurs, the system relies on the fact that if the .bitfield was not updated, the changes to the data and outboard files are treated as invisible. This avoids the massive performance penalty of calling fsync after every 16 KiB chunk, which would otherwise limit download speeds to a few megabytes per second.

Data enters the iroh-blobs store through two primary workflows, each with different requirements for consistency and complexity:

1. Adding local files by name

Used when you have a local file but do not yet know its hash. The system must:

1. Compute the complete outboard and the root hash.
2. Atomically move the file into the store under the root hash.
3. Depending on file size, data and outboard are stored in either the file system or the database.

2. Syncing remote blobs by hash

Used when you know the hash but need to fetch the data from a remote node.

• Small blobs: Can be requested and written atomically.
• Large blobs: (Larger than the 16 KiB chunk group size) must be written incrementally. This requires tracking which chunks are locally present and which chunks can be proven to be present.

Once a blob is stored under its hash, neither the data nor the outboard will be changed.

The iroh-blobs file-based store uses a hybrid approach to balance performance between millions of tiny blobs and a few massive blobs. It avoids the overhead of a database for large files and the file-system metadata bottleneck for small files.

How it works:

• Small Blobs: Stored entirely inline within the database (using redb). This prevents file-system fragmentation and metadata overhead.
• Large Blobs: Stored as individual files in the file system. This allows the OS to handle large I/O operations efficiently and enables direct file access without going through the iroh node.
• Thresholds: There are configurable thresholds for both data and outboards. If a blob's size is below these thresholds, it is stored inline in the database.
  • Default Thresholds: 16 KiB for both inline outboard and data size.
  • Debugging Tip: You can force all data to be stored on disk by setting both thresholds to 0.

The iroh-blobs metadata table maps a BLAKE3 hash to an entry state. Understanding these states is key to how the store manages persistence:

Complete Entry State

When a blob is fully downloaded, it exists in one of these modes:

• inline: Both data and outboard are stored directly in the metadata database.
• owned: Data is in a canonical file system location (path is computed from the hash). Outboards are either owned or not needed (if data is $\le$ 16 KiB).
• external: Data is in a user-defined path (the path must be explicitly stored in the metadata).

Partial Entry State

When a blob is still being downloaded:

• Incomplete (No size): The system is still receiving chunks and does not yet have a verified total size.
• Incomplete (Verified size): The last chunk has been received, providing a verified size, but the entry is not yet marked as complete.

Blobs in iroh-blobs can be protected from deletion using different tagging strategies:

• Temporary Protection: On creation, blobs are assigned a temporary tag that prevents deletion for the duration of the current process.
• Persistent Protection: Blobs can be assigned a persistent tag, ensuring they are not deleted even after a process restart.
• Bulk Protection: Large groups of blobs can be protected by creating a special blob containing a sequence of hashes and tagging that blob as a hash sequence.

Deletion Behavior

• Explicit Deletion: You can explicitly delete blobs by hash, but this is intended for emergency use only (e.g., when data must be removed regardless of consequences).
• Completeness: Deletion is always complete; there is no mechanism to delete or protect specific ranges within a file.
• Background Process: Deletion runs in the background. While there is currently a small delay between untagging a blob and its actual deletion, the system is designed to eventually delete blobs immediately once they are no longer tagged.

To use iroh-blobs, you must integrate it with the iroh networking library. You use an iroh::protocol::Router to accept incoming connections and route them to the BlobsProtocol handler using the iroh_blobs::ALPN identifier.

Typically, you will:

1. Create an iroh::Endpoint.
2. Initialize a blob store (e.g., MemStore).
3. Create a BlobsProtocol instance using the store.
4. Register the protocol with the Router using .accept(iroh_blobs::ALPN, blobs).

use iroh::{protocol::Router, Endpoint, endpoint::presets};
use iroh_blobs::{store::mem::MemStore, BlobsProtocol, ticket::BlobTicket};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // create an iroh endpoint that includes the standard address lookup mechanisms
    let endpoint = Endpoint::bind(presets::N0).await?;

    // create a protocol handler using an in-memory blob store.
    let store = MemStore::new();
    let tag = store.add_slice(b"Hello world").await?;

    let _ = endpoint.online().await;
    let addr = endpoint.addr();
    let ticket = BlobTicket::new(addr, tag.hash, tag.format);

    // build the router
    let blobs = BlobsProtocol::new(&store, None);
    let router = Router::builder(endpoint)
        .accept(iroh_blobs::ALPN, blobs)
        .spawn();

    println!("We are now serving {}", ticket);

    // wait for control-c
    tokio::signal::ctrl_c().await;

    // clean shutdown of router and store
    router.shutdown().await?;
    Ok(())
}

A StreamPair encapsulates a bidirectional stream (a SendStream and a RecvStream) along with connection metadata and an EventSender for progress tracking. It provides methods to read requests and transition into specialized ProgressWriter or ProgressReader types.

Key methods:

• accept(conn, events): Creates a new StreamPair from an iroh::endpoint::Connection.
• read_request(): Reads the initial Request from the stream. This consumes only the bytes needed for the request header/metadata, leaving the rest of the stream for payload processing.
• into_writer(tracker): Consumes the StreamPair and returns a ProgressWriter for sending data.
• into_reader(tracker): Consumes the StreamPair and returns a ProgressReader for receiving data.

// Example of accepting a stream and reading a request
let pair = StreamPair::accept(&connection, events).await?;
let request = pair.read_request().await?;