repository·main·Indexed 18 days ago
https://github.com/facebookresearch/spdlA library for high-performance data loading architectures featuring flexible pipeline abstractions and specialized operations for array-based data. It includes libspdl, a C++20 media codec library for processing audio, video, and image data via composable pipelines with CUDA acceleration and coroutine support, as well as an asynchronous Python pipeline engine and the spdl.autoresearch framework for LLM-driven experiment workflows.
Asynchronous operations are the core foundation of SPDL. SPDL utilizes Python's asyncio module to orchestrate these operations. Understanding the Async I/O paradigm is critical for advanced SPDL usage, as improper implementation can lead to system-wide performance degradation.
Key concepts covered in the Async I/O documentation include:
asyncio event loop resolves these limitations.async def and await syntax.asyncio to manage both sync and async functions in a unified manner.For specific applications of this paradigm, see the documentation on Pipeline Parallelism, which describes how SPDL leverages Async I/O to orchestrate pipeline stages.
spdl.autoresearch is a pluggable framework designed for automated, LLM-driven experiment workflows. It provides a domain-neutral async scheduling engine capable of checkpointing and resuming workflows. The framework is structured into four layers:
spdl.autoresearch.core: The public core containing the async scheduling engine, WorkflowProtocol and WorkflowSpec contracts, JSON-backed state persistence helpers (write_engine_state, read_engine_state, load_or_initial), and shared domain types (failure records, hypothesis nodes, analysis results)._app/: A private framework dispatcher that resolves workflows via --workflow module.path:factory or via short names registered in the spdl.autoresearch.workflows entry-points group. It handles CLI argument parsing and launches either an interactive supervisor agent or drives the engine directly._common/: Shared utility modules. Note that this directory has no __init__.py, so you must import leaf modules directly (e.g., from spdl.autoresearch._common._state import read_state).spdl.autoresearch.pipeline_optimization: A public submodule providing a concrete implementation for optimizing SPDL data-loading pipelines. It uses create_workflow as its WorkflowFactory entry point.The spdl.io module is a standalone component designed for efficient data loading into array formats, specifically optimized for AI training and inference in cloud environments.
Its primary workflow involves:
It supports decoding images and videos into multiple color formats (such as RGB, YUV420p, and NV12) to allow for application-specific memory optimization and use-case support.
Migrating to SPDL offers several advantages over traditional sub-process-based data loading solutions:
spdl.dataloader.Pipeline allows stage-by-stage configuration of concurrency, enabling independent adjustments based on network bandwidth and CPU capacity.spdl.dataloader.Pipeline executes user-provided functions without restrictions on data types, allowing stages to aggregate or disaggregate data.SPDL (Scalable and Performant Data Loading) is a library designed for building efficient data preprocessing pipelines, specifically for ML/AI applications. It focuses on high-throughput data loading from storage to GPUs.
Key Features:
Parallelism Model: By default, SPDL uses multi-threading as its core parallelism mechanism. It is designed to work efficiently with modern Python versions and is optimized to benefit from free-threaded Python.
Autoresearch is an automated engine designed to optimize data loading pipelines. It automates the manual optimization loop (instrumentation, experimentation, metric analysis, hypothesis formation, and code application) by using a coding agent (such as Claude or Codex).
The engine analyzes pipeline metrics to identify bottlenecks, proposes code changes, and iteratively improves performance with minimal human intervention. For example, in video classification pipelines, it has demonstrated the ability to discover complex optimizations like subprocess isolation, video subclipping, concurrency reduction, and GC alignment to achieve significant throughput speedups.
Converting media data (e.g., MP4 video) into contiguous arrays for Machine Learning involves several distinct stages. Understanding these stages helps identify whether your pipeline is bottlenecked by I/O, compute, or memory.
Core Conversion Steps:
packets (encoded media chunks for specific streams).frames.Optional Steps:
Crossing a process boundary involves pickling and copying data, which is expensive. To minimize this cost:
uint8 tensor so they use the fast shared-memory path.spdl.pipeline.SharedMemorySegmentPool to reuse pre-allocated shared-memory segments.ProcessPoolExecutor. Instead, use the PipelineBuilder.to region API (or run the whole pipeline in a subprocess) to keep them together in the same worker process.In SPDL, simple multi-threading (mt) can sometimes lead to inconsistent performance and 'spiky' behavior, even if the model's backward path and optimizer steps appear faster. This is often caused by the CPU being unable to launch GPU kernels promptly due to OS scheduling conflicts (the 'noisy neighbour' problem).
By moving the multi-threaded pipeline into a dedicated subprocess (mtp), you isolate the data loading tasks. While this may consume slightly more CPU resources than simple multi-threading, it provides higher throughput (QPS) and more stable, non-spiky execution compared to running multi-threaded pipelines in the main process.
When implementing a Dataset class, do not just provide a __getitem__ method that returns Tensors. To make your dataset compatible with SPDL's Pipeline and allow for efficient parallel processing, you should decompose it into a composable structure:
Source class) that returns metadata or paths (e.g., str) instead of Tensors.ImageFrames, or converting to a Tensor).By breaking the implementation into these stages, users can wrap your components in a PipelineBuilder to run stages like download and decode_and_preprocess with different concurrency levels.
class Source:
def __getitem__(self, key: int) -> tuple[str, int]:
...
def load(data: tuple[str, int]) -> tuple[ImageFrames, int]:
...
# The original Dataset then becomes a composition of these parts
class Dataset:
def __init__(self, ...):
self._src = Source(...)
def __getitem__(self, key:int) -> tuple[str, int]:
metadata = self._src[key]
item = download(metadata)
frames, cls = decode_and_preprocess(item)
tensor = spdl.io.to_torch(frames)
return tensor, clsIn libspdl, packets and frames are managed using RAII (Resource Acquisition Is Initialization) to ensure safe memory management across the C++/Python boundary.
std::unique_ptr with custom deleters (AVPacketDeleter, AVFrameDeleter) to automate deallocation.AVPacket* and performs bulk deallocation upon destruction.PacketSeries with metadata like id, src, stream_index, time_base, and codec information.AVFrame* with a custom destructor for bulk deallocation.Ownership is transferred explicitly. When a C++ function takes a PacketsPtr or FramesPtr, ownership moves from Python to C++. Once transferred, the Python variable becomes invalid. This prevents accidental reuse and ensures that heavy resource cleanup happens in worker threads rather than during Python's garbage collection cycle.