distilabel

Distilabel is an AI Feedback (AIF) framework designed for building datasets with and for LLMs. It provides a programmatic approach to building scalable pipelines for synthetic data generation and AI feedback, based on verified research methodologies.

Key capabilities include:

• Synthetic Data Generation: Creating high-quality, diverse datasets for tasks like instruction following, dialogue generation, and traditional NLP (classification, extraction).
• AI Feedback (AIF): Using LLMs to judge and filter data to improve quality.
• Unified API: Integrating AI feedback from various LLM providers through a single interface.
• Scalability and Fault Tolerance: Designed for large-scale data synthesis (e.g., generating millions of preference pairs).

A Pipeline in distilabel returns a Distiset object, which is a specialized Hugging Face datasets.DatasetDict. It acts as a dictionary-like object where each key corresponds to a different configuration (subset) of the dataset, specifically representing the different leaf steps in the pipeline's Directed Acyclic Graph (DAG).

If a pipeline has only one leaf node, the configuration name is set to "default" instead of the step name to align with Hugging Face Hub standards.

from datasets import Dataset
from distilabel.distiset import Distiset

distiset = Distiset(
    {
        "leaf_step_1": Dataset.from_dict({"instruction": [1, 2, 3]}),
        "leaf_step_2": Dataset.from_dict(
            {"instruction": [1, 2, 3, 4], "generation": [5, 6, 7, 8]}
        ),
    }
)

Distilabel workflows are built using several core components. To get started, you should learn how to:

• Define Steps: Use Steps as the building blocks of your pipeline to generate, evaluate, or manipulate data.
• Define Tasks: Create Tasks, which are specialized steps that rely on Language Models (LLMs) for data generation.
• Define LLMs: Configure LLMs as either local models or remote APIs to power your tasks.
• Execute Pipelines: Combine steps and tasks into a Pipeline to create a complete workflow.

The distilabel library uses a Step architecture to build data processing pipelines. You can interact with steps through the Step class or extend the _Step base class.

Key components include:

• Step: The primary class used to define a unit of work in a pipeline.
• _Step: The base class for implementing custom steps.
• StepInput: The data structure used for inputting data into a step.

For detailed implementation guides and examples of creating custom steps, refer to the Tutorial - Step documentation.

For complex data generation and processing workflows, Distilabel provides several advanced capabilities:

• Data Management: Use the Distiset dataset object (based on the datasets library) for storage and manipulation, or export your data to Argilla for searching and feedback.
• Pipeline Optimization: Use the File System to pass data batches between steps, implement Caching to recover executions and save LLM costs, and use the CLI to explore or re-run existing pipelines.
• Advanced Generation: Implement Structured Data Generation (e.g., JSON, function calls) and Serve LLMs (via TGI or vLLM) using clients like InferenceEndpointsLLM or OpenAILLM to share resources across tasks.
• Reliability: Impose Pipeline requirements on steps to ensure necessary dependencies are installed.

When pipeline batches contain large amounts of data, it can be more efficient to write data to disk and read it back in subsequent steps rather than passing it through the queue.

You can enable this behavior in the pipeline.run() method by setting use_fs_to_pass_data=True.

To use cloud storage (like Google Cloud Storage), you must install the corresponding fsspec implementation (e.g., gcsfs). If storage_parameters is not provided while use_fs_to_pass_data is True, distilabel defaults to using the local file system.

from distilabel.pipeline import Pipeline

with Pipeline(name="my-pipeline") as pipeline:
  ...

if __name__ == "__main__":
    distiset = pipeline.run(
        ..., 
        storage_parameters={"path": "gcs://my-bucket"},
        use_fs_to_pass_data=True
    )