DataFlow Documentation

DataFlow-Instruct-10K is a unified multi-domain instruction dataset generated using the DataFlow framework. It is constructed through automated data preparation pipelines covering mathematical reasoning, code, and general text instructions.

Each pipeline follows a “generate-evaluate-filter-refine” workflow to synthesize high-quality "instruction-response" pairs. The dataset contains approximately 10K samples designed to enable base models to achieve performance levels comparable to full-scale instruction-tuned models with significantly fewer training samples.

DataFlow includes several ready-to-use pipelines for different data tasks:

• Text Pipeline: Mines Q&A pairs from large-scale raw text for SFT and RL training.
• Reasoning Pipeline: Enhances existing Q&A pairs with Chain-of-Thought (CoT), category labeling, and difficulty estimation.
• Text2SQL Pipeline: Converts natural language questions into SQL queries with explanations and CoT reasoning.
• Knowledge Base Cleaning Pipeline: Extracts and organizes knowledge from unstructured sources (Tables, PDF, Word) for RAG or QA generation.
• Agentic RAG Pipeline: Mines Q&A pairs that require external knowledge to answer, used for training Agentic RAG models.

DataFlow supports several specialized pipelines for different data tasks, as demonstrated by the following experimental categories:

• Text Pipelines: Includes Pre-training Data Filtering (e.g., filtering SlimPajama-627B) and SFT (Supervised Fine-Tuning) Data Filtering & Synthesis (using Condor Generator and Condor Refiner).
• Mathematical Pipelines: Focused on reasoning tasks (e.g., GSM8K, AIME, Olympiad).
• Code Pipelines: Produces refined code instruction datasets (e.g., DataFlow-Code-1K, DataFlow-Code-5K, DataFlow-Code-10K) from sources like Ling-Coder-SFT.
• Dialogue Synthesis Pipelines: Generates conversational datasets (e.g., DataFlow-Chat-15K).
• Reasoning Pipelines: Uses high-quality seed sets (like NuminaMath) to build reasoning datasets (e.g., DataFlow-Reasoning-10K).

The DataFlow ecosystem consists of four integrated layers:

• DataFlow-Skills: A collection of skills and tutorials for operator development, pipeline construction, and best practices.
• DataFlow-WebUI: A visual and AI Agent-assisted workspace for building, managing, and running pipelines via drag-and-drop and natural language.
• DataFlow-Ecosystem: A modular distribution layer that standardizes operator registration, allowing domain-specific modules (like DataFlow-MM or DataFlow-AI4S) to contribute libraries.
• RayOrch: A high-performance orchestration layer built on Ray, providing distributed compute scheduling and resource management for massive-scale data tasks.

The DataFlow framework provides several specialized pipelines for data processing and synthesis:

• Text Pipeline: Includes pre-training data filtering (e.g., for SlimPajama corpora) and SFT (Supervised Fine-Tuning) data filtering and synthesis using the Condor Generator and Condor Refiner pipelines.
• Math Benchmarks/Pipelines: Specialized workflows for mathematical reasoning data.
• Code Pipeline: Curates code instruction datasets (e.g., DataFlow-Code-1K, DataFlow-Code-5K, DataFlow-Code-10K) from sources like Ling-Coder-SFT.
• Conversation Synthesis Pipeline: Generates conversational datasets like DataFlow-Chat-15K.
• Reasoning Pipeline: Constructs reasoning-focused datasets like DataFlow-Reasoning-10K using seed datasets like NuminaMath.

DataFlow provides several pre-configured pipelines for common data tasks:

• Text Pipeline: Mines question-answer pairs from large-scale plain-text data for SFT and RL training.
• Reasoning Pipeline: Enhances QA pairs with extended chain-of-thought, category classification, and difficulty estimation.
• Text2SQL Pipeline: Translates natural language questions into SQL queries with explanations and reasoning.
• Knowledge Base Cleaning Pipeline: Extracts and structures knowledge from tables, PDFs, and Word documents for RAG or QA generation.
• Agentic RAG Pipeline: Identifies and extracts QA pairs from existing datasets that require external knowledge.
• DataFlow Agent: An intelligent assistant that performs data analysis, writes custom operators, and orchestrates pipelines automatically.

It is recommended to initialize a git repository for release and distribution purposes. Use the following commands to initialize and create an initial commit:

git init
git add .
git commit -m "Initial commit"

For easy deployment, you can use a pre-built Docker image or build your own from the provided Dockerfile. The Docker image includes CUDA 12.4.1 and vLLM pre-installed. Ensure you have the NVIDIA Container Toolkit installed for GPU support.

Option 1: Use Pre-built Docker Image

# Pull the pre-built image
docker pull molyheci/dataflow:cu124

# Run the container with GPU support
docker run --gpus all -it molyheci/dataflow:cu124

Option 2: Build from Dockerfile

# Clone the repository
git clone https://github.com/OpenDCAI/DataFlow.git
cd DataFlow

# Build the Docker image
docker build -t dataflow:custom .

# Run the container
docker run --gpus all -it dataflow:custom