DOCSEARCHbyAlgolia
Submit Docs

Data-Juicer Documentation

repository·main·Indexed 27 days ago

https://github.com/datajuicer/data-juicer

A modular and extensible data processing 'operating system' for foundation models. Data-Juicer provides tools to clean, synthesize, and analyze large-scale multimodal datasets for pre-training, fine-tuning, RAG, and agent systems. Key features include automatic hyper-parameter optimization (Auto-HPO) using 3-Sigma principles or WandB Sweeps, a quality classifier toolkit for document scoring, and specialized recipe-driven pipelines for agent log analysis and bad-case reporting.

Tokens
122.5K
Snippets
256
Records
560
Agent score
92%

What's inside Data-Juicer

  1. Overview of Agent Pipeline Post-Analysis Scripts

    main

    The Agent pipeline post-analysis scripts are used to process and analyze data after running the dj-process step. These scripts allow for verifying exports, computing statistical thresholds, analyzing cohorts, and generating HTML reports for bad-case analysis.

    Key Scripts:

    • generate_bad_case_report.py: Generates self-service HTML reports including tiering, signal attribution tables, and cohort views.
    • bad_case_signal_support.py: Provides static data for signal attribution (mapping signal codes to upstream metadata/stats).
    • dj_export_row.py: Parses __dj__meta__ and __dj__stats__ and merges them with corresponding *_stats.jsonl files.
    • verify_bad_case_export.py: Verifies if agent_bad_case_tier and agent_bad_case_signals fields are present in the export.
    • compute_percentile_thresholds.py: Aggregates metrics (tokens, latency, perplexity) by agent_request_model and computes percentiles or generates calibration JSON.
    • analyze_bad_case_cohorts.py: Summarizes data by model, prompt template (pt), or tier.
    • slice_export_by_tier.py: Exports subsets of data based on meta.agent_bad_case_tier.

  2. Overview of Data-Juicer Operators

    main

    Data-Juicer uses Operators (OPs) to perform data modification, cleaning, filtering, and deduplication. Operators are categorized into 8 functional types:

    • aggregator: Aggregates batched samples (e.g., generating summaries).
    • deduplicator: Detects and removes duplicate samples.
    • filter: Filters out low-quality samples.
    • formatter: Discovers, loads, and canonicalizes source data.
    • grouper: Groups samples into batches.
    • mapper: Edits and transforms individual samples.
    • pipeline: Applies dataset-level processing (input and output are full datasets).
    • selector: Selects top-ranked samples based on quality.

    Operators are further classified by:

    • Modality: Text, Image, Audio, Video, or Multimodal.
    • Resource: CPU or GPU requirements.
    • Usability: Alpha (basic implementation), Beta (includes unit tests), or Stable (optimized for Data-Juicer).
    • Model Support: API (e.g., ChatGPT), vLLM, or HF (HuggingFace Hub).

  3. Overview of Data-Juicer Data Scoring Capabilities

    main

    Data-Juicer offers several ways to score and evaluate datasets:

    1. Filter Operators: All filter operators include a compute_stats sub-process that calculates statistical measurements (e.g., perplexity, length, modality matching) which are aggregated by the Analyzer.
    2. LLM Scoring: Prompt-based operators like llm_difficulty_score_filter and llm_quality_score_filter allow for scoring using customizable LLM prompts.
    3. Quality Classifier Toolkit: A PySpark-based toolkit to reproduce and apply the GPT-3 quality classifier using Tokenizer, HashingTF, and LogisticRegression.

  4. Explore Data-Juicer Extensions and Ecosystem

    main

    Data-Juicer integrates with several specialized extensions and a wide range of frameworks and platforms:

    Extensions

    • data-juicer-agents: DJ Copilot and agentic workflows.
    • data-juicer-hub: Community recipes and best practices.
    • data-juicer-sandbox: Data-model co-development with feedback loops.

    Supported Frameworks & Platforms

    Data-Juicer is compatible with various data storage and machine learning ecosystems, including:

    • Storage/Data Formats: Apache Arrow, Apache HDFS, Apache Hudi, Apache Iceberg, Apache Paimon, Delta Lake, LanceDB.
    • ML Frameworks/Platforms: Alibaba Cloud PAI, Hugging Face, LLaMA-Factory, ModelScope, NVIDIA NeMo, Ray, AgentScope, DiffSynth-Studio, EasyAnimate, Eval-Scope.
    • Hardware: Huawei Ascend.

  5. Use Elastic Multi-Node Sharding for large datasets

    main

    The Elastic Multi-Node Sharding demo provides an execution mode for processing large JSONL datasets across multiple nodes without requiring a cross-node Ray cluster. Instead, it uses a shared POSIX/NAS/CPFS filesystem to coordinate independent nodes, where each node runs its own local Ray runtime.

    Workflow:

    1. Pre-split input into deterministic JSONL shards.
    2. Workers dynamically claim shards via the shared directory.
    3. Each node processes claimed shards using its local Ray executor.
    4. Results are validated and merged in the original order.

    Key Requirements:

    • All Workers must use the same Data-Juicer version and dependencies.
    • The --job-dir must be on a shared filesystem supporting atomic operations (e.g., NAS, NFS, CPFS).
    • Every Worker must see the job directory, input JSONL, and local media at identical paths.
    • Only shard-independent Mapper and Filter operators are supported. Deduplicators, Selectors, and other whole-dataset operations are not supported.

  6. Explore Data-Juicer interactive demo types

    main

    The following interactive demos are available for exploring Data-Juicer features:

    • overview_scan: Introduces basic concepts like features, configuration, and operators.
    • data_process_loop: Analyzes and processes datasets, comparing statistical information before and after processing.
    • data_visualization_diversity: Visualizes dataset diversity (e.g., verb-noun structure) using sunburst plots.
    • data_visualization_op_effect: Displays the effect of different Filter operators by adjusting thresholds.
    • data_visualization_statistics: Analyzes datasets to obtain up to 13 different statistics.
    • process_cft_zh_data: Demonstrates processing Chinese Alpaca-CoT data for IFT/CFT LLM fine-tuning.
    • process_sci_data: Demonstrates processing arXiv scientific literature for LLM pre-training.
    • process_code_data: Demonstrates processing Stack-Exchange code data for LLM pre-training.
    • tool_quality_classifier: Provides three text quality classifiers to score datasets.
    • tool_dataset_splitting_by_language: Splits a dataset into sub-datasets based on language.
    • data_mixture: Selects and mixes samples from multiple datasets into a new exported dataset.

  7. Explore Data-Juicer interactive demo categories

    main

    The following interactive demos are available in the demos/ directory:

    • overview_scan: Introduces basic concepts like features, configuration systems, and operators.
    • data_process_loop: Analyzes and processes datasets, providing statistical comparisons before and after processing.
    • data_visualization_diversity: Visualizes lexical diversity (e.g., verb-noun structures in CFT datasets) using sunburst charts.
    • data_visualization_op_effect: Visualizes the effect of different Filter operators at various thresholds based on dataset statistics.
    • data_visualization_statistics: Analyzes datasets to extract up to 13 different types of statistical information.
    • process_cft_zh_data: Demonstrates analysis and processing workflows for instruction-following and supervised fine-tuning (SFT) data using Alpaca-CoT Chinese data.
    • process_sci_data: Demonstrates workflows for processing scientific literature (e.g., arXiv data) for LLM pre-training.
    • process_code_data: Demonstrates workflows for processing code data (e.g., Stack-Exchange) for LLM pre-training.
    • tool_quality_classifier: Provides three types of text quality scorers for dataset evaluation.
    • tool_dataset_splitting_by_language: Splits a dataset into sub-datasets based on language.
    • data_mixture: Samples from multiple datasets to create a new mixed dataset.

  8. Use VBench metrics for video generative model evaluation

    main
    VBench metrics are provided based on the paper "VBench: Comprehensive Benchmark Suite for Video Generative Models". These metrics can be used to evaluate video generative models. For detailed implementation details, specific metric definitions, and advanced usage, refer to the official VBench GitHub repository.

  9. Explore the Data-Model Co-Development Taxonomy

    main

    Data-Juicer provides a taxonomy for the co-development of Multi-Modal Large Language Models (MLLMs), categorizing research and tools into four primary dimensions:

    1. Data4Model: Scaling: Focuses on increasing dataset size (Acquisition, Augmentation, Diversity) and improving subset effectiveness (Condensation, Mixture, Packing, Cross-Modal Alignment).
    2. Data4Model: Usability: Focuses on instruction responsiveness (Prompt Design, ICL Data, Human-Behavior Alignment), reasoning abilities (Single-Hop and Multi-Hop Reasoning), ethics (Toxicity, Privacy & IP), and evaluation benchmarks (Understanding, Generation, Retrieval, Reasoning).
    3. Model4Data: Synthesis: Uses models to create, map, filter, or evaluate data.
    4. Model4Data: Insights: Uses models to navigate, extract, analyze, or visualize data.

  10. Supported Post-Tuning Data Formats

    main

    Data-Juicer supports conversion for four primary post-tuning data formats used by frameworks like ModelScope-Swift and LLaMA-Factory:

    1. Swift Messages Format: Uses a messages list with role and content keys.
    2. Swift ShareGPT Format: Uses a system key and a conversation list with human and assistant keys.
    3. Alpaca Format: Uses system, instruction, input, and output keys.
    4. Swift Query-Response Format: Uses system, query, response, and a history list of pairs.

    Data-Juicer uses a unified intermediate representation for multi-turn dialogue processing.

  11. Review Data-Juicer release history and new features

    main

    The Data-Juicer release notes provide details on new operators (OPs), performance optimizations, and stability fixes. Key recent updates include:

    • v1.5.4: Added 9 human-centric video understanding operators (e.g., active-speaker detection, face demographic captioning) and FusedSequentialBatchOp for faster sequential processing.
    • v1.5.3: Added VLA (Vision-Language-Action) operators and the ray_repartition_pipeline for dataset-level block repartitioning in Ray mode.
    • v1.5.2: Introduced DocumentLineDeduplicator for cross-document line-level deduplication and semantic LLM operators (llm_extract_mapper, llm_condition_filter, llm_structured_ops).
    • v1.5.1: Added LaTeX-focused mapper OPs and support for compressed dataset formats like json[l].gz.
    • v1.5.0: Introduced a partitioned Ray executor and OP-level isolated environments for better fault tolerance and dependency management.
    • v1.4.5: Introduced Ray + vLLM pipelines for LLM/VLM inference.
    • v1.4.4: Added S3 I/O support for loaders and exporters.

  12. Access Data-Juicer Documentation and Resources

    main

    Detailed documentation, operator lists, and developer guides are available through the following official links: