NVIDIA NeMo Curator Documentation

NeMo Curator provides a modular, scalable pipeline for curating high-quality image datasets, specifically designed for training generative AI models like LLMs, VLMs, and WFMs. The system allows for large-scale processing of image-text datasets, including quality filtering, content filtering, and semantic deduplication.

Key capabilities include:

• Quality Control: Applying aesthetic filtering and removing low-quality images.
• Content Filtering: Removing inappropriate or NSFW content.
• Deduplication: Using semantic similarity to remove duplicate images from large collections.
• Feature Extraction: Generating embeddings (e.g., via CLIP) for search and retrieval.
• Scalability: Distributed processing across multiple GPUs and nodes using Ray and GPU-accelerated DALI.

This section provides guidance for running NeMo Curator in production environments, focusing on scaling from a single development workstation to multi-node clusters. It covers deployment topologies, hardware/software requirements, and integrations with external systems.

Note: If you are looking to install NeMo Curator on a single machine rather than a production cluster, refer to the Install (All Modalities) guide.

NeMo Curator provides a modular pipeline architecture for curating high-quality image datasets at scale. The workflow is designed to handle large-scale image-text datasets by applying quality filtering, content filtering, and semantic deduplication.

Key capabilities include:

• Data Loading: Efficiently reading JPEG images from tar archives using GPU-accelerated DALI.
• Feature Extraction: Generating embeddings using CLIP models (e.g., ViT-L/14).
• Filtering: Applying aesthetic quality scoring and NSFW content filtering.
• Deduplication: Removing duplicate images via semantic similarity and clustering.
• Export: Saving metadata to Parquet and exporting datasets with custom resharding into tar archives.

The architecture is built for scalability using Ray for distributed processing across multiple GPUs and nodes.

NeMo Curator provides a suite of capabilities for preparing high-quality speech data for Automatic Speech Recognition (ASR) and multi-modal model training. The toolkit is designed to handle large-scale speech datasets by providing processors for loading audio, performing ASR inference, assessing transcription quality, and integrating audio results with text curation workflows.

Key Capabilities:

• ASR Inference: Generate transcriptions using NVIDIA NeMo ASR models.
• Quality Assessment: Filter datasets based on transcription accuracy metrics like Word Error Rate (WER) and Character Error Rate (CER).
• Audio Analysis: Extract metadata such as duration and validate audio formats.
• Data Integration: Combine audio processing results with text curation pipelines for multi-modal workflows.
• Scalability: Efficiently scale audio processing across GPU clusters.

NeMo Curator provides guidance for transitioning from development workstations to production-scale environments. The setup documentation covers deployment topologies, hardware and software requirements, and integrations with external systems to support multi-node clusters.

For single-machine installations, users should refer to the Install (All Modalities) guide instead of the production deployment resources.

NeMo Curator is an enterprise-grade platform designed for scalable, privacy-aware data curation across multiple modalities: text, image, video, and audio. It is used to prepare high-quality, compliant datasets for LLM and generative AI training.

Key capabilities include:

• Scalability: Supports workflows ranging from local laptops to multi-node GPU clusters.
• Modality Support: Specialized pipelines for Text (cleaning, deduplication, classification), Image (embeddings, aesthetic/NSFW classification), Audio (ASR, WER calculation), and Video (distributed processing).
• Performance: Proven to handle massive scales, such as deduplicating 1.96 trillion tokens in 0.5 hours using 32 NVIDIA H100 GPUs.
• Deployment: Supports distributed, cloud-native, and on-premises environments with integration into modern MLOps pipelines.

The Setup & Deployment documentation provides guidance for transitioning NeMo Curator from a single-machine development environment to a multi-node production cluster. This includes information on deployment topologies, hardware/software requirements, and integrations with external systems.

If you are looking to install NeMo Curator on a single machine rather than a production cluster, refer to the Install (All Modalities) guide instead.

NeMo Curator provides hands-on tutorials for curating interleaved multimodal data (documents containing text, images, and metadata interlaced in reading order).

Available tutorial paths:

1. Getting Started: Focuses on loading, exploring, filtering, and saving interleaved data using MINT-1T PDF shards.
   • Key files: interleaved_data_quickstart.ipynb, interleaved_pipeline.py
2. PDF Extraction Pipeline (Nemotron-Parse): Demonstrates how to convert PDFs into structured interleaved Parquet files using Nemotron-Parse v1.2.
   • Key file: main.py

NeMo Curator is a framework designed to streamline the curation and pre-processing of large-scale datasets for training generative AI models. It supports multiple modalities, including text, image, audio, and video.

Quickstarts are designed for AI/ML engineers and researchers to test capabilities or run initial curation pipelines with minimal setup, typically taking less than 30 minutes. For production-grade deployments, cluster configurations, or detailed system requirements, refer to the Setup & Deployment documentation.

When deploying NeMo Curator on SLURM, you may need to reference these related infrastructure concepts:

• Resumable Processing: For handling interruptions and resuming work.
• Multi-Node Ray on SLURM: For scaling Ray clusters across multiple SLURM nodes.
• Execution Backends: To understand how different backends handle task execution.
• Container Environments: For managing dependencies via containers in SLURM.