DOCSEARCHbyAlgolia
Submit Docs

NeMo Data Designer Documentation

repository·main·Indexed 24 days ago

https://github.com/nvidia-nemo/datadesigner

A framework for generating high-quality synthetic datasets featuring statistical sampling, LLM-based generation with field dependencies, and built-in validation. The ecosystem includes the data-designer-engine for execution, data-designer-config for programmatically constructing generation pipelines via DataDesignerConfigBuilder, and a CLI for managing model providers, MCP providers, tool configurations, and plugin catalogs.

Tokens
278.3K
Snippets
420
Records
1.1K
Agent score
84%

What's inside NeMo Data Designer

  1. Overview of the Retriever SDG Plugin

    main

    The data-designer-retrieval-sdg plugin is a first-party Data Designer extension designed to transform source documents into high-quality retriever training and BEIR evaluation data. It automates a four-stage pipeline: document bundling/chunking, artifact extraction and QA generation, deduplication and judging, and finally, exporting compatible artifacts for training and evaluation.

    Key components include:

    • document-chunker (seed reader): Converts text files into sentence chunks with stable chunk_id values to ensure generated questions can always map back to their source evidence.
    • embedding-dedup (column generator): Uses embeddings to remove near-duplicate questions, ensuring training data variety.

    This plugin is compatible with [AutoModel] and is used to power NeMo embedding and reranking fine-tune recipes.

  2. Overview of NeMo Data Designer CLI capabilities

    main

    The NeMo Data Designer CLI is an interactive tool used to manage the configuration and ecosystem of the Data Designer platform. It allows users to manage:

    • Model Providers: Configure LLM API endpoints (e.g., NVIDIA, OpenAI, Anthropic, or custom providers).
    • Model Configs: Define specific model settings including inference parameters like temperature and max tokens.
    • MCP Providers: Set up Model Context Protocol (MCP) server configurations for tool integration.
    • Tool Configs: Define tools used by configured models and workflows.
    • Managed Assets: Download and manage persona datasets.
    • Plugin Catalogs & Packages: Discover, install, and uninstall plugin packages from various catalogs.

    By default, all configuration files and managed state are stored in ~/.data-designer/.

  3. Use NeMo Data Designer for generating structured synthetic data

    main

    NeMo Data Designer is an orchestration framework used to generate high-quality synthetic data at scale. It can be used to build iterative pipelines that produce diverse, schema-constrained structured outputs in formats such as JSON, YAML, and XML.

    By using techniques like prompt refinement, rejection sampling, and programmatic validation, you can generate verified datasets to train models for better adherence to schemas (e.g., improving JSONSchemaBench or StructEval-Text scores).

  4. What is a ResourceProvider?

    main

    A ResourceProvider is a bundle of runtime dependencies passed to generators during initialization. It ensures generators have access to all necessary external services and data without needing to manage them directly. It includes:

    • ModelRegistry
    • MCPRegistry
    • ArtifactStorage
    • Seed readers
    • Person readers
    • secret_resolver

  5. What is the `agent context` command?

    main

    The data-designer agent context command is a specialized tool that dynamically generates a structured reference from the library's source code. It is designed to provide an AI agent with all the necessary technical details in a single read, preventing the need for multiple tool calls to piece together the API surface.

    It surfaces:

    • Column types
    • Sampler parameters
    • Validator configurations
    • Constraints and processors
    • Available model aliases and their providers
    • Installed persona datasets
    • Specific files required for context

    Because the output is derived from the code at runtime, the context remains synchronized with the current version of the API.

  6. Understand the Data Designer Async Scheduling Architecture Contracts

    main

    The async scheduling architecture relies on specific durable names and semantics to ensure stability. While implementation details may change, the following contract families define the normative vocabulary for the system:

    • Generator metadata and public config DTOs (data-designer-config): Includes SchedulingMetadata, metadata validation errors, and exposed RunConfig fields.
    • Scheduler/request runtime protocols (data-designer-engine): Includes queues, controllers, policies, leases, runtime snapshots, and event DTOs.
    • User interface and operator presentation (data-designer): Consumes the config and engine contracts for the public DataDesigner interface, CLI, and integrations.

    Constraint: Config-layer contracts must not import engine runtime protocols. Engine contracts may consume config-layer DTOs.

  7. Use image-based columns in multimodal pipelines

    main

    NeMo Data Designer allows you to extend text-based pipelines into multimodal ones by using specialized image columns. The engine treats images as first-class citizens within a row, maintaining the link between the image and the sampler metadata that produced it.

    There are three primary types of image building blocks:

    1. Text-to-image columns: Generate visual examples from controlled metadata and prompt templates.
    2. Image-to-image columns: Use an existing or generated image as context to produce controlled variants (e.g., image editing).
    3. Image-to-text columns: Pass images into VLM-backed (Vision Language Model) columns for tasks like captioning, VQA (Visual Question Answering), labeling, or extraction.

    When using create() mode, image bytes are saved to disk (typically under images/<column_name>/ with UUID filenames) and the dataframe stores relative paths. This prevents large image payloads from bloating memory while allowing images to be passed into subsequent image-aware columns via ImageContext.

  8. How ThrottledModelClient manages concurrency and rate limits

    main

    The ThrottledModelClient is a wrapper for ModelClient that implements automatic concurrency control and AIMD (Additive Increase/Multiplicative Decrease) feedback loops. It intercepts every call to manage throttle slots based on the ThrottleDomain:

    • completion / acompletion $\rightarrow$ ThrottleDomain.CHAT
    • embeddings / aembeddings $\rightarrow$ ThrottleDomain.EMBEDDING
    • generate_image / agenerate_image $\rightarrow$ ThrottleDomain.IMAGE (if request.messages is None) or ThrottleDomain.CHAT (if request.messages is set).

    Lifecycle of a request:

    1. Acquire: On entry, it attempts to acquire a throttle slot. If it fails due to a timeout, it raises a ProviderError(kind=TIMEOUT).
    2. Execution: The inner client method is called.
    3. Release:
      • On ProviderError(kind=RATE_LIMIT): Calls release_rate_limited using the retry_after value from the error.
      • On other ProviderError or any BaseException (including asyncio.CancelledError): Calls release_failure to ensure concurrency slots are not leaked.
      • On success: Calls release_success to trigger additive increase.

    This wrapper ensures that even if an async request is cancelled, the throttle permit is released, preventing permanent reduction in available concurrency.

    class ThrottledModelClient:
    def __init__(
        self,
        inner: ModelClient,
        throttle_manager: ThrottleManager,
        provider_name: str,
        model_id: str,
    ) -> None:
        self._inner = inner
        self._tm = throttle_manager
        self._provider_name = provider_name
        self._model_id = model_id

  9. How the Compilation Pipeline works

    main

    The compiler.py module transforms a DataDesignerConfig into an execution-ready form through a multi-step pipeline:

    1. Enrichment: The configuration is enriched with seed columns and an internal UUID column.
    2. Static Validation: The system runs validation.py to check for errors before execution begins. This includes verifying Jinja references, code columns, processor targets, and constraint consistency.
    3. Error Reporting: If validation fails, the pipeline produces Violation objects with typed ViolationType to provide structured error reporting.

    This 'fail-fast' approach ensures that configuration errors (like missing references or invalid templates) are caught before expensive LLM calls are made.

  10. Evaluating Data Designer design choices

    main

    During the preview phase, evaluate if you have selected the most efficient Data Designer features for your requirements:

    • Specialized Column Types: If a text column consistently produces structured data or code, consider switching to a specialized column type.
    • Samplers vs. LLM Columns: If values are drawn from a fixed set or a known distribution, use a sampler instead of an LLM column to improve efficiency and control.

  11. Tune concurrency using AIMD and request-admission control

    main

    Data Designer uses an AIMD (Additive Increase / Multiplicative Decrease) request-admission controller to automatically find the optimal concurrency level for your inference server without manual tuning.

    How Concurrency is Calculated

    concurrent_requests = min(active_ready_model_cells, current_admission_limit, max_in_flight_tasks)

    • max_parallel_requests sets the per-model ceiling.
    • current_admission_limit is the runtime limit managed by AIMD.

    AIMD Behavior

    • Startup Ramp: If startup_ramp_seconds > 0, the system starts at one concurrent request and increases linearly toward max_parallel_requests over the specified duration.
    • Reacting to 429s (Rate Limits): On the first 429 error in a burst, the limit is reduced by a configurable factor (default: 25% reduction) and a cooldown is applied.
    • Recovery: After consecutive successes, the limit increases by a configurable step (default: 1) until it reaches the ceiling or a stabilized threshold.

    Example: With buffer_size=100 and max_parallel_requests=32, if startup_ramp_seconds=30, the system climbs from 1 to 32 requests over 30 seconds. If the server returns 429s, the ramp stops, concurrency drops (e.g., to 24), and AIMD recovery begins once the server stabilizes.

  12. Supported Data Designer plugin types

    main

    Data Designer allows you to extend its functionality without modifying the core library using three specific plugin types. Once installed via Python entry points, these plugins behave like native objects and are automatically discovered by the engine.

    • Column generator plugins: Used to create custom column types. These are passed to the config builder's add_column method.
    • Seed reader plugins: Used to load seed datasets from new sources (e.g., databases, cloud storage, or custom file formats).
    • Processor plugins: Used to transform data at different lifecycle stages (before batches, after batches, or after generation completes). These are passed to the config builder's add_processor method.