DOCSEARCHbyAlgolia
Submit Docs

Synthetic Data Vault (SDV)

repository·main·Indexed 25 days ago

https://github.com/sdv-dev/sdv

A Python library for creating high-fidelity tabular synthetic data using machine learning models, ranging from statistical methods to deep learning. SDV supports single-table, multi-table (relational), and timeseries data modalities. It provides a complete workflow for data synthesis, preprocessing, and evaluation, including tools like GaussianCopulaSynthesizer and a benchmarking framework to measure data quality and similarity.

Tokens
1.6K
Snippets
7
Records
10
Agent score
86%

What's inside SDV

  1. Overview of the Synthetic Data Vault (SDV) ecosystem

    main

    The Synthetic Data Vault (SDV) is a synthetic data generation ecosystem designed to learn the statistical properties of real datasets and generate new synthetic data that maintains the same format and properties.

    SDV supports three main data modalities:

    • Single-table datasets: Uses Copulas and Deep Learning (e.g., CTGAN) to handle multiple data types, missing data, and custom constraints.
    • Multi-table (relational) datasets: Uses Copulas and recursive modeling to handle complex relational structures defined via a custom JSON metadata schema.
    • Timeseries datasets: Uses statistical, Autoregressive, and Deep Learning models for multi-variate, multi-type timeseries, supporting conditional sampling.

    Additionally, SDV provides an evaluation framework to measure the quality of synthetic data across different modalities and a benchmarking framework to compare various generators using distributed computing and prepared datasets.

  2. Explore SDV documentation and resources

    main

    To use SDV effectively, you can access the following resources:

    • Getting Started: Initial setup and quickstart guides.
    • User Guides: Detailed instructions for specific tasks and workflows.
    • API Reference: Comprehensive documentation of the public API surface.
    • Developer Guides: Information for those looking to contribute or extend the library.
    • Release Notes: History of changes and updates.
    • Slack Workspace: Join the community for announcements, support, and feature suggestions.

  3. Load demo datasets for testing

    main

    To get started with SDV, you can use the download_demo function from sdv.datasets.demo to load sample data and its corresponding metadata. This is useful for testing single-table, multi-table, or sequential data workflows.

    from sdv.datasets.demo import download_demo

# Load a single table dataset
real_data, metadata = download_demo(modality='single_table', dataset_name='fake_hotel_guests')
    from sdv.datasets.demo import download_demo

real_data, metadata = download_demo(modality='single_table', dataset_name='fake_hotel_guests')

  4. Synthesize single-table data using GaussianCopulaSynthesizer

    main

    To generate synthetic data, you must create a synthesizer object, fit it to your real data, and then sample from it. The GaussianCopulaSynthesizer is a classical statistical model available in sdv.single_table.

    from sdv.single_table import GaussianCopulaSynthesizer

# Initialize the synthesizer with metadata
synthesizer = GaussianCopulaSynthesizer(metadata)

# Train the model on real data
synthesizer.fit(data=real_data)

# Generate synthetic rows
synthetic_data = synthesizer.sample(num_rows=500)
    from sdv.single_table import GaussianCopulaSynthesizer

synthesizer = GaussianCopulaSynthesizer(metadata)
synthesizer.fit(data=real_data)

synthetic_data = synthesizer.sample(num_rows=500)

  5. Install the SDV library

    main

    You can install the Synthetic Data Vault (SDV) using either pip or conda. It is recommended to use a virtual environment to avoid software conflicts.

    Using pip:

    pip install sdv

    Using conda:

    conda install -c pytorch -c conda-forge sdv

  6. Visualize column distributions in synthetic data

    main

    To visually compare the distributions of a specific column between real and synthetic datasets, use get_column_plot from sdv.evaluation.single_table. This returns a figure object (e.g., for use with Plotly).

    from sdv.evaluation.single_table import get_column_plot

fig = get_column_plot(
    real_data=real_data,
    synthetic_data=synthetic_data,
    column_name='amenities_fee',
    metadata=metadata,
)

fig.show()

  7. Run SDV performance benchmarks

    main

    The sdv.benchmark.run_benchmark function evaluates SDV's performance across a collection of demo datasets or custom datasets stored in a local folder. It returns a pandas DataFrame containing the dataset name and its corresponding score. If a dataset fails or times out, an error column is added with details.

    Arguments:

    • datasets (list): Names of demo datasets or custom datasets.
    • datasets_path (str, optional): Path to custom datasets. If omitted, names are treated as demo datasets.
    • distributed (bool): Whether to use Dask for execution. Defaults to True.
    • timeout (int): Maximum seconds allowed for modeling, sampling, and evaluating each dataset. Datasets exceeding this return a score of None.
    from sdv.benchmark import run_benchmark

# Example: Run benchmark on specific demo datasets using Dask and a 60s timeout
scores = run_benchmark(datasets=['DCG_v1', 'trains_v1', 'UTube_v1'], distributed=True, timeout=60)

  8. List available demo datasets

    main

    Use sdv.demo.get_available_demos to retrieve a table of all available demo datasets. The resulting table includes the dataset name and properties such as the number of tables, total rows, and total columns.

    from sdv.demo import get_available_demos

demos = get_available_demos()

  9. Evaluate synthetic data similarity with evaluate()

    main

    Use the sdv.evaluation.evaluate function to compare a dictionary of synthetic tables against a dictionary of real tables. The function returns a single maximization score indicating similarity: higher values represent better modeling quality. Note that scores are frequently negative.

    To use this, you need:

    1. samples: A dictionary containing table names and their corresponding synthetic DataFrames.
    2. tables: A dictionary containing table names and their corresponding real DataFrames.
    3. metadata: The metadata object used for modeling.

    For advanced visualizations and detailed reports, use the SDMetrics library directly.

  10. Evaluate synthetic data quality

    main

    You can evaluate how well your synthetic data matches your real data using the evaluate_quality function from sdv.evaluation.single_table. This returns an overall quality score (0-100%) and detailed breakdowns of column shapes and pair trends.

    from sdv.evaluation.single_table import evaluate_quality

quality_report = evaluate_quality(real_data, synthetic_data, metadata)