SDMetrics

Single Table Metrics in sdmetrics operate on individual tables by comparing two pandas.DataFrame objects: the real_data and the synthetic_data. These metrics are categorized into several functional groups:

• BayesianNetwork Metrics: Evaluates the likelihood of synthetic data being sampled from a distribution fitted to real data (e.g., BNLikelihood, BNLogLikelihood).
• GaussianMixture Metrics: Uses Gaussian Mixture Models to evaluate likelihood (e.g., GMLogLikelihood).
• Detection Metrics: Uses ML classifiers to distinguish between real and synthetic data; the score is the complement of the classifier's cross-validated score (e.g., LogisticDetection, SVCDetection).
• ML Efficacy Metrics: Evaluates how well an ML model trained on synthetic data performs on real data. Warning: These require datasets representing ML problems and scores depend on problem difficulty (e.g., LinearRegression, MulticlassMLPClassifier, and the generic MLEfficacy).
• Privacy Metrics: Evaluates the accuracy of an adversarial attacker model attempting to identify real data from synthetic data (e.g., CategoricalCAP, NumericalMLP, CategoricalEnsemble).
• MultiSingleColumn Metrics: Applies a single-column metric to all compatible columns and returns the average (e.g., CSTest for categorical, KSComplement for numerical).
• MultiColumnPairs Metrics: Applies a column-pair metric to all compatible pairs and returns the average (e.g., ContinuousKLDivergence, DiscreteKLDivergence).

Time series metrics in sdmetrics operate on individual tables representing sequential data. They are used to evaluate synthetic time series data by comparing it against real data.

There are two main categories of implemented metrics:

1. Detection Metrics: These train a Machine Learning Classifier (e.g., LSTMDetection using PyTorch) to distinguish between real and synthetic data. The score is the complement of the classifier's cross-validated score.
2. ML Efficacy Metrics: These compare the performance of a model trained on synthetic data versus one trained on real data (e.g., LSTMClassifierEfficacy). The output is the ratio of the synthetic-trained model's score to the real-trained model's score.

Warning: ML Efficacy metrics are only meaningful for datasets representing relatively easy machine learning problems. If real-data model performance is too low, the output may be meaningless.

Instead of passing individual arguments like sequence_key or context_columns, you can pass a metadata dictionary. This dictionary must follow the SDV Metadata JSON schema.

Using metadata is preferred because:

1. It allows the metric to accurately identify column types (e.g., distinguishing categorical columns from numerical ones).
2. It can automatically provide required information like the sequence_key.

If metadata is not provided, it will be inferred from the real_data, but this may lead to inaccuracies in column type detection.

from sdmetrics.timeseries import LSTMDetection
from sdmetrics.demos import load_timeseries_demo

real_data, synthetic_data, metadata = load_timeseries_demo()

# Update metadata with the sequence key
metadata['sequence_key'] = 'store_id'

# Compute the metric using the metadata argument
score = LSTMDetection.compute(real_data, synthetic_data, metadata=metadata)
print(score)

To compute a metric, call the .compute() method on the specific metric class. At a minimum, you must provide real_data and synthetic_data as pandas.DataFrame objects.

Some metrics require additional arguments:

• target: A string specifying the target column (required for ML Efficacy metrics).
• metadata: A dictionary following the SDV Metadata JSON schema. While optional (it can be inferred from the data), providing it ensures correct type detection (e.g., distinguishing categorical integers from numerical ones) and allows passing extra info like the target column.

from sdmetrics.single_table import LogisticDetection
from sdmetrics import load_demo

# Setup data
real_data, synthetic_data, metadata = load_demo()
real_table = real_data['users']
synthetic_table = synthetic_data['users']

# Basic usage
score = LogisticDetection.compute(real_table, synthetic_table)

# Usage with target and metadata for ML Efficacy
from sdmetrics.single_table import MulticlassDecisionTreeClassifier

# Option 1: Passing target directly
score = MulticlassDecisionTreeClassifier.compute(real_table, synthetic_table, target='country')

# Option 2: Passing target via metadata
users_metadata = metadata['tables']['users'].copy()
users_metadata['target'] = 'country'
score = MulticlassDecisionTreeClassifier.compute(
    real_table, 
    synthetic_table, 
    metadata=users_metadata
)