DataProfiler

The DataProfiler API is organized into four primary functional components that allow you to ingest, analyze, categorize, and verify data:

1. Data Readers: Responsible for ingesting data from various formats and sources into a usable structure.
2. Profilers: Used to extract statistical, structural, or semantic information from data (e.g., Structured, Unstructured, or Graph profiles).
3. Labelers: Used to assign semantic labels or categories to data elements (e.g., identifying PII).
4. Validators: Used to verify that data conforms to specific schemas, rules, or expected profiles.

An Expectation is a declarative statement used for data validation, profiling, and documentation. They are semantically meaningful to humans (e.g., expect_column_values_to_be_unique) and are implemented as classes that allow computers to evaluate data quality.

Data Profiler provides a specific set of expectations that can be used within the Great Expectations framework. You can find the full list of available Data Profiler Expectations on the package page.

DataProfiler uses two primary approaches for data analysis depending on the data type:

1. Structured Data: Numeric statistics (such as mean, variance, skewness, and kurtosis) are computed using streaming algorithms. This allows for efficient, incremental updates without needing to recompute from the raw dataset. Approximate quantile metrics (like the median) are calculated using histogram-based estimation to ensure scalability for large or streaming datasets.

2. Unstructured Data: DataProfiler employs a Convolutional Neural Network (CNN) to detect and label entities (e.g., names, emails, credit cards) within text. This is used for PII detection, schema inference, and data quality analysis.

A Graph Profile is used for graph-structured data. It provides metrics on nodes, edges, and attribute distributions.

Core Metrics

• num_nodes: Number of nodes.
• num_edges: Number of edges.
• categorical_attributes: List of categorical edge attributes.
• continuous_attributes: List of continuous edge attributes.
• avg_node_degree: Average degree of nodes.
• global_max_component_size: Size of the largest component.

Attribute Distributions

• continuous_distribution: For each continuous attribute, provides a name, scale (negative log likelihood), and properties (shape, loc, scale, mean, variance, skew, kurtosis).
• categorical_distribution: For each categorical attribute, provides bin_counts and bin_edges for a histogram.
• times: Time taken to generate the profile (ms).

When using DataProfiler, be aware of the following technical constraints:

• Quantile Estimation: Metrics like the median are approximate and based on histogram binning rather than exact sorting.
• Entity Detection: The CNN model assumes consistent formatting for sensitive entities (e.g., standardized SSN or credit card formats). Overlapping entity types (e.g., phone numbers vs. SSNs) may result in misclassification if context is insufficient.
• Model Accuracy: Because the model may rely on synthetic training data, its accuracy on highly diverse or natural unstructured text may be lower than on standardized formats.

Certain profile statistics, specifically the Order profile, are sensitive to the order in which data is updated. The profiler uses the last value of a previous data batch to evaluate the relationship with the first value of the subsequent batch to detect non-random ordering (e.g., ascending or descending).

To maintain an 'ascending' prediction across batches, the first value of a new batch must be greater than or equal to the last value of the previous batch.

A Data Profile is a dictionary containing statistics and predictions. The structure of the profile depends on the type of data being analyzed:

• Structured Profile: Contains global_stats (dataset-level data like row_count, file_type, encoding) and data_stats (column-level data including data_type, data_label, and detailed statistics like mean, stddev, histogram, and quantiles).
• Unstructured Profile: Designed for text/unstructured data. Contains global_stats (like memory_size and empty_line_count) and data_stats focusing on data_label entity counts/percentages and vocabulary statistics.
• Graph Profile: Designed for graph data. Contains properties like num_nodes, num_edges, avg_node_degree, and distributions for categorical and continuous attributes.

DataProfiler supports unstructured profiling for text. You can profile a TextData object, a string, a list[string], a pd.Series of strings, or a pd.DataFrame of strings.

When using a Pandas object for unstructured profiling, you must specify profiler_type='unstructured' in the Profiler constructor.

import dataprofiler as dp
import pandas as pd
import json

# Example 1: Profiling a text file via Data object
my_text = dp.Data('text_file.txt')
profile = dp.Profiler(my_text)
report = profile.report(report_options={"output_format": "pretty"})
print(json.dumps(report, indent=4))

# Example 2: Profiling a pandas Series of strings
text_data = pd.Series(['first string', 'second string'])
profile = dp.Profiler(text_data, profiler_type='unstructured')
report = profile.report(report_options={"output_format": "pretty"})
print(json.dumps(report, indent=4))

The profiler automatically infers whether to create a structured or unstructured profile based on the data. You can explicitly force a type using the profiler_type argument in the Profiler constructor.

• Structured: Used for tabular data (e.g., CSVs) providing column-by-column statistics.
• Unstructured: Used for text data (e.g., .txt files) providing entity recognition and vocabulary statistics.

import json
from dataprofiler import Data, Profiler

# Creating a structured profile
data1 = Data("normal_csv_file.csv")
structured_profile = Profiler(data1, profiler_type="structured")

# Creating an unstructured profile
data2 = Data("normal_text_file.txt")
unstructured_profile = Profiler(data2, profiler_type="unstructured")