YData Profiling

For deeper theoretical knowledge, refer to these resources:

• Books:
  • Machine Learning Pocket Reference by Matt Harrison.
  • Python Data Science Essentials by Alberto Boschetti and Luca Massaron.
• Miscellaneous:
  • Penn Machine Learning Benchmarks (PMLB): A collection of machine learning benchmarks.
  • dabl package: A related data analysis library.

The project follows a specific Git branching model to facilitate collaboration and automation. GitHub Actions automatically lints code on release branches and builds documentation on pushes to the master branch.

Branch Naming Conventions:

• develop: The main development branch.
• master: The production/master branch.
• feat/[FEATURE NAME]: Used for new feature development.
• docs/[branch name]: Used for documentation and examples updates.

The Alerts section of a report automatically lists potential data quality issues. These alerts can refer to specific columns, inter-column relationships, or the dataset as a whole.

Note: Alerts are automated suggestions; domain validation is required to determine if an alert represents a genuine data quality issue.

To configure default values, specific parameters, thresholds, or to disable specific alerts, consult the available settings documentation.

The missing data section provides a summary of missing values across variables, including the percentage of missing data per variable. It uses visual representations such as bar plots and heatmaps to help identify patterns of missingness.

Visualization configuration is available via the user settings.

Instead of configuring individual settings, you can use configuration shorthands to selectively enable or disable entire sections of the report. Setting these to None disables the corresponding computation or display.

Available shorthands:

• samples: Controls whether the dataset preview is shown.
• correlation: Controls whether correlation computations are executed.
• missing_diagrams: Controls whether missing value analysis is executed.
• duplicates: Controls whether duplicate rows are previewed.
• interactions: Controls whether interactions are computed.

from data_profiling import ProfileReport

# Disable samples, correlations, missing diagrams and duplicates at once
r = ProfileReport(
    samples=None,
    correlations=None,
    missing_diagrams=None,
    duplicates=None,
    interactions=None,
)

The Outliers section profiles potential dataset outliers by observing variance and deviation from the general distribution of numerical variables. This helps determine if anomalies are genuine data points or erroneous entries.

Warning: This feature is limited to users of the cloud hosted solution.

YData Profiling provides insights across several data structures:

• Tabular data: Provides statistics on data distribution, central tendencies, categorical frequencies, multivariate relations (correlations/interactions), and missing data.
• Time-series data: Captures trends, seasonality, cyclic patterns, missing data gaps, volatility, periodicity, and anomalies.
• Text: Provides statistics on word frequency distributions, common phrases, and unique words.
• Image data: Supported via the profiling engine.

Multivariate profiling explores relationships between variables through two main lenses:

• Correlation view: Computes correlation coefficients between pairs of numerical variables to identify linear relationships and potential multicollinearity.
• Interactions section: Explores potential nonlinear relationships and interactions between variables to identify hidden patterns not captured by traditional correlation analysis.

The Data Catalog within YData Fabric acts as a searchable repository for an organization's datasets. It serves as a unified view that captures schema and metadata to help data scientists and engineers manage and understand data.

It functions through several integrated layers:

1. Ingestion: Uses built-in connectors to pull data from multiple sources.
2. Metadata Capture: Automatically stores information regarding data types, lineage (source), update frequency, and relationships.
3. Profiling & Analysis: Provides an interactive layer to drill down into data distributions and interactions.
4. Quality Monitoring: Maintains data quality indexes (completeness, uniqueness, etc.) to ensure data reliability.
5. Governance: Enforces security through access controls and identifies sensitive information for regulatory compliance.

Beyond logical types (like integer or float), data-profiling uses a powerful type system backed by visions to allow analysis through higher-level lenses.

Currently recognized types include:

• Boolean
• Numerical
• Date (and Datetime)
• Categorical
• Time-series
• URL
• Path
• File
• Image

Users can customize summarizations and type definitions. The default typeset implementation can be found in the source code.

For a quick overview of the dataset, YData Profiling provides configurable preview sections:

• First n records of the dataset.
• Last n records of the dataset.
• Duplicate table: A table containing observed exact matches (duplicates).