whylogs Documentation

whylogs organizes data information through the following hierarchy:

• Column Profile: A collection of metrics representing a single feature or column in a dataset.
• Dataset Profile: A collection of Column Profiles along with summary statistics and metadata for an entire dataset.
• Dataset Schema: Defines how metrics are tracked. It maps dataset columns to specific data types and metrics. While schemas are created automatically during logging, they can be customized to define specific column types or metrics to track.
• Metadata: Data describing either the dataset itself or the information produced during whylogs processing.

Monitoring data quality involves using metrics and constraints:

• Metric: The summary statistics that compose a dataset's profile. Metrics are organized by namespaces (e.g., counts, types, distribution, cardinality, frequent items/frequent strings).
  • Metric component: The smallest unit of a metric. A single metric may consist of multiple components (e.g., an ints metric contains min and max components).
• Constraints: Rules created to assert that data stays within expected ranges. Applying constraints to a Dataset Profile allows you to generate reports or perform boolean checks to see if any constraints were violated.

The whylogs V1 storage format is a delimited format designed to pack multiple messages into a single file. This structure allows readers to use chunk offsets to jump directly to relevant metrics without scanning the entire file, unifying both column-level and dataset-level metrics.

Key components of the V1 message format include:

• DatasetHeader: Contains dataset properties (names, tags, timestamps), offsets for column and dataset metrics, and an index of metric names.
• Chunk: The V1 format allows a single column message to contain multiple chunks. Readers use offsets to target specific chunks for a given column.

To use whylogs effectively, understand these fundamental terms:

• whylogs: A data logging library that captures statistical properties of data and ML models.
• Batch vs. Streaming Mode:
  • Batch Mode: Processes a dataset in discrete collections of datapoints (batches).
  • Streaming Mode: Processes individual data points, which the underlying algorithms group into micro-batches.
• Data Sketching: Algorithms used by whylogs to efficiently extract information from large or streaming datasets in a single pass.
• whylogs file: Profiles generated by whylogs can be stored as protobuf binary files, a lightweight serialization format that maps to the memory representation of a whylogs object.

Understanding the WhyLogs hierarchy:

• Project: A collection of related datasets used for multiple models or applications.
• Pipeline: One or more datasets used to build a single model or application.
• Dataset: A collection of records (structured as a table with rows as records and columns as features).
• Feature: A column in a dataset (can be discrete or continuous).
• Statistical Profile: A collection of statistical properties (counters, summary statistics, cardinality, histograms, etc.) for a feature.
• WhyLogs Output: Profile summary files provided in JSON format (flat table, histogram, or frequency formats).

whylogs integrates with various tools across the data and ML lifecycle, including:

• AWS S3
• Apache Airflow
• Apache Spark
• Mlflow
• GCS

A complete list of integration examples can be found in the python/examples/integrations directory of the repository.

whylogs uses Profiling instead of sampling for data logging.

Key advantages of profiling include:

• Scalability & Efficiency: Requires only a single pass over the data with minimal memory overhead and is naturally parallelizable.
• Mergeability: Profiles are mergeable, allowing statistics from multiple hosts, data partitions, or datasets to be merged post-hoc.
• Accuracy: Unlike sampling, profiling allows you to accurately capture rare events and outlier-dependent metrics.

The whylogs.datasets module provides standardized collections of data designed for experimenting with machine learning monitoring tasks, including:

• Drift detection
• Outlier detection
• Data quality and validation
• Time series monitoring
• General data monitoring

The API supports fetching data in batches across different time periods and maintains a clear distinction between baseline (training/reference) and inference (production) datasets.

Available dataset categories include:

• weather
• ecommerce
• employee

whylogs v1 separates the logging process from the inspection process using two distinct concepts:

1. Profile: An object used while logging operations are actively underway.
2. Profile View: An object obtained from a Profile once logging is complete. Use the Profile View to inspect, visualize, merge, and upload profiles.

Note: When a profile is written to a binary file, it is automatically converted to a Profile View.

The Ecommerce dataset currently consists of a single version named base. This version is split into two distinct partitions used for monitoring or model evaluation:

1. Baseline:
   • Instances: 34,743
   • Period: 2022-08-09 to 2022-08-16
2. Inference:
   • Instances: 86,899
   • Period: 2022-08-19 to 2022-09-08

Both partitions contain 19 total features (5 Input, 1 Target, 2 Prediction, and 11 Extra).