huggingface/dataset-viewer

The dataset viewer's backend is a service that provides a REST API for visualizing and exploring datasets (computer vision, speech, text, and tabular) stored on the Hugging Face Hub.

Its core functionality is to automatically convert Hub datasets into Parquet format. To ensure high performance, the backend runs a server that pre-computes and stores API responses in a database, allowing for near-instantaneous queries even for large datasets.

The Dataset Viewer provides a table interface on Hugging Face dataset pages that allows users to navigate rows (in pages of 100), filter, search, and view statistics.

Important distinction: This repository contains only the backend service that provides pre-computed data via an API for all datasets on the Hub. The frontend viewer component is not part of this repository and is not open-source.

The search and filter service provides endpoints to interact with dataset splits.

• GET /search: Retrieves a slice of search results over a specific dataset split.
• GET /filter: Filters rows within a dataset split.

For detailed usage instructions on how to structure queries for these endpoints, refer to the official documentation at huggingface.co/docs/dataset-viewer/search and huggingface.co/docs/dataset-viewer/filter.

Croissant is a metadata format built on top of schema.org designed to describe machine learning datasets. It enables programmatic indexing, searching, and loading of datasets.

The Hugging Face Dataset Viewer automatically generates Croissant metadata (in JSON-LD format) for every dataset on the Hugging Face Hub that can be converted to Parquet format. This metadata includes the dataset's name, description, URL, and the distribution of the dataset as Parquet files, along with column-level metadata.

The Dataset Viewer architecture relies on two main components to provide near-instantaneous dataset exploration:

1. User-facing Web API: Handles incoming queries from users to explore and retrieve dataset information.
2. Server: Performs time-consuming preprocessing tasks ahead of time and stores the results in a database cache.

By pre-calculating responses and storing them in a cache, the system avoids generating responses on-demand, allowing the Web API to serve preprocessed results immediately.

The job queue is a list of tasks stored in a MongoDB database that workers execute to prepare data for the Web API. The jobs correspond to specific API endpoints:

• /splits: Refreshes a dataset and returns its splits and subsets. A new job is created for every split in the dataset.
• /first-rows: Retrieves the first 100 rows and columns of a dataset split.
• /parquet: Downloads the entire dataset, converts it to Parquet format, and publishes the files to the Hub.

Note: The /rows and /search endpoints do not use the job queue; their responses are generated on-demand.

Developers can consume the dataset viewer's REST API to perform various data exploration tasks across over 100,000 datasets on Hugging Face. Supported operations include:

• Metadata Retrieval: List dataset splits, column names, and data types.
• Size Information: Get the dataset size in terms of number of rows or bytes.
• Row Access: Download and view specific rows at any given index.
• Search: Search for specific words within the dataset.
• Filtering: Filter rows based on a query string.
• Statistics: Retrieve insightful statistics about the data.
• Parquet Access: Access datasets directly as Parquet files for use in analytics or processing frameworks.

The system automatically classifies string columns into one of two types:

string_label (Categorical)

A string column is treated as a category if:

1. The proportion of unique values $\le$ 0.2 AND the number of unique values < 1000.
2. OR the number of unique values $\le$ 10 (regardless of proportion).

Measures: nan_count, nan_proportion, n_unique, and frequencies (counts per label).

string_text (Textual)

If the conditions for string_label are not met, the column is treated as text. Statistics are computed based on character length.

Measures: min, max, mean, median, std of lengths, nan_count, nan_proportion, and a 10-bin histogram of lengths.

The cache is a MongoDB database used to store the results of jobs completed by workers.

Workflow:

1. A worker completes a job (e.g., a /first-rows request).
2. The result is stored (cached) in MongoDB.
3. When a user calls the corresponding endpoint, the Web API retrieves the preprocessed response from the cache instead of re-running the computation.

This mechanism allows even large datasets to be queried nearly instantaneously.

A dataset's Parquet version may be marked as partial: true in the API response in two scenarios:

1. Large Row Groups: The original dataset is in Parquet format, but contains row groups larger than the recommended 100-300MB (uncompressed) size. This size is preferred for efficient streaming in most data libraries.
2. Size/Format Constraints: The dataset is not in Parquet format, or the total dataset size exceeds 5GB.

When a dataset is partially converted, the generated Parquet files are placed in a split directory prefixed with partial- (e.g., partial-train instead of train).

When requesting rows via the API, image and audio data are not returned as raw bytes. Instead, they are represented by a URL pointing to the file.

Images are returned as a JSON object containing:

• src: A signed URL to the image file. Note that these URLs are temporary and will expire.
• height: The height of the image in pixels.
• width: The width of the image in pixels.

Audio follows a similar pattern where the field contains a URL to the audio file.

If you encounter an error where an asset is unavailable, you may need to re-call the /rows endpoint to obtain a fresh signed URL.

{
  "img": {
    "src": "https://datasets-server.huggingface.co/cached-assets/.../image.jpg?Expires=...&Signature=...&Key-Pair-Id=...",
    "height": 32,
    "width": 32
  },
  "fine_label": 19,
  "coarse_label": 11
}