DOCSEARCHbyAlgolia
Submit Docs

Hugging Face Datasets

repository·main·Indexed 12 days ago

https://github.com/huggingface/datasets

A lightweight library for downloading, preparing, and managing large-scale datasets for machine learning. It provides one-line access to the Hugging Face Hub and efficient, reproducible data processing via an Apache Arrow backend, supporting both in-memory Dataset and streamable IterableDataset abstractions.

Tokens
89.1K
Snippets
308
Records
365
Agent score
97%

What's inside Datasets

  1. Overview of 🤗 Datasets

    main

    🤗 Datasets is a library designed for accessing and sharing AI datasets across Audio, Computer Vision, and Natural Language Processing (NLP) tasks. It allows you to load datasets with a single line of code and provides powerful data processing and streaming methods to prepare data for deep learning models.

    Key technical features include:

    • Apache Arrow Backend: Uses the Apache Arrow format for zero-copy reads, enabling efficient processing of large datasets without memory constraints.
    • Hugging Face Hub Integration: Deeply integrated with the Hugging Face Hub for easy loading and sharing of datasets.
    • Streaming Support: Methods to stream datasets, which is useful for working with data that is too large to fit in memory.

  2. What is an `IterableDataset`?

    main

    Loading a dataset in streaming mode returns an IterableDataset.

    Key characteristics:

    • Iterative access: Designed for iterative jobs like model training.
    • No random access: You cannot access specific indices (e.g., dataset[5]). To get the $N$-th example, you must iterate through all preceding examples.
    • Specialized methods: It includes specific processing methods for streaming, such as shuffle, take, skip, and shard.

  3. Stream large datasets with streaming mode

    main

    If a dataset is too large to fit on your disk, use streaming=True in load_dataset(). This allows you to iterate over the data on-the-fly without downloading the entire dataset first.

    from datasets import load_dataset

# Stream the dataset without downloading everything
image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)

for example in image_dataset["train"]:
    print(example["image"])
    break

  4. How automatic split detection works

    main

    If no YAML configs block is provided, 🤗 Datasets attempts to infer splits using the following hierarchy:

    1. Directory Names: Files inside directories named train, test, or validation are assigned to those splits.
    2. Filename Patterns: Split names are inferred from filenames if they are delimited by non-word characters (underscores, dashes, spaces, dots, or numbers).
      • Train keywords: train, training
      • Validation keywords: validation, valid, val, dev
      • Test keywords: test, testing, eval, evaluation
    3. Custom Filename Split: For non-standard split names, use the pattern data/<split_name>-xxxxx-of-xxxxx.csv.
    4. Single Split: If no patterns match, all files are treated as a single train split.

  5. How fingerprints track dataset state

    main

    A fingerprint is a unique identifier assigned to a cache file that represents the current state of a dataset.

    • Initial State: The first fingerprint is a hash of the Arrow table (in-memory) or the Arrow files (on-disk).
    • Transformations: When a transform (e.g., Dataset.map, Dataset.shuffle) is applied, a new fingerprint is generated by combining the previous state's fingerprint with a hash of the latest transform.

    Important: Hashable Transforms

    To ensure a transform is correctly cached, it must be picklable using dill or pickle.

    • If the transform is hashable: The fingerprint is deterministic, and the cache is reused.
    • If the transform is non-hashable: 🤗 Datasets assigns a random fingerprint and issues a warning. This causes the library to recompute all transforms every time, as it cannot guarantee the state is the same.

    Tip: If caching is disabled, use Dataset.save_to_disk to persist your transformed dataset, otherwise it will be deleted from the temporary directory when your session ends.

    >>> from datasets import Dataset
>>> dataset1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> dataset2 = dataset1.map(lambda x: {"a": x["a"] + 1})
>>> print(dataset1._fingerprint, dataset2._fingerprint)
d19493523d95e2dc 5b86abacd4b42434

  6. How the TsFile data model and output layout work

    main

    Unlike standard tabular datasets where one row equals one record, the TsFile loader follows a device-centric model. Each row in the resulting dataset represents a single device (uniquely identified by a tuple of TAG columns).

    Schema Structure:

    • TAG columns: Scalar string columns. These identify the device.
    • time column: An Arrow list<timestamp> column containing the full timeline for that device.
    • FIELD columns: Arrow list<...> columns containing the measurement values for that device, sorted by time.

    If a device appears in multiple files, its data is concatenated and sorted by timestamp. Note that duplicate timestamps for the same device will raise a ValueError.

    <tag_1>:    string
<tag_2>:    string                       # one column per TAG
...
time:       list<timestamp[unit, tz]>
<field_1>:  list<original_type>          # one column per FIELD
<field_2>:  list<original_type>
...

  7. Use Dataset.with_transform for on-the-fly transformations

    main

    Unlike Dataset.map, which applies a function and saves the result to disk, Dataset.with_transform applies a transformation function dynamically whenever an item is accessed. This is ideal for data augmentations (like image cropping or color jittering) where you want a different variation every time the data is loaded, without consuming extra disk space.

    # Apply a transformation function on-the-fly
dataset = dataset.with_transform(my_transform_function)

# The transformation is applied when you access the data
print(dataset[0])

  8. How `BuilderConfig` and `DatasetBuilder` work together to build datasets

    main

    Building a dataset involves two main components: BuilderConfig for configuration and DatasetBuilder for the execution logic.

    BuilderConfig

    BuilderConfig is the configuration class for a DatasetBuilder. It stores metadata such as:

    • name: Short name of the dataset.
    • version: Dataset version identifier.
    • data_dir: Path to a local folder containing data files.
    • data_files: Paths to local data files.
    • description: Description of the dataset.

    You can extend BuilderConfig by subclassing it to add custom attributes (like class labels). You can populate these attributes by:

    1. Providing a list of instances in the DatasetBuilder.BUILDER_CONFIGS attribute.
    2. Passing keyword arguments to load_dataset, which will override predefined attributes.

    DatasetBuilder

    DatasetBuilder uses the BuilderConfig to transform raw data into a table of rows and typed columns. It relies on three main methods:

    1. _info: Defines the dataset attributes and Features (the schema/skeleton of the dataset).
    2. _split_generator: Downloads or retrieves files using a DownloadManager and organizes them into splits.
    3. _generate_examples: Parses the files and yields examples according to the schema defined in _info. This method uses a Python generator to handle large datasets efficiently without loading everything into memory.

  10. Interleave multiple datasets

    main

    The interleave_datasets function mixes several datasets by taking alternating examples from each. This works with both Dataset and IterableDataset objects.

    Sampling Probabilities: You can pass a list of probabilities to specify the distribution of samples from each dataset. The construction stops when the first dataset is exhausted (unless a different stopping_strategy is used).

    Stopping Strategies:

    • first_exhausted (default): Stops as soon as one dataset runs out of samples.
    • all_exhausted: An oversampling strategy. When a dataset is exhausted, it restarts from the beginning until all datasets have been fully traversed at least once.
    • all_exhausted_without_replacement: Ensures every sample in every dataset is seen exactly once.
    >>> from datasets import Dataset, interleave_datasets
>>> seed = 42
>>> probabilities = [0.3, 0.5, 0.2]
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=probabilities, seed=seed)
>>> dataset["a"]
[10, 11, 20, 12, 0, 21, 13]

# Using all_exhausted strategy
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 20]

  11. How Dataset and IterableDataset differ

    main

    The library provides two primary dataset abstractions:

    1. Dataset: Designed for fast random access and efficient memory usage via memory-mapping. It is ideal when the dataset fits on your local disk. You can access any row or column directly using indices or names.
    2. IterableDataset: Designed for streaming. It allows you to access data progressively without waiting for the entire dataset to download or load into memory. This is essential for datasets that are too large to fit on disk or in memory. It does not support random access (indexing by row), but it does support column-based iteration and creating subsets via .take().

    Use Dataset for standard workflows requiring random access, and IterableDataset (via streaming=True) for massive datasets that require immediate streaming.

  12. Understand the core components of the dataset building process

    main

    The 🤗 Datasets library uses two primary classes to manage the dataset building lifecycle:

    1. DatasetBuilder: The main class responsible for defining how a dataset is constructed, loaded, and managed.
    2. BuilderConfig: A configuration class used to define specific variations or parameters for a dataset (e.g., different subsets, languages, or feature configurations).

    Depending on the data source and storage format, you will typically interact with one of the following specialized builder subclasses:

    • GeneratorBasedBuilder: Used when the dataset is built by iterating through data using a generator function.
    • ArrowBasedBuilder: Used when the dataset is already stored in Apache Arrow format, allowing for more efficient loading.

    To implement a custom dataset, you will generally subclass DatasetBuilder (or one of its specialized versions) and define a BuilderConfig to handle different dataset configurations.