repository·main·Indexed 26 days ago
https://github.com/rom1504/img2datasetA high-performance tool for converting large sets of image URLs into structured datasets for ML training. It supports downloading, resizing, and packaging millions of images into formats such as WebDataset, standard folder structures, Parquet, and TFRecord. The tool provides a CLI and a Python API via the download() function, featuring support for incremental downloads, image filtering by size/aspect ratio, hash verification, and respect for X-Robots-Tag opt-out directives.
CommonPool is a large-scale dataset containing image-text pairs collected from Common Crawl. It is part of the DataComp benchmark for multimodal dataset design.
CommonPool is available in four scales:
To use the LAION-400M dataset with img2dataset, you must first download the metadata files. Use wget to retrieve the metadata release from The Eye repository and move it to your local directory.
wget -l1 -r --no-parent https://the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/
mv the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ .To use the CC12M dataset with img2dataset, you first need to download the metadata file and format it with the required column headers. The metadata is a 2.6GB TSV file.
wget https://storage.googleapis.com/conceptual_12m/cc12m.tsv
sed -i '1s/^/url\tcaption\n/' cc12m.tsvTo ensure data integrity (e.g., when using datasets like LAION-5B that provide original hashes), you can compute a hash for the raw images and verify them against a column in your input file.
Example to compute MD5 and verify against an MD5 column in the input:
--compute_hash "md5" --verify_hash '["md5","md5"]'
img2dataset following the specific instructions provided in the official DataComp repository. DataComp-1B is a filtered subset of the xlarge version of CommonPool.To use LAION-COCO with img2dataset, you must first download the metadata parquet files from Hugging Face. The following commands create a directory and download all 128 parts of the dataset using wget.
mkdir -p laion-coco && cd laion-coco/
for i in {0..127}; do
wget "https://huggingface.co/datasets/laion/laion-coco/resolve/main/part-$(printf "%05d" $i)-2256f782-126f-4dc6-b9c6-e6757637749d-c000.snappy.parquet"
done
cd ..To use the MS COCO train split with img2dataset, you first need to download the metadata file containing the image URLs and captions. The metadata is available as a Parquet file from Hugging Face.
wget https://huggingface.co/datasets/ChristophSchuhmann/MS_COCO_2017_URL_TEXT/resolve/main/mscoco.parquetFor massive datasets like Laion5B, use img2dataset in distributed mode via PySpark.
Infrastructure Recommendation:
c6i.4xlarge).img2dataset.pex file is located at a path accessible to all worker nodes.Setup Steps:
download() function with distributor="pyspark" within a Spark environment.Key Configuration Requirements:
os.environ['PYSPARK_PYTHON'] to the path of your .pex file.spark.executorEnv.PEX_ROOT to a local directory (e.g., ./.pex).spark.driver.host and spark.driver.bindAddress are set to the master node's IP so workers can communicate back.from img2dataset import download
import os
from pyspark.sql import SparkSession
def create_spark_session():
# This path must be available on all worker nodes
pex_file = "/home/ubuntu/img2dataset.pex"
os.environ['PYSPARK_PYTHON'] = pex_file
spark = (
SparkSession.builder
.config("spark.submit.deployMode", "client") \
.config("spark.executorEnv.PEX_ROOT", "./.pex")
.config("spark.driver.port", "5678")
.config("spark.driver.blockManager.port", "6678")
.config("spark.driver.host", "172.31.44.42")
.config("spark.driver.bindAddress", "172.31.44.42")
.config("spark.executor.memory", "16G")
.config("spark.executor.memoryOverhead", "8G")
.config("spark.task.maxFailures", "100")
.master("spark://172.31.44.42:7077")
.appName("spark-stats")
.getOrCreate()
)
return spark
spark = create_spark_session()
url_list = "s3://laion-us-east-1/laion-metadata/laion2B-en/"
output_dir = "s3://laion-us-east-1/laion-data/laion2B-data"
download(
processes_count=1,
thread_count=64,
url_list = url_list,
image_size=384,
resize_only_if_bigger=True,
resize_mode="keep_ratio",
skip_reencode=True,
output_folder=output_dir,
output_format="webdataset",
input_format="parquet",
url_col="URL",
caption_col="TEXT",
enable_wandb=True,
number_sample_per_shard=10000,
distributor="pyspark",
save_additional_columns=["NSFW","similarity","LICENSE"],
oom_shard_count=6,
)To prepare the metadata for LAION-Face, you must download the LAION-400M metadata release, the LAION-Face IDs, and a conversion script. Use the following commands to fetch the files and convert them into the required format using convert_parquet.py.
Note: This process requires the convert_parquet.py script from the official LAION-Face repository.
wget -l1 -r --no-parent https://the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/
mv the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ .
wget https://huggingface.co/datasets/FacePerceiver/laion-face/resolve/main/laion_face_ids.pth
wget https://raw.githubusercontent.com/FacePerceiver/LAION-Face/master/convert_parquet.py
python convert_parquet.py ./laion_face_ids.pth ./laion400m-meta ./laion_face_metaInstead of local storage, you can stream metadata directly to an S3 bucket using wget piped into aws s3 cp.
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-en-joined/resolve/main/part-$i-4cfd6e30-f032-46ee-9105-8696034a8373-c000.snappy.parquet -O - | aws s3 cp - s3://laion5b/metadata/laion2B-en-joined/part-$i-4cfd6e30-f032-46ee-9105-8696034a8373-c000.snappy.parquet; doneTo use the CC3M dataset, download the 500MB TSV file from the Google Research website. Before processing, you must add column headers to the TSV file. Use the following command to prepend caption and url columns to the file:
sed -i '1s/^/caption\turl\n/' cc3m.tsvTo use the SBU Captions dataset, first download and extract the metadata file containing the image URLs and captions using wget and tar.
wget https://www.cs.rice.edu/~vo9/sbucaptions/sbu-captions-all.tar.gz
tar -xvzf sbu-captions-all.tar.gz