Daft Documentation

The Image Classification Benchmark evaluates high-performance data processing for multimodal workloads. It classifies 803,580 rows (consisting of 80,358 unique images, each repeated 10x) using the ResNet18 model. The workflow involves downloading images, applying preprocessing transforms, and running inference to predict ImageNet labels across distributed GPU nodes.

Benchmark Specifications:

• Input Dataset: ImageNet benchmark dataset in S3 parquet format.
• Output Format: Parquet files containing image URLs and predicted labels.
• Infrastructure: 8 worker nodes using g6.xlarge instances.
• Framework Versions used in benchmark: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0.

The Document Embedding Benchmark evaluates high-performance data processing by generating embeddings for 10,000 PDF documents. The workflow involves extracting text, chunking content, and creating 384-dimensional embeddings using the sentence-transformers/all-MiniLM-L6-v2 model across distributed nodes.

Benchmark Specifications:

• Input Dataset: Digital Corpora PDF metadata (stored in S3 parquet format).
• Output Format: Parquet files containing embeddings, text chunks, and metadata.
• Infrastructure: 8 worker nodes using g6.xlarge instances.
• Benchmark Date: September 22, 2024.
• Framework Versions: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0.

The daft-scan module provides the core primitives for data scanning operations within the Daft engine. It defines how data is discovered, partitioned, and pruned during the scan phase of a query execution plan.

Key components include:

• ScanOperator trait: The interface for implementing custom scan logic.
• ScanTask struct: Represents a specific unit of scanning work.
• Built-in Operators:
  • GlobScanOperator: For scanning files using glob patterns.
  • AnonymousScanOperator: For scanning data from non-file sources or anonymous providers.
• Scan Primitives: Includes Pushdowns (for predicate/projection pushdown), PartitionField (for defining partition keys), Sharder (for distributing scan tasks), and logic for predicate rewriting to enable partition pruning.

The Audio Transcription Benchmark evaluates the performance of transcribing a large-scale dataset using the Whisper-tiny model. The workload includes audio resampling, feature extraction, and speech-to-text inference.

Benchmark Details:

• Dataset: 113,800 audio files from Common Voice 17 (S3 parquet format).
• Output: Parquet files containing transcriptions and metadata.
• Infrastructure: 8 worker nodes using g6.xlarge instances.
• Benchmark Date: September 22, 2024.
• Framework Versions: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0.

The Video Object Detection Benchmark evaluates high-performance multimodal processing by detecting objects in 1,000 videos using the YOLO11n model. The workflow involves extracting frames, running object detection, and cropping detected objects across distributed GPU nodes.

Benchmark Specifications:

• Input Dataset: Hollywood2 video dataset (S3 binary files)
• Output Format: Parquet files containing object detections, bounding boxes, and cropped images
• Infrastructure: 8 worker nodes using g6.xlarge instances
• Benchmark Date: September 22, 2024
• Framework Versions: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0

Daft is benchmarked against Spark, Modin, and Dask using the TPC-H standard analytical query benchmark. The benchmarks demonstrate Daft's ability to handle terabyte-scale datasets with reliable out-of-core execution (disk spilling) and high performance without specialized tuning.

Key performance highlights:

• Reliability: Only Daft and EMR Spark reliably run terabyte-scale queries on out-of-the-box configurations.
• Speed: Daft is consistently faster than competitors, showing 3.3x speedup over EMR Spark, 7.7x over Dask, and 44.4x over Modin (for completed queries).

Daft demonstrates two key scaling properties during TPC-H 1000 Scale Factor testing:

1. Single-Node Out-of-Core: Daft can process 1TB+ of analytical data on a single 61GB instance (16x more data than available memory).
2. Linear Scaling: Query times scale linearly with the number of nodes (e.g., a 4-node cluster is approximately 4x faster than a 1-node cluster), allowing for predictable cost-to-performance scaling.

Daft supports two primary execution modes depending on your resource requirements:

1. Native Runner (Swordfish): The default mode. It runs using your local machine's resources. This is ideal for development and smaller workloads.
2. Distributed Execution: Used when you need to scale compute power across multiple machines. This is suitable for large-scale AI and multimodal workloads.

To scale to a distributed cluster, you can use Kubernetes or Ray.

Daft uses dynamic execution to automatically adjust batch sizes based on the operation type to optimize memory and performance:

• Multimodal Downloads: Uses smaller batch sizes (typically a factor of max_connections) to prevent memory exhaustion when downloading large files.
• Vectorized Operations: Uses larger batch sizes for operations like byte decoding/encoding, aggregations, and scalar projections to leverage SIMD hardware optimizations.
• Image Resizing: Uses medium batch sizes for operations like .resize().

Daft supports a wide range of DataType objects used to define the schema of DataFrames. These range from standard scalar types to complex multimodal types suitable for AI workloads.

Supported categories include:

• Standard types: Numbers, strings, and dates.
• Complex/Multimodal types: Tensors and images.