duckplyr

The code generation pipeline (scripts 00 through 07) automates the creation of duckplyr methods by introspecting dplyr's S3 methods for data.frame.

• Extraction: 01-dplyr-methods.R extracts the source code of data.frame methods from dplyr and saves them as text files in dplyr-methods/ to serve as fallbacks.
• Implementation: 02-duckplyr_df-methods.R generates R/<verb>.R files. These files contain the <verb>.duckplyr_df S3 method, which attempts a DuckDB execution via rel_try() and falls back to the embedded dplyr implementation on failure.
• Testing:
  • 03-tests.R ensures duckplyr_df results are equivalent to plain dplyr results.
  • 04-dplyr-tests.R adapts existing dplyr tests for duckplyr.
  • 05-duckdb-tests.R generates tests that validate the DuckDB relational API plans.
• Registration: 07-overwrite.R generates methods_overwrite_impl() to register duckplyr methods as the primary data.frame methods via vctrs::s3_register(), making the transition transparent to users.

duckplyr uses a lazy evaluation model. When you write dplyr code (like filter, mutate, or summarize), the results are not computed immediately. Instead, duckplyr builds a query plan.

Computation is triggered only when you attempt to access the data, such as:

• Querying a specific column (e.g., out$column_name)
• Printing the object
• Using functions that require materialized data

Key Behaviors

• Result Type: The output is a standard tibble (tbl_df, tbl, data.frame), making it fully compatible with the rest of the tidyverse.
• Ordering: Unlike standard dplyr, results are not guaranteed to be ordered by default (see ?config for details). However, once the data is materialized, the results are stable.
• Automatic Fallback: If a specific computation or operation is not supported by DuckDB, duckplyr will automatically fall back to using dplyr to ensure the code still runs correctly.

# Example of lazy evaluation and computation trigger
out <-
  flights_df() |>
  filter(!is.na(arr_delay), !is.na(dep_delay)) |
  mutate(inflight_delay = arr_delay - dep_delay) |
  summarize(
    .by = c(year, month),
    mean_inflight_delay = mean(inflight_delay),
    median_inflight_delay = median(inflight_delay),
  ) |
  filter(month <= 6)

# Nothing has been computed yet.

# Querying a column triggers the computation:
out$month
#> [1] 1 2 3 4 5 6

# Printing the object also materializes the results:
print(out)

The TPC-H benchmarking suite allows you to compare duckplyr performance against dplyr.

Step-by-step Pipeline

1. Setup: Generate and serialize TPC-H data.
2. Load: Load query sets.
3. Verify: Ensure correctness of the data/queries.
4. Benchmark: Run benchmarks for both duckplyr and dplyr, then compare the results.

Full Pipeline

You can run the entire process (setup, load, and comparison) in a single command using the master script.

# One-time setup: generate and serialize TPC-H data
source("tools/30-tpch-export.R", echo = TRUE)
source("tools/31-tpch-load-qs.R", echo = TRUE)

# Verify correctness
source("tools/33-tpch-compare.R", echo = TRUE)

# Benchmark duckplyr vs dplyr
source("tools/34-tpch-bench.R", echo = TRUE)
source("tools/35-tpch-bench-dplyr.R", echo = TRUE)
source("tools/36-tpch-bench-compare.R", echo = TRUE)

# Or run the full pipeline in one go:
source("tools/50-load-compare-bench.R", echo = TRUE)

You can query remote Parquet files directly from R without downloading them first by using the httpfs DuckDB extension. This allows for efficient, lazy querying of massive datasets over the internet.

To use this capability:

1. Install and load the httpfs extension using db_exec().
2. Use read_parquet_duckdb() to create a duckplyr data frame from a vector of URLs.
3. Perform transformations (like mutate, summarize, filter) lazily.

Important: Memory Protection Unlike local data frames, duckplyr prevents automatic materialization of remote data if the result is expected to be too large. This protects your system memory. If a query (like nrow()) would result in a large dataset, you must explicitly call collect() or as_tibble() to materialize the results.

# 1. Setup extensions
db_exec("INSTALL httpfs")
db_exec("LOAD httpfs")

# 2. Load remote files
urls <- c("https://example.com/data_1.parquet", "https://example.com/data_2.parquet")
flights <- read_parquet_duckdb(urls)

# 3. Query lazily
# This will error if the result is too large to protect memory
nrow(flights)

# 4. Explicitly materialize when needed
flights_materialized <- flights |> collect()