collapse

The collapse package provides highly optimized functions for data manipulation, particularly when working with a large number of groups. For many common operations, collapse outperforms dplyr, data.table, and arrow (when using unordered grouping).

Key patterns for high performance:

• Use fgroup_by(..., sort = FALSE) to avoid the overhead of sorting groups when order is not required.
• Use fselect() to subset columns before performing aggregations.
• Use specialized aggregation functions like fsum(), fmean(), fmedian(), fsd() (standard deviation), and fndistinct() (number of distinct values).

# Example: Fast aggregation across many groups
taxi |> 
  fgroup_by(vendor_id, pickup_location_id, passenger_count, sort = FALSE) |> 
  fselect(total_amount, fare_amount, tip_amount) |> 
  fsum()

Indexing allows you to treat data as a collection of time series. Use findex_by() to create an indexed object. Once indexed, you can perform time-series operations like G() (growth) or psmat() (convert to panel matrix) directly on the object, and the operations will respect the index structure.

# Create an indexed object
exportsi <- exports |> findex_by(c, s, y)

# Perform operations on the indexed object
exportsi |> G(0:1) |> head(5)

# Convert to a panel matrix
exportsi$v |> psmat() |> head(3)

# Check if the series is irregular
is_irregular(exportsi$v)

The GRP() function creates a GRP object, which is an efficient internal representation of groups within a dataset. This object stores group IDs, sizes, and starting indices, allowing for extremely fast subsequent operations like gcomp() (group composition) or gtrans() (group transformations).

g <- GRP(wlddev, ~ income + OECD)

The package includes several ways to learn and reference its functionality:

• Built-in Structured Documentation: Access the top-level overview and links to all other documentation pages by calling help('collapse-documentation') in R.
• Vignettes: Detailed guides are available, including a specific vignette on Documentation and Resources.
• Cheatsheet: A visual reference guide is available as a PDF.
• JSS Article: For academic reference, see the article published in the Journal of Statistical Software.

You can use collapse functions within a pipe workflow (e.g., using magrittr or dplyr). While dplyr verbs work, collapse specific verbs are significantly faster.

Key Manipulation Functions

• fgroup_by(data, ...): Efficiently creates a grouped tibble.
• fsubset(data, condition, grouping, selection): Fast selecting and subsetting.
• add_vars(data, ...): Adds new variables (like weights) to a dataset.
• fvar(data, w): Frequency-weighted group variance.
• roworder(data, ...): Reorders rows.
• fndistinct(data, g): Grouped distinct value counts.

Example Workflow

library(magrittr)
iris %>%
  fgroup_by(Species) %>%
  fndistinct

# Complex pipeline
iris %>%
  add_vars(w) %>%
  fsubset(Sepal.Length < fmean(Sepal.Length), Species, Sepal.Width:w) %>%
  fgroup_by(Species) %>%
  fvar(w) %>%
  roworder(sum.w)

The varying() function identifies which columns in a data frame change (vary) versus which remain constant (fixed) across a specified grouping variable. This is useful for understanding the structure of longitudinal or panel datasets.

# Identify which columns vary by iso3c
varying(wlddev, ~ iso3c)

Use set_collapse() to configure global execution parameters for the collapse package. This includes controlling how missing values are handled, whether results are sorted, and the number of threads used for parallel processing.

Common parameters:

• na.rm: Boolean indicating whether to remove NA values during computation.
• sort: Boolean indicating whether to sort the output.
• nthreads: Integer specifying the number of threads for parallel execution.

set_collapse(na.rm = FALSE, sort = FALSE, nthreads = 4)

The pivot() function allows for high-speed reshaping of data between long and wide formats, serving as an alternative to tidyr::pivot_longer and tidyr::pivot_wider.

Pivot Long

To convert data from wide to long format:

# collapse equivalent of tidyr::pivot_longer
pivot(flights, values = vars)

Pivot Wide

To convert data from long to wide format, use the how argument:

• how = "wider": Standard wide reshaping.
• FUN: A function to apply to values if multiple values exist for a cell (e.g., fsum or "sum").

# Example: Pivot wide with a summary function
pivot(flights, .c(month, day, dest), vars, "origin", how = "wider", FUN = fsum)

The fscale() function scales numeric variables, often used for standardization within groups.

fscale(iris$Sepal.Length, g = iris$Species)