DataFrames.jl Documentation

While DataFrames.jl is the core engine, several ecosystem frameworks provide convenience methods for data manipulation, similar to dplyr in R or LINQ in C#. These frameworks allow for more compact, readable code and are designed to assist both new and advanced users.

Available frameworks include:

• TidierData.jl: A macro-based interface inspired by the R tidyverse (dplyr and tidyr).
• DataFramesMeta.jl: A fast, macro-based interface providing convenient syntax for transformation functions.
• DataFrameMacros.jl: Provides macro-based enhancements.
• Query.jl: Provides a query-based interface.

By default, subsetting a DataFrame usually returns a copy of the columns, not a view or a direct reference. This means modifications to the subset will not affect the original DataFrame.

Exceptions (When it does NOT return a copy):

• When using the ! syntax in the first position: df[!, :A] or df[!, [:A, :B]].
• When using dot notation: df.A.
• When selecting a single row using an integer: df[1, [:A, :B]].
• When explicitly using view or @view: @view df[1:3, :A].

While combine, select, and transform support the do block syntax for convenience, this form is significantly slower. Avoid using do blocks in performance-critical code.

# Slow form - avoid in performance-sensitive code
combine(iris_gdf) do df
           (m = mean(df.PetalLength), s² = var(df.PetalLength))
       end

In DataFrames.jl, you can use compact predicate syntax instead of anonymous functions for cleaner and more efficient code. For example, instead of writing an anonymous function like x -> x >= 1, you can use the more compact >=(1).

Using this compact form has a performance benefit: the predicate is compiled only once per Julia session, whereas an anonymous function defines a new object every time it is introduced.

When dealing with columns in a DataFrame that have a small number of repeating levels, you can use level pooling to save memory and improve groupby performance. There are two primary types for this:

• PooledVector (from PooledArrays.jl): Best when your only goal is data compression. It acts as a drop-in replacement for Vector with minimal user-visible differences.
• CategoricalVector (from CategoricalArrays.jl): Best when you need full support for categorical variables. This includes handling unordered (nominal) or ordered (ordinal) categories. It supports AbstractString, AbstractChar, or Number (optionally with Missing). Use this when levels should respect a specific order for plotting, printing, or regression models.

If you have a vector of NamedTuples where different observations have different sets of columns (heterogeneous data), the standard DataFrame(source) constructor will fail. To create a DataFrame that includes all columns present in at least one observation (filling missing ones with missing), use Tables.dictcolumntable from the Tables.jl package.

using Tables
source = [(type="circle", radius=10), (type="square", side=20)]
DataFrame(Tables.dictcolumntable(source))

DataFrames.jl provides two versions of most manipulation functions to control whether the original data frame is modified or a new one is created:

1. Non-mutating (returns a new DataFrame): Functions without a ! in their name (e.g., transform, select, subset, combine). You should assign the result to a new variable: new_df = transform(source_df, operation).
2. In-place (modifies the existing DataFrame): Functions with a ! at the end of their name (e.g., transform!, select!, subset!). These modify the object directly and typically do not require assignment.

A GroupedDataFrame is a view of its parent DataFrame. If you mutate the parent DataFrame by adding or removing rows (e.g., using push!, subset!, filter!, or deleteat!), the GroupedDataFrame will become invalid and throw an AssertionError when accessed.

Workaround: If you need to append rows to the source DataFrame without breaking the grouping logic, create the GroupedDataFrame using a view of the parent instead of the parent itself.

# This will error if df is mutated:
gd = groupby(df, :id)

# This remains valid even if df is mutated:
gd = groupby(view(df, :, :), :id)

All manipulation functions (like select, combine, subset, transform) accept multiple operation pairs at once. You can pass them as:

• Multiple arguments: func(df, op1, op2)
• A vector: func(df, [op1, op2])
• A matrix: func(df, [op1 op2])

Note: All operations within a single function call use the state of the data before the function was called. You cannot use a column created in op1 as an input for op2 within the same call.

DataFrames.jl provides two special types used in selection operations to control how functions are applied to data:

• ByRow: Wraps a function to signal that it should be applied to each individual row (element) of the selection.
• AsTable: Signals that selected columns should be passed to a function as a NamedTuple, or signals a request to expand a transformation's return value into multiple columns.