DataFramesMeta.jl

The AsTable(cols) function allows you to treat a selection of columns as a single NamedTuple. This is useful for performing operations on a group of columns at once.

Usage Rules

• Right-hand side: When used on the right-hand side of an operation (e.g., :y = sum(AsTable(vars))), no other columns may be referenced in that specific expression. The command :y = sum(AsTable(cols)) + :d will fail.
• Selectors: AsTable supports column selectors like Not, Between, regular expressions, and lists of names/symbols.
• Escaping: Everything inside AsTable is escaped by default; you do not need to use the $ prefix for column names inside AsTable on the right-hand side.

# Example: Summing a list of variable names
vars = ["a", "b"]
@rtransform df :y = sum(AsTable(vars))

# Example: Using AsTable with a custom function
function fun_with_new_name(x::NamedTuple)
           nms = string.(propertynames(x))
           new_name = Symbol(join(nms, "_"), "_sum")
           s = sum(x)
           (; new_name => s)
       end

@rtransform df $AsTable = fun_with_new_name(AsTable([:a, :b]))

To apply operations row-by-row without manual vectorization, use the @byrow macro within DataFramesMeta.jl macros. Alternatively, you can use specific row-wise macros that use @byrow by default: @rtransform, @rtransform!, @rselect, @rselect!, @rorderby, @rsubset, and @rsubset!.

Usage Patterns

1. Inside a macro: Use @byrow as an argument to @transform, @transform!, @select, @select!, @combine, @subset, @subset!, @orderby, or @with.
2. Block syntax: To apply @byrow to multiple operations in a single block, place it at the beginning of the block.
3. Grouped DataFrames: When using @byrow with a GroupedDataFrame, the functions operate on individual rows and do not take the grouping into account (similar to ByRow in DataFrames.jl).

Note: @byrow is not a standalone macro and must be used within DataFramesMeta.jl macros.

# Example: Using @byrow inside @transform
@transform(df, @byrow :y = :x == 1 ? true : false)

# Example: Using @byrow in a block for multiple operations
@subset df @byrow begin
           :a > 1
           :b < 5
       end

If an entire argument is wrapped in $() or $, DataFramesMeta bypasss its internal anonymous function creation and passes the argument directly to the underlying DataFrames.jl function. This allows you to use the DataFrames.jl "mini-language", such as src => fun => dest pairs.

Example: Using multiple functions across multiple columns

You can use the . operator with a list of functions to apply multiple operations to multiple columns simultaneously:

@transform df $([:a, :b] .=> [sum mean])

Example: Using src => fun => dest pairs

You can pass complex transformation pairs directly into macros like @transform or @subset:

my_transformation = :a => (t -> t .+ 100) => :c
@transform df $my_transformation

@subset df $(:a => t -> t .>= 2)

Warnings:

• @orderby and @with do not transparently call underlying DataFrames.jl functions; escaping entire transformations here is considered unstable.
• Row-wise macros (@rtransform, @rsubset) will not automatically wrap src => fun => dest in ByRow.

using Statistics
using DataFrames

df = DataFrame(a = [1, 2], b = [30, 40]);

# Apply multiple functions to multiple columns
@transform df $([:a, :b] .=> [sum mean])

# Use a pre-defined transformation pair
my_transformation = :a => (t -> t .+ 100) => :c
@transform df $my_transformation

# Use a transformation pair in @subset
@subset df $(:a => t -> t .>= 2)

The $ syntax allows you to refer to columns via a Symbol, String, or column position using a variable or literal. This is essential for working with column names that are not hardcoded.

Referencing existing columns

You can use $ to reference a column name stored in a variable:

nameA = :A
df2 = @transform(df, :C = :B - $nameA)

Creating new columns programmatically

You can use $ on the left-hand side to name new columns using variables or complex expressions:

newcol = "C"
@select(df, $newcol = :A + :B)

# Using complex expressions requires parentheses
@by(df, :B, $("A complicated" * " new name") = first(:A))

Important Restrictions

• Mixing Types: You cannot mix integer column references (e.g., $1) with Symbol or String references in @transform, @with, or @eachrow. However, you can mix Symbols and Strings.
• Complex Expressions: If a column reference involves a complex expression (like string concatenation or a function call), you must wrap it in parentheses: $(get_column_name(x)).

df = DataFrame(A = 1:3, :B = [2, 1, 2])

nameA = :A
df2 = @transform(df, :C = :B - $nameA)

nameA_string = "A"
df3 = @transform(df, :C = :B - $nameA_string)

newcol = "C"
@select(df, $newcol = :A + :B)

@by(df, :B, $("A complicated" * " new name") = first(:A))