DOCSEARCHbyAlgolia
Submit Docs

data.table

repository·master·Indexed 26 days ago

https://github.com/rdatatable/data.table

A high-performance R package providing a fast, memory-efficient alternative to base R data.frame. It is designed for large-scale data manipulation with concise syntax (DT[i, j, by]), featuring high-performance file I/O via fread and fwrite, non-equi joins, in-place updates by reference, and internal parallelism. The package includes specialized functions for reshaping (melt, dcast), ordered data, and date/time handling (IDate, ITime) with zero dependencies beyond base R.

Tokens
17.7K
Snippets
34
Records
215
Agent score
87%

What's inside data.table

  1. Overview of data.table features

    master

    Core Capabilities

    • File I/O: High-performance delimited file reading with ?fread and writing with ?fwrite.
    • Parallelism: Many operations are internally parallelized using multiple CPU threads.
    • Joins: Supports ordered joins (rolling, nearest, etc.), overlapping range joins, and non-equi joins (using operators like >, >=, <, <=).
    • Memory Efficiency: Fast add/update/delete of columns by reference using no copies.
    • Reshaping: Fast pivoting/widening with ?dcast and unpivoting/lengthening with ?melt.
    • Zero Dependencies: Requires only base R.

  2. Leverage data.table inheritance from data.frame

    master
    Since data.table v1.5, data.table objects inherit from data.frame. This means they are compatible with any function or package that expects a data.frame, eliminating the need for manual conversion and saving memory. is.data.frame() will return TRUE for a data.table.

  3. Understand the scope of data.table functionality

    master

    The data.table package focuses on high-performance data manipulation and analysis. Understanding what is in and out of scope helps in evaluating feature requests and contributions.

    In Scope

    • Data manipulation and analysis: Reshaping/pivoting, aggregation/summarizing (via [,, by=...] and grouping sets), filtering rows, joins, adding/updating/deleting columns, and set operations (union/rbind, intersection, difference).
    • High-performance functions: frank, fcase, fifelse, transpose, chmatch, fsort, forder, uniqueN, etc.
    • Convenience functions: %like%, %notin%, timetaken, substitute2, etc.
    • Ordered data functions: rleid, shift, fcoalesce, nafill (locf/nocb), and rolling functions.
    • Date and time: IDate and ITime classes/functions.
    • Technical functions: address, tables, update_dev_pkg.
    • I/O: Reading/writing data from/to flat (plain text) files like CSV.

    Out of Scope

    • Plotting and graphics (e.g., ggplot2).
    • Out-of-memory data manipulation (e.g., data on disk or remote SQL DBs).
    • Machine learning (e.g., mlr3).
    • Reading/writing binary files (e.g., Parquet).

  4. Explore the data.table ecosystem

    master

    The data.table community includes several categories of packages that extend, utilize, or bridge the functionality of data.table. These include:

    • Extension packages: Add to the internal functionality of data.table (e.g., nc).
    • Application packages: Use data.table for specific tasks like machine learning (e.g., mlr3).
    • Bridge packages: Translate data.table syntax to other syntaxes or provide helpers for transitioning between object types (e.g., tidyfast, dtplyr).
    • Partner packages: Packages that follow the core philosophies of data.table but are not directly connected (e.g., collapse).

    For detailed information on these relationships, visit The Raft blog.

  5. Avoid deprecation warnings when using := with character vectors

    master
    In data.table v1.9.8, using with=FALSE together with the := operator is deprecated. To avoid warnings (and future errors), wrap the Left-Hand Side (LHS) of the := operation in parentheses when using a character vector of column names.

  6. Use the data.table subsetting syntax

    master

    The data.table subsetting operator [ follows the pattern DT[i, j, by].

    • i: The WHERE clause (filtering rows).
    • j: The SELECT or computation clause (calculating values or selecting columns).
    • by: The GROUP BY clause (grouping computations).

    Unlike base R data.frame, you do not need to prefix column names with DT$ inside the brackets, and you can use any R expression or function from any package within the j argument.

    library(data.table)
DT = as.data.table(iris)

# Syntax: DT[i, j, by]
# Example: Filter rows where Petal.Width > 1.0, calculate mean Petal.Length, grouped by Species
DT[Petal.Width > 1.0, mean(Petal.Length), by = Species]

  8. Configure CPU thread usage for `data.table`

    master

    Starting in v1.12.2, the default number of logical CPUs used by data.table was reduced from 100% to 50% to prevent system slowdowns.

    You can control thread usage via the following methods:

    1. Environment Variables:
      • R_DATATABLE_NUM_PROCS_PERCENT: Sets the percentage of CPUs to use.
      • R_DATATABLE_NUM_THREADS: Sets the specific number of threads.
      • OMP_THREAD_LIMIT: Respected by data.table in addition to OMP_NUM_THREADS.
    2. R Function:
      • Use setDTthreads() to configure threads during a session. This function supports a percent= argument.
    3. Inspection:
      • Use getDTthreads(verbose=TRUE) to see current thread configuration details.

  12. Use difftime() for POSIXt subtraction to avoid unit errors

    master
    Subtracting two POSIXt objects by group can lead to incorrect results because data.table may not correctly handle differing units attributes across groups. To ensure accuracy, it is recommended to call difftime() directly instead of relying on the subtraction operator.