DOCSEARCHbyAlgolia
Submit Docs

Arquero

repository·main·Indexed 23 days ago

https://github.com/uwdata/arquero

A high-performance JavaScript library for query processing and transformation of array-backed data tables, inspired by relational algebra and dplyr. It provides core abstractions for Data Tables and Verbs to perform multi-step transformations via method chaining. Supports integration with Apache Arrow and provides extensibility through custom functions, aggregate functions, and window functions.

Tokens
30.1K
Snippets
71
Records
215
Agent score
80%

What's inside arquero

  1. Explore Arquero Op Functions

    main

    Arquero provides a wide range of operation functions (op) used within table transformations. These are categorized into several functional groups:

    • Standard Functions: Includes Array, Date, JSON, Math, Object, and String functions.
    • Aggregate Functions: Used for summarizing data (e.g., sum, mean, count, max, min, stdev, corr).
    • Window Functions: Used for calculations over a window of rows (e.g., row_number, rank, lag, lead, fill_down).

  2. Use Two-Table Expressions for joins

    main

    For join verbs, Arquero supports two-table expressions. These accept two rows as input: one from the 'left' table (a) and one from the 'right' table (b).

    Limitations:

    • Aggregate and window functions are not allowed within two-table expressions.
    • Bound parameters (via params()) are accessed using a third argument (default name $):
    // Basic two-table expression
table.join(otherTable, (a, b) => op.equal(a.key, b.key))

// Two-table expression with parameters
table
  .params({ threshold: 1.5 })
  .join(otherTable, (a, b, $) => op.abs(a.value - b.value) < $.threshold)

  3. Use Window Functions in Arquero

    main

    Window functions are applicable over ordered table rows. When used within a table expression context (like derive), they operate on the current window of rows. If invoked outside a table expression context, column inputs must be column name strings, and the operator will return a corresponding table expression.

    Common use cases include ranking, calculating cumulative distributions, and accessing preceding or following values in a sequence.

  4. What are Arquero core abstractions and how do they work?

    main

    Arquero is built around two core abstractions:

    1. Data Tables: These model data as column-oriented structures where each column is an array of values. Tables can be created from arrays, typed arrays, array-like objects, or Apache Arrow columns.
    2. Verbs: These are table methods used to transform data. Verbs return new tables, which allows for method chaining to perform multi-step transformations (e.g., .derive().select().orderby()).

    While each transformation produces a new table, many verbs are optimized to reuse underlying columns to minimize memory duplication.

    import { all, desc, op, table } from 'arquero';

const dt = table({
  'Seattle': [69,108,178,207,253,268,312,281,221,142,72,52],
  'Chicago': [135,136,187,215,281,311,318,283,226,193,113,106],
  'San Francisco': [165,182,251,281,314,330,300,272,267,243,189,156]
});

dt.derive({
    month: d => op.row_number(),
    diff:  d => d.Seattle - d.Chicago
  })
  .select('month', 'diff')
  .orderby(desc('diff'))
  .print();

  5. Use Math operations in Arquero

    main

    Arquero provides a suite of mathematical operations via the op namespace that are equivalent to standard JavaScript Math methods. These can be used within table transformations like derive or filter.

    Note on Comparison Functions: op.greatest(...values) and op.least(...values) return the maximum or minimum among the provided input values for a single row. They are not aggregate functions. To compute a maximum or minimum across multiple rows, use op.max or op.min respectively.

  6. Use Aggregate Functions for summarizing values

    main

    Aggregate functions in Arquero are used to summarize values within a table expression (e.g., inside a .rollup() or .groupby() context). If invoked outside of a table expression context, the function will return a table expression instead of a scalar value.

    Common aggregate operations include:

    • Counting: op.count() (total rows), op.distinct(field) (unique values), op.valid(field) (non-null/NaN), and op.invalid(field) (null/NaN).
    • Extremes & Sums: op.max(field), op.min(field), op.sum(field), op.product(field).
    • Central Tendency: op.mean(field) (or op.average(field)), op.median(field), op.mode(field), and op.quantile(field, p).
    • Statistics: op.stdev(field) (sample), op.stdevp(field) (population), op.variance(field) (sample), op.variancep(field) (population), op.corr(f1, f2) (correlation), op.covariance(f1, f2) (sample), and op.covariancep(f1, f2) (population).

  7. Use Arquero Standard Functions via the op object

    main
    Arquero provides a standard library of table expression functions through the op object. These functions are designed to behave identically whether they are invoked inside a table expression context or as standard JavaScript functions. This allows for flexible data manipulation both within Arquero's fluent API and in standalone logic.

  8. Use Table Expressions in Arquero verbs

    main

    Most Arquero verbs (like derive, filter, groupby, etc.) accept table expressions. These are functions defined over table column values.

    An input argument (commonly named d) represents a row of the data table, and its properties correspond to the column names. Table expressions can include standard JavaScript expressions and invoke functions from the op object (standard, aggregate, or window functions).

    Arquero supports several ways to define these expressions:

    • Function definitions: Traditional functions, arrow functions, or destructured arguments.
    • String literals: Passing the expression as a string (e.g., "d => op.sqrt(d.value)").
    • Implicit row identifier: If using a string literal without a function definition (e.g., "sqrt(d.value)"), the row identifier defaults to d.
    table.derive({
  raise: d => op.pow(d.col1, d.col2),
  'col diff': d => d.col1 - d['base col']
})

  9. Perform left, right, or full outer joins

    main

    Arquero provides shorthand methods for common outer joins to avoid manually configuring the options object in join():

    • Left Outer Join: Use join_left(other[, on, values, options]). Preserves all rows from the left table. Shorthand for join(other, on, values, {left: true, right: false}).
    • Right Outer Join: Use join_right(other[, on, values, options]). Preserves all rows from the right table. Shorthand for join(other, on, values, {left: false, right: true}).
    • Full Outer Join: Use join_full(other[, on, values, options]). Preserves all rows from both tables. Shorthand for join(other, on, values, {left: true, right: true}).
    table.join_left(other, 'keyShared')
    table.join_right(other, ['keyL', 'keyR'])
    table.join_full(other, (a, b) => op.equal(a.keyL, b.keyR))

  10. Use Arquero in the Browser

    main

    You can load Arquero directly in the browser via a CDN. It will be available under the global aq object.

    Basic Setup:

    <script src="https://cdn.jsdelivr.net/npm/arquero@latest"></script>

    Using Apache Arrow support: The default browser bundle does not include Apache Arrow. To use toArrow() or loadArrow(), you must import apache-arrow before Arquero:

    <script src="https://cdn.jsdelivr.net/npm/apache-arrow@latest"></script>
<script src="https://cdn.jsdelivr.net/npm/arquero@latest"></script>

    Bundling Note: When using a module bundler (like Rollup) for browser applications, ensure the bundler uses the browser property from Arquero's package.json. For Rollup, pass browser: true to the node-resolve plugin.

  11. Build and develop Arquero locally

    main

    If you are contributing to or developing Arquero locally, follow these steps:

    1. Clone the repository.
    2. Install dependencies: npm i
    3. Run tests: npm test
    4. Run performance benchmarks: npm run perf
    5. Build output files: npm run build
    npm i
npm test
npm run perf
npm run build

  12. Handle external variables in table expressions using params()

    main

    Table expressions are parsed and rewritten, which means they do not support closures. They cannot access variables defined in the enclosing JavaScript scope.

    To use external variables, use the params() method to bind a value to the table context. You can then access these parameters by adding a second argument to your table expression (the default name is $):

    table
  .params({ threshold: 5 })
  .filter((d, $) => d.value < $.threshold)