deepscatter

A Tile is the fundamental unit of operation in Deep Scatterplots. It represents a collection of points grouped together in a quadtree structure.

Key characteristics:

• Data Representation: It corresponds to an Apache Arrow RecordBatch, but can exist as a metadata-only object before the actual data is fetched.
• Tree Structure: Each tile holds its place in a tree (typically a quadtree) and maintains references to its parent and children.
• Operations: Most batched operations—including network requests, GPU calculations, data transformations, and render calls—are performed at the tile level.
• Lifecycle: A tile can be instantiated with just a key, then populated with metadata via populateManifest(), and finally loaded with actual data via get_arrow() or get_column().

// Example of the Tile concept in use (conceptual)
const tile = new Tile('0/0/0', null, deeptable);
await tile.populateManifest();
const column = await tile.get_column('x');
// The tile now contains the data for that column.

You can create complex filters by composing existing selections using a Lisp-like syntax. The composition parameter accepts an array where the first element is the operator and the subsequent elements are the arguments (other selections or compositions).

Supported Operators:

• Binary: AND, OR, XOR (takes two arguments).
• Unary: NOT (takes one argument).
• Plural: ANY, ALL, NONE (takes indefinite arguments).

Use the union() and intersection() methods on an existing DataSelection to easily create new composed selections.

// Creating a composite selection manually
const composite = new DataSelection(deeptable, {
  name: 'complex-filter',
  composition: ['AND', selectionA, ['OR', selectionB, selectionC]]
});

// Using helper methods
const unionSelection = selectionA.union(selectionB, 'combined-name');
const intersectionSelection = selectionA.intersection(selectionB, 'overlap-name');

Deep Scatterplots uses boolean aesthetics to control visibility and rendering priority. You can use Filter to include or exclude points based on a condition, and Foreground to determine which points are rendered with high resolution (the 'foreground') versus lower resolution (the 'background').

Boolean aesthetics support three types of encodings:

1. Constant Channels: A fixed boolean value applied to all points.
2. Op Channels (Operations): Mathematical or logical comparisons against a value or range. Supported operations include:
   • eq: Equal to a
   • gt: Greater than a
   • lt: Less than a
   • within: Within a distance a of b (calculated as Math.abs(p - b) < a)
   • between: Between a and b
3. Lambda Channels: A custom JavaScript function used to evaluate each point.

Note: When using OpChannel with dates, they are automatically converted to numeric timestamps for comparison.

// Example conceptual usage of boolean encodings
// (Note: Exact instantiation depends on the Scatterplot API)

// 1. Constant encoding
const filter = { constant: true };

// 2. Op encoding (e.g., points between two values)
const rangeFilter = { op: 'between', a: 10, b: 20 };

// 3. Lambda encoding (custom logic)
const lambdaFilter = { lambda: (d: any) => d.value > 5 };

The Encoding object maps data dimensions to visual channels. Supported channels include:

• x, y: Position on the plot using NumericScaleChannel.
• color: Color encoding using ColorScaleChannel (supports LinearColorScale like 'viridis' or CategoricalColorScale).
• size: Point size using ConstantChannel or LambdaChannel.
• filter, filter2, foreground: Boolean logic for visibility using BooleanChannel (supports operators like gt, lt, eq, within, between).
• jitter_radius, jitter_speed: Controls for point jittering.
• jitter_method: The animation/distribution style ('None', 'spiral', 'uniform', 'normal', 'circle', or 'time').

Channels can be defined using scales (mapping a field to a range) or lambdas (applying a function to a field).

const encoding: Encoding = {
  x: { field: 'longitude', transform: 'linear' },
  y: { field: 'latitude' },
  color: {
    field: 'category',
    range: ['red', 'blue', 'green'] // CategoricalColorScale
  },
  filter: {
    field: 'age',
    op: 'gt',
    a: 18
  }
};

A StatefulAesthetic is a container that persists for the lifetime of a Scatterplot. It manages the lifecycle and memory resources for a specific encoding channel (e.g., color or size).

Key Concepts

• Dual States: It maintains two states: current and last. This allows the engine to perform smooth transitions (interpolations) between a previous visual state and a new one.
• Resource Persistence: Instead of re-allocating memory buffers every time data changes, StatefulAesthetic reuses existing buffers by maintaining consistent IDs during updates.
• Transition Detection: The update(encoding) method compares the new encoding with the current one. If they differ, it flips the states (moving current to last) and sets needs_transitions = true to trigger animations.

API Summary

• current: Returns the active aesthetic state.
• last: Returns the previous aesthetic state used for interpolation.
• update(encoding): Updates the aesthetic with a new encoding configuration. If the encoding is identical to the current one, it marks needs_transitions = false to prevent unnecessary animations.

Aesthetics in Deep Scatterplots (like X, Y, or Size) map data fields to visual properties using scales.

Key Concepts:

• Domain: The range of input values from your data (e.g., the min and max of a column).
• Range: The range of output values used for rendering (e.g., pixel coordinates or color values).
• Encoding: An object that configures how a field is mapped. It can specify the field, a custom domain, a custom range, a transform, or a constant value.
• Categorical vs. Continuous:
  • Categorical aesthetics use dictionary columns (Apache Arrow) and map values to discrete points using ordinal scales.
  • Continuous aesthetics use numeric or date columns and map values using linear, log, or sqrt scales.

Applying Encodings:

When an encoding is applied to a data point (Datum), the system looks for the field value. If the field is missing or the encoding specifies a constant, that constant is returned. Otherwise, the value is passed through the calculated scale.

A Deeptable manages the production and manipulation of tiles for large datasets. You can initialize it in several ways depending on your data source:

1. From Quadfeather: Use fromQuadfeather to load a quadtree created by the quadfeather package. This requires a baseUrl pointing to the data.
2. From an Arrow Table: Use fromArrowTable to wrap an existing Apache Arrow Table into a Deeptable.
3. Manual Construction: Use the new Deeptable() constructor for custom configurations, providing a baseUrl, rootKey, and optionally a tileManifest or extent.

Note that a Deeptable is asynchronous; you should await the ready property (which returns the initialization promise) before performing operations.

// Option 1: From Quadfeather
const dt = await Deeptable.fromQuadfeather({
  baseUrl: 'https://example.com/data',
  plot: myScatterplot
});

// Option 2: From an Arrow Table
const dt = Deeptable.fromArrowTable(myArrowTable, myScatterplot);

// Wait for initialization
await dt.ready;