DOCSEARCHbyAlgolia
Submit Docs

Mosaic

repository·main·Indexed 23 days ago

https://github.com/uwdata/mosaic

An extensible framework for scalable linked data views that pushes computation to DuckDB. It includes @uwdata/mosaic-core for managing data flow via Coordinators, Parameters, and Selections; @uwdata/mosaic-sql for SQL query construction; and @uwdata/mosaic-inputs for data-driven UI components. The ecosystem provides multiple DuckDB server implementations in Go, Rust, and Python, supporting Apache Arrow and JSON data transfer via WebSockets and HTTP.

Tokens
68.6K
Snippets
116
Records
389
Agent score
79%

What's inside Mosaic

  1. Overview of Mosaic packages

    main

    The Mosaic ecosystem is divided into several functional areas:

    Core Architecture (packages/mosaic)

    • mosaic-core: Contains the central coordinator, parameters, selections (for linking scalar values or query predicates), and filter groups with materialized views.
    • mosaic-sql: An API for constructing and analyzing SQL queries. Query objects can be coerced into SQL strings.

    vgplot Libraries (packages/vgplot)

    vgplot is a visualization grammar API that integrates multiple Mosaic packages. For most applications, you can use @uwdata/vgplot alone.

    • vgplot: A high-level API for building interactive dashboards. It re-exports much of mosaic-core, mosaic-sql, mosaic-plot, and mosaic-inputs.
    • mosaic-inputs: Standalone components like input menus, text search boxes, and sortable data tables.
    • mosaic-plot: An interactive grammar of graphics built on Observable Plot. It allows marks (plot layers) to push heavy processing (like binning or regression) down to the database.
    • mosaic-spec: Allows defining Mosaic applications declaratively using JSON or YAML.
    • widget: A Jupyter widget for generating web-based visualizations from Pandas, Polars, or DuckDB connections.

    DuckDB Servers (packages/server)

    • duckdb-server (Python): A local DuckDB instance supporting Web Sockets or HTTP, returning Apache Arrow or JSON. (Recommended for most users).
    • duckdb-server-rust (Rust): Similar to the Python server but with additional support for HTTP/2.
    • duckdb-server-go (Go): Similar to the Python server with experimental support for multi-tenant access control and function blocklisting.
    • mosaic-duckdb (Node.js): A Promise-based Node.js API. Note: The maintainers recommend using the Python-based duckdb-server instead due to quality issues with the Node.js DuckDB client.

  2. Overview of mosaic-inputs

    main

    The @uwdata/mosaic-inputs package provides standalone, data-driven UI components designed for interactive data exploration. These components include:

    • Input menus
    • Text search boxes
    • Sortable, load-on-scroll data tables

    These widgets are designed to integrate seamlessly with Mosaic Params and Selections. They can be used in two ways:

    1. Standalone: Operating independently to drive parameters.
    2. Data-backed: Drawing values directly from a backing data table.

    Note: mosaic-inputs is included as part of the vgplot API.

  3. Overview of @uwdata/mosaic-core components

    main

    The @uwdata/mosaic-core package provides the fundamental building blocks for creating linked, interactive data applications. Its primary components include:

    • Coordinator: A central entity that manages the flow of data and interactions.
    • Parameters (Param): Used for linking scalar values across different Mosaic clients.
    • Selections (Selection): Used for linking query predicates (e.g., filtering criteria) across clients.
    • Filter Groups: Manage materialized views of pre-aggregated data to optimize performance.

    The Mosaic coordinator supports multiple query transport methods:

    • socket and rest clients: For sending queries over a network to a remote backing server.
    • wasm client: For querying a local DuckDB-WASM instance directly in the browser.

  4. Introduction to Mosaic vgplot

    main

    Mosaic vgplot is a grammar of interactive graphics that combines concepts from Vega-Lite, ggplot2, and Observable Plot. It allows you to create interactive visualizations where graphical marks act as Mosaic clients, producing queries for the data they need.

    Key features:

    • Interoperability: Built on Mosaic, it works seamlessly with other Mosaic clients like Mosaic Inputs.
    • Rendering: Uses Observable Plot to render SVG output.
    • Declarative: Supports both a JavaScript API and standalone JSON/YAML specifications (via mosaic-spec).
    • Simplified Imports: For most applications, you can simply import @uwdata/vgplot, which re-exports much of mosaic-core, mosaic-sql, mosaic-plot, and mosaic-inputs.

  5. Overview of @uwdata/mosaic-duckdb

    main

    The @uwdata/mosaic-duckdb package provides a Promise-based Node.js API for interacting with DuckDB. It includes a data server capable of transferring Apache Arrow and JSON data via WebSockets or HTTP.

    Important Recommendations

    • Preferred Alternative: Due to persistent quality issues with the Node.js DuckDB client and the Arrow extension, it is highly recommended to use the Python-based duckdb-server package instead.
    • Browser Usage: This package is intended for local DuckDB servers in Node.js. If you need to use DuckDB in a web browser, use the wasmConnector provided in the mosaic-core package instead.

  6. Overview of mosaic-plot

    main
    The mosaic-plot package provides a Mosaic-powered grammar of graphics built upon Observable Plot. It uses 'Mmarks' (plot layers) that act as individual Mosaic clients. This architecture allows plot layers to push heavy data processing tasks—such as binning, hex binning, and regression—directly to the database. It also supports optimizations like M4 for line and area charts. Additionally, it provides built-in interactors for linked selection, filtering, and highlighting by leveraging Mosaic Params and Selections.

  7. Overview of vgplot

    main

    vgplot: A Mosaic-powered Visualization Grammar

    vgplot is a JavaScript API designed for creating interactive visualizations and dashboards. It functions as a visualization grammar that integrates multiple Mosaic packages (such as @uwdata/mosaic-core, @uwdata/mosaic-inputs, and @uwdata/mosaic-plot) into a single, convenient, and composable API.

    Key distinction:

    • JavaScript API: Use the vgplot package directly in your code for programmatic visualization construction.
    • Declarative Specification: If you prefer using JSON or YAML formats instead of a JavaScript API, use the mosaic-spec package.

  8. Use mosaic-sql for SQL query construction and analysis

    main
    The @uwdata/mosaic-sql package provides an API for the convenient construction and analysis of SQL queries. It allows you to build Query objects that can be coerced into standard SQL query strings, facilitating programmatic query generation and inspection within the Mosaic ecosystem.

  9. Use the Mosaic DuckDB Server

    main

    The duckdb-server package provides a Python-based server that runs a local DuckDB instance. It supports querying data over Web Sockets or HTTP and returns results in either Apache Arrow or JSON format.

    Note: This is a local server implementation. If you prefer to run DuckDB directly in the browser using DuckDB-WASM, use the wasmConnector from the mosaic-core package instead.

  10. What is a Selection and how does it work?

    main

    A Selection is a specialized Param used to manage a collection of Boolean-valued predicates for interactive data filtering. It uses a resolution strategy to merge multiple clauses into a single predicate for a client.

    Resolution Strategies

    • single: Only the most recent clause is kept.
    • union: Combines all predicates using Boolean OR (disjunction).
    • intersect: Combines all predicates using Boolean AND (conjunction).

    Key Concepts

    • Cross-filtering: When cross: true is enabled, selection clauses are modified to omit clauses where the client is part of the input argument to the predicate() function. This allows selections to affect other views without filtering the view currently performing the interaction.
    • Empty State: By default, a selection with no clauses selects all records. To select no records instead, set the empty: true option.
    • Inclusion: A selection can include clauses from upstream selections. It will relay new clauses/activations from those selections but can still apply its own independent resolution strategy.
    • Clauses: A clause is an object containing the source (the component providing the clause), clients (associated clients to skip during cross-filtering), a predicate (the query logic), a value (the data representation), and an optional schema (metadata).

  11. What is Mosaic and how does it work?

    main

    Mosaic is an extensible architecture designed for linking data visualizations, tables, input widgets, and other data-driven components. It is built to handle massive datasets (millions or billions of records) by pushing computation to a backing database, specifically DuckDB.

    Core Mental Model

    Instead of components managing their own data, interface components "publish" their data needs as declarative queries. A central coordinator manages these queries, optimizes them, and performs cross-filtering. The coordinator then proxies these queries to a DuckDB instance, which can be:

    • Server-side: Accessed via network clients (socket or rest).
    • Client-side: Accessed via DuckDB-WASM using a wasm client.

    This architecture allows for scalable, interactive exploration of data across multiple linked views.

  12. How API Context works in vgplot

    main

    In the JavaScript vgplot package, an API context is the environment in which all vgplot methods are evaluated. It provides the necessary infrastructure for Mosaic to function in the browser.

    A context consists of three main components:

    1. API methods: The standard vgplot functions (e.g., vconcat, plot).
    2. Coordinator: The Mosaic coordinator that manages data flow and state.
    3. namedPlots: A map that links plot names to plot instances. This is used by components like legends to look up scale information.

    By default, calling vgplot methods uses a global context with a shared coordinator (vg.coordinator()) and a shared namedPlots map (vg.namedPlots).