DOCSEARCHbyAlgolia
Submit Docs

VegaFusion Documentation

repository·main·Indexed 19 days ago

https://github.com/vega/vegafusion

VegaFusion provides Rust, Python, and JavaScript libraries for analyzing and scaling Vega visualizations. It serves as a low-level engine for high-level tools like Vega-Altair, enabling the handling of large datasets efficiently. The ecosystem includes vegafusion-core for task graph planning, vegafusion-runtime for execution, vegafusion-server for gRPC access, and vegafusion-wasm for client-side or embedded WebAssembly runtime.

Tokens
41.6K
Snippets
137
Records
210
Agent score
65%

What's inside VegaFusion

  1. Overview of vegafusion-common

    main
    The vegafusion-common crate serves as a shared library containing various data types and functions required by multiple independent downstream crates within the VegaFusion ecosystem. It provides the foundational types used across the project to ensure consistency between different components.

  2. What is vegafusion-runtime?

    main

    The vegafusion-runtime crate is responsible for evaluating task graphs generated by vegafusion-core. It acts as the execution engine that bridges Vega specifications with high-performance data processing.

    Its core responsibilities include:

    • Expression Compilation: Compiling Vega expression ASTs into DataFusion expressions.
    • Signal Evaluation: Evaluating signal expression tasks within the Vega lifecycle.
    • Data Task Evaluation: Executing Data tasks and transform pipelines. These transforms are implemented against the DataFrame trait provided by the vegafusion-dataframe crate.
    • Specification Pre-transformation: Pre-evaluating Vega specification transforms and inlining the resulting transformed data to optimize the specification before it reaches the renderer.

  3. What is VegaFusion?

    main

    VegaFusion provides Rust, Python, and JavaScript libraries designed to analyze, accelerate, and scale Vega visualizations. It is built as a set of low-level building blocks intended for integration into higher-level Vega systems.

    Note for Vega-Altair users: If your goal is specifically to scale Vega-Altair visualizations for large datasets, you should refer to the Vega-Altair documentation regarding the "vegafusion" data transformer rather than the low-level VegaFusion library documentation.

  4. What is vegafusion-core?

    main

    The vegafusion-core crate is the central logic engine for VegaFusion. It is responsible for transforming an input Vega specification into a VegaFusion task graph.

    Key responsibilities include:

    • JSON parsing: Handling the input Vega specification.
    • Planning: Determining the optimal execution plan for the specification.
    • Protocol buffer definitions: Providing the generated types and definitions for the task graph.
    • Vega expression parsing: Interpreting Vega expressions within the specification.

  5. Overview of the vegafusion Python library

    main
    The vegafusion Python library provides a way to embed the VegaFusion Runtime and select Connections directly into Python environments. It uses PyO3 to bridge Rust logic to CPython, allowing high-performance Vega-related operations to be executed within Python workflows.

  6. Overview of VegaFusion

    main
    VegaFusion provides Rust, Python, and JavaScript libraries designed to analyze and scale Vega visualizations. It serves as a provider of low-level building blocks that higher-level Vega systems, such as Vega-Altair in Python, can integrate with to handle large datasets efficiently.

  7. Use vegafusion-server for gRPC access to VegaFusion

    main
    The vegafusion-server crate provides a binary application that exposes vegafusion-runtime functionality over a gRPC interface. This allows clients to interact with the VegaFusion runtime remotely using a standardized protocol. The gRPC service definitions (protocol buffer interface) are defined in the vegafusion-core crate.

  8. What is the Chart State workflow?

    main

    The Chart State workflow is designed to support interactive charts where transforms must be re-evaluated repeatedly based on user interaction. This is particularly useful for charts implementing features like crossfiltering, where a filter transform needs to be re-applied to the input dataset as the user interacts with the visualization.

    This workflow serves as the foundation for Vega-Altair's JupyterChart when used in conjunction with the vegafusion data transformer.

  9. What are VegaFusion checks?

    main
    VegaFusion checks are specialized Python scripts designed to be executed within Continuous Integration (CI) environments. They are distinct from the standard pytest test suite. Their primary purpose is to perform specific validation tasks that are unsuitable for regular unit tests, most notably verifying that lazy imports are functioning correctly.

  10. Understanding Optimized Expressions in VegaFusion

    main

    VegaFusion optimizes a subset of the Vega expression language. When a Vega specification includes expressions that are not eligible for optimization, VegaFusion will include them in the client-side Vega spec to be evaluated by the standard Vega library in the browser.

    Optimization support is categorized as:

    • ✅ Full optimization support: The feature and all its options are optimized.
    • 🟡 Partial optimization support: Supported with specific limitations.
    • ❌ Not eligible: The feature is not currently optimized and will run in the browser.

  11. Compare VegaFusion with altair-transform

    main

    altair-transform

    altair-transform is a Python library that provides pandas implementations of the Vega expression language and Vega Transforms. It is useful for two primary workflows:

    1. Extracting Data: Constructing a Pandas DataFrame that represents the result of a Chart's transforms (the input to the Chart's mark).
    2. Pre-Aggregating Large Datasets: Evaluating transforms to create a new Chart instance that refers only to the evaluated dataset. This is particularly useful for aggregation charts (like histograms) to reduce the amount of data transferred to the browser.

    Key Differences from VegaFusion:

    • Interactivity: altair-transform does not support evaluating transforms on the server for interactive workflows (e.g., linked histogram brushing), whereas VegaFusion was designed for this.
    • Development Status: altair-transform is not under active development.
    • Compatibility: Support for the workflows mentioned above was added to VegaFusion in version 1.0.

  12. Use `pre_transform_extract` to separate transformed datasets from specs

    main

    The pre_transform_extract method generates a transformed Vega specification similar to pre_transform_spec, but with a key difference: instead of inlining the transformed datasets directly into the specification, it returns the datasets separately in Apache Arrow table format.

    This pattern is recommended when dealing with large datasets. By extracting the data from the spec, you can transmit the specification and the data independently, which is often more efficient when using the Arrow format for large-scale data transfer.

    # Conceptual usage in Python
# The method is available on the VegaFusionRuntime class

# transformed_spec: The Vega spec with data references
# extracted_datasets: A collection of datasets in Arrow format
transformed_spec, extracted_datasets = runtime.pre_transform_extract(spec)