datashader

Datashader is a graphics pipeline system designed to create meaningful representations of large datasets quickly and flexibly. It works by breaking image creation into explicit steps, allowing computations on intermediate representations. This enables accurate visualizations without manual parameter tuning.

Key technical features:

• Performance: Computation-intensive steps are written in Python and transparently compiled to machine code using Numba.
• Scalability: Supports distribution across CPU cores using Dask and across GPUs using CUDA (via cuDF).
• Hardware: Optimized to work on standard hardware while scaling to distributed and GPU systems when available.

Datashader is a data rasterization pipeline that automates creating meaningful representations of large datasets. The process consists of three main steps:

1. Projection: Each record is projected into one or more bins of a nominal plotting grid shape, based on a specified glyph.
2. Aggregation: Reductions are computed for each bin, compressing the dataset into a smaller aggregate array.
3. Transformation: The aggregates are processed to create the final image.

Datashader is designed to be composable and can function as a standalone tool or as a pre-processing stage for other plotting libraries to enable visualization of much larger datasets.

For detailed tutorials, examples, and integration guides with other HoloViz tools, visit the official HoloViz website or the PyViz examples site. If you encounter usage issues or have questions, use the Datashader Discourse community forum.

• Tutorials & Integrations: holoviz.org
• Project Examples: examples.pyviz.org
• Community Support: Datashader Discourse

For experienced users, the documentation covers advanced topics including:

• Plotting Pitfalls: How Datashader avoids common errors encountered when plotting large datasets with traditional small-dataset techniques.
• Extending Datashader: How to add new components and functionality to the library.
• Performance: Optimization hints for maximizing performance in your applications.
• Inspection Reductions: Using reduction techniques to inspect data rather than just aggregating it.

Datashader supports a wide variety of data visualization tasks. The user guide provides detailed instructions for the following data types and use cases:

• Points: Plotting x,y locations and scatterplots.
• Timeseries: Plotting timeseries and other curves.
• Trajectories: Plotting trajectories (e.g. connected GPS points) in a plane.
• Grids: Plotting 2D-gridded data such as rasters and quadmeshes.
• Trimesh: Plotting irregular triangular grids.
• Networks: Plotting network graphs.
• Polygons: Using Datashader for geographic and other spatial applications.
• Geography: Specialized pointers for geographic and spatial applications.
• GeoPandas: Integration and support for GeoPandas objects.

Most Datashader examples are provided as Jupyter notebooks. After setting up the environment and data as described in the setup guide, navigate to the datashader-examples directory and launch Jupyter.

To ensure notebooks work reliably without an internet connection (preventing issues like Loading BokehJS ... hanging), start the Jupyter server with BOKEH_RESOURCES=inline and an increased iopub_data_rate_limit.

cd datashader-examples
# Standard launch
jupyter notebook

# Recommended launch for offline/unreliable connections
BOKEH_RESOURCES=inline jupyter notebook --NotebookApp.iopub_data_rate_limit=100000000

After running benchmarks, you can interact with the results using the following commands:

• List timings: Use asv show main to list the benchmark timings stored for the main branch.
• Web interface: ASV includes a simple webserver to display results in a browser. Run asv publish followed by asv preview, then open the URL provided in your browser.

After installing Datashader, you can download the example suite and its associated data into a local directory named datashader-examples using the Datashader CLI.

To run the examples, you must install additional dependencies. If you are using an existing conda environment, update it using the provided environment.yml. Otherwise, create a fresh environment.

# Fetch examples and data
datashader examples
cd datashader-examples

# If using an existing active conda environment:
conda env update --file environment.yml

# OR, to create a new dedicated environment:
conda env create --name datashader --file environment.yml
conda activate datashader

To add new benchmarks, create classes in the benchmarks/benchmarks directory.

Requirements:

• Naming: Any class member function starting with the prefix time (e.g., time_shade_categorical) will be identified as a timing benchmark.
• Data Setup: Use the setup() member function to create required data. This ensures data creation time is excluded from the benchmark results. Note that setup() is called once for each invocation of each benchmark and data is not cached.
• Parameterization: Define lists of parameter names and values at the top of the benchmark class. ASV will run the benchmark for every unique combination of these parameters.

To run a specific subset of benchmarks, use the -b flag with a regex matching the benchmark name:

asv run -b ShadeCategorical

To execute all benchmarks against the default main branch, navigate to the benchmarks directory and run asv run.

Note: The first execution will create a machine file and set up isolated virtual environments. Each benchmark is run multiple times to ensure statistical validity.

To quickly run all benchmarks exactly once (e.g., to check for errors without waiting for statistical convergence), use asv dev instead of asv run.

cd benchmarks
asv run

To compare performance between different branches (e.g., main vs a feature_branch):

1. Commit your code: Ensure your new code is committed to your local feature branch.
2. Configure ASV: Edit asv.conf.json to include your branch in the branches list:

   "branches": ["main", "new_feature_branch"]

3. Run benchmarks: Execute asv run to benchmark both branches.
4. Compare:
   • Use asv show to list the benchmarked commits.
   • Use asv compare <commit1> <commit2> to view a side-by-side performance comparison.

asv compare commit1 commit2

Datashader uses ASV (Airspeed Velocity) for benchmarking. You must install asv version 0.4.2 before running benchmarks. Choose the command corresponding to your environment manager:

For Conda:

conda install -c conda-forge asv==0.4.2

For virtualenv:

pip install asv==0.4.2 virtualenv