datamapplot

repository·main·Indexed 21 days ago

https://github.com/tutteinstitute/datamapplot

A library for creating presentation and publication ready static or interactive data map plots. It provides tools for generating visualizations from 2D coordinates, including alpha-shape boundary polygons, edge bundling using the Hammer algorithm, and a comprehensive suite of UI widgets (such as TitleWidget, SearchWidget, and TopicTreeWidget) for interactive HTML representations. It includes a ConfigManager for global settings and SelectionHandlers for processing interactive data selections.

Tokens
31K
Snippets
81
Records
128
Agent score
76%

What's inside datamapplot

  1. Manage widget dependencies (JS and CSS)

    main

    Widgets can declare specific JavaScript and CSS dependencies using a dependencies list. Dependencies follow the format type:name (e.g., js:d3 or css:histogram).

    • js:name: Loads a local JavaScript file from the project's static directory.
    • css:name: Loads a local CSS file from the project's static directory.
    • http...: If the dependency string starts with http, it is treated as an external CDN library.

    Dependency Loading Priority:

    1. Core DataMap files (datamap.js, data_selection_manager.js).
    2. Widget-specific JS files.
    3. Widget-specific CSS files.
    4. External CDN libraries.
    class HistogramWidget(WidgetBase):
        dependencies = ["js:d3", "js:histogram", "css:histogram"]
    
    # Example of how dependencies are parsed internally
    # 'js:d3' -> type: 'js', name: 'd3'
    # 'https://cdn...' -> type: 'external', name: 'https://cdn...'
  2. Compare `use_system_fonts` with Offline Mode

    main

    DataMapPlot provides two distinct ways to handle offline requirements:

    Featureuse_system_fontsOffline Mode (cached)
    Use caseStatic plots without internetInteractive plots with cached resources
    Setup requiredNoneRun dmp_offline_cache tool
    Font selectionSystem fonts onlyCached Google Fonts
    Works withcreate_plot()create_interactive_plot()
    Storage needsNone~10MB for cache files

    Use use_system_fonts for simple static plots. For interactive plots that need to work offline, use the offline_mode configuration which caches JavaScript dependencies and fonts.

  3. How widgets and the datamap communication hub work together

    main

    In datamapplot, widgets are self-contained components that interact through a central communication hub: window.datamap.

    Instead of direct widget-to-widget communication, widgets register themselves with the datamap instance. Events (such as a selection in a HistogramWidget) flow through the datamap selection system, allowing other widgets (like a LegendWidget) to react to those changes. This architecture ensures that widgets remain decoupled while still enabling complex, interactive data visualizations.

  4. Understand the Widget System Architecture

    main

    The datamapplot widget system is transitioning from a flag-based legacy system to a widget-driven architecture.

    Legacy System Flow (use_widgets=False)

    In the legacy mode, functionality is controlled by explicit boolean parameters (e.g., enable_topic_tree=True). The Jinja2 templates use these flags to conditionally render JavaScript initialization code. This method is functional but decouples the widget instance from the actual JavaScript component creation.

    New Widget System Flow (use_widgets=True)

    The goal of the new architecture is to make JavaScript initialization dependent on the presence of widget instances rather than boolean flags.

    Current Implementation Status:

    • Widget-Owned JS Initialization: Widgets now include a javascript property and dependencies class variable to handle self-initialization via IIFEs triggered by the datamapReady event.
    • Data Flow: Data is extracted from widget instances via collect_widget_data(), prepared via encode_widget_data(), and passed through the rendering pipeline to the template context, eventually exposing data to the browser via window.widgetHistogramData or window.widgetColormapData.
  5. Use Selection Handlers to implement interactivity

    main

    The datamapplot.selection_handlers module provides a set of classes designed to handle user interactions and data selection within plots. By implementing or using these handlers, you can extend the functionality of your data maps to include features like word clouds, summaries, or tag-based filtering.

    Available selection handler classes include:

    • SelectionHandlerBase: The base class for all selection handlers.
    • DisplaySample: Likely used for displaying specific data samples upon selection.
    • WordCloud: Generates a word cloud based on selected data.
    • CohereSummary: Provides summaries using Cohere (requires integration with Cohere).
    • TagSelection: Handles selection based on specific tags.
  6. How widgets initialize their own JavaScript

    main

    In the current architecture, each widget is responsible for its own JavaScript initialization. Widgets implement a javascript property that returns a self-invoking function (IIFE). This function handles:

    1. Container Selection: Finding the specific DOM element using self.get_container_id().
    2. Data Readiness: Waiting for the datamapDataLoaded or datamapReady events before executing setup logic.
    3. Instance Storage: Storing a reference to the widget instance in window.datamap.widgets['{widget_id}'] so it can be accessed by other parts of the application.

    Pure HTML/CSS widgets (like TitleWidget or LogoWidget) return an empty string for their javascript property.

    class TopicTreeWidget(WidgetBase):
        @property
        def javascript(self):
            container_id = self.get_container_id()
            return f"""
            (function() {{
                const container = document.querySelector('#{container_id}');
                if (!container || !window.datamap) return;
                
                function setup() {{
                    const topicTree = new TopicTree(container, window.datamap, ...);
                    window.datamap.widgets['{self.widget_id}'] = topicTree;
                }}
                
                if (window.datamap && window.datamap.labelData) {
                    setup();
                } else {
                    document.addEventListener('datamapDataLoaded', setup);
                }}
            }})();
            """
  7. Create a new DataMapPlot example

    main

    Examples serve as both documentation and end-to-end tests. When adding a new example to the examples/ directory, ensure it follows these requirements:

    1. Descriptive Docstring: Include a title and explanation.
    2. Standalone: The script must be runnable on its own.
    3. Realistic Data: Use data that is realistic but manageable.
    4. Specific Use Case: Demonstrate a particular feature or capability.

    Workflow:

    1. Create a new .py file in examples/ (e.g., plot_new_feature.py).
    2. Implement the example following the structure above.
    3. Verify it runs without errors.
    4. Add it to the documentation and (optionally) include it in the CI pipeline.
  8. Basic usage of DataMapPlot

    main

    DataMapPlot provides two primary functions for generating plots: create_plot for static plots and create_interactive_plot for interactive HTML plots. Both functions require the coordinates of the data map and an array or list of labels for the data points. You can pass additional keyword arguments (**style_keywords) to customize the aesthetic output.

    import datamapplot
    
    datamapplot.create_plot(data_map_coords, data_map_labels, **style_keywords)
  9. Add custom elements to a plot using Matplotlib

    main

    DataMapPlot returns standard Matplotlib Figure and Axes objects. You can use these objects to apply any standard Matplotlib commands to add extra elements or modify the plot.

    Note: Custom modifications are performed at your own risk as they may interact with the library's internal rendering logic.

  10. Build documentation locally

    main

    DataMapPlot uses Sphinx with Read the Docs. Documentation is written in reStructuredText (.rst) or Jupyter Notebooks (.ipynb) and is located in the doc/ directory. To build the HTML documentation on your local machine, install the documentation dependencies and use make html within the doc directory.

    To view the built documentation, open _build/index.html in your web browser.

    # Install documentation dependencies
    pip install -r doc/requirements.txt
    
    # Build HTML documentation
    cd doc
    make html
    
    # View the documentation (open _build/index.html in a browser)
  11. Run DataMapPlot tests using Makefile

    main

    DataMapPlot uses a Makefile to manage various test suites. You should run backend tests before frontend tests to ensure the necessary HTML is generated for interactive testing.

    Available Test Rules

    Use make to see the full menu of available rules.

    • test: Run all tests.
    • test-static: Run Python-based backend and static frontend tests.
    • test-backend: Run Python-based backend tests (Unit and Backend).
    • test-ui: Run all interactive frontend tests (Playwright).
    • test-ui-fast: Run only fast interactive tests.
    • report-static: Open the mpl static test report.
    • report-interactive: Open the Playwright test report.
    • update-static-baseline: Update static baseline images.
    • update-interactive-baseline: Update interactive baseline images.
    # Run unit and backend tests (Must be run before frontend tests)
    make test-backend
    
    # Run static visual regression tests
    make test-static
    make report-static
    
    # Run interactive browser tests
    make test-ui
    # OR run only fast tests
    make test-ui-fast
    make report-interactive
  12. Set up the DataMapPlot development environment

    main

    To develop for DataMapPlot, you need Python 3.10+ (3.10, 3.11, or 3.12 recommended), Git, and Node.js. Follow these steps to configure your environment:

    1. Clone the repository
    2. Install Python dependencies: Install the package in editable mode, then install testing and documentation requirements.
    3. Install Node.js dependencies: Required for running interactive Playwright tests.

    Note: Use a virtual environment for Python dependencies.

    # 1. Clone the repository
    git clone https://github.com/TutteInstitute/datamapplot.git
    cd datamapplot
    
    # 2. Install Python dependencies
    pip install -e .
    pip install -r test-requirements.txt
    pip install -r doc/requirements.txt
    
    # 3. Install Node.js dependencies (for interactive tests)
    cd datamapplot/interactive_tests
    npm ci
    npx playwright install --with-deps