DOCSEARCHbyAlgolia
Submit Docs

Redash Toolbelt

repository·master·Indexed 41 days ago

https://github.com/getredash/redash-toolbelt

Official API client and utility suite for managing Redash instances. Version 0.1.9 provides a Python client for managing queries, dashboards, data sources, users, and alerts, as well as the redash-migrate CLI tool for migrating data between Redash instances via the REST API.

Tokens
5.3K
Snippets
17
Records
23
Agent score
99%

What's inside Redash Toolbelt

  1. Use redash-toolbelt CLI scripts

    master

    Once installed, redash-toolbelt provides several command-line interface (CLI) scripts to manage your Redash instance. These scripts allow for data migration, auditing, and exporting.

    gdpr-scrub                  Search for a string term in your Redash queries
                            and query results. The script returns a list of
                            URLs in your instance that contain references to
                            the search term you provide

find-tables                 Search the text of queries against a data source
                            to see which table names are used in queries of
                            that source. This script relies on regex that is
                            tested against ANSI SQL.

clone-dashboard-and-queries Completely duplicate a dashboard by copying all 
                            its queries and visualizations.

export-queries              Export all the queries of your Redash instance
                            as text files.

redash-migrate              Move data from one instance of Redash to another.
                            See docs/redash-migrate/README.md for more info

  2. Understand how meta.json tracks object mappings

    master

    The meta.json file maintains mappings between origin and destination object IDs. This allows the tool to maintain relationships (like a dashboard pointing to a specific query) even after IDs change during the migration.

    In the mappings, the key is the ID from the origin and the value is the ID at the destination.

    Example mapping for queries:

    "queries": {
    "108171": 6,
    "219541": 7,
    "381761": 8
}

    In this example, origin query 108171 was successfully migrated to destination query 6.

  3. Handle Data Sources and secret fields

    master

    When running the data-sources command, redash-migrate creates "stub" data sources at the destination.

    Important: These stubs mirror the name and type of the origin data sources, but all secret fields (passwords, key files, auth tokens) will be blank. This is a security measure of the Redash REST API. You must manually fill in these secret fields via the Redash UI at the destination before queries can run successfully.

  4. When to use (and not use) redash-migrate

    master

    Use redash-migrate if:

    • You are moving from Hosted Redash to Open Source (OSS) Redash.
    • You need to move data between instances but do not have access to the underlying databases.

    Do NOT use redash-migrate if:

    • Moving between major Redash versions: The tool relies on the REST API being identical between instances. If you are moving from V8 to V10, upgrade your V8 instance to V10 first, then use redash-migrate.
    • You have database access: If you can access the origin and destination databases, it is more efficient to perform a database dump and restore rather than using the API-based migration tool.

  5. Develop modifications to redash-migrate

    master

    To develop and test changes to the redash-migrate script, follow these steps to set up a local development environment using a Python virtual environment and Poetry:

    1. Clone the repository.
    2. Create a Python virtual environment (requires Python 3.6+).
    3. Activate the environment.
    4. Install poetry to manage the build process.
    5. Apply your changes.
    6. Build the package using poetry build, which generates a .tar.gz file in the dist/ directory.
    7. Install the newly built archive using pip to test your changes locally.
    # Clone redash-toolbelt to your disk
git clone https://github.com/getredash/redash-toolbelt.git

# Create a Python virtual environment using version 3.6+
python3 -m virtualenv ${env_name}

# Activate your virtual environment
. ${env_name}/bin/activate # macOS
${env_name}\scripts\activate # Windows

# We use Poetry to build redash-toolbelt for distribution on PyPi
pip install poetry

# Make your changes to the script. When it's time to test your changes.

# Build the package with Poetry. This will create a .tar.gz file in the /dist directory
poetry build

# Use to pip to install the .tar.gz archive created in the previous step
pip install dist/redash_toolbelt-x.x.x.tar.gz

  6. Initialize redash-migrate with meta.json

    master

    Before starting a migration, you must initialize a meta.json file in your working directory. This file tracks the state of the migration and stores authentication details for both the origin and destination instances.

    Run the following command to start the initialization process. You will be prompted to provide authentication details and the admin user IDs for both instances:

    redash-migrate init

  11. Configure meta.json settings

    master

    The meta.json file contains critical configuration settings.

    ⚠️ Warning: Do not share your meta.json file. It contains administrator API keys for your Redash instances. If compromised, reset your admin API keys immediately.

    Key settings include:

    • origin_url: The base URL for your origin Redash instance (e.g., https://app.redash.io/acme).
    • origin_admin_api_key: The API key for an admin user on your origin instance.
    • destination_url: The base URL for your destination Redash instance (e.g., http://localhost).
    • preserve_invite_links: A boolean (default true). If true, user invitation links are written to meta.json, allowing users to set passwords directly. If false, users must use the "I forgot my password" workflow.

  12. Avoid rate limits during migration

    master

    Because redash-migrate generates a high volume of network requests, it may trigger rate limiting on your destination Redash instance.

    To prevent this, it is recommended to disable rate limiting on the destination instance for the duration of the migration. On Redash V10, you can do this by setting the following environment variable:

    REDASH_RATELIMIT_ENABLED=false