DOCSEARCHbyAlgolia
Submit Docs

Querybook Documentation

website·Indexed 33 days ago

https://www.querybook.org/docs/

Documentation for Querybook, a data analysis platform featuring DataDocs, query peer review, and support for all SQLAlchemy compatible query engines. Includes guides for Kubernetes deployment via Helm, GitHub integration, AI assistant plugins, and developer instructions for DB migrations and Elasticsearch re-initialization.

Tokens
28.9K
Snippets
79
Records
259
Agent score
67%

What's inside Querybook

  1. Overview of Querybook Event Logging

    Querybook provides instrumentation support to log client action events and API events to analyze user behavior (e.g., tracking feature usage, UI clicks, and page views). This is currently an experimental feature and plugins used for customization may break in future updates.

    Each event log entry contains:

    • created_at (datetime): Timestamp of the event.
    • uid (int): ID of the user who performed the action.
    • event_type (EventType): The type of action (e.g., API, VIEW, CLICK).
    • event_data (dict): JSON containing additional information about the action.

  2. Overview of Querybook features

    Querybook is an open-source big data IDE that uses a notebook interface (DataDocs) to organize rich text, queries, and charts. Key capabilities include:

    • Smart Query Editor: Provides autocompletion, syntax highlighting, and table metadata on hover/click.
    • Visualizations: Built-in support for line, bar, stacked area, pie, horizontal bar, donut, scatter, and table charts.
    • Templating: Supports dynamically generated queries using Jinja2 templating with on-the-fly variables.
    • Scheduling: Automatic DataDoc updates at set intervals with options to export to external apps.
    • Query Analytics: Automatic analysis of executed queries to provide data lineage, example queries, and search ranking.

  3. Overview of Querybook Plugin Types

    Querybook supports a wide variety of plugins that allow administrators to extend functionality without modifying the core source code. Available plugin types include:

    • Query Engine Plugin: Add custom query engines or override default engine behaviors.
    • Auth Plugin: Implement custom authentication methods or post-authentication behaviors (e.g., automatic environment assignment).
    • Engine Status Checker: Customize how backend query engine information is exposed to the frontend.
    • Exporter Plugin: Provide ways to move query results to external sites (e.g., Python or R exporters).
    • Notifier Plugin: Send custom completion messages via channels like Email or Slack.
    • Job Plugin: Add new job schedules or schedule tasks from the Task plugin.
    • Metastore Plugin: Configure how metastores are populated and load table information.
    • Result Store Plugin: Implement custom storage for query results/logs (e.g., Google Cloud or local file system) beyond the default S3 and SQLAlchemy options.
    • Task Plugin: Implement custom asynchronous tasks (e.g., profile picture refreshes).
    • Web Page Plugin: Inject custom JS and CSS into the webapp via webpage_plugin/custom_script.ts.
    • Lineage Plugin: Fetch lineage information from custom data lineage backends.
    • Event Logger Plugin: Store event logs (API requests, instrumentation) in custom locations beyond the console or MySQL.
    • DAG Exporter Plugin (Experimental): Create workflows from Query Cells in DataDocs.
    • API Plugin: Create new or modify existing API endpoints.
    • Monkey Patch Plugin: Override or modify existing modules or functions.
    • AI Assistant Plugin: Enable LLM-powered features like text-to-SQL, title generation, and query auto-fix.

  4. Overview of Querybook Table Upload

    Querybook allows users to create SQL tables by uploading local CSV/TSV files or by using the results of a query execution. The process involves loading the data into a Pandas DataFrame and exporting it to the target query engine. Note that this is currently an experimental feature.

  5. Overview of Query Peer Review

    The Query Peer Review feature allows teams to collaboratively review queries before they are executed. This process helps maintain quality standards and prevents potential issues from reaching production or impacting the system.

  6. Overview of Querybook Notifiers

    Notifiers allow users to receive notifications upon the completion of their queries. If no notifiers are configured, Querybook uses a NoopNotifier, which logs the notification message to the server logs and suggests enabling a notifier.

  7. Use Advanced Charting features in Querybook

    Querybook allows users to transform data and create charts directly from query results. Key capabilities include:

    • Data Source Management: Choose a specific execution for the chart or set it to update automatically every time the query runs.
    • Data Transformation: Aggregate data using averages, counts, sums, min/max, and medians, or switch rows and columns.
    • Chart Types: Supports Stacked Area, Bar (including stacked), Horizontal Bar (including stacked), Pie, Doughnut, Scatter, and Table.
    • Display Customization: Select any column as the x-axis, customize displayed series, and configure titles, labels, and legend positions.

  8. Utilize Column Hover Tooltips and Value Auto-Completion

    The query editor includes two productivity enhancements for schema exploration:

    • Column Hover Tooltips: Hover over a column name in the editor to view detailed information about that column.
    • Value Auto-Completion: When writing a WHERE clause, the editor suggests distinct values from column statistics to improve filtering accuracy.

  10. Use the AI Assistant for query generation and fixing

    Querybook provides an LLM-powered AI assistant that supports three primary tasks: generating titles for query cells, converting natural language text to SQL (Text-to-SQL), and automatically fixing query errors.

  11. Understand Querybook primary architectural components

    Querybook consists of three primary functional components:

    • Web Server: Handles HTTP requests, manages WebSocket messages, and serves static web assets.
    • Worker: Executes long-running queries, processes scheduled DataDocs, updates Elasticsearch documents, and analyzes query lineage.
    • Scheduler: Reads task schedules from the database and dispatches them to Celery workers.

  12. Querybook Big Data IDE Core Features

    Querybook is a Big Data IDE designed for discovering, creating, and sharing data analyses, queries, and tables. Key capabilities include:

    • Analyses: Organize queries, charts, and rich text into cohesive documents.
    • Query Composition: Features autocompletion and hovering tooltips for writing queries.
    • Dashboards: Build dashboards using scheduling and charting within DataDocs.
    • Collaboration: Supports live query collaboration with other users.
    • Documentation: Allows adding custom documentation to tables.
    • Query Insights: Provides lineage, sample queries, frequent user data, and search ranking based on historical query runs.