quary

repository·main·Indexed 25 days ago

https://github.com/quarylabs/quary

A data modeling tool and VS Code extension for executing models and tests, visualizing tables and views, and managing data workflows. It includes a CLI, a dbt project playground called jaffle_shop, and a developer framework (quary-extension-bus) for implementing type-safe Webview communication and database service interfaces.

Tokens
22.3K
Snippets
44
Records
139
Agent score
81%

What's inside quary

  1. Overview of pbjson-types

    main

    The pbjson-types crate is a modified version of the pbjson crate by InfluxData. It is designed to be used as a library to provide types for Protocol Buffers (Protobuf) JSON mapping.

    A key modification in this version is that for Struct types, it uses BTreeMap instead of HashMap to ensure that the order of fields is preserved.

  2. Understand the jaffle_shop project structure and limitations

    main

    The jaffle_shop project is designed as a simple playground rather than a production-ready template.

    Key Characteristics:

    • Uses Seeds: It uses dbt seeds to load raw data from CSVs, whereas production projects typically use sources to point to existing data in a warehouse.
    • Simplified Patterns: To maintain simplicity, it does not implement standard file naming patterns, pull request flows, CI/CD integrations, or advanced dbt features like macros, packages, or hooks.
    • Data Model: The project transforms raw data (customers, orders, and payments) into analytics-ready models based on the provided entity-relationship diagram (ERD).
  3. Understand the Quary project structure

    main

    A Quary project is organized into specific directories for managing data models, sources, and testing:

    • models/staging/schema.yaml: Defines your external data sources.
    • models/: Contains the core data models for your project.
    • seeds/: (Optional) Contains seed data files used for initial database setup.
    • tests/: Contains custom test cases designed to validate your data models.
  4. Set up and run the quary-extension Web Extension

    main

    To develop and run the quary-extension as a VS Code Web Extension, follow these steps:

    1. Install Dependencies: Run npm install in the project root.
    2. Install Recommended VS Code Extensions:
      • amodio.tsl-problem-matcher
      • dbaeumer.vscode-eslint
    3. Debug the Extension:
      • Place breakpoints in src/web/extension.ts.
      • Press F5 (using the Run Web Extension launch configuration).
    4. Execute Commands: Open the Command Palette (F1) and execute Hello world to trigger the extension code.

    To apply changes after editing src/web/extension.ts, you can either use the debug toolbar to relaunch or reload the VS Code window using Ctrl+R (Windows/Linux) or Cmd+R (macOS).

    npm install
  5. Add a custom table and source to a Quary project

    main

    You can extend your project by adding arbitrary SQL tables and defining them as sources in YAML.

    1. Add a table

    Use the QUARY: Statement command to execute a SQL statement against the database:

    CREATE TABLE IF NOT EXISTS employee_band_table (employee_id INTEGER, band_id INTEGER);

    2. Define a source

    To make the new table available to models and tests, create a new project file and add a sources definition:

    sources:
      - name: employee_band
        path: employee_band_table
    // 1. Run via QUARY: Statement
    CREATE TABLE IF NOT EXISTS employee_band_table (employee_id INTEGER, band_id INTEGER);
    
    // 2. Add to a project file
    sources:
      - name: employee_band
        path: employee_band_table
  6. Use Quary development and deployment commands

    main

    Quary provides several commands for managing your data workflow. These can be used via the UI or the CLI.

    Development Commands

    • Render Model (QUARY: Render Model): Visualise a specific model.
    • Do Statement (QUARY: Do statement): Execute arbitrary commands directly against your database.
    • Test (QUARY: Test): Execute tests against all data models.
    • Render Tables (QUARY: Render Tables): Visualise tables and views currently in the database.
    • Render Sources (QUARY: Render Sources): View the configured sources.
    • Render Full Schema (QUARY: Render Full Schema): View the complete execution script.
    • Generate Query Helper (QUARY: QUARY: Generate Query Helper): Use AI assistance to generate models.

    Deployment Commands

    • Run Models (QUARY: Run): Deploys your data models to your database.
  7. Run the jaffle_shop dbt project

    main

    The jaffle_shop project is a self-contained dbt playground used for testing scripts and core dbt concepts. It uses seeds (CSV files) instead of sources to provide raw data for customers, orders, and payments.

    To run the project, follow these steps:

    1. Install dbt following the official documentation.
    2. Clone the repository and navigate to the project directory:
      cd jaffle_shop
    3. Configure a profile named jaffle_shop in your dbt profiles file to connect to your data warehouse (e.g., Postgres or another supported warehouse).
    4. Verify the connection:
      dbt debug
    5. Load the demo data (materializes CSVs as tables in your target schema):
      dbt seed
    6. Execute the models:
      dbt run
      Note: If dbt run fails, you may need to adjust the SQL in the models/ folder to match your specific database's SQL dialect.
    7. Run data tests:
      dbt test
    8. Generate and view documentation:
      dbt docs generate
      dbt docs serve
    $ cd jaffle_shop
    $ dbt debug
    $ dbt seed
    $ dbt run
    $ dbt test
    $ dbt docs generate
    $ dbt docs serve
  8. Run Quary models and tests via Command Palette

    main

    You can execute models and tests using the VS Code Command Palette (Cmd + Shift + P or Ctrl + Shift + P).

    • QUARY: Run: Executes all models in the project.
    • QUARY: Test: Executes all tests.

    Note: You must run the models first using QUARY: Run before running tests to ensure the necessary data is present in the database.

  9. Set up the Quary VS Code extension

    main

    To use Quary in Visual Studio Code, you can use a browser-based instance by pressing the . shortcut on a GitHub repository. Once the editor is open, install the recommended Quary extension via the pop-up notification or by selecting it from the Recommended tab in the Extensions view.

    When running Quary for the first time, you will be prompted to select a database configuration. For the sample project, select sqlite-in-browser.

  10. Render tables and views in Quary

    main

    To inspect the current state of your database, use the following commands from the Command Palette:

    • QUARY: Render Tables: Displays the tables and views currently in the database.
    • QUARY: Render Model: Prompts you to select a specific model to render individually. This is useful for rapid development.
  11. Add a new table and source to a Quary project

    main

    To extend your project with new data, follow these two steps:

    1. Add a table to the database

    Use the QUARY: Statement command to execute arbitrary SQL against your database. For example:

    CREATE TABLE IF NOT EXISTS employee_band_table (employee_id INTEGER, band_id INTEGER);

    2. Register the source in your project

    Create a new project file (YAML) and define the source so it can be used in models and tests:

    sources:
      - name: employee_band
        path: employee_band_table
    CREATE TABLE IF NOT EXISTS employee_band_table (employee_id INTEGER, band_id INTEGER);
    sources:
      - name: employee_band
        path: employee_band_table