VulcanSQL Documentation

repository·develop·Indexed 21 days ago

https://github.com/canner/vulcan-sql

An Analytical Data API Framework that turns SQL queries into RESTful APIs, optimized for AI agents and data-driven applications. It provides standardization via OpenAPI and performance via DuckDB caching. The framework includes a catalog-server, a CLI, and various extensions for external API calling, authentication via Canner Enterprise, debug tools, and data source drivers for BigQuery, Canner, and ClickHouse.

Tokens
91.3K
Snippets
344
Records
442
Agent score
69%

What's inside VulcanSQL

  1. Overview of VulcanSQL API plugins

    develop

    VulcanSQL provides a suite of API plugins designed to enhance functionality, streamline data processing, and ensure a secure, optimal user experience. These plugins allow developers to control how data is delivered and how the API behaves under load.

    Key areas of extensibility and control include:

    • Response Format: Standardizing data output (e.g., json, csv) for client consistency.
    • Pagination: Managing large datasets by breaking them into smaller, manageable chunks.
    • CORS (Cross-Origin Resource Sharing): Managing secure data access across different domains.
    • Rate Limiting: Controlling request frequency to prevent abuse and maintain performance.
    • Access Logging: Monitoring and analyzing user activities for security and efficiency.
  2. Overview of Data Privacy features in VulcanSQL

    develop

    VulcanSQL provides several built-in mechanisms to ensure data confidentiality, integrity, and availability. Developers can implement the following security layers to protect sensitive information:

    • Authentication: Verifying the identity of users, devices, or systems.
    • Authorization: Controlling the level of access granted after identity is verified.
    • Dynamic Data Masking: Obfuscating sensitive information by replacing it with fictional or scrambled data (useful for third-party sharing or UI display).
    • Column-Level Security (CLS): Restricting access to specific columns within a database table.
    • Row-Level Security (RLS): Limiting access to specific rows within a database table based on user permissions or specific criteria.
  3. Introduction to VulcanSQL Catalog Server

    develop
    VulcanSQL allows you to build a self-service catalog page. This enables users—including those without SQL knowledge—to extract data via APIs crafted by data analysts and engineers. The catalog provides a user interface to explore available APIs, view metadata, interact with data in real-time, and export results.
  4. Overview of @vulcan-sql/core extension components

    develop

    The @vulcan-sql/core package provides 10 distinct extension points that allow developers to customize how VulcanSQL interacts with data sources, processes filters and tags, validates input, manages persistence, serializes data, reads profiles, and provides templates.

    Key extension pairs include:

    • Filter Customization: Use FilterBuilder to define new filter recognition logic and FilterRunner to implement the execution logic for those filters.
    • Tag Customization: Use TagBuilder to define new tag recognition logic and TagRunner to implement the execution logic for those tags.
  5. What is VulcanSQL?

    develop
    VulcanSQL is an Analytical Data API Framework designed for AI agents and data applications. It allows data professionals to transform SQL queries into secure, standardized RESTful APIs that interact with databases, data warehouses, or data lakes. It uses OpenAPI documentation to ensure that AI agents can consistently understand and interact with the exposed data.
  6. Create Data APIs with VulcanSQL and SQL Templates

    develop

    VulcanSQL allows you to build scalable data APIs using SQL templates combined with Jinja templating. This approach enables you to expose data from warehouses or data lakes without backend programming skills.

    Key features include:

    • Automatic API Best Practices: Built-in support for OpenAPI documentation, rate limiting, pagination, and CORS.
    • Caching: Uses DuckDB as a caching layer to improve performance and reduce costs.
    • Security: Built-in user authentication, authorization, and data masking.
    • API Catalog: An extension that allows non-technical users to discover and explore available endpoints.

    The SQL syntax used in your templates depends on the database you are connected to (e.g., DuckDB, BigQuery, etc.).

    {% set country_codes = context.params.country_code %}
    
    SELECT * FROM read_csv_auto('WHO-COVID-19-global-data.csv')
    WHERE
        Date_reported >= {{ context.params.start_date | is_required }} AND
        Date_reported <= {{ context.params.end_date | is_required }}
    
        {% if country_codes %}
        AND Country_code IN (SELECT UNNEST(string_split({{ country_codes }}, ',')))
        {% endif %}
  7. Understand Response Format selection logic

    develop

    The plugin determines the response format type based on a hierarchy of inputs: the API URL path extension, the HTTP Accept header, and the configured options.default value.

    Key behaviors:

    • Path Priority: If the URL path specifies a format (e.g., /api/data.csv), that format is used, provided it is included in the options.formats list.
    • Header Negotiation: If no path extension is present, the plugin uses the Accept header to negotiate the format from the list of supported options.formats.
    • Default Fallback: If the Accept header is not set or cannot be negotiated, the plugin falls back to options.default.
    • Errors: If the path specifies a format that is not in the options.formats list, an error is returned.
    | Accept | API URL Path | options.default | options.formats | response format type |
    | --- | --- | --- | --- | --- |
    | `application/json;q=0.9` | `/api/data` | `json` | `["json","csv"]` | json |
    | `application/json;q=0.8, text/csv;q=0.9` | `/api/data` | `json` | `["json","csv"]` | csv |
    | Not set | `/api/data` | `json` | `["json","csv"]` | json |
    | Not set | `/api/data` | `json` | `["csv","json"]` | csv |
    | Not set | `/api/data` | `json` | Not set | json |
    | `application/json;q=0.9` | `/api/data.csv` | `json` | `["json","csv"]` | csv |
    | `application/json;q=0.9, text/csv;q=0.9` | `/api/data.json` | `csv` | `["csv"]` | Error |
    | `application/json;q=0.9, text/csv;q=0.9` | `/api/data` | Not set | Not set | json |
  8. Use Filters with the pipe operator

    develop

    VulcanSQL uses the Nunjucks template engine, allowing you to apply filters to variables using the pipe operator (|). Filters can be chained together for complex transformations.

    -- Basic usage
    SELECT {{ context.params.name | upper }} AS name_upper
    
    -- Chaining filters
    {% set items = ['foo', 'bar', 'bear'] %}
    SELECT {{ items | join(",") | upper }} AS items_joined_upper
  9. Use the `raw` filter for logic and conditions

    develop

    The raw filter is a custom VulcanSQL filter that outputs the literal value of a variable or dynamic parameter without further transformation. It is primarily used to capture a value into a variable so it can be used for logical checks (like if statements) within the template, preventing the templating engine from trying to evaluate the raw value as a template expression.

    {% set gender = (context.params.gender | upper | raw) %}
    {% if gender in ['MALE', 'FEMALE'] %}
        SELECT concat('Yes,', {{ context.params.name }}, 'is', {{ gender }}) as message;
    {% endif %}