DOCSEARCHbyAlgolia
Submit Docs

VeloDB Cloud 4.x SQL Manual

website·Indexed 34 days ago

https://docs.velodb.io

SQL reference manual for VeloDB Cloud 4.x and Apache Doris 3.x, covering scalar functions, conditional operators, and cluster management statements for compute, storage, and instance administration.

Tokens
1.5M
Snippets
6.4K
Records
10.2K
Agent score
60%

What's inside VeloDB

  1. Overview of VeloDB Agent Skills

    VeloDB Agent Skills embed Apache Doris and VeloDB best practices into AI coding assistants to prevent generic SQL generation. The skills include 49 specialized reference files and 37 built-in validation rules covering table design, partitioning, indexing, cluster sizing, and troubleshooting.

  2. Overview of Doris Streaming Jobs

    A Doris Streaming Job is a continuous import task that queries Table-Valued Functions (TVF) or upstream data sources in real time and writes the data into Doris tables. It supports two primary operational modes:

    1. TVF Mode: Continuously writes data from a TVF (such as S3 TVF or CDC Stream TVF) into a single specified Doris table using DO INSERT INTO ... SELECT FROM TVF() syntax.
    2. Multi-table CDC Mode: Synchronizes full and incremental data from multiple upstream database tables (MySQL or PostgreSQL) into a Doris database, automatically creating downstream tables during the first synchronization.

  3. Overview of Async Materialized Views

    An async materialized view is a query acceleration solution that precomputes and stores query results in a physical table (internal type MTMV). This avoids the overhead of repeatedly executing complex SQL by allowing subsequent queries to hit the materialized result set directly. It is primarily used for query acceleration, simplifying ETL workflows, accelerating lakehouse external table queries, and improving write efficiency. It is suitable for queries following the SPJG (SELECT-PROJECT-JOIN-GROUP-BY) pattern where eventual consistency is acceptable.

  4. Overview of Hive HLL UDFs

    The Hive HLL UDFs allow users to generate HLL operations in Hive tables that are compatible with Doris HLL. This is primarily used to reduce data import time to Doris by eliminating the need for dictionary construction and pre-aggregation, and to save Hive storage by compressing data. Note that HLL statistics are approximate with an error rate of approximately 1% to 2%.

  5. Overview of HNSW Vector Search in Apache Doris

    Starting with Apache Doris 4.0, HNSW (Hierarchical Navigable Small World) is supported as an Approximate Nearest Neighbor (ANN) index. HNSW is designed for high-performance online vector search, providing high recall and low latency by organizing vectors into a multi-layered graph structure. This reduces query time complexity to logarithmic, making it efficient for large-scale datasets.

  6. Overview of Group Commit Modes

    Group Commit optimizes high-concurrency small batch writes (specifically INSERT INTO ... VALUES and Stream Load) by merging multiple small loads into one large transaction. There are three available modes:

    • Off Mode (off_mode): Group Commit is disabled.
    • Synchronous Mode (sync_mode): Commits multiple loads in one transaction based on the load and the table's group_commit_interval property. The client waits for the transaction to commit before returning. Use this for scenarios requiring immediate data visibility.
    • Asynchronous Mode (async_mode): Writes data to a Write Ahead Log (WAL) and returns immediately. Data is committed asynchronously based on the group_commit_interval. This is ideal for latency-sensitive, high-frequency writes. Note: Large single loads automatically switch to sync_mode to prevent WAL disk overflow.

  7. Overview of the Doris query optimizer (Nereids)

    The Doris query optimizer, known as Nereids, is a modern optimizer built on the Cascades framework. It utilizes a combination of Rule-Based Optimization (RBO) and Cost-Based Optimization (CBO) to generate efficient execution plans for complex queries, including multi-level subquery nesting and multi-table joins. It operates on a unified logical execution plan tree to ensure stability and flexibility in extending optimization rules.

  8. Overview of VeloDB Warehouse Users and Roles

    Warehouse users and roles are SQL identities used to control access for people, applications, BI tools, and data pipelines connecting to a warehouse. They are distinct from organization-level members and roles. A warehouse user is defined as a SQL identity in the format 'username'@'host'. To manage users and roles, you must have the ADMIN privilege or a global-level GRANT privilege.

  9. Overview of MySQL to Doris SQL Mapping Sync

    SQL Mapping Sync enables real-time synchronization of a single MySQL table to an existing Doris table. It uses a combination of a Streaming Job and the cdc_stream Table-Valued Function (TVF) to implement an INSERT INTO ... SELECT * FROM cdc_stream(...) pipeline. This method is ideal for scenarios requiring column pruning, field renaming, data transformation, or end-to-end exactly-once semantics. Unlike Auto Table Creation Sync, the target Doris table must be created in advance.

  10. Overview of Stream Load for VeloDB Cloud

    Stream Load is a synchronous import method used to import local files or data streams into Doris over HTTP. It guarantees atomicity for a batch of import tasks: either all rows are imported successfully, or none are.

    Key Constraints and Recommendations:

    • File Size: Suitable for files smaller than 10 GB. For larger files, split them first or use the dedicated Doris Streamloader tool for multi-concurrency import capabilities.
    • Data Formats: Supports CSV, JSON, Parquet, and ORC.
    • CSV Nulls vs Empty Strings:
      • Null values are represented by \N (e.g., a,\N,b).
      • Empty strings are represented by no characters between delimiters (e.g., a,,b).

  11. Overview of Custom Analyzer components

    A Custom Analyzer in VeloDB Cloud 26.x defines how text is split into searchable terms. The process follows a linear pipeline: Original Text $\rightarrow$ Character Filter $\rightarrow$ Tokenizer $\rightarrow$ Token Filter $\rightarrow$ Terms.

    • char_filter: Pre-processes characters (replacement/normalization). 0 to N can be used.
    • tokenizer: Splits text into terms. Exactly 1 is required.
    • token_filter: Processes resulting terms (e.g., lowercasing). 0 to N can be used.
    • analyzer: The final object that assembles the components into a pipeline.

  12. Overview of VeloDB MCP Service query paths

    The VeloDB MCP Service allows AI agents (e.g., Claude Code, Cursor, VS Code, Codex) to connect to VeloDB Cloud. It provides two distinct paths for data retrieval:

    1. Metadata discovery and read-only SQL: Used when no semantic model is present. The agent discovers the schema (databases, tables, columns) and generates raw SQL queries.
    2. Semantic queries: Used when an administrator has published a semantic model. The agent uses governed metrics and dimensions, and the service handles the SQL generation to ensure consistency.

    A semantic model is optional; a deployment reporting no_models can still perform metadata discovery and execute read-only SQL.